Verifying And Validating Simulation Models Topic 03

Verifying And Validating

Simulation Models

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 1

 Verifying And Validating Simulation Models Topic 03

Verification And Validation References

VERIFICATION AND VALIDATION OF SIMULATION MODELS, Robert G. Sargent, Proceedings
of the 2011 Winter Simulation Conference, S. Jain, R.R. Creasey, J. Himmelspach, K.P. White,
and M. Fu, eds, http://www.informs-sim.org/wsc11papers/016.pdf

Discrete-Event System Simulation, Banks, Carson, Nelson, Nicol, Fifth Edition, Prentice Hall.

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 2

 Verifying And Validating Simulation Models Topic 03

Verifying And Validating Simulation Models

To be useful, simulation models need to:

Work correctly for the intended purpose

Be acceptable to users who intend to use it to make decisions

The processes of ensuring that simulation models are useful are often referred to as:

Verification

Validation

Use of models that are not both verified and validated can lead to:

Incorrect and possibly dangerous and/or expensive results

Lack of acceptance by users who need it to make decisions

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 3

 Verifying And Validating Simulation Models Topic 03

Verification
Sargent defines verification as the process of ensuring that the computer program of the
computerized model and its implementation are correct

In other words, verification is the process of ensuring that the model you build works the way
you intended it to work

Nonetheless, verified models may be totally incorrect in that the way you intended the model to
work may be incorrect

Sargent, R. G. 2000. Verification, Validation, and Accreditation of Simulation Models. In Proc.

2011 Winter Simulation Conf.

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 4

 Verifying And Validating Simulation Models Topic 03

Validation
Schlesinger defines validation as substantiation that a computerized model within its domain
of applicability possesses a satisfactory range of accuracy consistent with the intended
application of the model)

Note that a model validated for one purpose may not be valid for another

Schlesinger, S., R. E. Crosbie, R. E. Gagn, G. S. Innis, C. S. Lalwani, J. Loch, R. J. Sylvester, R.

D. Wright, N. Kheir, and D. Bartos. 1979. Terminology for Model Credibility. Simulation
32(3):103-104.

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 5

 Verifying And Validating Simulation Models Topic 03

Model Building Steps - Step 1 - Observation

Banks suggests that the first step in model building consists of observing the real system and
the interactions among its various components and of collecting data on their behavior.
Banks further suggests that you may need to repeat this step at various times in the model
building process

Banks, Carson, Nelson, Nicol, Discrete-Event System Simulation, Prentice Hall.

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 6

 Verifying And Validating Simulation Models Topic 03

Model Building Steps - Step 2 - Construction Of Conceptual Model

Banks suggests that the second step in model building is the construction of a conceptual
model a collection of assumptions about the components and the structure of the system,
plus hypotheses about the values of model input parameters

To validate the conceptual model (part of the whole validation process) it is important to keep a
record of these assumptions and hypotheses

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 7

 Verifying And Validating Simulation Models Topic 03

Model Building Steps - Step 3 Model Implementation

Banks further suggests that the third step is to implement the conceptual model using
appropriate simulation software

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 8

 Verifying And Validating Simulation Models Topic 03

Verifying Simulation Models

Before replicating the model:

Document the specific tests you will perform in advance of performing them

Create range names for parameters and initial values

Check all input parameters for correctness

Check all formulas

Add additional formulas with intermediate values to ensure those values are correct

Calculate additional summary statistics so as to ensure they are correct

Force the model to possibly unusual conditions by adjusting range values and make sure that you
get desired results

Present the model for review to other simulation modelers; note that just explaining to another
individuals how the simulation works can help identify errors

After replicating the model repeat the above tests to determine the effect of the tests on
summary statistics

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 9

 Verifying And Validating Simulation Models Topic 03

Verifying Other Simulation Models

Additional steps (not appropriate for spreadsheet based simulation models) for verifying
simulation models developed using simulation software:

Visually animate the model

Visually review the model

Visually review all simulation logic

Start up the simulation monitor and step through the simulation:
Watch as many variables as possible
With just one work item in it at a time and make sure it flows properly
With a few more work items and make sure the simulation still flows properly
Check time and content of future events in the monitor

Step through any visual logic you have

Run the simulation with a variety of combinations of parameters and check the results of each
combination for reasonableness

Present all of the above one at a time to other modelers

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 10

 Verifying And Validating Simulation Models Topic 03

Validation Of Assumptions
For statistical assumptions

Make sure the data you obtain is accurate and representative as inaccurate data is often the norm
because:
Data was retrieved from the wrong field or table
Data was never recorded properly in the first place
Data may have been rounded inappropriately
Data may have been under or over exaggerated to make individuals or departments look better (or worse)
Data may be recorded with different units (Celsius, Fahrenheit, pounds, kilograms, . . .)
The sample of time during which the data was collected may not be representative of what is being analyzed

Apply standard statistical hypothesis tests

For model process or logic assumptions, review assumptions with stakeholders and Subject
Matter Experts (SMEs)

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 11

 Verifying And Validating Simulation Models Topic 03

Validation Against Historical data

With the help of stakeholders and SMEs, build a test data set (in a data input file) for which you
know the results; that test data set is generally based on historical inputs
Use that test data set to test the model
Compare the outputs with the test data set to reality; if differences are not explainable then you
most likely have a problem you need to address
Run the test data set for which you know the results whenever you make changes to the
simulation model

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 12

 Verifying And Validating Simulation Models Topic 03

Stakeholder Validation
A simulation model will not be very useful if stakeholders are not convinced that it represents
their process
To obtain that validation:

Inform stakeholders in advance of need for their validation of model

Get input from stakeholders and SMEs as to important details of the model (throughout the modeling
process)

Log assumptions

Track data sources

Animate simulation model

Run model with real historical data and compare it to actual results

Have review meetings that includes all of the above

Respond to stakeholder and SME feedback

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 13

 Verifying And Validating Simulation Models Topic 03

Graded Homework Questions

After watching the video and reading Robert Sargent's article (see list of references at beginning
of these slides), in your own words, briefly answer the following questions:

Why is verification and validation important?

What is IV&V and what are the benefits of using it as opposed to other approaches for verifying and
validating models?

What is the difference between conceptual model validation, computerized model verification, and
operational validation?

List and very briefly explain at least six different validation techniques

Briefly explain what a Turing Test is (see en.wikipedia.org/wiki/Turing_test ) and then how Turing
Tests be used to validate a simulation model

Briefly explain why documenting verification and validation activities is important

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 14

 Verifying And Validating Simulation Models Topic 03

Graded Homework Problems

Develop a written verification plan for your implementation of Topic 2 Problem 12

Philip Troy, 2008 - 2015 philip.troy@mcgill.ca www.PhilTroy.com 15