Comparison and Evaluation of

Alternative Designs
Adapted from J. Banks

2 Statistical techniques used

in comparing Systems:
Independent Sampling the systems are
responding in negative correlation between
each other. This has been observed in some
inventory problems.
Correlated Sampling or Common Random
Numbers responds in the same direction for
each input of random variates (monotonic).
This is common for certain simple queuing
problems.

Comparing 2 Systems:
It is necessary to make use of
confidence intervals when comparing
two systems.
The confidence interval should be the
difference between two systems.
(Y1 Y2 ) t / 2 S .E.( difference ) 1 2 (Y1 Y2 ) t / 2 S .E.( difference )

Three possible scenarios when computing for

the confidence interval of the differences
(90%, 95%, or 99%):

a 1 2 b

a 1 2 b

a 1 2 b

When using independent

sampling:
Independent sampling with equal
variances
Independent sampling with unequal
variances

When using correlated

sampling:
Dedicate a random number stream for a
specific purpose. Use as many as needed.
Use attributes of an entity to consistently
apply same service times, order quantity, etc.
(which are dependent on the entity)
Use a specific stream for activities with
cycles. Examples are changes in shifts.
Synchronize if possible. Otherwise, use
independent random numbers

Average Response Time for Each Model

Observed difference
Replication
1
2Ind
2Corr w/Sync 2Corr D(1,2C w/s) D(1,2C)
1
2
5
4
4
-2
-2
2
3
6
4
4
-1
-1
3
2
7
4
6
-2
-4
4
3
8
3
5
0
-2
5
4
5
6
4
-2
0
6
5
6
4
4
1
1
7
6
7
4
5
2
1
8
2
8
5
6
-3
-4
9
4
7
6
4
-2
0
10
3
6
5
5
-2
-2
Sample Mean
3.4
6.5
-1.1
-1.3
Sample Variance
1.64
1.05
2.29
3.01
Standard Error
0.518652099
0.47853944 0.54863467
*Using the formula for unequal variances
Using t(0.05/2) = 2.0 for Independent, V=63
63.84325
System 1 Versus 2Ind
-4.137304198
1 vs 2Ind -2.0626958
Using t(0.05/2) = 1.7 for correlated, V=9
System 1 versus 2 Corr w/ Sync
-1.913517056 1 vs 2C w/s -0.2864829
System 1 versus 2 Corr
-6.417
1 vs 2 C -0.3673211

Comparison of Several
Designs
Possible Goals of an Analyst:
Estimate each performance measure
Compare each performance measure to
a present system (control)
All possible comparison
Selection of the best

Using Bonferroni Approach for

Comparison:
When making statements about several
alternatives, an analyst would like to be
confident that all statements are true
simultaneously.

j E

This method can be used in

three ways:
Individual C.I.s of a single system with
multiple performance measures. The alpha is
simply the product of all alphas used in the
comparison. They are assumed to be
independent.
Comparison to a present system. Construct a
1-alpha confidence interval for each
comparison.
All possible comparison. Use the equation
above. This assumes that correlated sampling
was used.

Selecting the Best

Two objectives:
Determining the best of the alternatives
Determining how much the best is
relative to the rest of the alternatives
NOTE: It might be possible that selecting
the second best is more practical, less
costly, more feasible, but still very
insignificantly close to being the best.

Understanding the Effect of

the Design Alternatives:
Use the power of design of experiments
Some of the tools under DOE are:

Factorial Designs (useful for understanding the

effects of the alternatives)
Screening, these are fractional factorial and
Placket Burman (useful for trimming down the
unimportant alternatives)
Response Surface, these are Central Composite
and Box Benhken (useful for identifying the
optimal setup in an alternative)

Metamodeling
Constructing a relationship between the
performance measure, Y, and the design
variables, X.
Some common relationships are: Simple
Linear Regression, Nonlinear relationships,
Multiple Linear Regression.
To verify whether these relationships are
reliable for predicting the effect on the
performance measure, it is necessary to test
the significance of the regressions (ANOVA is
used).