04
'rI
DTI
71..
4..
S

.~.
07

2
oi
i3
VF.
A.
,'I
DIRECT COMPARISON OF INTUITIVE, QUASIRATIONAL AND ANALYTICAL COGNITION Kenneth Hammond, Robert M. Hamm,
Report No.
248
,
I.
June 1983
JUL 1 2 1983
This research was supported by the Engineering Psychology Programs, Office of Naval Research, Contract N0001477C0336, Work Unit Number NR 197038 and by BRSG Grant #RR0701314 awarded by the Biomedical Research Support Program, Division of Research Resources, NIH. Center for Research on Judgment and Policy, Institute of Behavioral Science, University of Colorado. Reproduction in whole or in part is permitted for any purpose of the United States Government. Approved for public release; distribution unlimited.
K "
248
S
TP P
OFRPR
LR
OEE
Technical
PERFORMING 040, REPORT NUMBER
S.
N0001477C0336
10. PROGRAM ELEMENT. PROJECT, TASK AREA&WORK UNIT NUMBERS
Center for Research on Judgment and Policy Institute of Behavioral Science University of Colorado, Boulder, CO 80309
CONTROLLING OFFICE NAME AND ADDRESS
11.
12.
REPORT DATE
June 1983
13. NUMBER OF PAGES
Arlington,
VA
22217
15.
71
SECURITY CLASS. (of thia reporti)
Unclassified
15q. E CL ASSI FI CATI o
SCHEDULE
16. DISTRIBUTION STATEMEN ' (of this Report)
N/Dow
N GnAOING
17.
DISTfIBUTION STATEMENT (of the ob,,tract entered In Block 21, If different from Report)
IS. SUPPLEMENTARY
NOTES
19.
KEY WORDS (Continue on reveree aide It necessary and Identify by block rumber)
achievement, cognitive control, confidence, consistency, environmental models, experts, intuitive/analytical, Lens Model, cognitive continuum theory, social judgment theory, task characteristics, transportation,
ko
20.
safety
ABSTRACT (Continui on reverse side if necessary and Identify by block number)
The relative efficacy of intuitive and analytical cognition in analytically competent persons was directly compared. More subjects performed best in the intuitive mode when inconsistency was removed from their judgments, an indication that the subjects possessed implicit knowledge that they did not utilize in the analytical mode. More subjects made larger errors in the analytical mode than in the intuitive mode. Subjects' confidence was generally inappropriately placed. DD
I
,
1473
SN 0102. LF 014.6601
Direct Comparison
Hammond, Ham, Grassia, and Pearson Ack nowledgments from
our Barbara
research Chokol,
we
received
indispensable
assistance
Richard
Cutler, Daniel
Cronin,
Miro Supitar gave us valuable advice and served on our expert panels. Allen, John Bright, Charles R. Keim, and Marilyn
John H.
subjects. We thank Harvey Atchison, (0 Heiderstadt, Zimmerman. Steven Holt, Robert Clevenger, John Dolan, Tucker, Kells Edward Haase, Don and Theodore
William
Waggoner,
in the Denver metropolitan area allowed members of their staffs to participate as subject5 in our research. To the twentyone subjects whose judgments of this report, we are especially grateful: E. safety are described in
Donnelly,
Johnson,
Robert L. Kenny, G.
Phillip McCabe,
Muhicn,
C. Rautenstraus,
work our research would not have been possible. Finally, we deeply appreciate the skill and Doreen Victor and attention to detail which
Mary Luhring have put forth in editing this paper and the
* mi" L
Direct Comparison
Hammond, Hamm, (rassia, and Pearson
Page 1
'
~TAIBl
..
QUASIRATIONAL
*L
a major topic in review of approaches to this topic, Although intuitive biased, (for cognition and to see see Hammond, is largely believed in to which be systematically overconfident, produce Slovic, judgments Fischhoff
L
are 1977; "none
,iiA
Comparison of the efficacy of intuitive and analytical cogniff fi6Thaf'ben judgment and decision research since its inception (for a McClelland & Mumpower, 1980). inaccurate, persons
reviews
applications
reasoning
deviates
research that is restricted in the following ways: 1. Comparisons are indirect. The most widely used research method compares the intuitively derived judgments of subjects with analytically Theorem Slovic, and derived multiple answers generated by regression Hammond, equations Stewart, may be,
Kahneman, 1975).
& Tversky,
1982;
Brehmer, they do
& Steinmann,
Informative as
these
comparisons
not directly compare the achievement of intuitive and analytical Direct comparisons are needed in
",
.,
r
r~
Direct Comparison Hammond, Hamm, Grassia, and Pearson order to know which mode of human crognition is superior. comparisons see Tweney,
Page 2
Doherty
Research subjects lack analytical competence. Research subjectstypically college sophomoresgenerally lack
knowledge of any analytical principles that would enable them to erigage in analytical cognition. opportunity to do so. Moreover, they are seldom given the time or
models to represent analytical cognition constitutes a special That is, intuitive cognition is set in competition with a
comparison.
over
their
Although exhaust
such our
comparisons
are
important,
range
of interest in person
they do not inform us about what happens when the same different forms of cognitive activity.
of engaging in analytical cognition. 3. Intuition and analysis remain obscure concepts. Researchers in cognitior almost never explain what they mean by
intuition, although they take great pains to differentiate precisely among formal,
S
A3 a result it is not.
is
customary
to for
explain
analysis
Brooks (1978),
example, compares "analytic and nonanalytic concept formation." and Tversky (1982)
Kahneman
Page 3
reached by an informal and unstructured mode of reasoning, without the use 40 of analytical methods Cohen, or deliberate 1981) who calculations" criticize (p. 494). Lven philosophers (e.g., psychological research
regarding intuitive judgments of probability fail to say what they mean by O this term. cognition As a result, the task properties that induce are not specified, either form of
not impossible. 4. Methodological and substantive contion are not differentiated. Successful methodological (cf., 1981). Hammond, cognitive competence; 1966, efforts neither require alone substantive as well as
Topical
knowledge
however.
Judgment and decision researchers contrast cognitior (efforts to combine rules for subjects' 1977).
intuitive methodological
with analytically derived normative methodological Bayes' Theorem), & Hogarth, on irrespective 1981; the of
Einhorn
study
proolemsolving,
generally
their subjects'
manipulation of the substantive materials of the or experts' 1980; behavior (see, for example,
truths Simon
& Simon,
analytical
however, they
Page 4
5. Evaluations of intuitive methodological cognition are normconti gent. Considerable uncertainty exists about which normative rules should be used to evaluate intuitive methodological cognition. The sharply Kyburg &
divergent views of philosophers on this topic (see, Smokler, 1980, Kyburg, 1974, and Levi, 1980)
in the research
intuitive inductive
normative
evidence that both frequentists and subjectivists have failed to solve the
* fundamental
problem
of
induction.
Until
they
succeed,
the research
to which
psychologist who attempts an empirical evaluation of the extent intuitive cognition conforms
results contingent upon the normative rules prefer (1981) they different rules
cite other difficulties with normative rules in their note that choice of and an a
optimal model is "conditional on certain specific time horizon" (p. 55) and
environmental assumptions
"naive" (p.
0P to
upon
empirical
events
(cf.
Scribner,
1977,
empirical
achievement
is
T
that deserves as much or more interest as conformity with one, normative rules. (See Hammond
optimality," p. accuracy".) 6.
215ff;
Evaluations of intuitive methodological cognition are task contingent. Current conclusions about intuitive competence in human beings working are with
largely
restricted
to
the
performance
of
subjects
methodological problems that have been constructed to be by the methods of the subjective interpretation cognitive
readily of the
But, as noted
above,
problems
Therefore, examination
examination of performance in task conditions specifically constructed conform example, to normative methodological their closely 1977 to (1977, rules. JohnsonLaird a.d Wason, "by attempting to
note in more
relate
improved" "It
is essential to emphasize that the cognitive approach has The is.,ue of why
been concerned primarily with how tasks are represented. tasks 57). are Indeed,
represented in particular ways has not yet been addressed" (p. the general absence of the description and classification of has long been emphasized as a serious shortcoming in 1954; Edwards, 1971; Slovic
cognitive
tasks
Direct Comparison Hammond, Hamm, Grassia, and Pearson & Lichtenstein, (Hastie, 1983), 1971; Linhorn & Hogarth, 1981)
Page 6
and continues to be
In sum, in addition to making indirect comparisons directly compare an analytically competent subject to person's employ
substantive as well as methodological cognition, and to evaluate In the present study, therefore, we do
intuitive
and
analytical
cognition
in
the
same
intuitive
and analytical cognition can be brought to bear on the same task; 3. specify the properties of intuitive and analytical cognition properties of tasks that evoke each; 4. 5. include substantive as well as methodological aspects of cognition; contrast the efficacy of intuition and analysis in terms Jf achievement; 6. and empirical and the
 
 . /.,.
Direct Comparison
Hainond, Hamm, Grassia, and Pearson Theoretical Bac__round
Page 7
and analysis but rather a continuum marked by intuition at one pole Unlike the traditional dichotomous premise, a
compromise 1973;
Hammond
Hammond et &
Hammond, 1981.)
Note 1;
McClelland
Shanteau,
Modes of Cognition Intuition and analysis can be distinguished by the (a) cognitive control (in intuitioi, rapid, low; relative degree (b) of:
rate of in
i.e.,
microseconds;
(in intuition,
low;
in analysis,
intuition, a weighted average; (e) type of error (in large errors); not method; (f)
in analysis,
type of confidence (in intuition, confidence in answer but not answer). (For further
distinctions,
see
Hammond,
Note
1.)
The
compromise
form
of cognition,
both types of
1.
 
Task Properties That Induce Different Modes of Cognition If, task (a) (b) ,

the
properties
differentially
induce intuition and analysis include: large [> 5]; in analysis, small); in (in cue
which cues are displayed (in intuition, (c) the type of cue
analysis,
sequential)
measurement
characteristics in
variable,
distributed;
analysis dichotomous,
Quasirationality is induced to the extent that tasks contain properties both types of polar task conditions. We
properties must be present in order to locate a task at continuum, * nor do we know their
IObjectives * _Four aspects of the cognitive continuum theory are examined in the
ll
context
of
interest in the relative efficacy of differences safety in in the were the empirical
accuracy
judgments of Highway
intuitive,
quasirational
modes.
engineers
chosen as subjects because of their frequent professional use Judgments of highway safety were chosen
Page 9 and Pearson below, the properties of this task should induce quasi of the middle range of the
rationality, cognitive
intuitioninducing
taskjudging
highway
aestheticsand an analysisinducing taskjudging highway capacitywere also empicyed, but space prevents their discussion here.) because of the important theoretical role of cognitive
Second,
controlpredicted
and high in analysiswe examine differences in cognitive control in the three modes. More specifically, to by in accuracy Hammond
control
numerous
multiplecue
interpersonal 1976;
interpersonal Brehmer
learning
Holzworth,
press;
& Hammond,
psychoactive drugs on interpersonal learning and interpersonal conflict psychotic patients (Gillis, 1975, 1978), and
1982;
in press).
to determine whether they provide different contributions to achievement under the three modes of cognition within the same person. One of the advantages of the direct comparison analytical cognition is that the errors made of in human intuitive and
cognition can be observed and compared with the comparison formal niodel. cannot be made
errors
when intuitive cognition is compared only with a the answer provided by the formal model
by definition,
Therefore
Page 10
our third objective is to compare the errors produced by analytical as well as intuitive and quasirational connition. Brunswik has already argued intuitive cognition produces and demonstrated (195b, pp. 8993) that
answer because "intuitive perception must integrate many avenues of or cues...none of which
therefore ask whether intuitive cognition produces a errors centered on the correct answer;
precisely correct answers together with highly incorrect answers; quasirationality produces a distribution of errors lying
generated by the polar modes of cognition. Fourth, the relation between confidence and performance discover whether confidence matches performance (Oskamp, is examined to
1965;
Lichtenstein, cognitive
mode in which he performed best and least confident in the mode under which he performed most poorly? hypothesis. Since The the theory method outlined of above suggests an additional rapid, "avenues
intuitive
cognition
produces
nonretraceable answers and since these answers are based on multiple of approach or cues," intuitive in subjects answers. the should The
mode,
which
subject's
attention
fociised
organization of cues.
Direct Comparison
Hammond, Hamm, Grassia, and Pearson Method Subjects Twentyone male highway engineers, subjects. Since engineeers 3070 years of age,
Page 11
served as research
are professionally trained to cope with problems components, subject may of the be intuitive directly a and analytical The
compared. complex
highways,
problem
of
one
to
judged the safety of each segment solely on the basis of the irn the film strips (see Figure 1).
engineers were neither asked nor given the opportunity to materials explicitly.
Page 12
.44
4,,.
S..
In the condition designed to induce quasi 0 highway rationality, the same
; "I
forty
segments were presented as bargraph profiles which displayed values 2). The bargraph presentation meets the
for
inducing quasirational cognition by combining intuitionproperties. On the one hand, the task remains
analysisinducing
intuitioninducing
because
the
number of
cues
is still large;
Page 13
10
13
Lane Width
XXXXXXXXXXXXXXX
10
Shoulder Width
XXXXXXXXXXXXXXX
0
Percent No Passing Zone XXXXXXXXXXXXXXXXXXXX
41
80
0 1 XXXXXX
Grade
0 47 XXXXXXXXXXXXXXXXXXXXXXX
75
Traffic Volume
0 470 XXX
10000
0
Traffic Mix
21
35
XXXXXXXXXXXxxxxxxxxxxX
0 .7 XXXXXXX
30 42 XXXXXXXXXXXXXXX
60
0 4.4 XXXXXXXXX
18
Figure 2.
. , ,/ .
, .4,.
, . . . . . . . .
... .. ..
..
, ,
Direct Comparison
Page 14
Hammond, Hamu,
materials
explicitly
and were not asked to explicate or the number of observable cues as numbers, is
expressed them
required
measure
perceptually.
The
In construct
the and
analysisinducing justify a
condition
each equation
engineer
was
requested
to
mathematical
were
certain amount of time would be allowed for the completion of the formula, that the time period would to one be extended until he completed the task. of three subgroups within the
assigned
analytical
to work through the task with a minimum of instructions and requests from researchers. Six engineers
in this group were encouraged to complete their the other six were aloud" subgroup (n give
=
target
of
requested to think aloud as they constructed their analytical efforts could be monitored during
In the "maximal to
guidance" subgroup (n = 3) engineers constructed their equations according a detailed written procedure (see Appendix B) designed (Samples
likelihood of a systematic approach to the problem. remarks . regarding their cognitive activity in
Appendix C.)

..
K
SDirect Comparison
Hammond, Hamm, Grassia, and Pearson These three subconditions were used because they offered * information. The a
Page 15
variety
of
minimal guidance condition revealed how engineers would go on of their the own. The "think of aloud" condition
engineers'
process (see
Appendix
maximal guidance condition showed whether a structural approach was feasible. It was not subconditions. our purpose to determine the effects of the different
LD
and engineers who were given a 45minute target in minimal guidance took time than the others.
among the subgroups. Dependent Variables The effect of the intuitive, quasirational, were examined with respect and analytical conditions
to (a) each engineer's degree of achievement in the differential contributions of achievement, knowledge
different types of errors made in each of the three modes (d) the relation of confidence to performance in each mode. Achievement estimate of was measured and the by the correlation rate of
between
an
engineer's
safety
accident
each
criterion).
77
aA~..AA
A1
q17
Knowledge and cognitive control were derived from the parameters Lens Model Equation (Hanmnond et al., 1975):
ra =GR R + C[1
where ra achievement,
/I
R
the correlation between judgments and criterion values corrected for attenuation due to less than perfect linear predictability in each
=
=
environmental
R
C
correlations
=
between
residuals
(trivial
G Re Rs.
R. = 1.00)
criterion were perfectly predictable from the cues (i.e., Re = 1.00). Cognitive control is appropriately measured by Rk in the there was little equation since
evidence in this study of lack of fit of the linear model. for a detailed discussion of the distinction
Direct Comparison Hammond, Hamm, Grassia, and Pearson  ifferentiel errors produced n examination by each engineer are
of the distribution of deviations of judgments around the correct examination of mistakes made in the process of formulating the
equation and evaluation of their consequences for accurate prediction. Confidence in answers in the intuitive and quasirational modes is
measured by the mean of the engineer's confidence, his judgments. method session.
for all three modes was measured by questions at the conclusion of the
Procedure
Statistical Properties of the Task The same set of forty hiqhway segments was used in all criterion 9 for the conditions. The
accuracy of judgments is the accident rate, averaged over Accident rate is defined as the total
of accidents (involving fatalities, injuries, or property damage only) Due to its extremely high
accident rate, one highway was dropped from the analysis. Highways were measured on ten dimensions, chosen for inclusion in the
"study on
the
the information they considered essential for evaluating the safet.y of a (see Table 1 for list).
Eight of these measures were available from highway mile and number of
dimension
or
cue
Visual examination of the scatterplots of the relations between each cue the criterion indicated little if any nonlinear covariation.
was supported by the results from calculation of the contribution terms and interactions to accident rates.
Cues
Beta weights
Lane Width Sthoulder Width Percent No Passing Zone Curves per Mile Grade Traffic Volume Traffic Mix (% of Trucks) Intersections per Mile Average Speed Limit Obstacles per Mile
.023 . 042 . 143 .152 .055 . 198 . 017 .247 . 316 .478
Direct Compaarison
Hammond, Hamm, Grassia, and Pearson linear multiple regression for estimated each cue model of
Page 19
the
=
corrected
shrinkage given
the
(corrected for
shrinkage = .667).
for
each An
task
condition
was
appropriate
to
the
activity
induced.
abstract
(unsafe) was used for the film strips to induce intuitive cognition. bargraph formula, presentation and in the task requiring the
construction of a
employed because this specificity is compatible with calculation and thus with analytical cognition. Transformations to a common scale were made for
Trials
All engineers were presented with the tasks in the the film strips; It second, was not the bar graphs; appropriate work to same order: first,
because
analytical have
requiring
fashion
would
strongly
influenced
whereas the reverse is not true (see Jones and Harris, ten of the forty highways
*for
each engineer.
V' I.
T7,v
T
1,,
rr
Paq
20
IID
40N
LAi
0i
CD r
0
CLO
00~ MA ~~
CnC~ C3 L
L 'JO
4 Ch
L 0h a
(00
IL.
OD0
UO
%0
00
CA
Mr4
UA
L O
0~
CD
'.YO (. 0
V)
0t
0tC'
(nJ
U0
(.0.
LO L II~ I" In C)
*n
j
LU 0 0.JL a_ 0L LL. 4
V)
54.)
0 '.JC<
LN
LA
N L)
(DC
Direct Comparison Hammond, Time Response times were recorded in all three artifacts * behavior; of the task presentation modes, Ham, Grassia, and Pearson
Page 21
but
are
conditions rather than descriptions of the subjects' film strip required more time than
that is,
presentation of a
0
In the subgroups. analytical The mode the response time varied within the three eighteen engineers in the think aloud and minimal guidance to fortyfive
but seven of them took at least an hour at this task (maximum = 115 No time constraints were imposed on the engineers in the maximal
RESULTS Relative Efficacy of Three Modes of Cognition Achievement Individual differences in achievement. the correlation between Achievement (ra) is measured by
an engineer's judgments about the safety of each of Data from repeated For
the 39 highways and the accident rates of those highways. judgments in the
the engineer's formula was applied to each of the 39 thus produced were correlated with the accident
answers
AL
Wide individual differences occurred in all modes of cognition (idble 3). In the intuitive mode, the engineer's achievement correlations ranged from In the quasirational mode, ra ranged ra s
achievement
quasirational analytical
modes,
.457; .032
quasirational
regression analysis,
individual differences
",
mode,
however,
judgment
policies
were
not
modeled
but
Individual
differences in achievement
in part, by the wide variety in It turned out, however, are ,,ot related to
S.
the
structure
of
those
that the various structural features achievement. In the intuitive mode, age, Rs years of work,
by
or years of education),
otherwise there were no significant relations between experience and measures of achievement, knowledge and consistency.
various
cognitive modes thus supports the decision to study separately the performance
~ ~ ~ ~ ~ ~ ~0
7r

~'~,
.
.

Page 23
CI C)
CpC
C)
.jC) C)
C)
C)
C) Cj
CD
C0 0
C) )
C0
C)
C)
C)
C)
C)
C)
CD
0D c:)
CD
C)
cyC)
q0
IC) O
C) 160
C)
ct
0)
PC)L
c~f
cj Ci
LO 00 40)
L)
)0)al
00
C)
Qu
C7)
"
0 C)
C) 0
0o
0 )0
C) 00
0d
c C.*
C)
4)
C)
CD O
C)
0N
C.Q
CIJ
C.) C\J
'C) C4
C) C4
C) CD
10
C C')
CD LA C.0
C)
C) CD)
C0 C)C
Cl
O 0C)
L)) 0. 4
0) 04
to
M4
qr MV
r" 
a)
%4 00)
1.0 rLA
1.1) to
LiOl
e.J
00~
14
LO
04 M~
ON
CJ
r9
a4c) 
00
CD
 M_
4w
U)t
CL 0.)
qr
tic
~~C:)
~N
'J
(V) 00 c)
C. 1.
C\J
0
t4
(Y)
qCO
N
C) C) 0
ko.
(v) N10
N_
'
co
LA C:)
LA
1.0 LA
M
ct
P%
C.)
C.
LA)
Itd
)
(V)
U
C)
C)
MA
%.0
C)
LA
.0
H 
S.
0) 4J

1,
;t LA4
1.0 r=
LA
qt
MO NI
M~
6j
rIt
A
N C_
4*
MI C
LO C
CO'
ON
to
co
C.
co
cn
LA
N
0)
4.N
C) LA NWU Cy
14 L
to)
1.0
'Ln
P1.0
C)
IN
)
LA
Ci
cO
CO j
M.
cm. Dall
WD1 Nw
CO W.
_
LA
(.IM
CO

to
C)Nco
N

0)
l_
LA 1.0
.'C'.
O
I' C'..
4
. 4
CO C) 4
I. I
%04
w.
co
()
10
O4
nC.
C)
toJ '4
LA
LA
N
It
FlA L")
C)i rCAr
r_ C.
M.
10
L.. I~
CIO
L.LiO
MJ M
N
M'.
q3"
%0
10
00
C)
CO
1.0
M)
LcO
a)


__
Page 24
0
S0
4j~
C)
C)
C
i
CD
0
0D C0
C
0=
C
C)
4)
0
9
Lt 00r
t
M Ln l,
44
N1
CO
CO LO 0) LO N
LO
i
CC)
ro
CD
C 0
0C
CC
C)
C)
C=)
C)
CO
CD
C)
4.)
M CLO
Cy
it Kr 00
) 00 N
CY% N
C MN0A N
V. "
O~l LA
0 N
*~
0O
~ Cl
0) a
P4 C
'0 N
4
LA i
NMt
"
4) C: .0
0 C r'0U
O% 0.
it 0.D NA
iO
'.0
0t C
0" qt) .
V4 LA LO
O '.MO rt
0\ N U'.
on
E
Lu
r4P'4
LA
0)
'.0
4
0~)
0"
U,
Cr3
'o
CM
it (NJ
rl
N.
00
N 0Y)
ck C
(1) 0 '0
to
I0
CI
4. 14
*
MO U)
. .
Cv)
LA
M4 LA
.
11% II)
.
wU
LOA
64
Cl
Cj(\
Z o:
(00
Page 25
C,,
CD,,
"4
I,
0
S.
00
a) >
E
*U,
cu
CU%
4),
S.
S.a)
__ __ ___
cuI
5AV.
J
Cl!
C"p
Cy
Page 26 and Pearson 1952; pp. Luce, 1959; Tversky, 1972; 1980; Shanteau, Hammond,
115127;
Note 1). Achievement in three comparisons of each modes of cognition. in In order to make direct the of
subject's
perfornance
data thus consisted of rank ordering each engincer's performance across modes. Means are provided for statistical tests. A chisquare test of the superior to the others hypothesis that one mode of cognition was general information but are not included in the
resulted in a failure to reject the null hypothesis however, only four engineers wnose achievement was Comparison of the six orders of achievement (p < .01).
There were,
intuitive mode.
data suggest that all nine engineers who performed best in the analytical
nmode performed better in the intuitive than the quasirational mcde. Components achievement (ra) of is achievement. As the Lens Model Equation indicates, .ontrol (Rs). reasonsthat
their knowledge and their cognitive control over the application of Ztheir compensatory mannerwe determined the relative LI
contribution of these two components for each engineer in each cognitive mode.
k&
in Each Condition
Ist
Order
I >Q>A I >A>Q
2 2
Q>I>A O>A>I
6 2
A A>I>Q A>Q>I
and
quasirational
data
sets,
the
values
of
the
parameters of the Lens Model Equation were calculated from the best fit models of the environment and of each engineer's judgments.
analytical
data set, the best fit linear model was used for the environmental formula he
system, but the engineer's judgment policy was represented by the produced. In some cases the formulas had nonlinear features.
An additional
Direct Comparison Hammond, Hamm, Grassia, analysis was carried out generated form the
Page 28 and Pearson using a linear formula; best fit model of the answers
engineer's
from those produced by the first procedure and will not be reported here. Knowledge. of the Table 6 shows that the knowledge component, G, engineers is higher in for thirteen
twentyone
(X2 = 8.86;
order also shows a distribution significantly different from chance (p < .01). No engineer has poorest knowledge in the intuitive that if the attenuation mode. Thus we conclude
of ra by the inconsistency in the task and in each most engineers would have achieved Most The
about the same as the number whose quasirational judgments were poorest. Cognitive control. The cognitive control component, Rs, is of course
in the analytical mode than in the other modes mechanically had of a generated Rs in from the The is
because the answers in the analytical mode were the engineers' formulas. Far more engineers
larger 21;
quasirational mode than in the intuitive mode relation between cognitive control and
(16
repeated
trials
discussed in Appendix E.
.. '"
"
k .
.~
X. Z
Page 29
Ist
Urder
13
9 4
>>I>A [Q>A>I
6 0
A A
2
0
Fewest
engineers this
wvere is
most not
although
finding
Comparison of knowledge as measured by G, however, the engineers explicated their knowledge and
though
applied it rigorously, i.e., with perfect consistency, in the analytical mode, their achievement would have been higher in the intuitive and quasirational
SW7
7
191
Direct Comparison Hammond, Hamm, Grassia, modes if Page 30 and Pearson This result supports the
they had been equally consistent there. that engineers possess and can
conclusion
Differential Distributions of Errors under Different Cognitive Modes Investigating the covariation between judgment exhaust the possible 1956, of methods pp. 8991; for evaluation of and criterion does not
Brunswik use of
cognition
Based on the
distributed
and
also
than errors made in the analytical mode; 2. The quasirational mode between cognition. the produces of an error in distribution intuitive and that lies
distribution
errors
analytical
subtracting
the
used in the intuitive mode ran from 1 ( scales per in the quasirational miles
unsafe) to 10 (= safe),
and analytical mode ran from 0 to 32 accidents To make the error distributions
1
million
vehicle
traveled.
i
*.
Direct Comparison Hammond, Hamm, Grassia, and Pearson comparable, 32 scale. judgments Further,
Page 31
made in the intuitive mode were rescaled onto the 0 to engineers' analytical formulas did not produce
some
they intended their formula to produce for the safest road onto zero, and number they intended for the most dangerous road onto 32. Hypothesis 1: The error distributions in the analytical condition
should
deviate from normal more often than they do in the intuitive condition. Procedure. deviation For each engineer and for each mode, the mean and standard
of each engineer's error distribution were used to produce a normal The null hypothesis is that the does observed distribution of an
distribution. engineer's
errors
not differ significantly from the constructed normal the normal distribution was divided into six
distribution.
Therefore,
categories expected to have equal numbers of observations located at . 97, mean), . 43, 0, +.43 and +.97 standard
deviations
and the number of the errors that fell into each category was observed. In the intuitive, quasirational (two, of five, deviations of the and analytical modes there
engineers
distributions a
Additionally,
comparison
error distributions from normal revealed that no mode tended 0 deviations hypothesis than that the the other modes. These mode
analytical
would
Direct Comparison Hanunond, Hamm, Grassia, and Pearson Hypothesis judgments than I also asserts that more should of have the engineers'
Page 32
analytical
intuitive
judgments
error
positive kurtosis;
should lie between those of the other modes. Procedure. distribution, Since i.e., a one off, positive in we which kurtosis indicates a peaked frequency
the answers are often nearly correct yet the degree of kurtosis in each
examined
engineer's error distribution. Results. the The results in Table 7 indicate that the error distribution for
analytical mode was nmore peaked than for the intuitive mnode for seventeen The quasirational error distribution was more error distribution for seventeen the engineers error from would W1
of the 21 engineers (p < .01). peaked (p < .01). distribution than the intuitive
However, more
for only ten of the engineers was peaked than the quasirational.
examination of kurtosis thus support the hypothesis that more engineers produce answers that were
less frequently exactly correct in the intuitive mode. Finding that the quasirational error
distribution was not different from the analytical was unexpected. A second way of testing the hypothesis concerning the shape of error
distributions in each cognitive mode is to look at each engineer's performance as a whole rather than correct, achievement at (ra, each answer individually. If the formula is
the correlation between answers and accident rate) should generate more answers
are nearly correct than intuitive judgments do. all answers should be wrong and ra
Direct Comparison Hammond, Hamm, Grassia, and Pearson Table 7 Number of Engineers with Various Pairwise Orders of Kurtosis of Error Distribution
Page 33
A >I
17 4
X2 p <
6.86 .01
I >A
*2
A>
Q> A
10
11
X2
NS
O0O
>I
I >Q
17
4
6.86
p < .01
between
answers
and
accident and no
rates for the five best performances in the analytical mode (.643 to .731) for the five best performances in the intuitive mode (.618 to * overlap. From this .636) show
standpoint the best analytical performances are clearly However, the median achievement
superior to the best intuitive performances. in the analytical mode was .467,
hypothesis that analytical cognition is more often very precise yet more often widely in error.
tic
Direct Comparison Hammond, Hamm, Grassia, and Pearson Table 8 Achievement in Each Mode
Page 34
Intuitive Span of Best Five Engineers' Achievements Median Achievement (all 21 engineers) Range of Achievement (all 21 engineers)
Quasirational
Analytical
Experimenter's Observations Regarding "Widely Incorrect" Answers. The above analyses describe different modes of cognition; the distributions of errors produced by
predicted would occur in the analytical mode are best illustrated by of errors made by three engineers in the analytical mode: 1.
Engineer #2 made a careless arithmetic error in producing weights He first assigned a weight of .10 to each of the tei, cues. cues to .12. Finally, intending
his formula.
Thus he gave the highest weight to the cues to which he wished to weight. The effects and his of mean his error was were "catastrophic": 44.176 on a scale his he
error
. .
'
. ..   _
Although provided with the information that certain cues could assume of zero, Engineer #4 produced a formula in which some of these cues of fractions. As a result, his answer was
indeterminate on 5 of the 39 highways. 3. Engineer #10 produced a linear weighted average formula, used the wrong sign on two cues. this His error in which he was
inadvertently
achievement his
consequently a very low . 048. would have been .293. Since the processes of directly observed, we
achievement
intuitive explain
and
analytical
cognition
cannot
be
cannot
occasional
poor
filmstrip and bargraph tasks by citing the kind of careless, in execution of a complex symbolic procedure as we did above. occurring in the intuitive and
were actually engaging in a stepbystep cognitive procedure similar observed in the analytical mode, each trial in the task.
complete consistency when a faulty formula is applied to thirtynine highways, it is not guaranteed On the by the conditions since no of intuitive and quasirational
cognition.
contrary,
judgments in either the intuitive or the quasirational modes, expected to make consistent errors.
the task.
Ul
Direct Comparison Hammond, Hamm, Grassia, and Pearson Summary regarding differential errors. was examined, When the distribution of
Page 36
errors
the results provided no support for the general hypothesis that intuitive mode more
errors would be found to be normally distributed in the often than in the analytical
intermediate. engineer's
However,
performances
in the analytical mode were more often more accurate but occasional large errors
than their performances in the intuitive mode, were produced in the a,,alytical mode.
little different from the analytical mode in this regard. engineers' analytical
(misplaced decimals,
were found in the other modes of cognition. One reason that the anticipated differences in the degree of normality of error distributions were not found may be that the formulas produced by the in form; that is, many engineers developed
'It
that were analogous to a robust weighted average (eleven by a strict see Appendix A). Formulas of this
large errors as the weighted averages that the evidence suggests '.he engi.neers were using in the intuitive and quasirational modes.
IuD
oI
Al'
"
J '  :
mm
:
( 
'
'
"..
"
Direct Comparisor Hammond, Hamm, Grassia, and Pearsonz Confidence Ansvier Confidence Compared with Method Confidence Cognitive continuum theory predicts that subjects will be more in answers produced by intuitive cognition than in
Page 37
confident
Conversely,
confidence
answer and method confidence separately, and (b) the order of each mode. These
*
m
Answer Confidence
Intuitive
Quasirational
Analytic
m
:'
Method Confidence
Note:
Figure 3.
Direct Comparison Hammond, Hamm, Grassia, and Pearson Figure 3 illustrates the expected method confidence among modes of order of answer confidence
Page 38
and
of
cognition.
relatively high in the intuitive mode and decrease as cognition analytical. The reverse should be true for method confidence.
Figure 4 represents the expected withinmode order of * confidence and method confidence. In
engineers'
answer
express more method confidence than answer confidence for the mode the analytical pole on the cognitive continuum.
Q
Answer
IQ
Confidence
Method Confidence
1i
Note:
Figure 4.
Predicted Relationship Between Answer Confidence and Method Confidence for Each Pair of Cognitive Modes
Procedure. *Q All confidence ratings in this study were indicated on where 10 = high confidence. a 1toIO scale
In the intuitive and quasirational modes, answer by the mean Answer of the confidence in the
confidence is indicated for each engineer *ratings he made for each of his
judgments.
confidence
0~
Direct Comparison Hammond, Hamm, Grassia, and Pearson analytical mode and method, confidence for all th~ree modes are the means of responses
Page 39
represented
by
which the engineer answered after making judgments or constructing a formula. Results Orders of Answer and Method Confidence. confidence across modes is shown in Table 9. The rank order of engineers'
order (10 engineers) was exactly the predicted order subjects indicate more answer
1
quasirational mode, and more in the quasirational mode. As predicted, twenty engineers had more
answer confidence in the and sixteen had more in however, only eleven had
intuitive than in the quasirational mode (p < .001); the intuitive than in the analytical mode (p < .05);
OW
more answer confidence in the quasirational than in the analytical mode (NS). For method confidence, the modal confidence order was not predicted. was I > A > Q, which
Thirteen and a half engineers (ties counted as 1/2 in each in the analytical mode
of the indicated orders) had more method confidence than in the quasirational mode (NS, p < .20);
but only three had more method (p < .01 i, the wrong
direction), and only 3.5 had more method confidence in the quasirational than in the intuitive mode (p < .01 in the wrong direction).
* L
Page 40
14
Count of Engineers Who Had Each Possible Confidence Order among the Three Modes
A> I >

A>Q>1
When an engineer had equal confidence in two mndes, .5 count was assigned to each cf the two orders. For method confidence,
there were two ennineers who had ties. was missing for one engineer.
Method Confidence
data
*:.
Meth:.d Confidence Compared with Answer Confidence Comparison confidence shows of each that engipeer's the majority method confidence with his answer
answer than in met.hod for the intuitive and quasirational modes confidence in method
than in answe'r for the analytical mode (see Table 10). however, that this finding is not significant,
*,
Page 41
Ii
Answer Confidence is greater Equal Method Confidence is greater 8 6 12 15
A''L
7 2 12
In sumhary, the prediction that the engineers would have more in their
confidence
answers in the intuitive mode was born out, but the prediction that method in the analytical
the eng;neers would have more confidence in their mode was contradicted. In the
within mode, the predicted pattern of greater method confidence in the analytical mode and greater
intuitive ,rode was observed, but did not teach statistical significance. engineers method. had greater
had anticipated.
Direct Comparison Hammond, Hamm, Grassia, and Pearson Appropriate Placement of Confidence Appropriate placement of confidence can be evaluated in terms of experts' confidence in
Page 42
whether
are they more confident in the mode in which their achievement is higher? Table 11 displays for both answer confidence and method confidence the
relation between relative confidence and relative achievement for each pair of modes (intuitive versus quasirational, intuitive versus analytical, and
quasirational versus analytical). In each of the fourcelled blocks, the upper left and lower contain the right cells
"achievement; the upper right and lower, left cells, engineers whose confidence was misplaced. For example, consider the block relating relative answer
confidence to relative achievement for the intuitive and quasirational modes. Of the thirteen engineers who had higher intuitive achievement than
confidence;
they had higher answer confidence in the intuitive mode than in the However, none of the eight engineers who achieved higher
quasirational mode.
in the quasirational mode than the intuitive mode had appropriata confidence.
.O'
C
SA
La.) WA
A <
00.
LLJ
M L~AJ
ODLn
V
rU 4( ' 0 00
V
)
U.
o
0' 1
C; A
0 A 0. L 0 0
',~
( 0j A A cl A 0* A C'l
. M4)'
i.
f,,. Ia,.' 0
*
0L
4A
0404
0 M4 1 0 . I, a. >
*
C.,
4.
t V
IC0
V .U 4. a) 002
u. C
W
4C) C
1 4
*)
V M. r.
) U . . . . ... '
4" C
r.
^4a
__ __
 0 EGI C U
. 4)
 
EUJ
4
C
0
39
4A.
X__
C0,
X.__
4.)a)
'0 cc
40
z04.)C
A
C .0A
M M A
ccE.4*aL
nU Enc
4Cr 0 M,
$.
u
0) =
(a 4) 4'
"a
(a
(D
r_
C (1) 4,
E'~EU IV 0 u 4
4 Q
0.0
<
4
L.a) A
w5 j~
OEU
CS.
4
CD
A~
0=.0~
j ed o 0~
k",,0
Li Aa.
A[ C'. If
0C
C..0 C'.L.)
4
00 C.. C.D.
aJ
S ." W
U.a).
LS %
LJ(:M~ 4[
w' x.
k
UC
Page 44
Both answer and method confidence revealed a pattern of overconfidence in the intuitive mode as compared to the quasirational mode (p < .001 and high answer
p < .01,
Most engineers also had inappropriately and method confidence (p < .01)
confidence
No pattern of overconfidence
quasirational and analytical modes, although the engineers were In fact, a null model of random confidence judgments was
the order of their confidence when the quasirational and analytical modes are compa2ed. Since the engineers received no feedback about their achievement (ra),
the appropriateness of their answer confidence and method confidence must also be evaluated in terms of knowledge (G). the intuitive The same pattern of overconfidence in
(Table 12). Summary Regarding Aprpriateness of Confidence When appropriateness of confidence is evaluated achievement in terms of empirical
(r ) most engineers were found to be (a) more confident about a their answers and also about the method used to produce their answers in the intuitive intuitive cognition. mode than in and the other two modes; in and (b) overconfident in and quasirational
cognition The
underconfident
analytical
La
Page 45
LaJ
MC) V
LUJ
C)E
LUJ
.'4 A
LUJ
64

(.) 0

400
C
La..
Ui.
*'c
A Lfl
'
C;..
00 C..)
L SA
MV % ()) (v 0 Q) 'b." 6
0'y0
___ _
0i
X_ rrsr
Q)
A CD'
A0)
lid
0K....Jwa0
DW ..
>aI
LZ0 L"U
Q$LC
4).)
(U
C~
00 0.
LU~~ U IA
U C 1 U ) u
CL) Q
'~~~~4
cm
LAJ 0 LL. 3 CD
u)
A.
.I C'.J 0)
4 rV 4J
4) C 0
u)0)L $
LUJ 0C L&a.. M
C)
A '
A.
("
A CD
,0
~ ) 0)
)0
ca
S. )J
0 4J. 4)
44A

4J
=. *04~ CU
u
a *.,
WC
A 4 C14
* 4 I
4C
0M
LOl 4I Cj
)4J
vU. CA
0 )
J rCI
0
L
EU4
0) 0 r
04J
V0
01
IC%
00 Cr 0)
4J' 'C'C
40
EnA
04). 0
0O A .0
OM
d.J.E :5 0~ C (V
EU4M 0) C U 0)+ U tv Cc
0 L *rLSL
4'I
La)
A A)
*u) +J
.0 UC.)
_y A
'0A I
14
'
4
V)OK)C=0
U
0~
'4
(A ~r.
S0r0)tC
EU..
LOC
u)(
0)
C0
S.
0:
CN.
0 S.S 3: ( 4 )
a C
Ln to
V )0.
t
CL M
CL
M0() 04='
Direct Comparison Hammond, Hamm, Grassia, and Pearson Sunwnary and Discussion In an effort to intuitive and broaden and deepen comparisons of the
Page 46
efficacy
of
analytical
cognition;
used in the study should be analytically competent by virtue of (c) the concepts of intuition and analysis, as
the task properties that induce each, should be given clear meaningin the same
(d) substantive and methodological cognition should be included study; (e) conclusions be regarding the
relative efficacy of these modes of and thus (f) of not subject to disputed not be
inductive problems
and
conclusions the
should
restricted to calculus.
logic
standard
probability
(a)
each to
subject
to
engage
in
cognition
regard
cognition
distinguishing
methodological
of empirical achievement, and (f) employing problems that do not use of the conventional probability calculus.
Direct Comparison Hannond, Ham, Grassia, and Pearson The results indicate that within these broadened conditions: 1. Achievement in a cognitive mode degree of consistency present. is greatly influenced
Page 47
by
the
quasirational cognition, even when the substantive knowledge that is analytically is relatively less correct. Conversely, the
employed
inconsistency
(low cognitive control) of intuitive cognition will lead to value, unless such inconsistency is
was
found
to
be When
empirically
knowledge.
inconsistency was removed from the engineers' judgnents, intuition and quasi rationality
analysis for most of the subjects. 3. often Analytical cognition was more often very highly accurate, yet more
inaccurate when compared with intuitive cognition for most of Large errors were often made in the analytical mode, but
the subjects.
seldom in the intuitive or quasirational modes. 4. were When performance is evaluated in terms of achievement, engineers
most
confident in the intuitive mode, not only in their answers (as However, this
'a
These results demonstrate that a cognitive continuum theory has utility: .knowledge, the it enables the findings of this
errors,
research as a whole.
judgment and decision making to the analysisinducing tasks of problem solving may also be understood of to be located along a cognitive continuum.
Specification
explain a variety of results in cognitive research and thus reduce the current isolation of research approaches. It may well turn out that, like the have previously
engineers in our study, we know more about our field than we been able to express.
:I
Direct Comparison
Hammond, Hamm, Grassia, and Pearson Reference Notes
Page 49
1. Hammond, K. R.
decision making (Center for Research on Judgment and Policy). Unpublished manuscript, University of Coloradc, 1982.
II
Adelman,
L.
substantive,
properties on the relative effectiveness of different forms of feedback in multiplecue probability learning tasks. Human Performance, Anderson, B., Deane, D., 1981, 27, 423442. K. R., McClelland, G. H., & Shanteau, J. Organizational Behavior and
Hammond,
Definitions, sources,
Praeger, 1981.
Policy conflict as a function of policy similarity and policy Scandinavian Journal of Psychology, 1972, 13, 208221.
complexity. Brehmer, B.
conflict.
Brehmer, B.
Journal of Psychology., 1979, 21. 209221. Brehmer, B., & Hammond, K. R. Cognitive factors in interpersonal conflict. Socialpsychological perspectives.
"Brooks, L.
In
New York:
Direct Comparison Hammond, Harm, Grassia, &nd Pearson Bruns'ik, E. The conceptual framework of psychology. 10).
Page 51
Perception and the representative desgn of psychological Berkeley: University of California Press, 1956. The
1983, 220,
261268. Edwards, W. Bayesian and regression models of human information processinga myopic pe'spective. 0 Performance, 1971, 6, 639648. Behavioral decision theory: 1981, Processes of 32, 5388. Organizational Behavior and Human .
Einhorn, H. J., & Hogarth, R. M. P judgment and choice. Einhorn, H. J., & Hogarth, Braunstein (Eds.),
In G. Ungson & D.
Decision ,making:
An interdisciplinary inquiry.
Boston:
Fisch, H. U.,
Wadsworth, 1982.
Hammond, K. R., & Joyce, C. R. B. On evaluating the severity British Journal
of depression: of Psychiatry,
An experimental 1982,
study of psychiatrists.
140, 378383.
mmm
Direct Comparison
Hammond, Hanun, Grassia, and Pearson Fox, J. Making decisions uwder the infltuence of memory.
Page 52
Lsychological
Review, 1980, 81, 190211. Gillis, J. S. Interpersonal conflict: Effects of chlorpromazine and In K. R. Hammond & C. R. B. Theory and
Joyce (Eds.), Psychoactive drugs and social judgment: research. Gillis, J. S. New York: Wiley, 1975.
Sources of judgmental impairment in paranoid and nonparanoid Journal oF Abnormal Psychology, 1978, 87, 587596.
Representative versus systematic design in clinicai Psychological Bulletin, 1954, 51, 150159. Probabilistic functioning and the clinical met~iod.
Psychological Review, 1955, 62(4), 255262. Hamn, ond, K. R. (Ed.). The psychology of Egon Brunswik. New York: Holt,
Rinehart, & Winston, 1966. Hammond, K. R., & Brehmer, B. Quasit'ationality and distrust: Implications qP1
In L. Rappoport & D. Sunmmers (Eds.), Human interaction. New York: Holt, Rinehart, and Winston,
& Mumpower, J.
Direct Comparison Hammond, Hamm, Grassia, and Pearson Hammond, K. R., Stewart, T. R.,
Page 53
.0
judgment theory.
In M. F. Kaplan & S. Schwartz (Eds.), Human judgntmerit New York: Academic Press, 1975. Psychological Review,
& Summers, D. A.
Cognitive control.
1972, 79, 5867. Hammond, K. R., & Wascoe, N. E. (Eds.). social and behavioral science: design. Hastie, R. 511542. Holzworth, J. Intervention in a cognltive conflict. in press. A theoretical analysis of insight into a Readings in Organizational Behavior San Francisco: Social inference. New directions for methodology of
A comparison
of Piagetian and judgment theory. Psychology, Kahneman, D., 1982, 34a, 4/9488. & Tversky, A.
Slovic, P.,
New York:
.....
.....
....
.... ....
....
....
....
........
....
...
7,
'

'.

7'
Direct Comparison Hammond, Hamm, Grassia, and Fearscn Kahneman, D., Coqnit.ion, & Tversky, A. 1982, 11, Oi the study of statistical intuitions.
Page 54
The logical foundations of statistical inference. D. Reidel, & Smokler, H. Wiley, 1980. 1974. Studies in subjective probabi!lit
(2nd ed.).
New
Larkin, J., McDermott, J., Sinon, D. P.. & Simon, H. A. performance in solving physics problems. 13351342. Levi, I. The enterprise of knowledge. Cambridge, MA.
Massachusetts
Institute of Technology Press, 1980. Lichtenstein, S., probabilities: Fischhoff, P., & Phillips, L. The state of the art to 1980. Calibration of In
).
Kahneman, P. Slovic
& A. Iversky (Eds.), judgment under uncerta;nt,: Cambridge, England: Luce, R. D. Oskamp, S.
Journal of Corsulting
Page 55
reconsidered.
In M. F. Kaplan & S. Schwartz (Eds.), Humen judgment and decision processes. 0 New York: Academic Press, 1975. & Lichtenstein, S. Behavioral decision theory.
approaches to the study of information processing judgment. Organizational Behavior and Human Performance, 1971, 6, 649744. Tversky, A. Elimination by aspects: 79, 281299. & Mynert, C. (Eds.). On scientific thinking. New A theory of choice. Psychological
Review, 1972,
9.
ill
.................................................
"'
"
"__"
"_
"L "
L'
". '
"x
'.
,".
."
'
V'
"'
Direct Comparison  Appendix A Hammond, Hamnm, Grassia, and Pearson APPENDIX A Characteristics of Engineers' FQrmulas for
Page 1
Eleven engineers developed formulas that involved only additive combinations of the cues (median ra = .467). linear functions of the cues (median ra Four of these involved only 281). For example:
Accident Rate = 12.8(23  (SWIDTH + LWIDTH))/15 + 12.8(OBSPM + CURVEPM + INTPM)/29 + 6.4(TRAFMIX + PCTNPZ)/115 One expressed nonlinear functions with graphs (ra = .518); six used tables to express nonlinear functions of cues (median ra = .496). Another engineer used only a multiplicative combination of variables (ra .670):
Accident Rate
= .25(Fs*
FLW FL*
*
FNpZ F~p
*
FGR FGR
FIPM FIP
FOPM
*
FASL
TV)
where each F is a onedimensional table expressing a linear or nonlinear function of the cue. The remaining nine engineers produced formulas that involved a hierarchy
*of
combined a subset of factors in a table and then added the result together with other factors. Five of these hierarchical formulas used adding and For example:
Accident Rate = CURVEPM + (60  AVESL)/1O (LWIDTH + SWIDTH  23)/2 + (TRAFVOL/10000)(INTPM + OBSPM  3) + (TRAFMIX * GRADE * PCTNPZ)/10000 Three engineers used adding as well as tables (median ra
r used multiplying and tables (ra .641). aa
.390).
One
0
Direct Comparison  Appendix B Hammond, Ham, Grassia, and Pearson APPENDIX B Maximal Guidance Procedure
Page 1
This procedure was designed to give the engineer a maximum of guidance in constructing his formula for safety in order to prevent the commission of minor errors and to ensure that his formula adhered to principles of measurement theory with which he might not be familiar. consisted of the following steps: The procedure 0

Specify the answer scale. Specify the scale for each input dimension and its overall relation to the answer scale, and identify possible interactions with other
dimensions.

Group the input dimensions according to their redundancy, similarity, or mutual interactions.

Express the formula as a hierarchy of groups of variables. Determine what organizing principle should be used at eacn level of hierarchy.


Specify the function form governing each dimension's input to its organizing principle.

98
"I,,
lnm=ii
:i
'9
Page 2
The engineer was guided through these steps by a series of forms which contained instructions for the steps and choice points, and on wnich intermediate steps were recorded. Two examples follow. The engineer also
received detailed tutorials about interactions and organizing principles as part of the maximal guidance procedure.
kit., .
Direct Comparison  Appendix B Hammond, Hamu, Grassia, and Pearson Form 1: Answer Dimension Form
Page 3
Task
.
_.
High
A "natural V" on a scale means that when it is called "0" there really is NONE of the quality being measured. If you had a natural 0, then it would make sense to say that an "8" is twice as much of a thing as a "4"; but if the 0 was arbitrary, it wouldn't have that sort of meaning. For example, if as if you had $5000, But 32 degrees F is temperature scale is natural 0. you have savings of $10,000, you have twice as much money because 10000 is twice 5000. Here the $0 is a natural 0. not twice as warm as 16 degrees F, because the 0 on the picked arbitrarily. In other words, it does not have a
Yes No .____
It is useful, when considering numbers that measure a dimension, to ask whether the intervals between the numbers have consistent meaning, or whether the numbers simply express order. For example, is the difference between a 1 and a 2 the same as the difference between an 11 and a 12? In the above measures of money or temperature, the intervals do have consistent meaning. However if we were to assign numbers to grades on a test, where A = 1, B = 2, C F 3, D = 4, E = 5, and F = 6, the interval between 2 and 4 would be different in meaning from the interval between 4 and 6. All the numbers convey is that A is better than B, etc. Do the intervals in the answer scale have a consistent meaning?
Yes No .
p.. i
S.
. .
.. .p.... .
Direct Comparison  Appendix B Haimond, Haemm, Grassia, and Pearson Form 3: Choice of Organizing Principle
0N
Page 4
This form is for use in deciding what organizing principle to use for producing either the final answer or an intermediate product to be plugged in at a higher level in the hierarchy. Output. Is this the top level, producing the final answer?
If so, what are the units of the final answer? What is its range? Low
_
Yes
No
High
.____
If this is not the top level, then the output of this organizing principle will be input for an organizing principle at a higher level.
What organizing principle is used at the next higher, level? What kind of itiput does it require?
,
.___
Units
___"
Range:
Lowest point
Highest point
;_
Organizing Principle. What organizing principle do you want to use here? (Refer to Sheet 2 for guidance in your choice, and to the Forms 2i and 2g that you used to describe these dimensions to see what kinds of interaction they have with each other.)
Check one: Averaging
.
Multiplying
Table (Configural)
Other
0A
Direct Comparison
Appendix C
Page 1
Hammond, Hamm, Grassia, and Pearson APPENDIX C Independent Confirmation of the Quasirational Properties of the SafelyJudgment Task
Evidence that the safetyjudgment task was not fully susceptible to analytical treatment can be seen in the following examples of spontaneous comments by engineers during their attempts to construct a formula:
General. "You kind of have to be able to juggle them all in your head, you know, in order to assign a weight to them." "I kind of understand what I did." "This is too hard! Can I form a committee?"
"I haven't the vaguest idea how this is going to come out."
"On the one hand, the lane width and shoulder width lends itself to giving the drivers a sense of security pause perhaps a sense of comfort on the road pause where the geometry is pause not like that. I don't know what to do with those." Oh boy,
,.1
.:
Direct Comparison. Appendix C Hammond, Hamm, Grassia, and Pearson Subject attempts to use average speed limit. (1)
Page 2
"But whether or not, you would have more accidents per mile typically on something that is 30 miles an hour or 60 wiles an hour is a little bit * ambiguous, and I don't know that I cculd really properly consider that. My inclination is to leave it out for the time being. out or conversely assume that it * Either you leave it I
(2)
really difficult for me to get a feel for how that influences So rather than trying to guess, I just left it off,
highway safety.
recognizing that, you know, in a further generation or a further refinement of the formula, you might be able to come to some conclusion about the effect and include it." I"4
7. 7
LUirect Comparison
Hammond, Hamm,
Page I
APPENDIX D Notes on Think Aloud Transcripts Engineer #1 organizing principlehas feel for multiplicative relationship but in end he merely rank orders cues and assigns weightsadditive %NPZhe finds this cue ambiguous, but finally resolves it as diagnostiche calls it "inverted", "reciprocal" (implying its function form), but then in his formula he uses "I00  %NPZ" (complement)he does not see this cue as that important ASLalso seen as ambiguousleans toward using it predictively, but then leaves it out of the formula because he is unsure how to use ithe mentions that, with regard to ASL, what is more dange~rous is a high
differential, and a low posted speed limit may increase this (some cars
obey the low SL while other cars disregard what they may see as unnecessarily low) [not enough information for engineer to know the characteristics of a particular highway regarding their influence on ASL; also, he is unaware of how to express such a relationship in a generalized formula; therefore, cue is simiply left out of formula] indicates possibilities for a more complex organizing principle (tilian additive), i.e., nonlinear  TV2  "but need data to make that judgment" Engineer #5 Engineer #5 initially breaks accidents down into the subcategories of property damage, injury, &nd fatalityelenents affect different types of accidents; these can combine differently to get the same accident
the relationship of TM to accidents is geometric (logarithmic?)"O5%, 610%, 1120%, 2135%" each expresses a range of ratings that are "about the same"
~
... .
Page 2
he aims for a norm/baseline as he develops his formula and attempts to produce results within the desired range. When the results are slightly off, he goes back to one of his matrices and adjusts the "range" for SW [actually he has adjusted the beta weight of this cue in his equation] Interview: how would he use historical research (which he mentions he would like to consult if he could)in the past he has "taken an average rate and modified it based on whether you provide certain features or detract from those features"; he would look at how "features stack up and influence rates" he indicates that he knows he would use a regression equation if the data he had
he also mentions that if the data were available, the initial three categories (property damage, injury, fatality) would be kept separatee.g., some barriers may reduce fatalities yet increase property damage. Therefore, each individual equation should be different for the three categoriesof these, the property damage formula is the simplest, but the other two are not as high(?} as property damage, and may be insignificant on the same scale (especially fatalities), causing the formula to be basically a property damage formula (which,, again, is simpler) he indicates a definite awareness of multiplicative relationships confidence in method"I would hesitate to create an equation for all roadway conditions because you just have to take a look at where the roadway is" capacity was easier for himwas well researched, timespace relationship rather than individual characteristics that could influence (because of variance?) Engineer #8 also starts with optimum value of each characteristic %NPZ"skrewed" relationship"can't get a handle on how to get that incorporated into this (needs to use a matrix?) uses trial and error to obtain the relationship (function) which "looks" right (intuitive feel for relationship?) converts all ranges to a 110 scale based on an optimum which may not be an end point SPD60 set as optimum irregardless of other factors [no interactions]
..
~/ i
...
_:
....
_ .
I....l/
Page 3
OPMrange (018) is too complicated to calibrate so "satisfices" by spreading 11lu ratings out along the range; therefore some cue values have the same rating (arbitrary?)i.e., 12 aria 13 both equal a rating of 10. uses a weighted average organizing principle he thought safety "was going to be easy because it would fit into one of the slots like I did before, but it doesn't." (He had done both aesthetics and capacity)he discovers that he cannot just subtract off the number of cars for each limiting factor like in capacity he observes that this is like most any formula"it gives you some kind "of relativity of the things." with regard to the memory task, he says there is "no way in the world" he could remember his weighting system (how analytic was he? or does this generalization hold?) Engineer #10 prioritizes cues first and then arranges them in a linear additive "manner, but not weighted average. originally ranks cues by class (cross section, geometry, traffic) and then give subrankings, but then in the weighting process each cue is examined independently and the overall rankings are altered [no cue interactions] realizes he has to normalize cue ranges, possible cue value on a scale from 110 i.e., give ratings to each
6'
he forgets that his answer should fall between 032initially comes up with a range in the hundreds, but he is unsure of the endpointshe says one could work them out . After trying out his formula on three examples, he realized that he had framed the wrong relationships, i.e, most of the cues had a negative relation (correlation) with safety rather than a positiv one. He compensated for this by switching the scale to low=safe/high=unsafe and then readjusting the two positively related cues to reflect this shift. All of his cues were used predictively except for %NPZ, which he regards negatively related to safety [and could be diagnostic or predictive%NPZ could indicate an unsafe road or it could frustrate drivers and cause more accidents (both negativef why low confidence? (5)"not enough thought" put into itwould like to test over a range of highways "knowing the data". * memory taskHe has problems remembering all the cues, much less the numbers he assigned...
* *as
"0
I .0
Direct Comparison  Appendi;x D Hammond, Hamm, Grassia, and Pearson Page 4
_En_ nee r #12 begins to combine cues even as he reads their definitions and ranges, e.g., %NPZ/TV/TM approaches safety rating from a "relative" standpoint since the chances of something being zero rare] slim" [answer for safety is on a relative scale though cue, are presented with their absolute ranges for The sampleexample of prior knowledge interferring with an analytic approach?] changes cues to relative scales alsoi.e., (LW+SW) ranges from 2 to +2 9.
wants an interaction between %NPZ and TM but their multiplication can produce a zero if one term is a zero wants to say, that if both %NPZ and TM are high, there is an interaction [nonlinear] mentions "duplication" (compare Engineer #5's complaint about "redu~idancy")have no passing on steep grades and curves, "so it's all kind of melded in together and somehow we're duplicating each other" "intuitively" knows the kind of relationships he wants to express, i.e., what kind of an effect a certain cue interaction should have on the overall safety rating, but cannot express itgoes through trial and error he winds up with a variable in the denominator that could be zerohe responds by saying "I won't allow it to be zero". * he constantly tries out the formula he has so farto make sure it sounds right and nothing is too far off (dealing with more concrete examples) he wants his worse case to be 32, so in weighting he thinks in approximate percentages of 32 SPDLdiagnostic"the lower the speed limit is, the worse the situation produces a so that to the in brackets
is."
ultimately, his strategy is to weight each "group", i.e., he "traffic number", by normalizing the ranges of involved cues they add up to a percentage of 32. [this comes out linear in spite of his earlier expression as nonlinearityhe tries to show some relationship by grouping and multiplying or diving by a common factor]
oI
Page 5
SPDLnot weighted as heavily because "in setting the speed limit, it considered all of the other things that were taken into consideration" [diagnostic; but he does not see it as an interaction here between SPUL and other cues] during the discussion he expresses an interaction between LW and OPM that he did not include in ,is formula some cues can be plugged directly into his formula (e.g., IPM), while others (e.g., SW) have been converted to a different scalenegative, neutral, or positive [SW, as he expresses it verbally, may be thirdorder relationship] Engineer #15 he recognizes that the safety rating must be in terms of million vehicle miles travelled and so TV will be a component of the rating rather than strictly a factor wants to develop weights that would predict the number of accidents which would then be correlated with the required tange (032) [he asks "can I do that?"] uses TV on both sides of his equation (now using as a factor)
40
he sees the complexity of the task but attempts to "keep it simple" weighting factors were the "factors of importance", which then had to be put in the same "context" (normalized) by giving them a safety ratingproduced a "relative safety value" (actually he just changed the range on each cue to 15) but relationships are not quite linear %NPZdiagnosticthe higher the percentage, the worse the condition
SPDpredictivethe lower the speed limit, the safer multiplies ranges by weighting factors to amplify them assumes a linear relationship between cues because he has "built any curve aspects into the judgments here" (the cuecriterion relationships) he comments that this empirical formula is probably too complicatedafter trying it out, one would probably find that he could get just as good results by dealing with four or five fectors rather than all of the dimensions. He nevertheless used all of them in his formula "because there was a judgment built into this whole thing in the first place that these factors were important."
.!.
Page 6
places TV in the denominator to give "a means of comparing different length segments" and "differing volumes"what he doesn't realize is that the rating is already in terms of million vehicle miles travelled (or maybe he just forgets this) in going over his steps he recalls wanting to "use a system where I come up with an intuitive judgment about the safety of the thing as judged by the dimensioTnsjgven' [underlining ours] "if you're going to have a practical formula, you've got to have a way where people can use it quickly." [In the process of developing a "Npractical formula" that can be applied easily does one oversimplify complex relationshipsthereby reducing validity? (and accuracy?)]
'9..
0.
""",.
'ar
Direct Comparison  Appendix E Ha'immond, Hamm, Grassia, and Pearson APPENDIX E Com'parison of Repeated Trials Reliability with Cognitive Control
Page 1
Fourteen of the engineers had higher Rc s than Rs 's in the intuitive mode. This finding suggests that these engineers were using information in a and/or that the pictorial presentation contained information
noniinear way,
not present in the ten cues provided in the bargraph and formula presentations. Both possibilities were investigated, and neither can be ruled The evidence suggests, however, that (a) the engineers
did not make systematic or substantial use of information in a nonlinear fashion; (b) in the filmstrip condition they occasionally used information (c) the engineers' occasional use of
additional information in the intuitive task does not account for the higher repeated trials reliabilities. The contribution of the nonlinear component of the Lens Model Equation was investigated to determine whether a significant relation existed between the portions of the variances of the criterion and the judgments that were not explained by their respective linear models. nonlinear knowledge was found. component (C 1  R_2,1 e the quasirational mode.

Rs
it
Page 2
That some engineers in the intuitive condition made use of information O present only in the film strips, at least on some trials, is almost certainly true. For instance, residential driveways entering the highway were evident
and could have been taken into account. The most plausible explanation for the higher repeated trials reliability in the filmstrip condition than in the bargraph condition is that the engineers recognized the repeated film strips and recalled their previous answers. There are two reasons for accepting this explanation: (a) it was
obvious to the engineers that they were viewing a film for a second time, and some of them remarked that they remembered their previous answer; (b) whereas
repeated trials reliability was higher than multiple R in the filmstrip condition, it was higher for only two engineers in the bar graph condition; and no engineer in this condition indicated that he recognized a repeated trial. In short, the fact that fourteen of the 21 engineers had higher
repeated trials reliabilities in the filmstrip condition is almost certainly the result of an artifact of memory.
'muJ
'U.ii>
'
Appendix F
Page I
Confidence Questionis
Answer Confidence.
confident are you that your formula will work well with any particular 2lane rural Colorado highway?" once after producing the formula and again at the end of the session. The mean of his responses is the measure of his answer 0
confidence in the analytical mode. Method Confidence. In the intuitive and quasirational modes i e mean of
the confidence ratings on the following three questions from the Self Report form is the measure of method confidence:
1. How confident are you that your method for making these safety judgments is correct? 2. 3. How well does this presentation help you make safety judgments? How accurate do you think one could be when making these judgmeitt in this way? The measure for analytical method confidence is the mean of the ratings on the following questions:
1. How confident are you that your formula is correct?
(answered at two
different times) 2. .4
4 .
How accurate do you think a formula for judging the capacity of highways can be?
,.
'
'..
**.?.L."77
OFFICE OF NAVAL RESEARCH Engineering Psychology Group TECHNICAL REPORTS DISTRIBUTION LIST
OSD CAPT Paul R. Chatelier Office of the Deputy Under Secretary of Defense OUSDRE (E&LS) Pentagon, Room 3D129 Washington, D. C. 20301 Dr. Dennis Leedom Office of the D~puty Under Secretary of Defense (C I) Pentagon Washington, D. C. 20301 Department of the Navy * Engineering Psychology Group Office of Naval Research Code 442 EP Arlington, VA 22217 (2 cys.) Aviation & Aerospace Technology Programs Code 210 Office of Naval Research 800 North Quincy Street Arlington, VA 22217 Communication & Computer Technology Programs Code 240 Office of Naval Research 800 North Quincy Street Arlington, VA 22217 Physiology & Neuro Biology Programs Code 441NB Office of Naval Research 800 North Quincy Street Arlington, VA 22217
Department of the Navy Tactical Development & Evaluation Support Programs Code 230 Office of Naval Research 800 North Quincy Street Arlington, VA 22217 Manpower, Personnel & Training Programs Code 270 Office of Naval Research 800 North Quincy Street Arlington, VA 22217 Mathematics Group Code 411MA Office of Naval Research 800 North Quincy Street Arlington, VA 22217 Statistics and Probability Group Code 411S&P Office of Naval Research 800 North Quincy Street Arlington, VA 22217 Information Sciences Division Code 433 Office of Naval Research 800 North Quincy Street Arlington, VA 2217 CDR K. Hull Code 230B Office of Naval Research 800 North Quincy Street Arlington, VA 22217
'9n
Department
of the Navy
Dr. Robert G. Smith Office of the Chief of Naval Operations, OP987H Personnel Logistics Plans 20350 Washington, D. C. Dr. Andrew Rechnitzer Office of the Chief of Naval Operations, OP 952F Naval Oceanography Division 20350 Washington, D. C. Combat Control Systems Department Code 35 Naval Underwater Systems Center Newport, RI 02840
Naval Research Laboratory Technical Information Division Code 2627 20375 Washington, D. C. Dr. Michael Melich Communications Sciences Division Code 1500 Naval Research Laboratory 20375 Washington, D. C. Dr. J. S. Lawson Naval Electronic Systems Command NELEX06T "Washington, D. C. 20360 Dr. Robert E. Conley Office of Chief of Naval Operations Command and Control OP094H 20350 Washington, D. C.
Training Analysis and Evaluation Group Orlando, FL 32813 CDR Norman E. Lane Code N7A Naval Training Equipment Center Orlando, FL 32813 Dr. Gary Poock Operations Research Departmen', Naval Postgraduate Schoo. Monterey, CA 93940 Dean of Research Administration Naval Postgraduate School Monterey, CA 93940 %P Mr. H. Talkington Ocean Engineering Department Naval Ocean Systems Center San Diego, CA 92152
"CDR Thomas Berghage Naval Health Research Center San Diego, CA 92152
!I.
Dejpartment of the Navy Mr. Paul Beckman Naval Ocean Systems Center San Diego, CA 92152 Dr. Ross Pepper Naval Ocean Systems Center Hawaii Laboratory P. 0. Box 997 Kailua, HI 96734 Dr. A. L. Slafkosky Scientific Advisor Commandant of the Marine Corps Code RDi Washington, D. C. 20380 Dr. L. Chmura Naval Research Laboratory Code 7592 Computer Sciences & Systems 20375 Washington, D. C. HQS, U. S. Marine Corps CCA40 (Major Pennell) ATTN: 20380 Washingtcn, D. C. Commanding Officer MCTSSA Marine Corps Base Camp Pendleton, CA Chief, C Division Development Center MCDEC Quantico, VA 22134 Human Factors Technology Administrator Office of Naval Technology Code MAT 0722 800 N. Quincy Street Arlington, VA 22217 Commander Naval Air Systems Command Human Factors Programs NAVAIR 334A 20361 Washington, D. C.
Department of the Navy Commander Naval Air Systems Command Crew Station Design NAVAIR 5313 Washington, D. C. 20361 Mr. Philip Andrews Naval Sea Systems Command NAVSEA 03416 20362 Washington, D. C. Commander Naval Electronics Systems Command Human Factors Engineering Branch Code 81323 20360 Washington, D. C. Larry Olmstead Naval Surface Weapons Center NSWC/DL Code N32 Dahlgren, VA 22448 Mr. Milon Essoglou Naval Facilities Engineering Command R&D Plans and Programs Code 03T Hoffman Building II Alexandria, VA 22332 Captain Robert Naval Medical Code 44 Naval Medical Bethesda, MD Biersner R&D Command Center 20014
92055
Dr. Arthur Bachrach Behavioral Sciences Department Naval Medical Research Institute Bethesda, MD 20014 Dr. George Moeller Human Factors Fngineering Branch Submarine Medical Research Lab Naval Submarine Base Groton, CT 06340
Department of the Navy Head Aerospace Psychology Department Code L5 Naval Aerospace Medical Research Lab Pensacola, FL 32508 Commanding Officer Naval Health Research Center San Diego, CA 92152 Commander, Naval Air Force, U. S. Pacific Fleet ATTN: Dr. James McGrath
Department of the Navy Dean of the Academic Departments U. S. Naval Academy Annapolis, MD 21402 Dr. S. Schiflett Human Factors Section Systems Engineering Test Directorate U. S. Naval Air Test Center Patuxent River, MD 20670 Human Factor Engineering Branch Naval Ship Research and Development
*[
"
Naval Air Station, North Island San Diego, CA 92135 Navy Personnel Research and Development Center Planning & Appraisal Division San Diego, CA 92152
Dr. Robert Blanchard Navy Personnel Research and
Center, Annapolis Division Annapolis, MD 21402 Dr. Hairy Crisp Code N 51 Combat Systems Department Naval Surface Weapons Center
Dahlgren, Mr. VA 22448 John Quirk
Development Center
Command and Support Systems San Diego, CA 92152 CDR J. Funaro Human Factors Engineeing Division Naval Air Development Center
Warminster,
PA
18974
Monterey,
CA
93940
Mr. Stephen Merriman Human Factors Engineering Division Naval Air Development Center Warminster, PA 18974 
Washington,
D. C.
20310
:1
Department of the Navy Dr. Edgar M. Johnson Technical Director U. S. Army Research Institute 5001 Eisenhower Avenue Alexandria, VA 22333 Director, Organizations and Systems Research Laboratory U. S. Army Research Institute 5001 Eisenhower Avenue
Foreign Addressees Dr. Kenneth Gardner Applied Psychology Unit Admiralty Marine Technology Establishment Teddington, Middlesex TWIl OLN England Director, Human Factors Wing Defence & Civil Institute of Environmental Medicine
!'4
Alexandria,
VA
22333
Technical Director U. S. Army Human Engineering Labs Aberdeen Proving Ground, MD 21005
TX
78235
Other Organizations Dr. Robert R. Mackie Human Factors Research Division Canyon Research Group 5775 Dawson Avenue Goleta, CA 93017 Dr. Amos Tversky Department of Psychology Stanford University Stanford, CA 94305 Dr. H. McI. Parsons Uuman Resources Research Office 300 N. Washington Street Alexandria, VA 22314 Dr. Jesse Orlansky Institute for Defense Analyses 1801 N. Beauregard Street Alexandria, VA 22311 Professor Howard Raiffa Graduate School of Business
Administration Harvard University Boston, MA 02163 Dr. T. B. Sheridan Deparu.ment of Mechanical Engineering Massachusetts Institute of Technology Cambridge, MA 02139 Dr. Arthur I. Siegel Applied Psychological Services, 404 East Lancaster Street Wayne, PA 19087 Dr. Paul Slovic Decision Research 1201 Oak Street Eugene, OR 97401 Dr. Harry Snyder Department of Industrial Engineering Virginia Polytechnic Institute and State University Blacksbtrg, VA 24061
Other Organizations Dr. Ralph Dusek t.'ministrative Officer Scientific Affairs Office American Psychological Association 1200 17th Street, N. W. Washington, D. C. 20036 Dr. Robert T. Hennessy NAS  National Research Council (COHF) 2101 Constitution Avenue, N. W. Washington, D. C. 20418 Dr. Amos Freedy Perceptronics, Inc. 6271 Variel Avenue Woodland Hills, CA 91364 Dr. Robert C. Williges Department of Industrial Engineering and OR Virginia Polytechnic Institute and State University 130 Whittemore Hall
Blacksburg, VA 24061 Dr. Meredith P. Crawford American Psychological Association Office of Educational Affairs 1200 17th Street, N. W. 20036 Washington, D. C. Dr. Deborah BoehmDavis General Electric Company Information Systems Programs 1755 Jefferson Davis Highway Arlington, VA 22202 Dr. Ward Edwards Director, Social Science Research Institute University of Southern California Los Angeles, CA 90007 Dr. Robert Department Vanderbilt Nashville, Fox of Psychology University TN 37240
Inc.
.,1
Other Organizations Dr. Charles Gettys Department of Psychology University of Oklahoma 455 West Lindsey Norman, OK 73069
Other Organizations Dr. Babur M. Pulat Department of Industrial Engineering North Carolina A&T State University Greensboro, NC 27411 Dr. Lola Lopes Information Sciences Division Department of Psychology University of Wisconsin Madison, WI 53706 Dr. A. K. Bejczy Jet Propulsion Laboratory California Institute of Technology Pasadena, CA 91125 Dr. New Box Las Stanley N. Roscoe Mexico State University 5095 Cruces, NM 88003
Dr. Kenneth Hammond Institute of Behavioral Science University of Colorado CO 80309 SBoulder, Dr. James H. Howard, Jr. Department of Psychology Catholic University 20064 Washington, D. C. Dr. William Howell Department of Psychology Rice University Houston, TX 77001
SDr.
Christopher Wickens Department of Psychology University of Illinois Urbana, IL 61801 Mr. Edward M. Connelly Performance Measurement Associates, Inc. 410 Pine Street, S. E. Suite 300 Vienna, VA 22180 Professor Michael Athans Room 35406 Massachusetts Institute of Technology Cahbridge, MA 02139 Dr. Edward R. Jones Chief, Human Factors Engineering McDonnellDouglas Astronautics Co. St. Louis Division Box 516 St. Louis, MO 63166
Mr. Joseph G. Wohl Alphatech, Inc. 3 New England Executive Park Burlington, MA 01803 Dr. Marvizi Cohen Decision Science Consortium Suite 721 7700 Leesburg Pike Falls Church, VA 22043 Dr. Wayne Zachary Analytics, Inc. 2500 Maryland Road Willow Grove, PA 19090 Dr. William R. Uttal Institute for Social Research University of Michigan Ann Arbor, MI 48109 Dr. William B. Rouse School of Industrial and Systems Engineering Institute of Technology Atlanta, GA 30332
SGeorgia
'9A
Other Organizations Dr. Richard Pew Bolt Beranek & Newman, 50 Moulton Street Cambridge, MA 02238 Dr. Hillel Einhorn Psychological Documents (3 copies) ATTN: Dr. J. G. Darley N 565 Elliott Hall University of Minnesota Minneapolis, MN 55455
Inc.
A%
Graduate School of Business University of Chicago 1101 E. 58th Street Chicago, IL 60637 Dr. Douglas Towne University of Southern California Behavioral Technology Laboratory 3716 S. Hope Street Los Angeles, CA 90007 Dr. David J. Getty Bolt Beranek & Newm an, 50 Moulton street Cambridge, MA 02238
Inc.
Dr. John Payne Graduate School of Business Administration Duke University Durbsin, NC 27706 Dr. Baruch Fischhoff Decision Research 1201 Oak Street Eugene, OR 97401 * Dr. Andrew P. Sage School of Engineering and Applied Science University of Virginia Charlottesville, VA 22901 Denise Benel Essex Corporation "333 N. Fairfax Street Alexandria, VA 72314
7,
Mult mai mult decât documente.
Descoperiți tot ce are Scribd de oferit, inclusiv cărți și cărți audio de la editori majori.
Anulați oricând.