CATEGORY E Product Shipment, Commercial Soft-

ware, and Usage--Software Devel-

opment and Maintenance Aids, and
Applications Software

This category includes software-based critical tools Human Aspects H. Ledgard

for implementing military systems, the CAD/CAM of Computing Editor
systems, the critical software development tools de-
scribed in subsection 3.2, and the many commercial ap-
plication packages that have absolutely no military
Studying Programmer
significance. Therefore, the CNCTEG recommends that
this category be controlled initially, and that a high
Behavior
priority effort be placed on further subdividing it so that
many of the products in the category may be freed from
Experimentally: The
control, thus reducing the administrative workload re-
lated to export licenses for software having no military
Problems of Proper
significance.
Methodology
Ruven E. Brooks
CATEGORY F Product Shipment, Commercial Soft- The University of Texas Medical Branch,
ware, and Usage--Widely Available
Galveston, Texas
Operating Systems, Network Ori-
ented Software, Database Manage-
ment Software, and User Interface
The application of behavioral or psychological
Software
techniques to the evaluation of programming languages
and techniques is an approach which has found
The CNCTEG recommends that no export controls
increased applicability over the past decade. In order to
be applied to this category. The product shipment--
use this approach successfully, investigators must pay
commercial software category consists of products that
close attention to methodological issues, both in order
are widely available throughout the free world in easily
to insure the generalizability of their f'mdings and to
replicable form. In the case of usage, the products are
defend the quality of their work to researchers in other
not actually transported into the communist country.
fields. Three major areas of methodological concern,
Some controls may be applied indirectly in this area as
the selection of subjects, materials, and measures, are
a part of Category B. Additional work is recommended
reviewed. The first two of these areas continue to
to remove these controls when practical.
present major difficulties for this type of research.
In general, the CNCTEG believes that if a copy of a
Key Words and Phrases: psychology of
generally available program is known to be available in
programming, software psychology
the communist countries, whether critical or not, then no
CR Category: 4.6
great value lies in controlling the export of additional
copies.

Received 10/79; revised 11/79; accepted 12/79

References
1. Brown, H. Interim DoD Policy Statement on Export Control of
United States Technology. Office of the Secretary of Defense, Aug.
26, 1977.
2. Computer Network Critical Technology Expert Group (F.R.
Spitznogle, Chairman). Computer networks: An assessment of the
critical technologies and recommendations for the controls on the
exports of such technologies. Final rep., prepared for the U. S.
Department of Defense, April 30, 1979.
3. Dale, A.G. Database management systems development in the
USSR. Comptg. Surv. 11, 3 (Sept. 1979), 213-226. Permission to copy without fee all or part of this material is
4. Davis, N.C. and Goodman, S.E. The Soviet blocs unified system granted provided that the copies are not made or distributed for direct
of computers. Comptg. Surv. 10, 2 (June 1978), 93-122. commercial advantage, the ACM copyright notice and the title of the
5. Defense Science Board Task Force (J.F. Bucy, Chairman). An publication and its date appear, and notice is given that copying is by
analysis of export control of U.S. technology--a DoD perspective. permission of the Association for Computing Machinery. To copy
ODDR&E, Feb. 4, 1976. otherwise, or to republish, requires a fee and/or specific permission.
6. Goodman, S.E. Software in the Soviet Union: Progress and Author's present address: R.E. Brooks, Dept. of Psychiatry and
problems. Advances in Computers 18 (1979), 231-287. Behavioral Sciences, The University of Texas Medical Branch, Gal-
7. U.S. Congress. Public Law 96-72, The Export Administration veston, TX 77550.
Act of 1979, Sept. 29, 1979. 1980 ACM 0001-0782/80/0400-0207 $00.75.

207 Communications April 1980

of Volume 23
the ACM Number 4
I. Introduction
Over the past decade, a number of studies have
appeared which use behavioral or psychological meth-
odologies to evaluate programming language constructs
and techniques; some examples include Boies and Gould
[11, Gould and Drongowski [7], Shneiderman et al. [18],
Shneiderman [19], and Sime et al. [20, 21, 22]. Studies of
this kind involve direct observation and manipulation of
the behavior of individual programmers, in contrast to
studies on programming products [11, 14] or on the social
dynamics of programming groups [23].
Such studies have a large potential for impacting the
construction, selection, and use of programming tools
and techniques in a number of different ways. First, they
may serve to affirm or refute behavioral assumptions
underlying particular programming concepts. As an ex-
ample, the work of Sime et al. [22] lends support to the
claim that for nonprogrammers nesting is superior to
jumps as a method for representing conditionals. Often,
such work may lead to further refinement or explication
of the behavioral assumption; in the study just men-
tioned, scope markers for nesting which carded redun-
dant information were found to be superior to those
which only marked the scope end.
A second impact that behavioral studies may have is
the provision of quantitative information on the relative
effectiveness of programming techniques. The work of
Sackman [15] on time-sharing versus batch processing
provides comparative figures on labor and hardware
costs for each operating mode, as well as showing that
the differences were small compared to individual dif-
ferences. Similarly, the work of Shneiderman et al. [18]
provides an order-of-magnitude estimate of the benefits
of flowcharts.
Finally, behavioral studies may provide verification
of explicit models of programmer behavior [2, 19]. Once
such models are verified, they can be used in a number
of ways, such as a basis for selecting programming
language features or to generate new programming tools
and techniques. As an example, verification of a theory
which asserted that variable naming was more important
to comprehensibility than control structures would lead
to increased work on variable naming systems.
1.1 The Need for Good Methodology
The goal of this article is to explicate some of the
methodological problems that research on behavioral
aspects of programming entails and to suggest some
tactics for addressing these problems.
It would seem that the first duty of this kind of
research, as a newly emerging approach to an existing
set of problems, is to demonstrate powerful and useful
findings; if a crude methodology is adequate to this task,
the development of a more refined, sophisticated meth-
odology is a task that can wait for further maturation in
the area. There are, however, two significant reasons
208
why workers in this area should have a particularly
strong, early concern for a well-developed methodology.
The first is that while behavioral research in com-
puter science is relatively new, other disciplines, most
notably psychology, have developed and elaborated the
methodological techniques for over a century. In partic-
ular, psychometric theory, analysis of variance, and,
more recently, the multivariate techniques are a powerful
set of analytic tools for unraveling complex behavioral
phenomena. Researchers in these other behavioral dis-
ciplines have come to expect the use of such tools as an
integral part of behavioral research and to judge re-
search, in part, by the skill with which these tools are
used. Behavioral research, even if done in a computer
science context, will inevitably be judged by these same
standards of methodological rigor. In order to maintain
creditability with behavioral researchers in other areas,
behavioral researchers in computer science must pay
close attention to methodological issues.
A second reason for an early concern with method-
ology has to do with the impact that methodology avail-
ability has on the choice of experimental questions. In
theory, the process of designing an experiment begins
with the identification of an important theoretical ques-
tion, selected without regard to its experimental meas-
urability. Further assumptions are made as needed to
bridge between the theoretical question and an actual
experimental situation. Since it is this translation process
that requires the greatest amount of both ingenuity and
effort, there is a subtle pressure to give preference to
those theoretical questions that are the easiest to answer
experimentally. A strong concern for methodology does
not per se counteract this tendency; however, it does
encourage researchers to be aware of the role methodo-
logical issues play in the design of experiments.
2. Major Methodological Issues
In the paradigmatic situation in which these meth-
odologies are employed, a researcher has some hypoth-
esis about the behavioral effect of a programming lan-
guage construct or technique. In order to test this hy-
pothesis, he or she must construct a situation in which
the behavioral effect can be observed and measured.
Creating such situations typically involves the selection
of appropriate subjects, the administration of stimulus
materials likely to evoke an experimental effect, and the
use of an appropriate measure of the extent to which
effects have occurred. These issues of subjects, materials,
and measures are the fundamental issues in the choice
of methodology.
2.1 Choosing Subjects
In selecting subjects the experimenter must satisfy
two, sometimes contradictory criteria. On one hand, he
or she would like the subjects to be representative. The
purpose of doing a study is to make a statement about
behavior which will be true for some large population
Communications April 1980
of Volume 23
the ACM Number 4
such as all Fortran programmers or all real-time pro-
grammers. Usually, the subjects are only a small sample
of the population, but they are selected by some method
which justifies the claim that they are representative of
the population as a whole. Thus, an experimenter may
draw all his subjects from one firm on the basis that
employees from the firm are typical of programmers in
general.
The other criteria that experimenters must satisfy is
that the subjects be relatively uniform in regard to their
characteristics and abilities at the point at which they are
selected to participate in the experiment. Suppose that
the goal of an experiment is to measure the effect of two
control structures on comprehensibility and that two
different groups of subjects are used. If one group con-
tains a disproportionate number of high ability subjects,
then the experimental outcome may reflect this differ-
ence, rather than the effect of control structure.
These two criteria become contradictory when the
parent population is very hetergeneous. In this situation,
a representative sample perforce reflects this heteroge-
neity. The most widely used method for handling this
kind of situation is simply to use enough subjects so that
the chances will be extremely small that all the low or
high ability subjects will fall into the same group. Un-
fortunately, in the study of programming behavior, the
number of subjects required can be very large. Studies
of different aspects of programmer performance have
found that ability differences are from 4 to 1 [7] to 25 to
1 [15] across experienced programmers with equivalent
backgrounds. For some commonly used experimental
designs, this range of variability can imply a need for
hundreds of subjects in order to obtain significant results.
The use of too few subjects under these conditions entails
a strong risk that subject differences will obscure exper-
imental effects. Considering their report of large subject
differences, this may explain the failure to obtain statis-
tically significant results in Experiments II and III of
Shneiderman et al. [18].
One tempting solution to obtaining uniform subjects
is to use students in programming classes, as was done
by Shneiderman et al. [18] and Gannon [6]. There are
two problems with doing this. The first one is that it is
by no means a solution to the individual differences
problem; Shneiderman et al. report large individual dif-
ferences both within and between programming classes.
The second problem is that at the current time, there is
little justification for the assumption that results found
with beginning programmers are applicable to experi-
enced ones. In order for such an assumption to be
justified, it must be the case that the experienced pro-
grammer follows the same procedures and practices in
problem solving as does the naive programmer, albeit at
a faster rate. In the case of chess, a task with certain
analogies to programming, very experienced players
(masters) use entirely different procedures than novice
players, and the work of Youngs [26] on debugging
suggests that experienced programmers use different de-
209
bugging strategies than beginners do. Hence, in the
absence of a body of studies that fmd the same results
with experienced and student programmers, researchers
are not justified in generalizing between the two groups.
The caution given in the previous paragraph is prob-
ably equally applicable to intermediate programming
students. Shneiderman [19] has shown that even a few
months' difference in experience can have a significant
effect on performance. An intermediate programming
student may have taken 5 or 6 programming courses and
have spent a maximum of perhaps a thousand hours on
programming, including writeups and class time. A pro-
grammer with only three years of experience has prob-
ably spent five times as much time programming, so that
the intermediate student programmer probably resem-
bles the beginning programmer more than he or she does
the experienced one. The results of studies, such as
flowcharts or the work of Gannon [6] on data typing,
which use such intermediate students are therefore in
need of replication using experienced programmers.
Since control of variability by sample size alone is
rarely feasible, the experimenter will frequently want to
take other measures. A potentially attractive approach is
based on assessing the abilities of subjects prior to their
participation in the experiment and then either grouping
them on the basis of ability (stratification) or adjusting
the measurements of their performance in the experiment
for their initial ability level (covariance analysis). Unfor-
tunately, a problem lies in the selection of an ability
measure. The obvious choice, length of experience, was
used to select subjects in those studies in which the
difference was first found, and recent work using a wider
range of experience [17] also failed to find a significant
relationship. Their study did, however, suggest that va-
riety of experience has some predictive value, particu-
larly for programmers with less than three years of
experience.
Given the uncertain status of biographic measures,
an alternative approach would be to use a pretest of
programming skills to assess subject ability. The diffi-
culties of this approach lie in deciding what to use as the
pretest measure. If the question under investigation con-
cerns comparing programming languages with GOTOs
with those with block structuring, then what sort of
control structure should be used in the test materials?
Perhaps because of this difficulty in selecting the pretest,
no studies have yet appeared which use this pretest
approach.
The solution to the subject variability problem that,
to date, appears most successful is the use of within-
subject experimental designs. These designs are based on
exposing each subject to all levels of the experimental
variables under investigation; for example, in an experi-
ment to investigate the effects of four different structur-
ing techniques, each subject would use all four tech-
niques. The advantage to this approach is that the anal-
ysis is then based on the relative performance of each
technique with each subject.
Communications April 1980
of Volume 23
the ACM Number 4
 A drawback of within-subject designs is that they
often require the preparation of large amounts of stim-
ulus materials. If an experiment is being done to compare
five commenting techniques and each subject reads dif-
ferent programs with all five techniques, then a total of
55=25 different program listings would be needed.
Despite their requirements for stimulus materials,
within-subject designs have been used successfully in a
variety of studies [8, 9, 17].
2.1.1 A c o m m e n t on subject selection. In this discus-
sion, the wide range of intersubject variability has been
treated as a source of variance that must be compensated
for in experiments on other factors. At the same time,
however, the source and properties of this variance rep-
resent important research questions in their own right.
Specification of the exact behaviors and processes which
differ between better and poorer programmers and the
establishment of etiologies for these differences are re-
search problems that are as intriguing as those concerned
with differences in programming languages and meth-
odologies. In addition, progress on this individual differ-
ences issue offers the potential for better controls on
individual variation for research in other areas.
2.2 Choosing Materials
The most important criterion in the choice of exper-
imental materials is that they tap the experimental dif-
ference being tested; if, for example, the experiment is
designed to assess the impact of good structure on pro-
gram modifiability, the materials used should vary in
level of structuredness. The major problem in achieving
this criterion is to design materials that are usable with
the resources available for the experiment, but which are
representative of some wider class of programs. If the
programs which are created are too abnormal or patho-
logical, then readers of the studies are likely to be
unwilling to apply their results to the real world; if a test
of branching constructs requires programs that are half
branch statements to obtain significant results, then the
reader might reasonably conclude that branching con-
structs make very little difference at best.
Though none of the programs used in published
studies appear to be exceptional in regard to their inter-
nal structure, they do come under criticism in regard to
their lengths. At a time when several systems with more
than a million lines of source code exist, programs of less
than 500 lines will usually turn out to be toy or student
programs. An argument can be made that programs in
the 50- to 100-line range are the same size as the modules
that constitute the larger systems, and that, therefore, the
results obtained from using these smaller programs are,
at least partially, generalizable to larger systems. By now,
however, there is universal acknowledgment that writing
a large system is not just a matter of scaling up the
manpower required for a small system; it requires a set
of techniques which are qualitatively and quantitatively
different, and the factors which cause such projects to
210
succeed or fail are entirely different than those involved
in small systems.
Unfortunately, all of the published studies in this
area have used programs of less than 500 lines in size.
To legitimately generalize from these experiments with
small programs to large ones requires a psychological
characterization or model of the effects of program size.
Since such models are not yet available, all of the work
done to date is in need of replication using larger pro-
grams before any claims can be made about the gener-
ality of the results.
In addition to being appropriate to the experimental
difference being investigated, stimulus materials must
meet several other criteria. One of the more difficult of
these is insuring comparability across different experi-
mental conditions. If an experiment requires the use of
several different programs in different conditions, then
the programs must be comparable in all significant re-
spects except those which are experimental manipula-
tion. The difficulty lies in deciding, at the present level
of knowledge about the psychology of programming,
which aspects are actually significant. In the absence of
adequate theory, an obvious approach for insuring com-
parability is to match the programs on their intrinsic
characteristics, such as their length, the language in
which they are written, and the kinds of data structures
and variables which they use. For most studies, it will be
extremely difficult to fred a set of programs which match
well on all the possible measures.
An alternative to attempting to match on multiple
characteristics is to use the measures of program com-
plexity developed in software science [10, 12]. This work
is aimed at deriving measures of the difficulty or com-
plexity of a program from the operations performed
within the program. The measures are intended to be
independent of the task the program is performing and
of the language in which the program is written, but to
correlate well with other indicators of program complex-
ity.
Experimental work indicates that these measures can
be used to predict both time and accuracy to make
modifications in small programs [4, 17]. Researchers
considering these measures should, however, also be
aware that they leave nearly half of program variability
unaccounted for, and that they appear to be most effec-
tive with inexperienced programmers and with unstruc-
tured programs.
A fmal consideration in selecting materials is that
they be of an appropriate level of difficulty to produce
data with desirable statistical characteristics. Experiment
I, in the study of Shneiderman et al. [18], provides an
illustration of what happens if the problems are too easy.
In that experiment, scores for the two experimental
groups were 94 and 95 respectively, while the maximum
possible score was 100. With the means so close to the
upper limit, a number of subjects must have received the
maximum score. Not only has this probably acted to
reduce the magnitude of the experimental difference, but
Communications April 1980
of Volume 23
the ACM Number 4
it also invalidates any statistical analyses, such as the t-
test, which make the assumption of an underlying normal
distribution. Similar results would occur in a study in
which many o f the subjects received scores of zero.
2.3 Selecting an Experimental Measure
From the point of view o f their effects on program-
mer behavior, most innovations in programming lan-
guages or techniques can influence the ease with which
programs can be constructed a n d / o r the ease with which
existing programs can be understood. The experimental
tasks used in these studies will, therefore, be aimed at
measuring changes in either or both o f these properties.
Shneiderman [19] discusses a range of such tasks and
presents experimental evidence in favor of a particular
one, "memorization-recall." The aim of the discussion
here is to elaborate on some of the issues he raises.
2.3.1 Program construction measures. An extremely
powerful claim for any programming technique or lan-
guage feature is that it improves the program writing
process, either by reducing the effort needed to write
programs or by producing a better fmal product. The
obvious approach to assessing such a claim is to have
subjects write programs with and without the new tech-
nique; for example, the advantages o f the case statement
over the switch statement could be assessed by having
subjects solve the same problem using languages which
differed only in the feature in question.
A wide variety of measures are available for making
this assessment. If the claim under test is that the new
feature reduces programming effort, then an important
measure is the time taken to write the program. Unfor-
tunately, this measure suffers from the problems, dis-
cussed in a later section, that are common to time
measures in general. An additional problem o f insuring
accurate measurement arises if subjects are allowed to
set their own work periods. For these reasons, the time
measures must be supplemented with other measures,
such as the number of debugging runs or the ratio o f
total lines written to final program size. Sackman [15]
illustrates the use of a number of such measures.
If the claim is that the new feature improves program
quality, then the measurement issue becomes more dif-
ficult. For a situation in which "quality" is considered
equivalent to execution efficiency, then runtime provides
an empirical measure of program quality. At the other
extreme, if quality is interpreted to involve issues of
programming style, then no objective measures are read-
ily available and the experimenter will have to be con-
cerned with issues of rating scale construction and rater
reliability.
2.3.2 Debugging and modification measures. While
this class of measures may not be used to directly test
claims about the ease of debugging or modification, a
more frequent use will probably be to verify assertions
about program comprehensibility. Such tests involve an
211
inference: To modify or debug a program, the program-
mer must first understand it. In practice, this inference
may be difficult to guarantee; Gould [8] contains exam-
ples of debugging tactics which require little or no un-
derstanding of the program involved, and many modi-
fication tasks, such as altering output formats, can also
be accomplished without understanding. To use modifi-
cation or debugging tasks successfully to measure com-
prehension, the experimenter should be sure that the
task cannot be accomplished either by looking for state-
ments of a particular type or by simple checks on the
properties or variables. The use of multiple bugs or
modifications in a single program serves as an additional
safeguard that comprehension is not limited to a single
section of the program.
An additional problem peculiar to modification tasks
is that the specification of the modification may, itself,
give away information about the program. The following
modification was used by Sheppard et al. [16]:
Subroutine EUCLD currently finds the greatest common divisor
of two integers (M, N). Modify the subroutine so that it finds the
greatest common divisor of three integers (L, M, N).
Clearly, this modification task will not reveal much
about how well subjects understood what the E U C L D
subroutine did in the original program.
2.3.3 Memorization-recall and reconstruction. This
class of tasks involves presenting a programmer with a
program for some time interval, removing the program,
and then asking the programmer to reproduce it. Appro-
priate scoring systems are used to quantify the functional,
as well as literal, accuracy o f the reproductions. Shnei-
derman [19] has advocated this technique for evaluating
the comprehensibility of programs, and he has demon-
strated its sensitivity to differences in program structure
and in comment location.
The implicit assumption behind the use of these tasks
is that, with restricted study times, the easier a program
is to comprehend, the easier it will be to learn. This
assumption can be based on one or two models. The
model Shneiderman used is based on analogy to the
work of Chase and Simon [3] on chess. It assumes that
the programmer learns the program from the bottom up
by organizing his or her knowledge into ever-larger units.
The ease of comprehending the program will affect the
effectiveness of this reorganization process. On the basis
of this model, Shneiderman advocates the term "mem-
orization-recall" for this kind of task.
An alternative model is that during the learning
phase programmers extract enough information to re-
construct the program. This information is a mixture o f
low-level details such as variable names, constants, and
expressions and higher level information about algo-
rithms and global structure. The ease of comprehension
of the program will affect the ease with which the higher
level information can be extracted.
These two models are not exclusive, and which model
Communications April 1980
of Volume 23
the ACM Number 4
is a better description of actual events will depend on
the experimental instructions and on individual strate-
gies. The memorization-recall model will be a better fit
if subjects are encouraged to be as literally accurate as
possible, while the reconstruction model is more likely
to be appropriate if subjects are told to write a program
that is as close as possible to the original.
From the point of view of the experimenter, the
choice of a model will have a decided impact on the
construction of experimental materials and on the inter-
pretation of study results. The memorization-recall
model is consistent with an interpretation of "under-
standing" a program as complete knowledge of all the
details of the program's construction. Appropriate scor-
ing techniques will be ones that check for literal accuracy,
such as those used by Shneiderman.
The problem with the memorization-recall model is
that it is applicable only to isolated modules or to toy or
student programs. Even though a programmer is thor-
oughly familiar with say, a typical compiler, he or she
will certainly be unable to reproduce it literally. For
systems of realistic size, instructions which encourage
subjects to behave consistent with a reconstruction model
will probably be more appropriate. The drawback of
such instructions will be that it will be necessary to
develop a scoring scheme that compares programs on
underlying structure, rather than on literal equivalence.
2.3.4 Question answering. If the goal of the experi-
ment is primarily to assess comprehension of a program,
then an attractive task is simply to have the subject study
a program and then to ask him or her questions about it.
These tasks may be used alone or in conjunction with
construction, debugging, or memorization tasks.
The kinds of questions used can range from com-
pletely open-ended ("How does this program work?") to
completely structured multiple-choice. Open-ended and
short-answer questions have the advantage in that it is
fairly easy to construct a comprehensive set of questions.
On the other hand, scoring is often difficult; unless an
elaborate formal scoring scheme has been created, it is
often difficult to tell how much more accurate one
description of how a program works is from another.
A variant on the open-ended question is, instead, to
ask the programmer to talk aloud as he or she looks at
the program. These protocols can then be used as the
basis for constructing models of the problem-solving
process such as those of Newell and Simon [13].
The primary drawback to multiple-choice questions
is that construction of a sufficiently large set of questions
often requires considerable effort, since the questions
must satisfy a number of criteria. First, while the correct
response should be the one that is chosen most often, the
other responses are also chosen a moderate fraction of
the time. Second, if the goal of the experiment is to
measure comprehension of the entire program, then the
questions must cover all aspects of the program to ap-
proximately the same degree. Finally, the content of one
212
question must not inadvertently reveal the answer to
another question. Given the difficulty of balancing all
these requirements, extensive pretesting is virtually a
necessity in the use of multiple-choice questions.
2.3.5 Hand execution. The accuracy or speed with
which a programmer can hand execute or simulate a
program has been used as a measure of program quality
[24]. A major problem with the use of this task for this
purpose or for measuring comprehensibility is that there
is no easy way to insure that the hand execution involves
knowledge of the overall structure or organization of the
program, since hand (or machine) execution can be
accomplished on a statement basis. Combination of this
task with question answering about the overall structure
would yield greater confidence in interpreting the exper-
imental results.
2.3.6 A comment on time measures. The previous
discussion has not advocated any particular experimental
task as being uniformly superior. The choice of a partic-
ular task will depend on the experimental question and
on the subject population and resources available to the
experimenter: Most studies will use several different
tasks. A general comment can, however, be made about
the use of time as a variable.
A first consideration that must be kept in mind for
use of this measure is that the time must be measured in
such a way as to exclude irrelevant behavior; for exam-
ple, the experimenter is rarely interested in the time
taken to understand the task instructions and would like
to exclude this from total program time. While a simple
solution to this problem would be to time from the point
at which the subject actually begins to write code, some
subjects may begin coding before they understand the
problem and then go back to the problem description. A
potential remedy for this problem is to instruct subjects
that they will be allowed to read the problem description
only once and that they must be sure they understand it
before they begin to generate code. After subjects have
read the problem, they would be asked to answer ques-
tions about it, and whenever a subject gives incorrect
answers, he or she would be asked to go back and reread
the problem description.
A similar difficulty in the use of time measures occurs
because not all parts of a program are relevant to the
hypothesis under test. If the hypothesis is about control
structures for iteration, then including the time spent on
I / O in the program writing time may give false results,
particularly if the I / O turns out to be unusually difficult.
One solution to this problem, used by Brooks [2], is to
supply the programmer with a partially completed pro-
gram containing all the code except that which is of
experimental interest. Another alternative would be to
time the programmer only when he is working on the
relevant sections of the program; the difficulty in imple-
menting this method would be determining reliably what
the programmer is working on at given instant.
Communications April 1980
of Volume 23
the ACM Number 4
 In addition to the problems of determining exactly
what to time, time measures often have the additional
characteristic of having skewed distributions, with the
slowest subjects taking several times as much time as t h e
mode. In order to use most statistical techniques which
rely on a roughly normal distribution, this problem must
be corrected. One way is to simply throw out the extreme
values on the reasonable premise that those subjects
either did not understand the problem or that they, in
fact, belonged to a different population than the rest of
the subjects. Another method is to use statistical trans-
formations, such as a square root or arcsin [25], to make
the distribution more normal.
3. Prospects for Future Improvements
The intent of this paper has been to review some o f
the methodological problems faced in the behavioral
evaluation of programming language constructs and
techniques. O f the three topics surveyed, two, the selec-
tion of subjects and the selection o f materials, currently
remain sources of difficulty in performing behavioral
studies.
The impacts of the persistence of these two areas as
problems are two-fold: First, since the experimenter does
not have good control over them, they introduce addi-
tional error variance into experiments. This variance
may act to obscure the true differences arising from the
particular programming feature or technique being stud-
ied.
More importantly, the lack of knowledge about the
description and specification of differences among sub-
jects and programs has a damaging effect on the gener-
alizability of the experimental findings. If there is no
effective, replicable way to describe the characteristics of
subjects or programs used in an experiment, then there
is no way to determine whether the results of the exper-
iment apply to real world situations.
Desirable as it might be to come up with such descrip-
tors, prospects for their development in the immediate
future do not appear good. The essential problem is that
there is a strong interaction between the characteristics
of the programmer and those of the program; the diffi-
culty in modifying, rewriting, or understanding a pro-
gram may be as much a function of the programmer's
familiarity with that type of program as they are of
intrinsic characteristics of the program itself. As Curtis
et al. [4] point out, measures such as Halstead's or
McCabe's that are based only on control flow or opera-
tional characteristics of the program can never tap these
additional sources o f variation. An analogous problem
is likely to occur with measures o f programmer ability
that are based on gross experience or training measures.
What approaches, then, show promise? Any successful
characterization of the program-programmer interaction
will probably be based on a model of the process or
processes used by a programmer in interacting with a
program. The development of such theories or models of
213
the cognitive processes involved in programming is,
therefore, likely to be a prerequisite to progress on these
methodological issues.
Acknowledgment. This paper benefited from the
comments of B. Shneiderman on an earlier draft.
Received 8/79; revised 11/79; accepted 1/80
References
1. Boies, S.F., and Gould, J.D. Syntactic errors in computer
programming. Human Factors 16 (1974), 253-257.
2. Brooks, R. Using a behavioral theory of program comprehension
in software engineering. Proc. 3rd Internat. Conf. Software Eng.,
IEEE, New York, 1978, pp. 196-201.
3. Chase, W.G., and Simon, H.A. Perception in chess. Cognitive
Psychol. 4 (1974), 55-81.
4. Curtis, B., Sheppard, S.B., and Millman, P. Third time charm:
Stronger prediction of programmer performance by software
complexity metrics. Proc. 4th Internat. Conf. on Software Eng.,
IEEE, New York, 1979, pp. 356-360.
5. Garmon, J.D. An experiment for the evaluation of language
features. Internat. J. of Man-Machine Studies 8 (1976), 61-73.
6. Garmon, J.D. An experimental evaluation of data type
conventions. Comm. A C M 20, 8 (Aug. 1977), 584-595.
7. Gould, J.D., and Drongowski, P. An exploratory study of
computer program debugging. Human Factors 16 (1974), 258-276.
8. Gould, J.D. Some psychological evidence on how people debug
computer programs, lnternat. J. of Man-Machine Studies 7 (1975),
151-182.
9. Green, T.R.G. Conditional program statements and their
comprehensibility to professional programmers. J. of Occupational
Psychol. 50 (1977), 93-109.
10. Halstead, M.H. Elements of Software Science. Amer. Elsevier
Pub. Co., New York, 1977.
11. Knuth, D.E. An empirical study of FORTRAN programs.
Software Practice and Experience 1, 2 (1971) 105-133.
12. McCabe, T.J. A complexity measure. 1EEE Trans. on Software
Eng. SE-2 (1976), 308-320.
13. Newell, A., and Simon, H.A. Human Problem Solving. Prentice-
Hall, Englewood Cliffs, N.J., 1972.
14. Rubey, R.J. A comparative evaluation of PL/1. Datamation 20
(Dec. 1968).
15. Sackman, H. Man-Computer Problem Solving: Experimental
Evaluation of Time-Sharing and Batch Processing. Auerbach, New
York, 1970.
16. Sheppard, S.B., Borst, M.A., Curtis, B., and Love, L.T. Predicting
programmers' ability to modify software. TR-388100-3, General
Electric Co., Arlington, Va., 1978.
17. Sheppard, S.B., Borst, M.A., Millman, P., and Love, T. Modern
coding practices and programmer performance. Computer 12 (1979).
18. Shneiderman, B., Mayer, R., McKay, D., and Heller, P.
Experimental investigation of the utility of detailed flowcharts in
programming. Comm. A C M 20, 6 (June 1977), 373-381.
19. Shneiderman, B. Measuring computer program quality and
comprehension, lnternat. J. of Man-Machine Studies 9 (1977), 465-
478.
20. Sime, M.E., Arblaster, A.T., and Green, T.R.G. Reducing
programming errors in nested conditionals by prescribing a writing
procedure. Internat. J. of Man-Machine Studies 9 (1977), 119-126.
21. Sime, M.E., Green, T.R.G., and Guest, D.J. Psychological
evaluation of two conditional constructions used in computer
languages, lnternat. J. of Man-Machine Studies 5 (1973), 105-113.
22. Sime, M.E., Green, T.R.G., and Guest, D.J. Scope marking in
computer conditionals--a psychological evaluation. Internat. J. of
Man-Machine Studies 9 (1977), 107-118.
23. Weinberg, G., and Shulman, E. Goals and performance in
computer programming. Human Factors 16, 1 (1974).
24. Weissman, L. A methodology for studying the psychological
complexity of computer programs. Unpub. Ph.D. diss., Univ. of
Toronto, Canada, 1974.
25. Winer, B. Statistical Principles in Experimental Design. McGraw-
Hill, New York, 1971.
26. Youngs, E.A. Human errors in programming, lnternat. J. of
Man-Machine Studies 6 (1974), 361-376.
Communications April 1980
of Volume 23
the ACM Number 4