Models and Managers: The Concept of a Decision Calculus

Author(s): John D. C. Little

Source: Management Science, Vol. 50, No. 12, Ten Most Influential Titles of "Management
Science's" First Fifty Years (Dec., 2004), pp. 1841-1853
Published by: INFORMS
Stable URL: https://www.jstor.org/stable/30046155
Accessed: 03-02-2019 17:57 UTC

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

INFORMS is collaborating with JSTOR to digitize, preserve and extend access to Management
Science

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
MANAGEMENT SCIENCE infm-
Vol. 50, No. 12 Supplement, December 2004, pp. 1841-1853 DOI 10.1287/mnsc.1040.0267
ISSN0025-1909 1 EISSN 1526-5501 04 15012S 11841 c 2004 INFORMS

Models and Managers:

The Concept of a Decision Calculus
John D. C. Little
Sloan School of Management, Massachusetts Institute of Technology, Cambridge, Massachusetts

A manager tries to put together the various resources under his control into an activity that achieves his
objectives. A model of his operation can assist him but probably will not unless it meets certain require
ments. A model that is to be used by a manager should be simple, robust, easy to control, adaptive, as comple
as possible, and easy to communicate with. By simple is meant easy to understand; by robust, hard to get absu
answers from; by easy to control, that the user knows what input data would be required to produce desire
output answers; adaptive means that the model can be adjusted as new information is acquired; completenes
implies that important phenomena will be included even if they require judgmental estimates of their effe
and, finally, easy to communicate with means that the manager can quickly and easily change inputs and obta
and understand the outputs.
Such a model consists of a set of numerical procedures for processing data and judgments to assist manager
decision making and so will be called a decision calculus. An example from marketing is described. It is an
on-line model for use by product managers on advertising budgeting questions. The model is currently in tria
use by several product managers.
History: Received November 1969.

1. Introduction activities of governments, universities, hospitals, and

The big problem with management science modelsother organizations.
is
that managers practically never use them. ThereThehaveterms "manager" and "decision" will be used
been a few applications, of course, but the frequently.
practice Let it be noted now that a "manager" is
frequently
is a pallid picture of the promise. Much of the diffi- a fuzzy, shifting mix of people and a "deci-
culty lies in implementation and an especiallysion" is usually a murky event, identifiable only in
critical
aspect of this is the meeting between manager retrospect.
and
model. I believe that communication across this inter- The paper is organized under the following head-
face today is almost nil and that the situation standsings: (1) Introduction, (2) What's wrong? (3) How
as a major impediment to successful use of modelsdo managers use models? (4) What might be right?
by managers. Furthermore I want to suggest that the(5) An example from marketing, (6) What happened
to science? and (7) Conclusions.
requirements of the interface have implications for the
design of the model itself.
2. What's Wrong?
As an area to illustrate the ideas presented, weSome of the reasons that models are not used more
shall use marketing. This field well demonstrates
widely by managers appear to be:
the problems and opportunities at hand. Marketing
(1) Good models are hard to find. Convincing mod-
has high managerial content in the sense that deci-els that include the manager's control variables and
sions are often non-routine and usually require aso contain direct implications for action are relatively
bringing together of people, ideas, data, and judg-difficult to build, particularly in the areas that are of
ments from diverse sources. Although something is greatest concern. Some progress, however, is certainly
known about underlying processes, much uncertainty being made. In marketing, for example, see Mont-
remains to confront the manager. Data is prolific but gomery and Urban (1970).
usually poorly digested, often irrelevant, and some (2) Good parameterization is even harder. Measure-
key issues entirely lack the illumination of measure-ments and data are needed. They require high quality
ment. At the same time, marketing is of interest forwork at the design stage and are often expensive to
its own sake. This is not only because of its key and carry out.
sometimes controversial role in the society but also (3) Managers don't understand the models. People
because fundamental knowledge in marketing has tend to reject what they do not understand. The man-
application beyond business into the marketing-like ager carries responsibility for outcomes. We should
1841

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
 Little: Models and Managers: The Concept of a Decision Calculus
1842 Management Science 50(12S), pp. 1841-1853, @2004 INFORMS

variables
not be surprised if he prefers a simple to absurd
analysis thatvalues if critical phenomena a
he can grasp, even though it may omitted.
have a Onequalitative
popular answer to this difficulty is no
structure, broad assumptions, and only a little
to optimize. rele-
Sometimes this is the right thing to do
vant data, to a complex model whose assumptions
we should say out loud the model provides only par
may be partially hidden or couched indecision
of the jargon andinformation and that the res
making
whose parameters may be the result must come from
of obscure elsewhere. However, in most cases
statis-
tical manipulations. we want to be able to evaluate and compare. This
Typically the manager is willing and eager to
embryonic accept
optimization and incompleteness can be
flawless work that delivers the future to him with pitfall.
certainty. Unfortunately as he digs into any study The above list of obstacles of implementation
performed by human researchers in an ordinary ORcould be extended but should suffice to ward off
group, he finds assumptions that seem questionable,
complacency.
terminology that is confusing, and a certain tendency
to ignore a variety of qualitative issues the manager
3. How Do Managers Use Models?
feels are important. The manager feels that to get deep
into the model and find out what is really going on is is an impression, albeit anecdotal, of how man-
Here
agers actually use models.
totally out of the question because he lacks the time
and background. The solution to this predicament is The OR Group of a major oil company recently did
often for him to pick on some seeming flaw in the a survey on the use of mathematical programming
model, usually a consideration left out, and make thatproduction scheduling at their refineries. Refinery
in
scheduling was a pioneer application of mathematical
the basis for postponing use into the indefinite future.
In this situation the operations researcher's re- programming and has been an active research area for
sponse is often to conclude that his model is not 10-15 years. At one refinery the dialog between the
complete enough. Therefore he goes back to work to interviewer and the local OR analyst went somewhat
as follows:
make things more complicated and probably harder
to understand. Meanwhile the manager continues to Interviewer: "Do you make regular mathematical
use intuitive models that are much simpler than the programming runs for scheduling the refinery?"
one rejected. Analyst: "Oh yes."
I might point out the professional OR/management Interviewer: "Do you implement the results?"
science fraternity also escalates the model builder into Analyst: "Oh no!"
complexity. A favorite pastime in the trade is to tell a Interviewer: "Well, that seems odd. If you don't
model builder, "You left such and such out." implement the results, perhaps you should stop mak-
Some people have asked why it is necessary for a ing the runs?"
manager to understand the models he uses. After all, Analyst: "No. No. We wouldn't want to do that!"
most of us drive cars and few of us understand Interviewer: "Why not?"
the details of an internal combustion engine.Analyst:
An "Well, what happens is something like
this: I make several computer runs and take them to
R&D manager is not expected to follow the technical
the plant manager. He is responsible for this whole
niceties of the work being done in his labs. However,
I would argue that the kind of understanding multi-million
that dollar plumber's paradise.
"The
is required is defined relative to the job. As drivers plant manager looks at the runs, thinks about
we had better understand what will happen when themwefor a while and then sends me back to make a

turn the steering wheel even though we do not few more with conditions changed in various ways.
need
to know how to repair the brakes. The R&D managerI do this and bring them in. He looks at them and
had better be able to tell effective teams from inef-
probably sends me back to make more runs. And so
fective ones, but this will not usually require forth."
him to
Interviewer: "How long does this keep up?"
be a specialist in laboratory technique. The marketing
manager should understand a marketing model in Analyst:
the "I would say it continues until, finally, the
sense of knowing very well what to expect from plant manager screws up enough courage to make a
it but
decision."
need not know the details of its computer program.
(4) Most models are incomplete. Having complainedWhat is the plant manager doing here? Before
about complexity as a bar to understanding,speculating
I now on this, let me recount some experiences
decry incompleteness. This means that I hope withwe people using MEDIAC, a media planning model
can invent simple models that have the capacity developedto by L. M. Lodish and myself (1969). The first
include quite a few phenomena. step in using the model is preparing the input data.
This
Incompleteness is a serious danger if a model is requires a fair amount of reflection about the
problem at hand, a certain effort spent digging out
used for optimization. Optimization can drive control

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
Little: Models and Managers: The Concept of a Decision Calculus
Management Science 50(12S), pp. 1841-1853, d2004 INFORMS 1843

numbers, and usually subjective estimates of several by control theory: Action in control sys-
is supported
quantities. Thereafter, the model is run andtems is usually triggered by a discrepancy between
a schedule
is generated. observed and desired results.
The user looks at the schedule and immediately
The model builder can take advantage of the idea
starts to consider whether it makes sense to him or of working with differences by having his model give
not. Is it about what he expected? Sometimes itand is receive much of its information in these terms.
and, if so, usually that is that. Oftentimes, however,
The following quotes from Mathes may (or may
the schedule does not quite agree with his intuition. It be helpful to any group of managers and man-
not)
may even differ substantially. Then he wants to knowagement scientists working together to apply models:
why. A process starts of finding out what it was about
The difference (managerial) people know only two
the inputs that made the outputs come out as they
degrees of probability, zero and one, and the similarity
did. This usually can be discovered without too much
(scientific) people recognize every degree of probabil-
difficulty by a combination of inspection, considera-
ity except zero and one.
tion of how the model works, and various sensitivity
analyses. The difference people tend to act before they think,
Having done this, the user decides whether he is if they ever think; whereas the similarity people think
before they act, if they ever act.
willing to go along with the results as they come out.
If not, he can, for example, change the problem for-
mulation in various ways or possibly change his sub-
jective estimates. Sometimes he finds outright errors
4. What Might Be Right?
If we want a manager to use a model, we should
in the input data. Most of the time, however, if he
make it his, an extension of his ability to think about
has been careful in his data preparation, he will agree
and analyze his operation. This puts special require-
with the reasons for the answers coming out as they
ments on design and will often produce something
did and he has, in fact, learned something new about
rather different from what a management scientist
his problem. The whole process might be described
might otherwise build. I propose a name to describe
as an updating of his intuition. The model has served
the result. A decision calculus will be defined as a
the function of interrelating a number of factors and,
in this case, not all the implications of the interrela-
model-based set of procedures for processing data
tions were evident to him when he started. and judgments to assist a manager in his decision
making.
Notice, incidentally, that he has by no means turned
over his decision making to the computer. He remains From experience gained so far, it is suggested that
a decision calculus should be:
the boss and demands explanations from his elec-
tronic helper. (1) Simple. Simplicity promotes ease of understand-
I believe the same type of process is going on ing. Important phenomena should be put in the
with the plant manager in the earlier example. Hemodel and unimportant ones left out. Strong pressure
often builds up to put more and more detail into a
is involved in an analysis-education-decision pro-
cess built around man-model-machine interaction in model. This should be resisted, until the users demon-
strate they are ready to assimilate it.
which the man does not lose responsibility or control
and instead of understanding less understands more.(2) Robust. Here I mean that a user should find it
difficult to make the model give bad answers. This
Such an interaction should, I believe, be the goal
for much of our normative model building. can be done by a structure that inherently constrains
Further advice to scientists about managers answers
has to a meaningful range of values.
(3) Easy to control. A user should be able to make
been reported by Mathes (1969). He asserts that man-
agers and scientists have different approachesthe to model behave the way he wants it to. For exam-
problem solving and that this fact hinders commu-ple, he should know how to set inputs to get almost
any outputs. This seems to suggest that the user
nication between them. Managerial people analyze
problems on the basis of differences or changes in sit- could have a preconceived set of answers and simply
uations. Scientific people look for similarities or com- fudge the inputs until he gets them. That sounds bad.
mon elements. Should not the model represent objective truth?
Certainly the model builder by the nature of his Wherever objective accuracy is attainable, I feel con-
role is seeking generalizations that fit many problems. fident that the vast majority of managers will seize it
And certainly the manager looks at differences: this eagerly. Where it is not, which is most of the time, the
year vs. last year, his organization vs. the competition,view here is that the manager should be left in control.
forecast vs. actual, etc. Pounds (1969) has observedThus, the goal of parameterization is to represent the
that such mechanisms form one of the manager's operation as the manager sees it. I rather suspect that
principal means of problem finding. The same ideaif the manager cannot control the model he will not

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
 Little: Models and Managers: The Concept of a Decision Calculus
1844 Management Science 50(12S), pp. 1841-1853, @2004 INFORMS

use it for fear it will coerce him into actions he does A person develops his understanding of a problem
not believe in. However, I do not expect the manager and its solution as he works on it. The critical time is
to abuse the capability because he is honestly looking not that of the decision deadline but of the next step
for help. in the user's thinking process.
(4) Adaptive. The model should be capable of being Perhaps equally as important as the operational
updated as new information becomes available. This convenience of conversational programs is their con-
is especially true of the parameters but to some extent tribution to learning. Good on-line models are user-
of structure too. instructing and introduce a person to the issues of the
(5) Complete on important issues. Completeness is in problem and the model much faster than would oth-
conflict with simplicity. Structures must be found that erwise be possible. A person can rapidly get a feel
can handle many phenomena without bogging down. for how the model works through direct experience.
An important aid to completeness is the incorporation This is in sharp contrast to batch processing with its
of subjective judgments. People have a way of making long time lags and imposing tribal rituals of punched
better decisions than their data seem to warrant. It is cards, systems programmers, and computer operators.
clear that they are able to process a variety of inputs In summary, we are learning techniques of model
and come up with aggregate judgments about them. design and implementation that bring the model to
So, if you can't lick 'em, join 'em. I say this withoutthe manager and make it more a part of him. We are
taking away from the value of measurement. Many, if calling such a model a decision calculus.
not most, of the big advances in scientific knowledge
come from measurement. Nevertheless, at any given
5. An Example from Marketing
point in time, subjective estimates will be valuable
An on-line marketing-mix model for use by product
for quantities that are currently difficult to measuremanagers is currently being developed. The present
or which cannot be measured in the time available
version emphasizes advertising budget decisions. It
before a decision must be made.
will be described below in an inductive, narrative
One problem posed by the use of subjective inputs
way. Its mathematical specification is given in the
is that they personalize the model to the individ- appendix.
ual or group that makes the judgments. This makesThe product or brand manager is an ideal customer
the model, at least superficially, more fragile and less
for a decision calculus. He has substantial responsi-
to be trusted by others than, say, a totally empiri-bility for all of the marketing control variables for a
cal model. However, the model with subjective esti- brand. He is busy and will not use a model unless
mates may often be a good deal tougher because it is it does something for him. He is at ease making
more complete and conforms more realistically to the
judgments and, being a single person accountable for
world.
results, he can gather inputs and make judgments
(6) Easy to communicate with. The manager should without the elaborate coordination required in many
be able to change inputs easily and obtain outputs other complex decision processes.
quickly. On-line, conversational I/O and time-shared The work is being done in cooperation with three
computing make this possible. different product managers at two different compa-
Every effort should be made to express input nies. The variety in companies and managers has been
requests in operational terms. The internal param- helpful for getting perspective on the man-model
eterization of the model can be anything, but the interface and in keeping the model structure gen-
requests to the user for data should be in his lan- eral. The development has proceeded in evolution-
guage. Thus, coefficients and constants without clear ary steps. First a very simple advertising budgeting
operational interpretation are to be discouraged. Let model was brought up and used to demonstrate con-
them be inferred by the computer from inputs that cepts. Then a more complex model for advertising
are easier for the user to work with. Expressing inputs budgeting, one with sufficient detail to be of practical
and outputs as differences from reference values often value, was brought up. This version will be described
helps. here. Experience with it is influencing the design of a
On-line systems come through as being very effec- more elaborate model.
tive in bringing the model to the manager. Some writ-
ers, for example Dearden (1964), have belittled the 5.1. Model Structure
importance of immediate response. They argue that We seek a simple, robust, easy to control model of
decisions made once a year or even once a month sales response to advertising. As a first step brand
hardly require systems that deliver the answers in sec- sales is partitioned into product class sales and brand
onds. Anyone who has used a conversational system market share. That is, we separately model what i
perceives that this argument misses the point. Practi- happening to the whole industry or product class of
cally no decision is made on a single computer run. which the brand is a part and what is happening t

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
Little: Models and Managers: The Concept of a Decision Calculus
Management Science 50(12S), pp. 1841-1853, @2004 INFORMS 1845

Figure 1 Input Data for Fitting a Sale Response The constants min,
to Advertising max, 8 and y are implicitly deter-
Function
mined by the input data.
Equation (1) represents a versatile but nevertheless
max share at end restricted set of response relations. Actually I am will-
ing to use anything. The curve could go down or up
share , end share with + 50% adv.
or loop the loop for all I care. It should be changed
when and if a product manager wants it changed.
- -- Meanwhile, he can give four numbers, each of which
initial _> maintenance adv. has operational meaning to him and which together
share
will specify a curve. It is doubtful that, as of today,
we could specify a sales response curve in any greater
min share at end detail than represented by a smooth curve through
four appropriately chosen points.
I now claim that the above structure is robust. Sup-
~ one period
pose we do a two level spending test and run a regres-
time sion that is linear in advertising in order to estimate
response. Such a regression might make reasonable
statistical sense but by itself would have absurd nor-
the brand's share within the class. Such a breakdown
has a number of advantages, not the least of which mative implications (advertising = 0 or oo); it would
not
is that marketing people usually think this way. Con- be robust. However, if the regression results are
sider a given time period. We suppose: used to estimate the +50% point and a reasonable
max and min are chosen we can expect reasonable
1. If advertising is cut to zero, brand share will
answers. This would be difficult to prove in general,
decrease, but there is a floor, min, on how much share
but with a specific manager and product it can usually
will fall from its initial value by the end of one time
period. be demonstrated satisfactorily by sensitivity analysis.
2. If advertising is increased a great deal, say to To be sure, more sophisticated models and data
something that could be called saturation, brand share analyses can easily be suggested. A quadratic term
will increase but there is a ceiling, max, on how much could be put in the regression, for example, but its
can be achieved by the end of one time period. coefficient would almost certainly be unstable and
3. There is some advertising rate that will maintain normatively alarming. A Bayesian analysis or an
initial share. adaptive control model like that of Little (1966) might
4. An estimate can be made by data analysis or restore order, but the intellectual cost of such com-
managerial judgment of the effect on share by the endplications is high. Even if more sophisticated studies
of one period of a 50% increase in advertising overare done, they could probably be translated into a set
the maintenance rate. of operational terms like the above. In any case we
Figure 1 gives a pictorial representation of this should start simply.
information. The same data can also be represented A person might well ask: Is the structure too
as four points on a share response to advertising robust? Conceivably a model could be so constrained
curve, as in Figure 2. A smooth curve can then be put that output would be almost decoupled from input.
through the points; for example, the function This is hardly the case here. The value specified for
the share increase with +50% advertising is certain
share = min + (max - min)(adv)'/[8 + (adv) ]. (1) to be an important determinant of advertising rate.
The values of max and min play the role of keeping
Figure 2 A Smooth Curve of Share vs. Advertising Put Through the changes in a meaningful range.
Data of Form Shown in Figure 1 Incidentally, the sketch in Figure 2 shows an
share S-shaped curve. This is not required by (1). If y > 1,
the curve will be S-shaped, for 0 < y < 1, a concave
max function. The particular y will depend on the input
data.
+50% adv. -
A major omission in the description so far is con-
initial - -+
min
sideration of time delays. To take these into account,
the model assumes:
1. In the absence of advertising, share would even-
0 main- +50% advertisingI I >- tually decay to some long run minimum value (pos-
tenance
sibly zero).

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
 Little: Models and Managers: The Concept of a Decision Calculus
1846 Management Science 50(12S), pp. 1841-1853, v2004 INFORMS

The
2. The decay in one time period will be a treatment of this is analogous to that for share.
constant
fraction of the gap between current share Details
and the are long
given in the appendix.
run minimum, i.e., decay is exponential. A variety of other factors affect share and there-
3. The decay determines min for the timefore period.
indirectly or directly can affect the product man-
ager's-thinking
4. The advertising affectable share, (max min) about the advertising budget. Some
stays constant. of these factors are: promotions, competition, distri-
Let long run min denote the long runbution,
minimumprice, product changes, and package changes.
These
and persistence denote the fraction of the factors are all treated, but in a simple way, not
difference
between share and long run minimum that unlike the way a product manager might handle them
is retained
after decay. Under the above assumptions:without a model.
Upon examining the factors, we find that the prod-
persistence = min - long run min uct manager has a definite idea about what various
initial share - long run min changes are likely to do for him. If he plans a pro-
share(t) - long run min motion he does so with the expectation that some-
thing will happen to his sales and share. The same
= (persistence)[share(t - 1) - long run min]
holds for a product change or price change. Therefore
we ask him to construct an index of how he believes
+ (max - min)[adv(t)]l/(8 + [adv(t)]Y). (2)
these factors will affect brand share in each period.
The process
This is a simple dynamic model. It is explainable andcan be formalized by filling in a table
it behaves reasonably. It could be furthersuch general-
as Table 1, listing all factors deemed by the prod-
ized by permitting some of the constantsuctto manager
change to be relevant. The composite index of
with time, but that does not seem desirable
nonadvertising
at the effects is simply the product of the
moment. numbers in each column. Brand share will then be
the product of the nonadvertising effect index and the
But now what is meant by advertising? Dollars?
Exposures? A product manager worries about spend- share developed from the advertising response rela-
ing rates, media, and copy. Let us construct two time For clarity the latter will be called the unadjusted
tion.
varying indices: (1) a media efficiency index, and share:
(2) a
copy effectiveness index. Both will be assumed to
brand share(t) = [non adv effects index(t)]
have reference values of 1.0. The model then hypothe-
sizes that the delivered advertising, i.e., the adv(t) that - [unadj share(t)].
goes into the response function is given by
People often ask how product managers can make
adv(t) = [media efficiency(t)] judgments like the above. The answer is that man-
agers make such judgments all the time but in a less
" [copy effectiveness(t)]
formal and less numerical way. Whenever they take
" [adv dollars(t)]. (3) an action they form some belief about what will hap-
pen. As a result, it has not proven difficult for them to
The media efficiency and copy effectiveness indices which they feel reasonably comfort-
make estimates
can be determined subjectively, but better alternatives
able with.
exist. Copy testing is helpful and data onEssentially,
media cost,the model is now specified. However,
exposures by market segment, and relative value
as we have of time varying effects such as media
added
market segments can be used to develop a media
efficiency and the nonadvertising phenomena, we
index.
have created a problem for the inputs that deter-
So far we have taken up share response to adver- mine share response to advertising. What values of
tising, media efficiency, copy effectiveness, and share the time-varying effects are assumed in the share
dynamic. Consider next product class sales. Two
important phenomena here are seasonality and trend.
Table 1 Developing a Composite Index of Nonadvertising Effects
These and any similar effects can be combined into a
Period
product class index that varies with time. Thus
Index of effect on share 1 2 3 4
product class sales(t)
Promotions 1.00 1.10 0.98 1.00
= [reference product class sales] Price 1.00 1.00 1.00 1.00

- [product class sales index(t)]. Package 1.00 1.05 1.05 1.05

Competitive section 1.00 0.98 0.95 1.00
Other 1.00 1.00 1.00 1.00
In addition there may be a product class response
Composite 1.000 1.132 0.978 1.050
to brand advertising and corresponding time lags.

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
Little: Models and Managers: The Concept of a Decision Calculus
Management Science 50(12S), pp. 1841-1853, p 2004 INFORMS 1847

response inputs? To deal with this question 5.2. Conversational

we intro- I/O
duce the concept of a reference case. The We have said that the model should be easy to us
reference
case is a standard set of values against which It must changes
be easy to put data into the computer, easy
can be measured. The reference case includes a ref- to find out what is in there, easy to change it, easy
erence time period. This is not one of the numbered to make output runs, easy to search over control var
time periods of the calculation but one set apart to and make sensitivity analyses. Clerical erro
ables
serve as a standard. It can be patterned after a should
real be quickly correctable. The mechanical operat
period or can be constructed as a "typical" period. ing details should require as little training as possible
In any case each time varying effect is assignedThe
a best way to show how these issues are being
value in the reference period. From this data approached
the would be by demonstration at a com
sales response parameters min, max, y, and 8 are then
puter terminal. Short of this we can provide an exam
inferred.
ple. Table 2 shows the trace generated by a perso
To summarize the model:
putting in data for "Groovy," a struggling brand in
1. Share
the treacle market. Table 3 shows an input summary
printed back by the computer. Table 4 shows an out
brand share(t) = [non adv effect index(t)]
put run.
- [unadj share (t)] Some of the options open to the user are: saving
data in the computer for later use, changing individ-
unadj share(t) = long run min + [persistence]
ual data items, and printing selected items of input
S[unadj share(t - 1) or output. He can choose between long descriptive
- long run min]
questions or terse questions that type quickly. An
important facility is the search option: Any variable
(max - min) [wtd adv(t)]y
or parameter of the model can vary from an arbitrary
8 + [wtd adv(t)]' minimum to an arbitrary maximum in any number
wtd adv(t) = [media efficiency(t)] of steps. At each step a set of user specified out-
put items is calculated and printed. Thus a search
[copy effectiveness(t)] might be made over a control variable to look for
adv dollars(t) improved profit. Or a search might be made over an
reference value of numerator input parameter to find that value which conforms
best to the manager's feeling about the parameter's
2. Brand Sales
effect on sales.

brand sales(t) The traces of Tables 2-4 are largely self-explanatory,

except that the item SLOPE in Table 4 needs clarifi-
= [reference product class sales] cation. This item is intended to answer the question
" [product class sales index(t)] that a user is most likely to ask: Which way should
I change advertising to increase profit? But we must
" [brand share(t)]
ask: What profit? Profit in that period or, since sales
3. Profits changes persist into the future, profit over several
periods? We have chosen to anticipate the answer to
contribution to profit after adv(t) be "cumulative contribution after advertising" in the
= [contribution per sales unit(t)] last period of the calculation. But which advertising?
We expect the question might be asked about adver-
- [brand sales(t)] - adv dollars(t) tising in any period. Thus we calculate
The units situation has not been developed in detail SLOPE (t)
here and we have omitted the effect of brand adver-
= the change in cumulative contribution
tising on product class sales. These details are treated
after advertising in the last period, per
in the appendix.
unit change in adv dollars in t.
The basic equations defining the model are really
quite few. Nevertheless the structure permits consid- A positive SLOPE indicates that advertising increases
eration of share response to advertising, copy effec- will be profitable (in the above sense); negative,
tiveness, media efficiency, product class seasonalityunprofitable; and zero, indifference.
and trends, share dynamics, product class response to
advertising, and a variety of non advertising effects 5.3. Applying the Model
such as promotion, distribution, and price. I feel thatOne might think that ways to apply the model would
the structure meets the criteria of simplicity, robust-be obvious. Not really. The model has to be worked
ness, and ease of control. into the user's system. There are a number of ways

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
 Little: Models and Managers: The Concept of a Decision Calculus
1848 Management Science 50(12S), pp. 1841-1853, i 2004 INFORMS
in which this can and should be done. I shall describe Table 2 Trace of a User Putting Input Data for GROOVY into the Com-
puter. All User Responses are Bold
one which we have just been through: The model was
used to assist in the quarterly review of a brand plan.
GO /ADBUDG
The usual pattern of operations with a consumer
ADBUDG II - A multiperiod advertising budgeting model
product is to construct a brand plan. This is done
once a year. The plan lays out the whole market-1 Computer asks questions in standard form
ing program in considerable detail. However, as2 Computer
the asks questions in short form
ANS= 1
year progresses and various parts of the program1 Enter new data
are carried out, changes get made: New opportuni-
2 Use saved data
ties arise, actual results come in and are not quite
ANS= 1
as expected, and generally a variety of unforseen Brand
cir- name: GROOVY
cumstances occur. Consequently, a series of review
No. of time periods (MAX = 8): 4
and replanning points are scheduled, usually quar-
Length of period: QUARTER
terly. This does not preclude actions at other times,
Name of first period: 1 ST Q 70
which in fact take place, but it does at least schedule
times in which changes are definitely consideredGeographic
or, area: US

if already made, are consolidated in a revised forecast

Brand data for reference period.
of results. Seasonality, trend, or other non-adv.
Effect removed.
Our goals in applying the model were to start from
a "brand plan" view of the market, modify it to acco-
Market share at start of period (% of units): 1.86

modate the new information contained in year-to-date

Advertising that will maintain share (dollars/period): 486900
results, then evaluate new strategies and repredict
Market share at end of period
future outcomes. Here is what we did: If adv reduced to zero: 1.77

Step 1. Setting up the model according to the annual

Market share at end of period
brand plan. A set of input data was developed whichIf adv increased to saturation: 2.25
would reproduce as model output the results found Market share at end of period if adv increased
in the original brand plan. (If the brand plan had
50% over maintenance rate: 1.95

been constructed using the model, this step would

Market share in long run if adv reduced to zero: 0
not have been necessary.) The product class was iden-
Index of media efficiency (e.g., average efficiency = 1.0): 1.0
Index of copy effectiveness (e.g., average copy = 1.0): 1.0
tified. The seasonality and trends in product class
were worked out. The input data for sales response in which sales are to be measured
Units
(To be used for both brand and product class,
to advertising was estimated by a combination of
e.g., pounds, gallons, cases, thousands of dollars, etc.):
judgment and the examination of past time series of HOGSHEADS
advertising and sales data. (In this case there were no
Contribution profit (before adv expense)
spending levels test data but one of the side conse-
Expressed in dollars/sales unit: 0.68
quences of our study is that the company is seriously
considering such tests for the future.) A promotionAverage brand price (dollars/sales unit): 1.812

was planned for the second quarter and estimated Other brand data
to have a certain effect on share. A copy test, usingMarket share in period previous to period 1: 1.86

two different areas of the country, was under way. Product class data for reference period.
The brand plan proposed that the test be continuedSeasonality, trend and other non-adv
Effects removed.
for the year and so the copy index was held constant
Name of product class: TREACLE
at 1.0. Similarly no substantial media changes were
Product class sales rate at start of period
anticipated and the media efficiency was held at 1.0.
(units/period): 290000000
A certain set of spending rates for advertising was
envisaged and they were put into the model. A pack- Consider response to product class adv? NO

age and price change was under consideration but itAverage price for product class (dollars/sales unit): 1.88

had not gone into the plan. Time varying data. If time variation not specified,
The assembled data was put into the model and reference data will be copied into all periods.
fine adjustments were made in the parameters untilProduct class sales rate has seasonal or other non-adv
the model predicted the brand plan results exactly.time effect? YES
We then took the model as a reasonable indication of Index of product class sales (reference case = 1.00) for period:
1:0.943
the product manager's feelings about how the market
2:1.012
worked as of the time the brand plan was written.
3:1.065
Step 2. Updating the model on the basis of year-to-date
4: 0.959
results. Our analysis was done after the first quarter

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
Little: Models and Managers: The Concept of a Decision Calculus
Management Science 50(12S), pp. 1841-1853, x2004 INFORMS 1849

Table 2 (continued) levels for the rest of the year had been reduced. Sec-
ADBUDG II - A multiperiod advertising budgeting
ondly, the package and price change under consider-
model
ation had been decided upon and was scheduled to
Brand share has a non-adv time effect? YES
begin in the third quarter. In support of that, the trade
promotion
Index of non-adv effects (reference case= 1.00) for period was changed from the second quarter to
1: 1.0
the third quarter. Finally, more results were avail-
2:1.05
3: 0.98
able on the copy test and a sufficient difference had
4: 1.0 shown up between the two areas that it was planned
Media efficiency varies? NO
to implement the better one nationally in the fourth
quarter. An estimate of the effect of the new copy on
Copy effectiveness varies? NO
the copy index was made using the results of the test.
Contribution varies ? NO
All these changes were made to the input. Further-
Average brand price varies? NO
more a rough brand plan for the following year was
Average price for product varies? NO put into the analysis. Then the new plan was run. This
Brand adv rate varies? YES suggested there would be a substantial improvement
Brand adv (dollars/unit) in period in sales and profit compared to the previous case. It
1:486000 also showed that certain reallocations of advertising
2: 606000
spending during the year and certain changes in the
3: 876000
4: 414000 budget might well be warranted.
Step 4. Predictions of Future Results. After the above
1 Save data
2 Print data runs were made a few further adjustments to strat-
3 Change data egy were decided upon. Thus the whole plan was
4 Output run again. This run then became part of the quarterly
5 Restart review.
ANS= 1
Data file name: GROOVY-70
The above application illustrates the general way
we expect the model to enter into the product man-
ager's operation. However, each application is some-
what different. The previous one was very much of
data were in. Two principal events had occurred. First
of all, sales were off from their forecast value. Sec- a team operation with the product manager being
supported by specialists with marketing research and
ond, media expenditures had been lower than orig-
operations research skills. Although this is usually to
inally planned. The first question to be asked was
whether the lower sales could be attributed to the be expected, in another case the product manager has
run the model and made his recommendations almost
decreased media expenditures. Therefore, we ran the
single-handed. He found that it took two or three
model with the new first quarter's advertising level.
concentrated exposures to the model to become com-
According to the model, the change would account
fortable with it. In between he was pulled away by
for some but not all of the sales differences. The ques-
the press of other activities for a month or so at a
tion then arose whether the advertising had a greater
time. Finally, however, he was confronted by a spe-
effect on sales than we originally thought or whether
cific budgeting problem and sat down to work with
some other factors were causing sales to be off. The
product manager's opinion was that other factors the model intensively. Out of this effort came a report
were probably responsible. The next question and
wasa specific set of budget recommendations. His
whether the factors would continue to operate and particular
he concern was the conflict between a strat-
felt that there was no reason to believe otherwise. egy of budget cutting, short range profit taking, and
Consequently we adjusted the nonadvertising possible erosion of market position and a strategy
effects index to account for the loss in sales observed of maintaining or increasing budgets to try to pro-
in the first quarter and not otherwise attributed to thetect or build share. He worked out sets of assump-
advertising decrease. The same adjustment was then tions about market behavior and alternative company
continued through the year. actions and, using the model, traced out their pro-
At this point it was possible to rerun the brand plan jected consequences. Finally he wrote it up with his
with the new parameters. It put forth a rather pes- recommendations.
simistic view of the year. The following conversation then took place between
Step 3. Evaluation of new strategies. In the meantime, himself and his boss, the group product manager.
a number of new strategies had been proposed. First They went over the report at length and finally the
of all, because of the lower sales in the first quarter group manager said, "All right, I understand what the
and the implied poorer profit position, the advertising model says, but what do you really think?"

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
 Little: Models and Managers: The Concept of a Decision Calculus
1850 Management Science 50(12S), pp. 1841-1853, @2004 INFORMS

Table 3 Summary of Input Data for GROOVY Brand. It has been stored in a file named /GROOVY-70/

1 Save data
2 Print data
3 Change data
4 Output
5 Restart

ANS = 2

1 Standard print
2 Only specified lines

ANS = 1

/GROOVY-70/

1 Brand name: GROOVY

2 No. periods: 4.000
3 Per. length: Quarter
4 First per.: 1ST Q 70
5 Area: US
Reference per. - brand
7 Init. share (% of units): 1.860
8 Maint. adv (dol./per.): 0.486M
9 MIN share at end: 1.770
10 MAX share at end: 2.250
11 End share with +50% ADV: 1.950
12 Long run min share: 0.000
14 Media effcy.: 1.000
15 Copy effect: 1.000
16 Sales unit: Hogsheads
17 Contribution (dol./unit): .680
18 Brand price (dol/unit): 1.812
Other brand data
19 Starting share: 1.860
Reference per. - prod. class
21 Prod. class name: Treacle
22 Init. class sales rate (units/per.): 290M
29 Class price (dol./unit): 1.880
Time variations

Period 1 2 3 4
30 Class sales index: 0.943 1.012 1.065 0.959
31 Non-adv effect index: 1.000 1.050 0.980 1.000
32 Media effcy.: 1.000 1.000 1.000 1.000
33 Copy effect: 1.000 1.000 1.000 1.000
34 Contribution (dol./unit): 0.650 0.680 0.680 0.680
35 Brand price (dol./unit): 1.812 1.812 1.812 1.812
36 Class price (dol./unit): 1.880 1.880 1.880 1.880
37 Brand adv (dol./per.): 0.486M 0.606M 0.876M 0.414M

the
This is a good question because itmarket."
uncoversAs for the issue of
certain
might
important issues. First, has the be partisan,
product managerit must be re
lost
product
control, i.e., does the model really manager
reflect system
his view of is an ad
each may
the market? Second, the question man contain
is supposed
someto look out
implications that the product It manager
appears, however, the use of m
is using the
model in a partisan way to makethisa partisanship
case for a particu- because assum
explicit
lar position. Third, has the next level and subject to examinatio
of management
consideration
lost any control when the product of alternatives. For
manager's the same
case is reason,
buttressed by this new tool? although the next and higher levels of management
need to understand
The product manager was a little surprisedthe basic model
by ideas,
theonce this
is accomplished,
question but his answer was: "This isthe explicitness
what of the model and
I really
think. I've spent a lot of timeits inputs can actually make
considering thecommunications
assump- between
levels more
tions and results and feel they effective.
express my view of

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
Little: Models and Managers: The Concept of a Decision Calculus
Management Science 50(12S), pp. 1841-1853, b2004 INFORMS 1851

Table 4 Output for GROOVY

1 Save data
2 Print data
3 Change data
4 Output
5 Restart

ANS = 4

1 Standard output
2 Exclude specified lines
3 Include specified lines only

ANS = 1

1 Calculate current case

2 Search
3 Finished

ANS = 1

1 Output for GROOVY

2 Period length: Quarter
3 Starting period: 1ST Q 70
4 Area: US
5 Sales unit: Hogsheads
6 Data from file: /GROOVY-70/
8 Period 1 2 3 4
9 Market share: 1.868 1.999 2.002 2.009
(% of units)

10 Prod. class 273M 293M 309M 278M

sales (units/per.)

11 Prod. class 514M 552M 581M 523M

sales (dol./per.)

12 Brand sales 5.89M 5.87M 6.18M 5.59M

(units/per.)

13 Brand sales 9.22M 10.6M 11.2M 10.1M

(dol./per.)

14 Contribution 3.46M 3.99M 4.20M 3.80M

(dol./per.)

15 Brand adv 0.486M 0.606M 0.876M 0.414M

(dol./per.)

16 Cont. after 2.97M 3.38M 3.33M 3.39M

adv. (dol./per.)

17 Cumulative 2.97M 6.36M 9.69M 13.1M

cont. after adv.

23 Slope 1.634 1.169 0.228 -0.379

6. What Happened to Science

for a one-to-one corr
quantities
Science is concerned and quanti
with describing
fidelity and economy. We have
However, that propo
is not
managers describe that
the the
world as they
model s
repres
really afford to pass over issues like:
of comparison, how
world really work? rather
What iswhat the ma
the best way
it in a model? How Ifaccurate
you lookisata his
given
pre
do we measure accuracy?
has practically no p
Clearly these are important issues,
extrapolation alth
of the
is a fairly
tenabledetailed
position that says
fidelity are n
value from models,us
even
hopeif they do not
that carefu
world measurements. One of
a series can argue t
increasin
which
tative model can be used atas
the same tim
a qualitat
aid qualitative thinking. In this
controlled byrole
the the
use

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
 Little: Models and Managers: The Concept of a Decision Calculus
1852 Management Science 50(12S), pp. 1841-1853, q2004 INFORMS

Most of the models we are proposing herecommunicate

tend, atwith. Such a model, consisting of a
least initially, to be over parameterized of numerical
with respectprocedures for processing data and ju
to the available data. That is, we tend toments
put intomore
assist decision making, has been call
decision calculus.
phenomena than we know how to measure, but do
The model is
so anyway because we believe they are important. Asmeant to be a vehicle through which
a result, by suitably picking parameters awe manager can express his views about the opera-
can often
fit past data fairly easily. Therefore it maytions under his control. Although the results of using
be difficult
to develop a good a priori measure of the a model may sometimes
accuracy of be personal to the manager
the model. because of judgmental inputs, the researcher still has
We should, however, evaluate the model by track- the responsibilities of a scientist in that he should
ing performance, if this is at all applicable. As deci- offer the manager the best information he can for
sions are made, we usually forecast the future with making the model conform to reality in structure,
the model. We should see whether actual performanceparameterization, and behavior.
differs from the forecast. Ordinarily it will. Then Although it is really too early to tell, I would like to
the task is to determine why and correct the model predict how such a model will enter these companies
parameters and sometimes the model structure. Thisand how the companies will organize to make use
process will be greatly facilitated if the model con- of it. First of all, the managers have to learn how to
use the model. This requires technical assistance and
tains a variety of touch points with the real world, i.e.,
contains quantities which are observable in the real a teaching program. Technical assistance is required
world. The process will also be aided if we design for problem formulation and data analysis. As for a
and implement special measurement programs. One teaching program, our experience suggests that the
of the most obvious side benefits of model use is the best approach is to lead the potential user through a
pinpointing of critical measurements that should besequence of models of increasing scope and complex-
made. ity. This is essentially what we have done with the
model above and it is what Urban has done with his
The task of parameterizing the model is, of course,
difficult and important. A good methodology for this new product model, SPRINTER (Urban 1969). Often
is the one used by Armstrong for forecasting (Arm-a user, having learned a simple model, will start to
strong 1968). After he had specified what he hoped ask for just the additional considerations found in the
advanced models.
was a satisfactory structure, he proceeded as fol-
lows: First, all the parameters were set by judgment. As for organization, a matrix form seems to be
Then, he tried to estimate each through data anal-indicated. Under this setup the manager has line
ysis. He used as many independent data sets andresponsibility but also has a commitment from oper-
ations research and/or market research in terms of
approaches to analysis as he could invent and sepa-
rately appraised the accuracy of each. Then he com- somebody assigned to his area. The manager needs
a person to whom he can address questions about
bined the results up to that point by formal methods.
Using the now parameterized model, he made fore- model behavior and a person or persons who can help
casts and devised various means of evaluating their design measurements and do data analysis.
quality. One way was to make forecasts from new One of the most evident consequences of the expe-
rience to date has been that a model is a stone in the
data. Having done this, he readjusted his parameters
to use the information from the new data. The same shoe for better data. Under present planning proce-
dures many measurement problems are glossed over
sequence of initial parameterization, model use, new
data collection, and updating the parameters is anor suppressed. The model forces explicit considera-
tion of every factor it contains and so pinpoints data
adaptive procedure appropriate for most applications
needs.
of models to ongoing operations.

Acknowledgments
7. Conclusions This is the eighth in a series of twelve expository papers
In many respects, the biggest bottleneck in commissioned
the man- by the Office of Naval Research and the
Army Research Office under contract numbers Nonr-
agerial use of models is not their development but
4004(00) and DA 49-092-ARO-16 respectively. The under-
getting them used. I claim that the model builder
lying research was supported in part by a grant from the
should try to design his models to be given away.
Mobil Oil Corporation and in part by the Army Research
In other words, as much as possible, the Office
modelsunder contract number DA-31-124-ARO-D-209. The
should become the property of the manager,authornot the
wishes to acknowledge the contribution of several
technical people. I suggest that, to be used by a man- product managers who provided their time
unidentified
ager, a model should be simple, robust, easy and
to con-
enthusiasm for the example reported. An earlier ver-
trol, adaptive, as complete as possible, andsion
easy to paper was delivered at the TIMS/University of
of the

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms
Little: Models and Managers: The Concept of a Decision Calculus
Management Science 50(12S), pp. 1841-1853, @2004 INFORMS 1853

Chicago Symposium on Behavioral and

PtManagement Sci-
= brand contribution rate after advertising in t
ence in Marketing in June 1969. (dol./period)
Pt = mtst - xt (A.8)
Appendix. ADBUDG Model
ot = cumulative contribution after advertising for p
st = brand sales rate in period t (sales units/period)
1 to t (dol)
ht = brand share in period t
T = number of periods considered
ct = product class sales rate in period t (sales
units/period) t

st = hc, (A.1) t = EPs (A.9)

s=l

ht = unadjusted brand share in t

n, = nonadvertising effects index in t oTT(xt) = value of oT as a function of xt
,qt = the rate of change of oT with xt, called SLOPE
h, = nh, (A.2) and calculated by:

nT brand
a = persistence constant for unadjusted = [oT(xt +share
0.05x*) - aT(xt)]/0.05x* (A.1
/ = affectable range of unadjusted brand share
y = advertising response functionTheexponent
parameter setsfor
a, P, brand
y, 8, A and a', P', y', 8', A'
uniquely determined by
8 = advertising response function denominator constant the reference case data in the i
for brand
In the reference case nt = 1, elt = el, e2t = e* and ref
A = long run minimum brand share
ring to Figure 1, the items in Table 3, and A.2-A.
wt = weighted, normalized brand advertising
obtain in t
the following determining relations: A = (item
(item 9) = A + a[(item 7)-A]; (item 10) = A + a[(item 7)
ht = A + a(ht - A) + Swwt/(8 + k't) (A.3) p; (item 7)= A + a[(item 7)-A] + P/(8 + 1); (item 11)
elt = brand media efficiency in t a[(item 7)-A] + P3(1.5)Y/[8 + (1.5)1]. A similar set of re
e* = brand media efficiency reference value tions determines the primed parameters. If the optio
e2t = brand copy effectiveness in t to consider the effect of brand advertising on product
e* = brand copy effectiveness reference value sales is chosen, ct is set to the initial product class sales
xt = brand advertising rate in t (dol./period) of the reference period (item 22).
x* = brand maintenance advertising rate (dol./period)
References
wt = elte2xt/eIelx* (A.4)
Armstrong, J. S. 1968. Long-range forecasting for a consumer
dt = product class sales rate index in t durable in an international market. Ph.D. thesis, M.I.T., Cam-
bridge, MA.
it = unadjusted product class sales rate in t (sales
units/period) Dearden, J. 1964. Can management information be automated. Har-
ct = cedt (A.5) vard Bus. Rev. 42(March-April) 128-135.
Little, J. D. C. 1966. A model of adaptive control of promotional
a' = persistence constant for unadjusted product
spending. Oper. Res. class
14(November) 1075-1098.
sales
Little, J. D. C., L. M. Lodish. 1969. A media planning calculus. Oper.
p' = affectable range of product class sales rate (sales Res. 17(January-February) 1-35.
units/period) Mathes, R. C. 1969. "D" people and "S" people (letter). Science
y' = advertising response function exponent for product (May) 164 630.
class
Montgomery, D. B., G. L. Urban. 1970. Applications of Management
8' = advertising response function denominator constant Science in Marketing. Prentice-Hall, Englewood Cliffs, NJ.
for product class Pounds, W. F. 1969. The process of problem finding. Indust. Man-
A' = long run minimum product class sales (sales agement Rev. 11(Fall) 1-19.
units/period) Urban, G. L. 1969. SPRINGER: A model for the analysis of new fre-
vt = normalized product class advertising rate in t quently purchased consumer products. Working paper, Sloan
School of Management, M.I.T., Cambridge, MA.
Et = A' + a'(t-1 - A') + P'vt /(8' + vt) (A.6)
This article originally appeared in Management Science,
v* = maintenance advertising rate for product class sales April 1970, Volume 16, Number 8, pp. B-466-B-485, pub-
(dol./period)
lished by The Institute of Management Sciences. Copy-
vt = (v* - x* + xt)/v* (A.7) right is held by the Institute for Operations Research
and the Management Sciences (INFORMS), Linthicum,
mt = brand contribution per unit in t (dol./sales unit) Maryland.

This content downloaded from 49.248.211.170 on Sun, 03 Feb 2019 17:57:01 UTC
All use subject to https://about.jstor.org/terms