Prediction Markets: A New Tool for Strategic Decision Making

Author(s): Adam Borison and Gregory Hamm

Source: California Management Review, Vol. 52, No. 4 (Summer 2010), pp. 125-141
Published by: University of California Press
Stable URL: http://www.jstor.org/stable/10.1525/cmr.2010.52.4.125 .
Accessed: 20/10/2011 16:30
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.
University of California Press is collaborating with JSTOR to digitize, preserve and extend access to
California Management Review.
http://www.jstor.org
California
Management
S u m m e r 2 0 1 0 | V o l . 5 2 , N o . 4 | R E P R I N T S E R I E S
2010 by The Regents of
the University of California
Review
Prediction Markets:
A New Tool for Strategic Decision Making
Adam Borison
Gregory Hamm
125 CALIFORNIA MANAGEMENT REVIEW VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU
Prediction Markets:
A NEW TOOL FOR STRATEGIC
DECISION MAKING
Adam Borison
Gregory Hamm
U
ncertainty is at the center of many of the most important strate-
gic decisions made by private businesses and public agencies.
In the private sector, uncertainty dominates decision
making in key areas such as R&D, product introduction, mar-
ket entry, project development, and technology choice. Toyotas well-known
introduction of the Prius is a good example. During development and even early
in the introduction process, there were a variety of commercial and technical
problems. Success was by no means inevitable, but Toyotas CEO Watanabe sup-
ported the cars introduction despite this uncertainty. As he said in a 2006 Forbes
interview, I did not envisage such a major success.Some thought it would
grow rapidly, and others thought it would grow gradually. I was in the second
camp. The leader of this effort, Takehisa Yaegashai, has been quoted as saying
the likelihood of success at the time was estimated at 5%.
1
Of course, as it has
turned out, the Prius is now regarded as a considerable market success.
In the public sector, uncertainty plays a pivotal role in many policies and
regulations having far-reaching effectsranging from climate change to health
care to national security. Perhaps the most memorable and controversial recent
example is the invasion of Iraq, which was justied largely based on assess-
ments of the likelihood that Iraq possessed weapons of mass destruction. There
appeared to be substantial evidence, of varying qualities, to support this view.
As White House Press Secretary Ari Fleischer said in 2003, We have high con-
dence that they have weapons of mass destruction.
2
It is (perhaps surprisingly)
difcult to nd any quantitative likelihood estimates from before the invasion.
However, an interesting retrospective analysis by two military authors assigns a
probability, at the time, of 75% that Iraq had WMDs based on the evidence then
available.
3
Of course, subsequent events have proven otherwise.
Prediction Markets: A New Tool for Strategic Decision Making
UNIVERSITY OF CALIFORNIA, BERKELEY VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 126
Not all strategic decisions are as signicant (or as well publicized) as these.
However, strategic decisions are regularly made that have substantial impacts
on the decision maker, on the decision makers organization, and on the organi-
zations stakeholders. In the private sector, these stakeholders include not only
employees, but customers and suppliers, stock and bond holders, and regula-
tors. In the public sector, these stakeholders can include many, most, or even
all citizens. In making these decisions, responsible decision makers seek out the
best possible information to support their choicesand affected stakeholders are
increasingly demanding this as well.
Some strategic decisions are largely data-driven; that is, there is a wealth
of historical data on which to base assessments of the uncertainties that underlie
them. Decision makers then face the time-consuming but relatively straightfor-
ward task of collecting and interpreting this historical data. Good examples are
weather and commodity prices. There is a wealth of detailed historical data in
both these cases. If an important decision depends on the weather conditions at
a specic location and season, such as when siting a hydroelectric facility, a great
deal of historical data can be brought to bear on that decision. Although not
without complications, this is the easy case.
Other strategic decisions are largely judgment-driven; that is, there is
little or no directly-relevant historical data on which to base assessments of the
uncertainties that underlie them. Decision makers then face the challenging
task of eliciting and processing judgmenttheir own or the judgment of others.
Good examples are technology development and social change. There may be
past analogues of varying degrees, but there is no database of equivalent events.
If an important decision depends on the performance of a new technology such
as carbon sequestration, judgment rather than data is the focus.
Of course, this distinction between data-driven decisions and judgment-
driven decisions is a bit extreme. Judgment plays a role even where there is a
large amount of data. For example, the weather data noted above must be inter-
preted in light of judgment about climate change. Data plays a role even where
judgment dominates. For example, data
from other past technological innovations
may be applied to the performance of car-
bon sequestration technology. Recognizing
the importance of both data and judgment,
research has been conducted not only on
obtaining the best data and the best judg-
ment, but combining judgment with data
formally.
5
Whether alone or in combination with data, judgment plays a critical
role in most important decisions in both the public and private sectors. That
role appears to be growing if one examines the issues that dominate the cur-
rent business and policy environment: the cause, rate, and response to climate
change, the impact of globalization, the prospects for recovery from our current
economic/nancial crisis, the relative roles of the public and private sectors,
Adam Borison is a Senior Vice President at NERA,
on the adjunct faculty at Stanford, and a Board
Member of Agni Corporation, a biomass energy
venture. <adam.borison@nera.com>
Gregory Hamm is a Senior Economist at NERA, on
the adjunct faculty at Stanford, and President of
the Board of Alameda Municipal Power. <gregory.
hamm@nera.com>
Prediction Markets: A New Tool for Strategic Decision Making
CALIFORNIA MANAGEMENT REVIEW VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 127
the risks of regional conicts, the extent of environmental legislation, or the
advancement of technology.
An emerging toolprediction marketscan improve the current prac-
tice for assessing judgmental uncertainty in strategic decision making, thereby
improving the quality of decisions that affect us all.
Current Practice
How is judgmental uncertainty currently treated in strategic decision
making? Some decision makers deal with this uncertainty informally. They rely
on intuition, experience, or perhaps luck rather than analyzing uncertainty in a
rigorous fashion. Some decision makers take a somewhat more rigorous, yet still
qualitative approach. They rely on tools such as scenario planning, contingency
planning, stress testing, or sensitivity analysis that describe alternative futures
but do not assign relative likelihoods or probabilities.
These informal and qualitative approaches certainly have value, and
many decision makers who rely on these approaches are successful. However, it
is difcult to see how they can be fully relied upon and defended when decisions
depend not just on the possibility of an event, but on its probability. Of course,
in the extreme, this is true of all decisions affected by uncertainty. For example,
the wisdom of entering a new market generally depends on the likelihood that
the entry will be successful. The wisdom of devoting substantial resources to the
development of a new drug depends on the likelihood that the drug will be safe
and effective. The wisdom of committing to a new technology depends on the
likelihood that the technology will work economically.
Decision makers rely increasingly on formal quantication of judgmental
uncertainty to support their strategic decisions. In part, this is because the tech-
nology for collecting and processing information has gotten much better and has
become much more widespread over the years, making quantication more fea-
sible. In part, this is because evidence is accumulating that formal analysisand
quantication in particularimproves decision making, making quantication
more desirable. Russo and Schoemaker describe many of the traps that intui-
tive decision makers fall into that can be avoided with more formal analysis.
6
When it comes to developing quantitative estimates of judgmental uncer-
tainty, the current state of the art is based on advances in decision theory start-
ing in the middle of the last century. Over the past several decades, decision
theory has produced two well-established techniques for this purpose. The rst
is to use a structured probability encoding process to assess the opinion of individ-
ual experts. The second is to use a structured expert aggregation process to com-
bine these individual opinions. These two processes have been used in a wide
range of applications both in the public and private sectors, particularly when
faced with bet the organization decisions. In the case of probability encoding,
there is even a formal manual for the process.
8
Prediction Markets: A New Tool for Strategic Decision Making
UNIVERSITY OF CALIFORNIA, BERKELEY VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 128
Probability Encoding
Assessing judgmental uncertainty is not just a matter of asking an expert
what do you think the odds are of X? Individual judgment may be difcult to
capture formally, and typically suffers from a variety of cognitive (unintentional)
and motivational (intentional) biases. For example, on the cognitive side, most
people unintentionally overweight memorable recent experience in their assess-
ments; this is called availability bias. On the motivational side, many salespeople
low ball their assessments intentionally because they are rewarded for exceed-
ing them.
8
Probability encoding is a formal process for eliciting judgmental probabili-
ties from individuals that was developed and codied beginning in the 1960s,
largely in recognition of these biases. The formal encoding process helps ensure
that the individual understands the need for the assessment and the impor-
tance of a high-quality assessment. It also helps ensure that the probabilistic
event being assessed is well-dened and, if necessary, that the event is struc-
tured as better-understood component pieces. Finally, it helps ensure that biases
are uncovered and minimized, and that the individual will stand behind the
assessment.
9
As described by Spetzler and Stael von Holstein, probability encoding is a
ve-step process:
MotivatingRapport with the subject is established and possible motiva-
tional biases are explored.
StructuringThe structure of the uncertain quantity is dened.
ConditioningThe subject is conditioned to think fundamentally about his
judgment and to avoid cognitive biases.
EncodingThe subjects judgment is quantied in probabilistic terms.
VerifyingThe responses obtained in the encoding are checked for
consistency.
10
Research continues on renements and enhancements to this probability
encoding process. Some work focuses on the types of questions that should be
asked. For example, a recent article by Abbas et al. compares probability assess-
ment based on xing a value rst and then asking for a probability with xing
a probability rst and then asking for a value.
11
Other work focuses on scoring
rules based on actual event outcomes that provide incentives for honest esti-
mates.
12
Despite these more recent developments, the fundamentals of the prob-
ability encoding process have not changed substantially for more than twenty
years.
Figure 1 shows a real example of the output of this process for a continu-
ous variablethe future cost of a new environmental control technology. One
can easily imagine how this uncertainty could play a signicant role in both
business strategy and public policy. In this gure, each asterisk (*) represents a
response to a particular question, such as Which is more likelythat I roll a
die and get a six or that the new technology costs less than $20M? Once all the
responses are collected, a cumulative probability distribution is tted to the data
Prediction Markets: A New Tool for Strategic Decision Making
CALIFORNIA MANAGEMENT REVIEW VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 129
and veried with the individual. Once the encoding process is complete, the
resulting judgmental probability distribution serves as input into the strategic
decision-making process through decision analysis, real options, simulation, and
the like.
A similar process is also used for events with discrete rather than continu-
ous outcomes. In this context, a good example might be passage of a particular
piece of environmental legislation.
Expert Aggregation
The probability encoding process outlined above is appropriate and suf-
cient when a single individual has been identied as the source of information
on the judgmental uncertainty in question. Often, however, decision makers and
other stakeholders will not want to rely on a single individual. Collecting infor-
mation from multiple individuals will better reect the true state of knowledge
and will be easier to justify and defend. Under these circumstances, the probabil-
ity assessments of multiple individuals are combined using expert aggregation.
FIGURE 1. Encoded Probability Distribution on the Cost of a New Environmental Control
Technology
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
10 20 30 40 50 60 70 80 90
VARIABLE __________________________ UNITS _________________________ DATE ________________
EXPERT ____________________________ INTERVIEWER __________________ COMMENTS __________
NOx Response
Jack
$ Millions
Marion
Jun-30
Prediction Markets: A New Tool for Strategic Decision Making
UNIVERSITY OF CALIFORNIA, BERKELEY VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 130
In many ways, expert aggregation is a more complex issue than probabil-
ity encoding. The theory behind combining multiple assessments into a single
group assessment is not straightforward analytically, and managing interactions
among multiple individuals can be challenging organizationally. Consequently,
several approaches have been developed for expert aggregation in different con-
texts. This work started largely in the 1960s and continues to this day.
13
As explained by Clemen and Winkler, expert aggregation approaches
generally fall into one of two categoriesmathematical and behavioral.
14
In
the former, individual probability assessments are collected and then combined
mathematically according to a set of rules and parameters. In the latter, a group
of individuals interact according to a set process, share information, and revise
their probability assessments accordingly. In some cases, a consensus is achieved
and a group probability assessment emerges. In other cases, no consensus is
achieved and additional mathematical aggregation is required.
Again according to Clemen and Winkler, mathematical approaches have a
slight edge over behavioral ones. Among the mathematical approaches, a simple
average of the individual probability assessments appears to perform the best
or nearly so. Among behavioral approaches, the results are less clear. The well-
known Delphi technique appears to perform as well but no better than other
behavioral approaches.
Figure 2 shows a real example of the inputs and outputs in a mathemati-
cal expert aggregation process using a simple average. (Note that the variable
in questionpeak ground acceleration during an earthquakeis different from
Figure 1.) The inputs are the probability assessments of the individuals. In this
example, the assessments of three experts are shown. The output is the group
probability assessment that combines the individual views. As with individual
probability distributions, this aggregate distribution can be used directly to sup-
port strategic decision making.
Limitations
The decision theory approach described above is effectively the state of
the art for quantitative assessment of judgmental uncertainty for strategic deci-
sion making. Despite this, the approach suffers from some important limitations.
First, any method that relies on individual judgment must address
the well-documented biases that are associated with that judgment. As noted,
Tversky and Kahneman reported years ago on groundbreaking research in
identifying and characterizing these biases.
15
More recently, other authors have
described the foibles of individual judgment in a variety of contexts. Loftus
focuses on the problems that individuals have in remembering and reporting
even the most basic facts.
16
Camerer and Johnson
17
and Tetlock
18
show that
even so-called experts dont appear to fare much better in medical and politi-
cal contexts respectively. Formal probability encoding is designed to overcome
many of the biases in individual judgment. However, there is really no signi-
cant theory underlying the level of quality of the resulting assessmentsparticu-
larly their degree of calibration or consistency with actual event outcomes. There
Prediction Markets: A New Tool for Strategic Decision Making
CALIFORNIA MANAGEMENT REVIEW VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 131
are no principles regarding the degree to which biases are reduced through spe-
cic encoding procedures or the extent of procedures required to achieve a spec-
ied level of quality. This lack of a strong foundation is viewed as an important
shortcoming and area of improvement, even by leading academics and practitio-
ners in the eld. As Ferrell notes, Subjective probability elicitation and combi-
nation needs a rmer theoretical structure if [it is] to be used with condence.
19
Second, the empirical evidenceas distinct from the theoretical founda-
tionregarding the quality of the assessments is relatively sparse and mixed.
There is certainly empirical evidence, although not as much as you might
expect, that formal methods are an improvement over informal ones. For
example, Browne et al. report that formal probability encoding tools produce
demonstrably better results.
20
However, there is little evidence about the result-
ing quality of these assessments (as opposed to the improvement). Furthermore,
much of the evidence involves very specic applications and tools, making it
difcult to draw general conclusions. Cooke conducted one of the few broader
reviews on the quantitative assessment of judgmental uncertainty, largely in
the context of science-based probabilistic risk analysis.
21
His observations are
instructive. He reports on one review of 84 empirical studies where 47 reported
poor performance, while 37 reported good performance. He notes that litera-
ture citations show a much greater prevalence of poor performance (poor cita-
tions outweigh good citations by a factor of 6), but cautions against relying
FIGURE 2. Individual and Aggregated Probability Assessments
Note: Taken from Robert T. Clemen and Robert L. Winkler, Aggregation of Expert Probabilities, in Advances: Aggregate Probabilities (Draft), 2006,
chapter 9.
0 100 200 300 400 500 600 700 800
0.90
0.80
0.70
0.60
0.50
0.30
0.20
0.10
0.00
0.40
1.00
Peak Ground Acceleration (cm/sec
2
)
C
u
m
u
l
a
t
i
v
e

P
r
o
b
a
b
i
l
i
t
y
Expert A
Expert B
Expert C
Aggregated
Prediction Markets: A New Tool for Strategic Decision Making
UNIVERSITY OF CALIFORNIA, BERKELEY VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 132
too heavily on these results because bad news typically dominates good news.
His overall conclusion is that expert opinions in probabilistic risk analysis have
exhibited extreme spreads, have shown clustering, and have led to results with
low reproducibility and poor calibration. Despite this conclusion, he remains
a glass half full supporter of the approach.
Third, perhaps due to the two limitations noted above, the internal
credibility and external defensibility of the approach is a matter of concern.
Internally, decision makers have difculty justifying the use of judgmental
assessments. This is particularly true in contexts where decision making is
typically based on (more credible) historical frequency data. Externally, these
judgmental assessments have difculty getting widespread support among stake-
holders. For example, speaking of delays in the use of probabilistic risk analysis
(PRA) in nuclear power regulation, Bier notes, the subjectivity of PRA results is
largely responsible for the delay.
22
On the legal side, there has been a debate for
at least a hundred years on the role of expert testimony, specically probabili-
ties, in both civil and criminal cases. In the late 1800s, for example, the Supreme
Court of Mississippi made its feelings about subjective probability assessments
quite clear.
The courts . . . have gone quite far enough in subjecting the life, liberty, and prop-
erty of the citizen to the mere speculative opinions of men claiming to be experts in
matters of science, whose condence, in many cases, bears a direct similitude and
ratio to their ignorance. We are not disposed to extend this doctrine into the eld
of hypothetical conjecture and probability, and to give certainty as evidence to
that which, in its very nature, must be wholly uncertain and unsatisfactory.
23
More recently, there has been considerable study and debate on the role
of probabilities and probabilistic analysis in court, including high-visibility crimi-
nal cases.
24
The greater the scrutiny faced by public and private decision makers,
the more important the ability to justify and defend the approach taken to quan-
tifying judgmental uncertainty.
These limitations underlie an argument in favor of current practice that is
remarkably similar to the argument in favor of democracy as a political system;
that is, it is imperfect but the best we can do. This is clearly the tone taken in
Morgan and Henrion: One can only proceed with care, simultaneously remem-
bering that elicited expert judgments may be seriously awed, but are often the
only game in town.
25
This situation leaves many decision makers and stake-
holders in a quandary. They would like to approach strategic decisions involv-
ing judgmental uncertainty with more rigor, but they are reluctant to rely on
an approach that suffers from potentially serious limitations. This reluctance is
amplied where decisions are made, reviewed, or evaluated in a public context.
Prediction Markets
Until recently, the decision theory approachprobability encoding and
expert aggregationwas generally the best that strategic decision makers and
stakeholders could do with respect to judgmental uncertaintydespite the limi-
Prediction Markets: A New Tool for Strategic Decision Making
CALIFORNIA MANAGEMENT REVIEW VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 133
tations. However, a new approach has emerged in the past few years with the
potential to put decisions based on judgmental uncertainty on a better founda-
tion. The term associated with this approach is most commonly prediction mar-
kets, although predictive, information and decision markets are used as well.
26
The fundamental principle behind prediction markets is to create a
market where a number of individuals can place bets on a specic outcome
of interest, such as a political or social event. The market is closed when the
event occurs, the outcome is revealed and the participants win or lose based on
that outcome. The bets establish a price for a contract tied to the outcome, and
a group probability assessment of the outcome, a prediction, can be obtained
directly or indirectly from this price. For example, a prediction market could
include a bet on whether the President elected in 2012 will be a Democrat. To
bet on this outcome, the participant buys a contract that pays one dollar if a
Democrat wins the 2012 election. Until the outcome of the 2012 election is
revealed, the price of this contract will be something less than (or equal to) a
dollar because the most it can return is a dollar. Under suitable conditions, the
current equilibrium market price of the contract, say $0.60, represents the group
assessment of the probability that the President elected in 2012 will be a Demo-
crat, 60%. (This gure is hypothetical.)
Based on archeological discoveries, betting appears to be a human activ-
ity dating back thousands of years. Furthermore, there are examples dating back
hundreds of years of organized efforts to use betting as a kind of probabilistic
forecasting tool akin to a prediction market. The modern history of prediction
markets, however, goes back a little over twenty years to 1988 when the Uni-
versity of Iowa created the Iowa Electronic Market (IEM) for betting on the out-
come of the upcoming Presidential election. The IEM has continued to function
and expand modestly over the years.
Since the IEM was founded, the academic and commercial interest in
prediction markets has grown and the number and scope of such markets has
increased dramatically. There are now many variations on the basic theme.
Some markets involve real money, others involve virtual or play money. Some
involve simple contracts such as the one mentioned above, others involve more
complicated conditional contracts.
27
One very important distinction is that some
markets are public and others are private.
There are currently a handful of public prediction markets. These are
open to all comers, thereby truly working to take advantage of the wisdom of
crowds or the view that such markets work best with large numbers of diverse
participants.
28
As public markets, they generally focus on political, social, and
economic events of broad general interest. Public markets that use real money
are subject to considerable regulatory oversightparticularly in the United
States. This has restricted their development and created some turmoil in this
new industry. Many public market developers are motivated by the information
content that they then use for professional or business reasons. Others are moti-
vated directly by the nancial rewards of the market itself. Although it is dif-
cult to tell, participants in public markets appear to be motivated by a mixture
of curiosity, pride, and money.
Prediction Markets: A New Tool for Strategic Decision Making
UNIVERSITY OF CALIFORNIA, BERKELEY VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 134
In addition to the IEM, three good examples of public markets are
Intrade, the Hollywood Stock Exchange, and the CFO Prediction Market. Intrade
is an Ireland-based company that is currently the leader in public prediction
markets. In fact, Intrades tag line is The Prediction Market. Members wager
real money, and Intrade offers a very broad range of potential bets in categories
such as art, business, current events, entertainment, politics, science, weather,
and more. For example, it includes bets on the passage of CO2 cap and trade
legislation and the conrmation of cold fusion. Like most exchanges or brokers,
Intrade prots by earning a small fee on each transaction. Following up on the
hypothetical example at the beginning of this section, Figure 3 shows the actual
market data for the Intrade bet on the 2012 U.S. election. Based on this mar-
ket, the current group probability (as of October 25, 2009) is roughly 65% that
a Democrat will win the 2012 election. The Hollywood Stock Exchange has a
much narrower focus. Participants use play rather than real money to bet on
the success or failure of movies, and they can redeem their winnings for prizes.
The owners of the Hollywood Stock Exchange sell the information to the enter-
tainment industry. The CFO Prediction Market is run on CFO.com by CFO
magazine. Naturally, it focuses on variables of interest to a nancial executive
audience, such as economic growth. The value to the organizer is primarily pub-
licity. Winning participants are awarded modest prizes, although the primary
motivation appears to be personal pride at being smarter than the experts.
Private prediction markets are considerably more common than pub-
lic ones. These markets are open only to members of a specic group, usually
employees of a particular company or a subset of those employees with particu-
lar qualications. As such, they focus on events of interest to that specic group,
such as the completion date of an important project or the introduction of a new
product by a competitor. Market developers are motivated entirely by the infor-
FIGURE 3 Intrade Price for Democrat Winning 2012 Election
Source: www.intrade.com .
PRESIDENT.DEM.2012
Nov 01, 2008 Oct 25, 2009
Closing Price: 64.7 Vol: 0
600
400
200
0
Nov 08 Dec Jan 09 Feb Mar Apr May Jun Jul Aug Sep Oct
V
o
l
u
m
e
P
r
i
c
e
50
55
60
65
70
75
80
Prediction Markets: A New Tool for Strategic Decision Making
CALIFORNIA MANAGEMENT REVIEW VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 135
mation provided, and this information is communicated directly to the appro-
priate decision makers. Again, it is difcult to tell, but participants are typically
motivated by loyalty to their group, as well as pride and curiosity. These markets
involve play money and prizes; it appears that prizes are rarely a strong motiva-
tion in these private markets.
Several organizations have been quite public about their use of private
prediction markets. These markets seem to be particularly popular among tech-
nology rms in Silicon Valley. According to Bo Cowgill, a product manager who
oversees their prediction markets, Google has run hundreds of such markets
internally.
29
The major topics are company performance, new product perfor-
mance, and industry trends. Google also runs fun markets on sporting and
other events primarily for entertainment and familiarization. BestBuy, General
Electric, HP, Yahoo, and many other prominent rms have also publicized their
use of prediction markets. In government, the most famous (or infamous) pre-
diction market was the one set up by the Department of Defense on political
events that was aborted because of the public and Congressional reaction to the
idea of betting on terrorist attacks, assassinations, and the like.
Applications to Strategy
There is a great deal of popular enthusiasm over the potential for predic-
tion markets for strategic decision making.
30
There is also a small but growing
body of formal research, at least at a conceptual or experimental level, on the
design and implementation of prediction markets to support strategic decisions.
31
As noted above, academic and commercial interest in prediction markets
has grown substantially since the 1980s. Considerable research and experimen-
tation has been conducted on these markets and on what conditions make them
more or less successful in different contexts. Based on this work, three criteria
appear to be most important specically in the context of strategic decision
making.
The rst criterion deals with the design of the market itself and contracts
in that market. There are many potential avors of prediction markets. There
are different methods for determining the type and magnitude of potential bets,
for matching buyers and sellers and for determining the payoffs to winners. In
the context of strategic decision making, care must be taken in this design so
that contract prices can be interpreted without ambiguity as a group probability
assessment, and the probability assessment is as accurate and credible as possi-
ble. Some of the simple designs that are used in existing markets, and may work
adequately in those contexts, may not be suitable.
The second criterion deals with the selection of variables for the market.
These variables must be dened in a way that meets two potentially conicting
goals. They must be clear and relevant to the participants (otherwise, the qual-
ity of the results will be poor); and they must directly t the decision-making
context (otherwise, the usefulness of the results will be limited). In existing pub-
lic prediction markets, only the former consideration is paramount. In existing
Prediction Markets: A New Tool for Strategic Decision Making
UNIVERSITY OF CALIFORNIA, BERKELEY VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 136
private prediction markets, the latter criterion has dominated and markets have
sometimes suffered from lack of interest and activity as a result. A good example
of this challenge involves long-term events. From a strategic decision-making
point of view, an organization may be very interested in the long-term potential
of a new technology or product. On the other hand, participants may be largely
uninterested in betting on an event whose outcome remains unresolved for
years. In this particular case, selection of a portfolio of short-term and long-term
variables, along with careful market design, can help overcome this problem.
The third criterion deals with the selection of participants in the market.
Many existing markets, both public and private, rely on large numbers of par-
ticipantshundreds or thousandsto provide both liquidity and diversity. For
strategic decision making, this may not be feasible or desirable. The challenge
then is to nd a relatively small group of participantsas few as twentythat
is both sufciently knowledgeable and sufciently diverse. In practice, the impli-
cation is that markets aimed at strategic decision making should not be open to
all comers, as in existing public markets, nor should it be limited only to a par-
ticular organization, as they are in existing private markets. Participants instead
should include a mix of qualied individuals with different backgrounds, experi-
ences, perspectives, and afliations.
Our rm, National Economic Research Associates (NERA), recently estab-
lished a form of prediction market we call the Clean Energy Exchange. This market
is based on software from Crowdcast, a leading vendor, and includes a mix of
NERA and non-NERA participants. This market is intended to generate informa-
tion that participants can use for risk management and strategic planning. It is
currently focused on issues of interest to the energy industry, such as nuclear
and solar technology, carbon-related policy, and resource prices. The design and
management of this market is based on the criteria outlined above, and we are
working to address the resulting challenges. Figure 4 shows the current status
(as of October 25, 2009) of one market variablethe amount of nuclear capac-
ity under construction worldwide at the end of 2011. This market is constructed
specically to generate information in the form of a probability distribution. This
particular distribution shows that the consensus forecast among our participants
is a little over 54000MW, and there is roughly a 1% chance that the amount will
be as low as 14,000MW or as high as 95,000MW.
Advantages
With careful design and execution along the lines mentioned above,
prediction markets have three advantages that appear to address the limitations
discussed earlier regarding current practice built on decision theory. Specically,
prediction markets have advantages in their theoretical foundation, empirical
evidence, and credibility/defensibility.
On a theoretical level, prediction markets have three important features
that distinguish them from current practice. First, they are built on real rather
than hypothetical choices. In current practice, participants indicate hypotheti-
cally how they would choose or would allocate something of value if they had
Prediction Markets: A New Tool for Strategic Decision Making
CALIFORNIA MANAGEMENT REVIEW VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 137
the choice. In prediction markets, participants make real choices and make real
allocations of something of value to those choices. They walk the walk. In
principle, this will provide a better measure of the true beliefs of each partici-
pant. Second, the relative condence or strength of opinion of each participant
is captured directly and measured explicitly in the amount of money they are
willing to bet. In current practice, condence is either not measured at all or
is self-reported. In principle, this will provide a better measure of the relative
knowledge of each participant. Third, prediction markets can draw on the power
of rational expectations to argue that long-run results from such markets must
not be systematically biased. Otherwise, of course, there would be opportunities
for risk-free return. No such argument can be made for current practice. In prin-
ciple, this will provide for higher-quality assessments over the long-run.
On an empirical level, there is a limited but growing body of evidence
that prediction markets can be superior to other forecasting techniques. It is
widely reported that the IEM has been more accurate than most opinion polls.
32
This is encouraging but leaves open the question of accuracy versus political
experts using formal probability encoding procedures. Evidence that private
prediction markets outperform experts at rms such as Google and HP has also
been reported.
33
This is encouraging, although it is unclear how the assessments
from experts were obtained. Statistical evidence is more limited than anecdotal
evidence, but is also encouraging. For example, Pennock et al. report on the
results of research on public prediction markets in a letter to Science indicating
FIGURE 4. Nuclear Capacity Under Construction Worldwide at End of 2011
(NERA Prediction Market)
MW
14101 27537 40973 54409 67845 81281 94717
Prediction Markets: A New Tool for Strategic Decision Making
UNIVERSITY OF CALIFORNIA, BERKELEY VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 138
the assessments from these markets are very well-calibrated; that is, they are
highly consistent with actual outcomes.
34
Third, prediction markets have important characteristics that may make
them more credible and defensible than current practice. Current practice relies
heavily on one-on-one discussions with an individual or a group of individuals.
Courts, regulators and the public have concerns with probabilities that emerge
from such one-on-one discussions, and arguments that these assessments are
the best we can do carry only modest weight. On the other hand, despite the
current nancial crisis, there appears to be considerable belief in market mecha-
nisms and market prices for generating forecasts. Consequently, although it is
too early to tell, probability assessments that emerge from prediction markets
may have greater support both within and outside of organizations than assess-
ments based on current practice.
Limitations
Of course, prediction markets suffer from their own set of limitations. The
widespread application of these markets is relatively new, and there are a vari-
ety of conceptual and practical issues that have not been fully addressed at this
stage.
One limitation is simply the other side of the market coin. While there
may be many good empirical and theoretical reasons to believe in markets as
good predictors, there are also good reasons to believe that markets can fail in
small and large ways on this front. With respect to traditional markets, there is
a growing body of work in behavioral economics that is devoted precisely to this
observation. Thaler touches on many of the most important topics in this eld.
35
There is also considerable work on predictive errors in other relevant contexts,
such as pari-mutuel betting at race tracks.
36
Overall, the appropriate response
appears to be cautionmarkets typically generate useful information but they
are not perfect and can stray signicantly from the truth.
A second limitation is not that markets may be subject to unintentional
bias, but that they can be intentionally manipulated. Again, there is both empiri-
cal and theoretical research to support this view. In traditional markets, the
incentive for manipulation is direct and substantial. The manipulators stand to
make a lot of money from the market. In virtually all prediction markets, this
incentive is typically absent or minimal since the money involved is small (or
even nonexistent). However, there can be other reasons for manipulation. If
information from a prediction market, public or private, is going to be used for
business or policy decisions, it may be to the advantage of affected organiza-
tions to manipulate that information. In the case of a private prediction market
within a rm, for example, individuals or groups may have a strong incentive to
bias the assessment of the release date or projected sales of a particular product.
In the case of a public prediction market, one rm may have an incentive to
provide false information if they believe a competitive rm may use the market
information in its decision making. While the incentive for manipulation is gen-
Prediction Markets: A New Tool for Strategic Decision Making
CALIFORNIA MANAGEMENT REVIEW VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 139
erally smaller than in traditional markets, the difculty is likely to be smaller as
well. Again, the appropriate response appears to be caution.
Finally, a third limitation is that these markets can be most problematic
for precisely the long-term forecasting that is critical to much strategic decision
making. Prediction markets are best suited to events that are both near-term
(say, resolved within a year) and discrete (say, involving only yes and no
outcomes). Near-term means that participants are attentive and motivated,
and that bets can be thought of as being fundamentally on the event outcome,
not the direction of the market prior to learning that outcome. Discrete means
that the probabilistic information can be extracted from a single bet, where the
price of this contract is the consensus probability. It is more difcult to design
markets that address long-term events, say the price of nuclear technology in
2020 rather than the price of solar panels in 2010. It is also more difcult to
design markets for continuous variables, say the price of CO2 allowances rather
than the passage of CO2 legislation. This represents a challenge for prediction
markets in the strategic decision-making context; one that further research and
experience may address.
Conclusion
It is always exciting when a new mouse trap comes along. It is particu-
larly exciting when this new mouse trap has the potential for signicant positive
impact. This appears to be the case with prediction markets and their application
to strategic decision making. While it is important not to be carried away by the
latest fad and to recognize the limitations of this new tool, our own judgmen-
tal probability assessment is that prediction markets will play a major role in
improving the quality of strategic decisions in the future.
Notes
1. Adrian J. Slywotzky, The Upside: The 7 Strategies for Turning Big Threats into Growth Break-
through (New York, NY: Crown Publishing Group, 2007), p. 24.
2. Don Van Natta, Jr., and David Johnston, A Nation At War: Banned Weapons, New York
Times, April 14, 2003.
3. Maj. Ernest Y. Wong and Lt. Col. Rod Roederer, Should the US Have Attacked Iraq? Can
Decision Theory Shed Light on Polarizing Debate? OR/MS Today (December 2006).
4. For an example of combining judgment and data from model results, see Robert C. Blattberg
and David J. Hoch, Database Models and Managerial Intuition: 50% Model + 50% Man-
ager, Management Science, 36/8 (1990): 887-899.
5. For a discussion of methods for combining statistical data and judgment in the context
of time series forecasting, see M.J. Lawrence, R.H. Edmundson and M.J. OConnor, The
Accuracy of Combining Judgmental and Statistical Forecasts, Management Science, 32/12
(December 1986): 1521-1532; J. Scott Armstrong and Fred Collopy, Integration of Statisti-
cal Methods and Judgment for Time Series Forecasting: Principles from Empirical Research,
International Journal of Forecasting, 21 (1998): 69-293.
6. J. Edward Russo and Paul J.H. Schoemaker, Decision Traps: The Ten Barriers to Brilliant Deci-
sions and How to Overcome Them (New York, NY: Simon & Schuster, 1989).
7. Carl-Axel S. Stael von Holstein and James Matheson, A Manual for Encoding Probability Distri-
butions (Menlo Park, CA: SRI International, 1978).
Prediction Markets: A New Tool for Strategic Decision Making
UNIVERSITY OF CALIFORNIA, BERKELEY VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 140
8. For a more in-depth discussion of biases, see A. Tversky and D. Kahneman, Judgment
under Uncertainty: Heuristics and Biases, Science, September 27, 1974, pp. 1124-1131.
9. For more information on probability encoding, see Daniel G. Brooks and Timothy J. OLeary,
A Comparison of Encoding Techniques, OMEGA: The International Journal of Management
Science, 11/1 (1983): 49-55; Carl S. Spetzler and Carl-Axel S. Stael von Holstein, Probabil-
ity Encoding in Decision Analysis, Management Science, 22/3 (November 1975): 340-358;
Thomas S. Wallsten and David V. Budescu, Encoding Subjective Probabilities: A Psychologi-
cal and Psychometric Review, Management Science, 29/2 (February 1983): 151-173.
10. Spetzler and Stael von Holstein (1975), op. cit.
11. Ali E. Abbas, David V. Budescu, Hsiu-Ting Yu, and Ryan Haggerty, A Comparison of Two
Probability Encoding Methods: Fixed Probability vs. Fixed Variable Values, Decision Analysis,
5/4 (2008): 190-202.
12. Robert L. Winkler, Scoring Rules and the Evaluation of Probabilities, TEST, 5/1 (June
1996): 1-60.
13. For more on expert aggregation, see Robert T. Clemen and Robert L. Winkler, Combin-
ing Probability Distributions from Experts in Risk Analysis, Risk Analysis, 19/2 (April
1999): 187-203; Daniel Osherson and Moshe Y. Vardi, Aggregating Disparate Estimates
of Chance, Games and Economic Behavior, 56/1 (July 2006): 148-173; Thomas S. Wallsten,
David V. Budescu, Ido Erev, and Adele Diederich, Evaluating and Combining Subjective
Probability Estimates, Journal of Behavioral Decision Making, 10/3 (1997): 243-268.
14. Robert T. Clemen and Robert L. Winkler, Aggregation of Expert Probabilities, in Advances:
Aggregate Probabilities (Draft), 2006, chapter 9.
15. Tversky and Kahneman (1974), op. cit.
16. Elizabeth F. Loftus, Eyewitness Testimony: Psychological Research and Legal Thought,
Crime and Justice, 3 (1981): 105-151.
17. Colin F. Camerer and Eric J. Johnson, The Process-Performance Paradox in Expert Judg-
ment: How Can Experts Know So Much and Predict So Badly? in K.A. Ericsson and J.
Smith, eds., Towards a General Theory of Expertise: Prospects and Limits (New York, NY: Cam-
bridge Press, 1991), pp. 195-217.
18. P.E. Tetlock, Expert Political Judgment: How Good Is It? How Can We Know? (Princeton, NJ: Princ-
eton University Press, 2003).
19. William R. Ferrell, Discrete Subjective Probabilities and Decision Analysis: Elicitation, Cali-
bration, and Combination, in G. Wright and P. Ayton, eds., Subjective Probability (New York,
NY: John Wiley and Sons, 1994), pp. 411-451, at p. 448.
20. Glenn J. Browne, Shawn P. Curley, and P. George Benson, Evoking Information in Prob-
ability Assessment: Knowledge Maps and Reasoning-Based Direct Questions, Management
Science, 43/1 (January 1997): 1-14.
21. Roger Cooke, Experts in Uncertainty: Opinion and Subjective Probability in Science (Oxford:
Oxford University Press, 1991).
22. Vicki M. Bier, Challenges to the Acceptance of Probabilistic Risk Analysis, Risk Analysis,
19/4 (August 1999): 703-710.
23. Henry Wade Rogers, The Law of Expert Testimony (St. Louis, MO: Central Law Journal Com-
pany, 1891).
24. D.H. Kaye, The Laws of Probability, in Peter Murphy, ed., Evidence, Proof, and Facts: A Book
of Sources (Oxford: Oxford University Press, 2003), pp. 529-546.
25. Granger Morgan and Max Henrion, Uncertainty: A Guide to Dealing with Uncertainty in Quanti-
tative Risk and Policy Analysis (Cambridge: Cambridge University Press, 1992), p. 137.
26. For an excellent review of this approach, see Justin Wolfers and Eric Zitzewitz, Prediction
Markets, Journal of Economic Perspectives, 18/2 (Spring 2004): 107-126.
27. For a discussion of design variations and their impact on market information-gathering
performance, see Charles Plott, Markets as Information Gathering Tools, Southern Economic
Journal, 67/1 (July 2000): 1-15.
28. James Surowiecki, The Wisdom of Crowds: Why the Many are Smarter than the Few and How
Collective Wisdom Shapes Business, Economies, Societies and Nations (New York, NY: Doubleday,
2004).
29. Renee Dye, The Promise of Prediction Markets: A Roundtable, McKinsey Quarterly, 2
(2008): 84-93.
30. Clear evidence of current enthusiasm over the potential of prediction markets for strategic
decision making includes recent articles by Teck-Hua Ho and Kay-Yut Chen in California
Management Review (Ho and Chen, 2007), The Economist (2009), the McKinsey Quarterly (Dye,
Prediction Markets: A New Tool for Strategic Decision Making
CALIFORNIA MANAGEMENT REVIEW VOL. 52, NO. 4 SUMMER 2010 CMR.BERKELEY.EDU 141
2008) and Risk (Patel, 2006). Additional evidence is provided in Teck-Hua Ho and Kay-Yut
Chen, New Product Blockbusters: The Magic and Science of Prediction Markets, California
Management Review, 50/1 (Fall 2007): 144-158; An Uncertain Future: Prediction Markets,
The Economist, February 28, 2009, p. 68; Dye (2008), op. cit.; Navroz Patel, Deciding on a
Future, Risk, 19/2 (February 2006): 33-35; Peter A. Coles, Karim R. Lakhani, and Andrew
McAfee, Prediction Markets at Google, Harvard Business School Case #607088-PDF-ENG,
May 30, 2007.
31. For a comparison of prediction markets and the Delphi method, see Kesten C. Green,
J. Scott Armstrong, and Andreas Graefe, Methods to Elicit Forecasts from Groups: Delphi
and Prediction Markets Compared, Foresight: The International Journal of Applied Forecasting,
8 (Fall 2007): 17-20.
32. Joyce E. Berg, Forrest D. Nelson, and Thomas A. Rietz, Prediction Market Accuracy in the
Long Run, International Journal of Forecasting, 24/2 (April-June 2008): 285-300.
33. Kay-Yut Chen and Charles Plott, Information Aggregation Mechanisms: Concept, Design,
and Field Implementation for a Sales Forecasting Problem, California Institute of Technol-
ogy Social Science Working Paper 1131, March 2002.
34. David M. Pennock, Steve Lawrence, C. Lee Giles, and Finn Arup Nielsen, The Real Power
of Articial Markets, Science, 291, February 9, 2001, pp. 987-988 (letters).
35. R.H. Thaler, ed., Advances in Behavioral Finance (New York, NY: Russell Sage Foundation,
1993).
36. R.H. Thaler and W.T. Ziemba, Anomalies: Parimutuel Betting Markets: Racetracks and Lot-
teries, Journal of Economic Perspectives, 2/2 (Spring 1988): 161-174.
Subscribe, renew, and order reprints online at cmr.berkeley.edu
California Management Review
University of California F501 Haas School of Business #1900 Berkeley, CA 94720-1900
(510) 642-7159 fax: (510) 642-1318 e-mail: cmr@haas.berkeley.edu web site: cmr.berkeley.edu