A Sample of the Literature

on Heuristics* and Biases

What is this lecture about?

This

lecture is a Sample of the

Literature on Heuristics* and
Biases(Shortcuts in Thinking* and
Biases Related To Them)-the law of
small numbers (and its related
heuristic: representativeness)
and errors in statistical thinking
related to it demystified.
What is all this? I will present this
slide again in the end and ask you to
explain it.

The law of small numbers

demystified
A

study of the incidence of kidney cancer

in Counties of United States.
Kidney Cancer is lowest in counties which
are mostly rural and sparsely populated
and located in traditionally Republican
States.
You probably ignored the Republican Part.
Did you focus on the Rural Part?
Did you think that rural lifestyle leads to
low kidney cancer? Probably.

Now comes the surprise!

The

counties in the United States which

have the highest incidence of kidney
cancer also tend to be in mostly rural,
sparsely populated counties and
Republican States!
Maybe the poverty of the rural lifestyle
caused this!
Something is wrong here! Rural
Lifestyle cannot explain both high and
low incidence of kidney cancer!

What is the solution to this riddle?-lets begin to think.

Well,

what could be the solution to this

riddle?, Lets atleast begin the process
of exploration.
Infact, the key factor is not that the
counties were rural or predominantly
Republican, it is that rural counties
have small populations.
More about this explanation later*, first
lets investigate certain features of this
conondrum.

The Problem is In fact: Errors in

Statistical ReasoningFrom

whatever I told you above you

must have thought O My God! Here,
we have -another average lecture
on some academic topic, but the
lecture I am about to give you is not
academic at all, infact it is a lecture
whose basis is on the difficult
relationship between our mind and
statistics.

What was the shortcut that your mind got

trapped in at the beginning of this story?
We

automatically and effortlessly identify

causal connections between events,
sometimes even when the connection is
spurious.
What was the shortcut that your mind got
into? When told about the high incidence
counties, you immediately assume that
these counties are different from other
counties for a reason, that there must be
a cause that explains this difference.

Our Mind and Statistics.

Our

mind, specially the intuitive part

of our mind is inept when faced with
merely statistical facts, which
change the probability of outcomes
but do not cause them to happen.
How can we justify the above? Here
we goAs I had promised earlier there is an
explanation to this story of Kidney
Cancer. What is it?

The Explanation:
Imagine

the population of the United

States as marbles in a giant urn. Some
marbles are marked KC, for kidney
cancer. You draw samples of marbles
and populate each county in turn. Rural
Samples are smaller than other
samples. Extreme Outcomes (very high
and/or very low cancer rates) are most
likely to be found in sparsely populated
counties. This is all there is to the story.

Going Deeper into the Explanation.

Imagine

a large urn filled with marbles.

Half the marbles are red, half are white.
Next, draw 4 marbles from the urn, record
the number of red balls in the sample,
throw the balls back into the urn, and then
do it again, many times. If you summarize
the results, you will find that the outcome
"2 red, 2 white" occurs (almost exactly) 6
times as often as the outcome "4 red" or
"4 white." This relationship is a
mathematical fact.

A related statistical fact

A

related statistical fact is relevant

to the cancer example. From the
same urn, Jack draws 4 marbles on
each trial, Jill draws 7. They both
record each time they observe a
homogeneous sampleall white or
all red. If they go on long enough,
Jack will observe such extreme
outcomes more often than Jillby a
factor of 8.

Just an accident of sampling

The

small population of a county neither

causes nor prevents cancer; it merely
allows the incidence of cancer to be
much higher (or much lower than in the
larger-population. The deeper truth is
that there is nothing to explain. The
incidence of cancer is not truly lower or
higher than normal in a countv with a
small population it just appears to be so
in a particular year because of an
accident of sampling.

Law of large and small numbers

large

samples- deserve more trust than smaller samples.

you may find that the following statements apply to you:

"sparsely populated" did not immediately stand out as
relevant when you read the kidney cancer story.
You were at least mildly surprised by the size of the difference between
samples of 4 and samples of 7.
The following two statements mean exactly the same thing:
Large samples are more precise than small samples.
Small samples yield extreme results more often than large samples do.
The

first statement has a clear ring of truth, but until the second version
makes intuitive sense, you have not truly understood the first.
The bottom line: yes, you did know that the results of large samples are
more precise, but you may now realize that you did not know it very well.

Sampling Variation

you

wish to confirm the hypothesis that the

vocabulary of the average six-year-old girl
is larger than the vocabulary of an average
boy of the same age. The hypothesis is true
in the population; Girls and boys vary a
great deal, however, and by the luck of the
draw you could select a sample in which
the difference is inconclusive, or even one
in which boys actually score higher. Using a
sufficiently large sample is the only way to
reduce the risk. Researchers who pick too
small a sample leave themselves at the
mercy of sampling

Content versus reliability

In

a poll of 300 seniors 60% support

the President
summarize in exactly three words,
you would choose The elderly support
the President.'' These words provide the
gist of the story. Your summary would
be the same if the sample size had
been different. Of course, a completely
absurd number of sample size, small or
big, would draw your attention.

Halo Effect
believing

that small samples closely

resemble the population from which they
are drawn implies: we are prone to
exaggerate the consistency and
coherence of what we see, The
exaggerated faith of researchers in what
can be learned from a few observations is
closely related to the halo effect, the
sense we often get that we know and
understand a person about whom we
actually know very little.

There has to be a reason for everything

Take

the sex of six babies born in sequence at a hospital. The

sequence of boys and girls is obviously random; the events
are independent of each other, and the number of boys and
girls who were born in the hospital in the last few hours has
no effect whatsoever on the sex of the next. However we
will not consider the fact that if in the sequence all events
are independent and outcome boy and girl are
approximately equally likely, then any possible sequence of
six births is as likely as any other. We are pattern seekers,
believers in a coherent world, in which regularities (such as a
sequence of six girls) appear as a result of someone's
intention. Lions may appear on the plain at random times,
but it would be safer to notice and respond to an apparent
increase in the rate of appearance of prides of lions, even if it
is actually due to the fluctuations of a random process.

Search for certainty, search for

causality- a waste of time.
War

broke out in 1973. Squadrons flying from the

same base, one of which had lost four planes
while the other had lost none. An inquiry was
initiated. There was no prior reason to believe that
one of the squadrons was more effective than the
other, and no operational differences were found,
but of course the lives of the pilots differed in
many random ways, including, how often they
went home between missions.. Rationally the
command should accept that the different
outcomes were due to blind luck, a random
search for a non obvious cause was hopeless.

Finding Patterns in Sports

The

assumed hot hand in sports is

very usual, if in basketball a player
sinks three or four baskets in a row,
defense starts guarding him more,
his players start passing more to
him, even his coach thinks he has a
temporary hot hand, we are too
quick to perceive order and causality
in randomness.

In a small sample there will be more extreme results.

A

research study showed that more small

schools had done well, so the Gates foundation
started funding small schools and a causal story
can easily be linked to this saying that attention
to students is more in small schools, actually
larger schools empirically, if anything, do better
possibly because of greater curriculum options .
And so Unfortunately the causal analysis is
wrong, the actual fact is which could have been
pointed out had the Gates foundation taken
statistics seriously is that more small schools
had also done badly. Clearly on average small
schools are not better just more variable.

Journal Articles
In

further reading
1) if you read the article I have given you
Belief in the law of small numbers.Tversky, Amos;
Kahneman, Daniel, Psychological Bulletin, Vol 76(2),
Aug 1971, 105-110.
This article gives many examples which show that
the fact that extreme outcomes result from small
samples more often than large samples is a
statistical fact and can lead to a bias or statistical
error in misinterpreting, what is a result of sample
size, as a factor related to the content of the story.
Unfortunately we focus more on content than
reliability.

Journal Articles
In

further reading
1) if you read the article I have given you
On the psychology of prediction.Kahneman,
Daniel; Tversky, Amos Psychological Review, Vol
80(4), Jul 1973, 237-251.
You will find examples of how people see patterns
where none exist. People judge even extreme
and rare outcomes as more probable if the
content of the story indicates that a particular
outcome is more representative, even if that
outcome is extreme or rare. This is the heuristic
of representativeness.

Journal Articles
In

further reading
1) if you read the article I have given you

How to Make
Cognitive Illusions Disappear: Beyond Heuristics and Biases,
Gerd Gigerenzer, European Review of Social Psychology,Volume 2, Issue 1,
1991
Special Issue: European Review of Social Psychology .

You

will find that this statistical error of not considering

the sample size and focussing on the content of the
story, itself has limitations, as the author explains: by
considering a more detailed view of statistics ie by
considering, for example: relative frequency rather than
single frequency case, and also, whether or not the
observation was randomly selected or self selected
itself; this kind of a nuanced technical view of statistics
could possibly remove this statistical error itself.

What was this lecture about?Explain below statementThis

lecture is a Sample of the

Literature on Heuristics* and
Biases(Shortcuts in Thinking* and
Biases Related To Them)-the law of
small numbers (and its related
heuristic: representativeness)
and errors in statistical thinking
related to it demystified.