Chapter 2

Observational Studies and Experiments

Aims
In this chapter we want to learn how to identify two types of statistically based studies,
observational studies and experiments, by considering how the data are collected. We need to
know what conclusions can be made for each type of study.
By the end of this chapter you should:

know the difference between an observational study and an experiment

understand why we need to distinguish between observational studies and experiments

know how to collect data under a completely randomised design and under a randomised
block design

We deal with:

Observational studies
o causation, confounding variables, factors of interest, response

Experiments
o causation, experimental units, treatments, response, placebo, placebo effect,
blinding, double blinding
o two types of experimental design: a completely randomised design and a
randomised block design

Chapter 2

Page 1

Observational Studies

Important ideas:
Causation, observational study, confounding variable.

TV cuts chances of getting a degree

NZ Herald July 5 2005
Children who spend hours in front of the
television are less likely to have a university
degree in their mid-20s, a New Zealand study
has found. The research concluded that even
bright children who watched lots of television
had fewer qualifications from universities.
"These findings suggest that reducing
television
viewing
could
improve
the
education of New Zealand children," one of
the researchers, Bob Hancox, said last night.
The University of Otago study monitored the
TV habits of more than 1000 children, born in
Dunedin in 1972 and 1973, between the ages
of 5 and 15. It found that those who watched
the most television had earned fewer
qualifications by the time they were 26. Just
7% of the children watched less than an hour
a day. They were the most qualified in their
mid-20s.
Dr Hancox said those who watched the least
television had the best chance of going on to
university and earning a degree.
"This suggests that excessive television in
younger children has a long-lasting, adverse
effect on educational performance," Dr
Hancox said.

Previous studies have investigated the link

between television viewing and school
performance but the Otago research is the
first in the world to track a group of children
into adulthood. The study found that the
educational effects were not explained by
intelligence or socio-economic factors.

Moral of the story:

Causal claims cannot generally be made based on an observational study.

The results need to be considered with

caution, say some experts. Dr Ruth Zanker
at Christchurch Polytechnic said the children
who were left to watch hours of television
often had parents who did not set limits to
help create an academic environment.
"I don't know if you can necessarily isolate
television as the root of all evils," she said.
The study has sparked fears about future
generations, exposed to far more media than
those involved in the research.
Average time spent watching TV per day and
academic achievement:
2.76 hours left school with no qualification
2.5 hours School Certificate
2.3 hours post-school qualification
1.9 hours university degree

Can we trust this result?

Look at the way the study was designed and carried out.
Type of research:

experiment / observational study

Longitudinal
Unit:

Person

Factor of interest:

TV habits

Response:

Qualification type

Comments:
Cant give the idea that watching TV

causes/results in

fewer qualifications.

Can say that TV watching and qualifications are linked/associated/related

Possible confounding/lurking variables: ???

Chapter 2

Page 2

Experiments

Types of Experiments

Aspirin cuts risk of first heart attack

Completely Randomised Design

The treatments are allocated entirely by chance to the experimental units.

Science Service, Inc. Jan 30, 1988

An aspirin every other day could help keep
the doctor away by cutting in half the risk
of that first heart attack, scientists said this
week.

Researchers said the study showed aspirin's

"statistically extreme beneficial effect [in
preventing]
nonfatal
and
fatal
[heart
attacks]."

On the basis of results from the Physicians'

Health Study, involving more than 22,000
U.S. male physicians 40 to 84 years of age,
a research group at Harvard Medical School
and Women's Hospital in Boston concluded
that using aspirin reduced the risk of heart
attacks by 47 percent among these healthy
men.

Half of the subjects were randomly assigned

to take an ordinary aspirin every other day,
while the remainder received a placebo. None
of the physicians knew whether he was
taking the actual aspirin or the placebo.

Previous studies had shown that aspirin

can reduce the likelihood of a second heart
attack through its ability to thin the blood.
But this is the first major examination of
the drug's preventive effects in healthy
individuals.

There were 104 heart attacks among the

aspirin-treated subjects and 189 among
those receiving placebos.
Although aspirin is not a panacea for heart
patients, experts who have commented
publicly seem to agree that its beneficial
effects are another step toward reducing a
problem of great proportions.

Example:
Suppose 8 men were involved in the aspirin study. Randomly allocate treatments to the men.

1
A

2
A

3
A

Random Numbers

4
P
11369

5
P

6
A

7
P

8
P

23569 26339 42774 79623 92280 93246

Randomised Block Design

Group (block) experimental units by some known factor and then randomly allocate treatments
within each block.
Can we trust these results?
Look at the way the study was designed and carried out.
Type of study:

experiment / observational study

Unit:

Male

Treatments:

Aspirin and placebo

Response:

Heart attack?

Example:
Suppose 12 men were involved in the aspirin study and it is known that age (especially being
65 or older) is a risk factor for heart attacks. Divide the men into two blocks and then
randomly allocate treatments to men within each block.

Age:
Comments:
Can conclude aspirin

caused

the lower rate of heart attacks.

44

47

50

53

55

55

62

63

65

69

73

74

1
A

2
P

3
A

4
P

5
A

6
A

7
P

8
P

1
A

2
P

3
P

4
A

65169

37186

05143

32004

74092

Generalisability only people like those in this study

Random Numbers
Important ideas:
Causation, randomised experiments, treatment, response, placebo, placebo effect, blinding

760116

50572

What are some other risk factors for heart attack?

Moral of the story:

Causal claims can be made on the basis of well-designed and well-executed experiments.

Chapter 2

Page 3

Chapter 2

Page 4

Observational Studies

Experiments

What is an observational study?

What is an experiment?

A study where the data comes from observing and recording things as they are in the world, or as

A study in which the researcher controls (or manipulates or changes) the conditions experimental

they unfold over time, without the investigator actively changing anything is called an observational

units experience is called an experiment.

study.

A study in which the researchers determine which units (people) receive which treatments is an

A study in which researchers simply compare units at each level of the factor of interest is an example
of an observational study.

example of an experiment.

We use observational studies when we are interested in studying the effect of a factor of interest

We use experiments when we are interested in studying the effect of a treatment on individuals
or units.

on individuals or units

In an experiment the experimenter determines which units receive which treatment.

We compare the units that happen to have received each of the levels of the factor of interest.

We measure the response for each unit under a given treatment.

We measure the response for each unit under an observed factor level.

The aim is to make comparisons fair try to make groups as similar as possible except for the

comparisons fair try to make treatment groups as similar as possible except for treatments

factor of interest.

being used.

The underlying principle of experiments is to compare treatment groups. The aim is to make

Observational studies should use some form of random sampling to obtain representative
samples.

Only well-designed and well-executed experiments

can be used to reliably establish causation.

Observational studies are useful for identifying possible causes of effects,

but they cannot reliably establish causation.

Types of experiments include:

1.

Types of observational studies include:

1.

Treatments are allocated entirely by chance to the units. Random allocation is an attempt to

Cross-sectional

make the treatment groups as similar as possible we can only expect to achieve this when

A study which observes a group of individuals or units at a point in time. It is a descriptive

there are a large number of experimental units.

study, providing a snapshot at a particular point in time.

2.

Completely Randomised Design

2.

Randomised Block Design

Longitudinal
A study which observes the same group of individuals or units over a long period of time. A
longitudinal study is comprised of a series of cross-sectional studies.

Group (block) units by some known factor and then randomise within each block in an attempt
to balance out the unknown factors.

E.g., The Dunedin Longitudinal Study is a study based in Dunedin, New Zealand. 1037 children
born between 1 April 1972 and 31 March 1973 initially took part in the study. The study is still

Experiments involving people:

continuing.

Refer: Chance Encounters, pages 24 28

Chapter 2 - Notes

Blinding

Subjects dont know which treatment they receive

Double blinding

Subjects and administerers / diagnosticians have been blinded

Placebo

Inert (inactive) dummy treatment

Placebo effect

A common response in humans when they believe they have been treated

Refer: Chance Encounters, pages 18 24

Page 5

Glossary
Blinding

Population

Complete set of individuals, objects or units that we want information about.

Random allocation

The random process by which experimental units are allocated to treatments.

The treatments should be allocated to units in such a way that each treatment
is equally likely to be applied to each unit.

Randomised block design

An experiment in which the experimental units are grouped (blocked) by some

known factor and then treatments are randomly allocated within each block.

Randomised experiment

An experiment where conditions are changed purposefully and a random

process is used to decide who (or what entities) will be subject to what
conditions.

Subjects dont know which treatment they receive.

Block

A group of units which have something in common.

Blocking

Grouping units into similar groups (blocks) in an experiment.

Cases

The individual things data is being recorded about.

(Alternative names: individuals, units, entities)

Causal conclusion

A this is what made it behave like this conclusion.

Causation

The act of making something happen.

Response variable

The outcome variable on which comparisons are made.

Cause

Something that makes something happen.

Statistic

A number calculated from the data; often used to estimate an unknown

parameter.

Completely randomised
design

An experiment in which the treatments are allocated entirely by chance to the

units.

Treatment

Something researchers administer to experimental units.

Units

The individual things data is being recorded about.

(Alternative names: entities, individuals, cases)

Confounding variable

A variable that causes changes in both the response variable and the
explanatory variable.
(Alternative names: confounder, lurking variable)

Control

Units which do not have the characteristic of interest.

Control group

May receive no treatment, a placebo or some other existing treatment.

Controlling

Grouping subjects into similar groups in an observational study.

Cross-sectional study

A study which observes a group of individuals or units at a point in time.

Double blinding

Neither subjects nor any researches in contact with subjects know who
received which treatment.

Effect size

The size of the difference between treatment group centres.

Entities

The individual things data is being recorded about.

(Alternative names: individuals, units, cases)

Estimate

A number calculated from the data; used to estimate an unknown parameter

value.

Experiment

A study in which the researcher controls (or manipulates or changes) the

conditions experimental units experience.
A study in which the researchers determine which units (people) receive which
treatments is an example of an experiment. (See randomised experiment)

Experimental units

The individual things data is being recorded about in an experiment, e.g.,

people, cars, tomato plants, mice, etc.

Explanatory variable

A variable used to attempt to predict or explain a response.

Factor

Categorical variable; defines group membership.

Longitudinal study

An observational study which observes the same group of individuals or units

over a long period of time.

Lurking variable

See confounding variable

Observational study

A study where the data comes from observing and recording things as they
are in the world, or as they unfold over time, without the investigator actively
changing anything.
A study in which researchers simply compare units at each level of the factor
of interest is an example of an observational study.

Parameter

Numerical characteristic of a population or distribution. For example, a

population mean, .

Placebo

Inert dummy treatment.

Placebo effect

Common response in humans when they believe they have been treated.

Chapter 2 - Notes

Page 6

A challenge!
Have you grasped the

big ideas from Chapter 2?

If you think you have got everything under control then accept this
challenge and find out how good your understanding really is!

Answers: (See Section C: Lecture and Tutorial Answers or the fill-ins on Canvas)

d.

Media reports which describe two variables as being linked or

associated cannot be interpreted as a change in one variable will
cause a change in the other variable.

h.

If one of the treatment levels in an experiment involving people is a

placebo, blinding should still be used.

Challenge:
Identify the 2 false statements amongst the 13 following statements:
a. Only properly designed and well executed experiments can reliably establish causation.
b. An observational study is often useful for identifying possible causes of effects.
c. In an experiment, the experimenter decides which experimental units will receive which
treatment, whereas in an observational study, we simply compare units that happen to
have received each level of the factor of interest.
d. Media reports which describe two variables as being linked or associated can be correctly
interpreted as a change in one variable will cause a change in the other variable.
e. We should be very wary of a media report which claims the existence of a causal
relationship between two variables (i.e., this causes that type claims) based on the results
from an observational study alone.
f. Blocking in an experiment allows for comparisons to be made between different treatment
groups with the same level of blocking factor.
g. A study in which the treatments are allocated in a non-random manner is still called an
experiment.
h. If one of the treatments in an experiment involving people is a placebo then there is no
need to use any form of blinding.
i. In an experiment involving people, randomly assigning people to treatment groups is an
attempt to make the treatment groups as similar as possible except for the treatment being
used.
j. The placebo effect is the response in human subjects by the idea that they are being
treated.
k. A group of experimental units given no active treatment is called a control group.
l. Double blinding occurs when the experimental units and those diagnosing them are
prevented from knowing which treatment the units have received.
m. Results obtained from subjects treated with a placebo wont always be equivalent to
results obtained from comparable subjects receiving no treatment.

Chapter 2

Page 7

Sample Exam / Test Questions

1. A randomised study was conducted to determine the frequency of deep-vein thrombosis (DVT) as

3. An experiment was conducted to assess the effect of 2 new heart worm medications on the fat

a result of long-haul flights. 231 economy-class adult passengers with no history of DVT-related

content of shoulder muscles in German Shepherd dogs. 60 dogs were grouped by sex and then

problems took part. Passengers were randomly allocated to one of two groups where one group

each group was randomly allocated to 3 treatment groups: an untreated control group, a group

wore graduated elastic compression stockings, while the other group did not. All passengers made

receiving heart worm medication A and a group receiving heart worm medication B.

trips lasting more than 8 hours per flight and returned within 6 weeks.

Which one of the following statements is false?

Consider the following sentence:

a. The dogs were blocked on sex.

From this study it should be possible to make conclusions about whether the risk of DVT is

b. In order to control for the placebo effect, some dogs should also have been allocated to a
placebo control group.

reduced by the wearing of graduated elastic compression stockings by economy-class adult

passengers during long-haul flights."

c. It was not necessary to use double blinding in this study.

Which one of the following statements is true?

d. This is an experiment because the type of treatment given has been allocated to each dog.

a. The sentence is correct because the researchers randomly allocated whether or not

e. Random allocation of the dogs to the treatment groups was carried out in an attempt to

economy-class adult passengers wore graduated elastic compression stockings.

ensure that comparisons between the 3 treatment groups for each sex are fair.

b. The sentence is incorrect because the number of economy-class adult passengers in the
study was not sufficiently large.
c. The sentence is incorrect because even though the researchers randomly allocated
economy-class adult passengers to the wearing of graduated elastic compression stockings,

Answers: (See Section C: Lecture and Tutorial Answers or the fill-ins on Canvas)

1. (a)

2. (e)

3. (b)

the passengers were not blocked on gender.

d. The sentence is correct because blinding has been used in the study.
e. The sentence is incorrect because the researchers only observed whether a passenger
developed DVT.
2. A sample of 425 New Zealanders older than 40 years participated in a study which investigated the
effect of health levels on the ability of an individual to save for retirement.
It is incorrect to use the study to conclude that increasing the health levels of New Zealanders
older than 40 years will result, on average, in an increase in their ability to save for retirement
mainly because:
a. for those people in the sample, the ability to save for retirement was not controlled by the
researchers.
b. the sample size, n = 425, was not large enough for a population of more than 3 million.
c. it is too difficult to correctly classify a persons health level.
d. the difference between the proportion of the sample that were in the low level of health group
and the proportion of the sample who were in the satisfactory health level group was not
large enough.
e. the health level of those people in the sample was not controlled by the researchers.

Chapter 2

Page 8

Tutorial
1.

(iii) Which of the three studies would be described as experiments and which would be
described as observational studies?

Consider the following three studies:

Study 1: An animal researcher was interested in cats abilities to survive surprisingly high
falls if they had time to twist round and prepare for the impact. Vets in New York
City recorded incidents of cats falling out of apartment windows. The data was
divided into three groups: cats that fell from one or two storeys above the ground;
cats that fell from three to five storeys above the ground and cats that fell from
six or more storeys above the ground. The proportion of cats that survived in
each group was then compared.
Study 2: A random sample of 100 students is asked to keep a diary in which they record
their clothing expenditures for the next three months. The expenditures of males
and females are then compared.
Study 3: A sample of 50 shoppers at an appliance store is split into two groups. One
group is shown a television commercial for a new range of appliances that has
been filmed in the same style as previous television commercials for the store.
The second group is shown a television commercial for the same new range of
appliances that has been filmed in a totally new style. An hour after viewing the
commercial, each of the shoppers was asked what they could recall about the
new range of appliances and a score based on their recollection was recorded.
The recall scores were then compared for the two groups.
(i)

For each study, describe what groups are being compared.

(iv) For the studies that are observational, briefly explain why an experiment could not be
carried out instead.

2.

Which one of the following statements about experiments is false?

(1)

Blocking is used to ensure fair comparison with respect to factors that the
experimenter knows are important.

(2)

Random allocation of treatments to subjects guarantees comparability of treatment

groups, even when we only have small numbers of subjects.

(3)

Blinding can be used on the subjects of an experiment, the people administering

treatments and the people measuring the results.

(4)

Properly designed and executed experiments can reliably establish causation.

(5)

The placebo effect is the response caused in human subjects by the idea that they
are being treated.

Study 1:

Study 2:

Study 3:

(ii)

For each study, describe what is being measured to make the comparison.
Study 1:

Study 2:

Study 3:

Answers: (See Section C: Lecture and Tutorial Answers or the fill-ins on Canvas)
Chapter 2

Page 9

Chapter 2

Page 10