PPA 696 RESEARCH METHODS

EXPERIMENTAL DESIGNS FOR RESEARCH

Causality
Experimental Designs
Control Group Pre-test/Post-test Design
Threats to Internal Validity
Threats to External Validity
Post-Test only Control Group Design
CAUSALITY
To establish whether two variables are causally related, that is, whether a change in
the independent variable X results in a change in the dependent variable Y, you must
establish:
1) time order--The cause must have occurred before the effect;
2) co-variation (statistical association)-- Changes in the value of the
independent variable must be accompanied by changes in the value of the
dependent variable;
3) rationale-- There must be a logical and compelling explanation for why
these two variables are related;
4) non-spuriousness-- It must be established that the independent variable X,
and only X, was the cause of changes in the dependent variable Y; rival
explanations must be ruled out.
To establish causality, one must use an experimental or quasi-experimental design.
Note that it is never possible to prove causality, but only to show to what degree it is
probable.
EXPERIMENTAL DESIGNS

True experimental designs include:

Pre-test/Post-test control group design
Solomon Four-Group design
Post-test only control group design
Pre-test/Post-test control group design
This is also called the classic controlled experimental design, and the randomized pretest/post-test design because it:
1) Controls the assignment of subjects to experimental (treatment) and control groups
through the use of a table of random numbers.
This procedure guarantees that all subjects have the same change of being in
the experimental or control group. Because of strict random assignment of
subjects, it is assumed that the two groups are equivalent on all important
dimensions and that there are no systematic differences between the two
groups. Researchers may substitute matching for random assignment. Subjects
in the two groups are matched on a list of characteristics that might affect the
outcome of the research (e.g., sex, race, income). This may be cheaper but
matching on more than 3 or 4 characteristics is very difficult. And if the
researcher does not know which characteristics to match on, this compromises
internal validity.

2) Controls the timing of the independent variable (treatment) and which group is
exposed to it.
Both group experience the same conditions, with the exception of the
experimental group, which receives the influence of the independent variable
(treatment) in addition to the shared conditions of the two groups.
3) Controls all other conditions under which the experiment takes place.
Nothing but the intervention of the independent (treatment) variable is assumed
to produce the observed changes in the values of the dependent variable.
The steps in the classic controlled experiment are:
1) randomly assign subjects to treatment or control groups;
2) administer the pre-test to all subjects in both groups;
3) ensure that both groups experience the same conditions except that in
addition the experimental group experiences the treatment;
4) administer the post-test to all subjects in both groups;
5) assess the amount of change on the value of the dependent variable from the
pre-test to the post-test for each group separately.
These steps are diagramed as follows:
R
R

O1
O1

O2
O2

This diagram can be expanded upon as in the following table:

1st observation
(measurement) of the
dependent variable
O1 = Pre-test
Experimental Group's
Experimental Group average score on the
dependent variable
Control Group's average
Control Group
score on the dependent
variable
Scientific Random
Assignment of
Subjects to:

2nd observation
Exposure to the
(measurement) of the
Treatment (X)
dependent variable
(independent variable)
O2 = Post-test
Experimental Group's
X
average score on the
dependent variable
Control Group's average
score on the dependent
variable

The difference in the control group's score from the pre-test to the post-test indicates
the change in the value of the dependent variable that could be expected to occur
without exposure to the treatment (independent) variable X.
Control group
pre-test score

control group
post-test score

control group difference

on the dependent variable

The difference in the experimental group's score from the pre-test to the post-test
indicates the change in the value of the dependent variable that could be expected to
occur with exposure to the treatment (independent) variable X.
Experimental group
pre-test score

experimental group
post-test score

experimental group difference

on the dependent variable

The difference between the change in the experimental group and the change in the
control group is the amount of change in the value of the dependent variable that can
be attributed solely to the influence of the independent (treatment) variable X.
Control group difference
attributable to X

experimental group difference

difference

This can be illustrated by the following experiment to see whether participation in

small group discussions would improve medical students' ability to respond to
emotional needs of patients:
How many times did
How many times did
Scientific Random
Exposure to the
students use emotional
students use emotional
Assignment of Medical
Treatment (X)
words to describe
words to describe
Students to:
(independent variable)
patients
patients
Attended small group
Small group
Average of .68 times
Average of 2.02 times
discussions
discussions
per student in 3 case
per student in 3 case
plus regular course
(experimental group) studies
studies
work
Average of .89 times
Average of 1.13 times
Regular course work
Control Group
per student in 3 case
per student in 3 case
only
studies
studies

The control group used emotional words an average of .89 times per student (in three
case studies) on the pre-test and an average of 1.13 times per student (in three case
studies) on the post-test. The difference in the control group's score from the pre-test
to the post-test is +.24 times per student. This indicates the change in using emotional
words that could be expected to occur with regular course work only.
The experimental group used emotional words an average of .68 times per student (in
three case studies) on the pre-test and an average of 2.02 times per student (in three
case studies) on the post-test. The difference in the experimental group's score from
the pre-test to the post-test is +1.34 times per student. The experimental group's score
from the pre-test to the post-test indicates the change in using emotional words that
could be expected to occur with regular course work plus the small group discussions.
The difference between the change in the experimental group (+1.34) and the change
in the control group (+.24) is +1.10. This is the amount of change in using emotional
words that can be attributed solely to the influence of the small group discussions.
The controlled or true experimental design allows the researcher to control for threats
to the internal and external validity of the study. Threats to internal validity
compromise the researcher's ability to say whether a relationships exists between the
independent and dependent variables. Threats to external validity compromise the
researcher's ability to say whether this study's findings are applicable to any other
groups.
Controlling for Threats to Internal Validity
1) History: did some other current event effect the change in the dependent variable?
No, because both groups experienced the same current events.
2) Maturation: were changes in the dependent variable due to normal developmental
processes? No, because both groups experienced the same developmental processes.
3) Statistical Regression: did subjects come from low or high performing groups?
Differences between the two groups that could influence the dependent variable would
be controlled for as subjects were generally equivalent at the beginning of the
research.
4) Selection: were the subjects self-selected into experimental and control groups,
which could affect the dependent variable? No, the subjects were assigned by strict

random selection and all had equal chance of getting the treatment or control
condition.
5) Experimental Mortality: did some subjects drop out? did this affect the results?
About the same number of students made it through the entire study in both the
experimental and control groups, so there appears to be no bias.
6) Testing: Did the pre-test affect the scores on the post-test? Both groups got a pretest; but a pre-test may have made the experimental group more sensitive to the
treatment.
7) Instrumentation: Did the measurement method change during the research? The
measurement method and instruments did not change.
8) Design contamination: did the control group find out about the experimental
treatment? did either group have a reason to want to make the research succeed or
fail? The researcher must do some qualitative investigation to find out if there was
design contamination.
Controlling for Threats to External Validity
1) Unique program features: There may have been an unusually motivated set of
facilitators for the small group discussions.
2) Effects of Selection: Probably applicable to other medical students.
3) Effects of Setting: Medical schools have their own cultures; doubtful if this would
be applicable to other types of students.
4) Effects of History: No information given
5) Effects of Testing: No information given
6) Reactive effects of experimental arrangements: It would be best to replicate the
results in other medical schools.
Post-Test Only Control Group Design
This design follows all the same steps as the classic pre-test/post-test design except
that it omits the pre-test. There are many situations where a pre-test is impossible
because the participants have already been exposed to the treatment, or it would be
too expensive or too time-consuming. For large enough groups, this design can
control for most of the same threats to internal and external validity as the classic
controlled experimental design. For example, it eliminates the threat to internal
validity of pre-testing by eliminating the pre-test. It may also decrease the problem of
experimental mortality by shortening the length of the study (no pre-test).
For small groups, however, a pre-test is necessary. Also, a pre-test is necessary if the
researcher wants to determine the exact amount of change attributable to the
independent variable alone.
Public administrators would like to be able to use experimental designs for policy and
program evaluation. Did a regional economic development policy bring more
business to the economically depressed region? Did the Women-Infants-and-Children
(WIC) program lower the rate of malnutrition in young children?

Unfortunately, it is difficult for public administrators to meet the requirements of the

classic controlled experimental design.
-It is difficult to conduct program evaluations in a laboratory, where other
influences can be controlled.
-It is difficult to achieve random assignment, due to political and ethical
concerns.
-Policies may not be specific as to what changes they intend to produce.
-Often funds or other resources for a large-scale research project are lacking.
-Decision makers often operate on short time frames and cannot wait for an
experimental study to run its course.
-Research is not always begun before a program is implemented, and therefore
cannot take measurements of the "before" condition.
-Programs are not always implemented all at once, but rather gradually, over
time, which may diminish their effects.
-Programs may not be implemented as originally intended, or may not produce
the effects that researchers are looking for.
-Small treatments with modest goals are more amenable to a controlled
experimental design than large scale social programs with ambitious goals.