INTRO. DEV.

ECONOMICS
ECO 2117

ABEL BRODEUR

Week 2

1/ 43
Program Evaluation

Outline:
I (1) Program Evaluation
I Correlation does not imply causality

I (2) Causal Effect

I Randomized Control Trial (RCT)
I Regression Discontinuity Design (RDD)
I Instrumental Variables (IV)
I Difference-in-Differences (DID)

2/ 43
Objective:

How can we evaluate government policies?

I How can we get a causal effect?

I Understanding RCT, RDD, IV and DID

Reference:
I Angrist and Pischke, 2014. Mastering’ Metrics.

I Ravaillon, 2001. The Mystery of the Vanishing

Benefits: An Introduction to Impact Evaluation

3/ 43
Income and Democracy

Democracies have higher GDP per capita

I Does democracy increase GDP per capita?

I Democracy may reduce corruption

I Democracy may improve property rights

I Usually find a positive correlation in the data...

Does this mean that democracy cause higher GDP per

capita?

4/ 43
Income and Democracy

Democracies have higher GDP per capita

I Does democracy increase GDP per capita?

I Democracy may reduce corruption

I Democracy may improve property rights

I Usually find a positive correlation in the data...

Does this mean that democracy cause higher GDP per

capita? No!

5/ 43
Income and Democracy

Policy implications
I If democracy leads to more growth...

I Then making countries more democratic and related

policies may determine wealth
I If there is no causal effect, then policy implications
are very different!

6/ 43
Mystery of Vanishing Benefits

Let’s take (another) example from a researcher at World

Bank
I Imaginary example of conditional cash transfer (CCT)

I Mrs. Speedy is conducting the empirical evaluation

I See syllabus for the paper

I Again, not compulsory to read it

7/ 43
Mystery of Vanishing Benefits

Government of country Labas introduces a policy

I Cash transfers to poor families with school-age
children
I To keep receiving the transfer, families must keep
their children in school until 18 (PROSCOL)
I Goals are to reduce poverty with transfers and
reduce future poverty by increasing education of kids

Is the policy working?

I Are transfers going to poor families?

I Are children going to school?

8/ 43
Mystery of Vanishing Benefits

Statistics 1 year after policy:

I Poorest 30% in PROSCOL receive 70% of the
transfers
I School attendance is 80% in PROSCOL and
non-PROSCOL

Is the policy working?

I ...

9/ 43
Mystery of Vanishing Benefits

Findings:
I Poorest 30% in PROSCOL receive 70% of the
transfers
I School attendance is 80% in PROSCOL and
non-PROSCOL families

Is the policy working?

I Not sure whether 80% is good or bad

I We cannot compare PROSCOL and non-PROSCOL

families: they are different

10/ 43
Mystery of Vanishing Benefits

Comparison Group:
I PROSCOL and non-PROSCOL families are different

I Mrs. Speedy argues that non-PROSCOL families are

exactly like PROSCOL families if they would not have
received the cash transfers

Potential bias:
I Omitted variable bias: variables not included in the
analysis

11/ 43
Definitions

Do you guys know what is a variable? Please say yes...

12/ 43
Independent and Dependent Variables
A dependent variable is the variable we are trying to
explain
I In this case: Wealth/GDP per capita

I We seek to explain variation in an outcome

(i.e. dependent variable)

An independent variable is used to explain variation in the

dependent variable
I In this case: Democracy

Control variable is used to control for something in a

regression (ceteris paribus)

13/ 43
Regression

α is the intercept: refers to the value of Y when X is zero

β is the slope: the rate of change in Y for a one-unit

change in X (regression coefficient)

This implies a straight line. But since we never have a

perfect line, we have an error term (epsilon)

14/ 43
Potential Biases to Regression Analysis

(I) Omitted variable bias:

I Variables that can explain why a democratic country
is richer (other than democracy)
I Examples: latitude, access to the coast, population,
etc.

15/ 43
Potential Biases Regression Analysis

(II) Measurement error

I Can be in the dependent and/or independent
variables!

Dependent variable:
I 1) not measured perfectly (e.g. GDP)

I 2) do not measure what we want (e.g. use IQ for

talent)

16/ 43
Potential Biases to Regression Analysis

(III) Reverse causality

I Growth leads to democracy, not the other way around

I The dependent variable explains the independent

variable

17/ 43
What?!?

Boy it’s complicated... should I drop this class? Heard

[discipline] is easier

I Relax! But get prepared to study if this is hard/tricky

for you

18/ 43
Program Evaluation

Outline:
I (2) Causal Effect
I Randomized Control Trial (RCT)
I Regression Discontinuity Design (RDD)
I Instrumental Variables (IV)
I Difference-in-Differences (DID)

19/ 43
Program Evaluation
Difference between causality and correlation
I Stephen Pinker’s book: The Blank Slate

I Pinker cites an old Russian folktale in which a Tsar

discovers that, of his many provinces, the one that
has the highest disease rate also has the most
doctors
I So he orders all the doctors killed!

If we find a positive correlation between GDP growth and

democracy, that does not mean that democracy is
causing economic growth!!! Maybe it’s true, maybe it’s
not, we cannot infer much from this correlation...

20/ 43
Program Evaluation

How can we know whether more doctors prevent

disease?
I Research question requires different data and
identification (ID) strategy
I Important to know which ID strategy and data to use

21/ 43
Definitions

Dependent variable, independent variable, omitted

variable bias, reverse causality, measurement error

Identification strategy: method used in a study to get a

causal effect
I We will rely mostly on four methods throughout

I First method is RCT

22/ 43
Randomized Control Trials (RCT)

To measure the causal impact of T (treatment) on a given

outcome y
I We assign randomly T

I We just need to compare averages for the treatment

and control groups
I Example: lottery

I Only experimental method used in social sciences

23/ 43
Randomized Control Trials

We should always check whether the randomization

worked well
I What does it mean “worked well”

I We can check that treatment is independent from

many control variables

24/ 43
Randomized Control Trials

Advantages:
I No bias

I Easy to explain to people outside academia

Disadvantages:
I Expensive (time and money)

I Ethics (not giving treatment to control group that

could save/improve their life)
I External validity

25/ 43
Randomized Control Trials

Disadvantages:
I Hawthorne Effect: individuals react and modify their
behavior in response to their awareness of being
observed
I Henry Effect: a control is aware of his/her status as a
member of the control group and is able to compare
his/her performance with that of the treatment group
(may actively work harder to overcome the
“disadvantage” of being in the control group)

26/ 43
Validity

External validity: Can we generalize the results to other

settings?

Internal validity: How convincing is the study? Did we get

a causal effect?

27/ 43
Regression Discontinuity Design

Assignment to treatment is based on a clearly defined

index or parameter with a known cutoff for eligibility
I Possible when units can be ordered along a
quantifiable dimension which is systematically related
to the assignment of treatment
I Compare individuals just above and below the cutoff
for eligibility

Note that the effect is measured at the discontinuity.

Estimated impact around the cutoff may not generalize to
entire population

28/ 43
Regression Discontinuity Design

Rules with cut-offs are common in social programs

I Anti-poverty programs targeted to households below
a given poverty index
I Scholarships targeted to students with high scores on
standardized test

Beneficiaries and non-beneficiaries are usually different.

BUT: for people near the threshold, differences are slight

29/ 43
Regression Discontinuity Design

Are these thresholds applied?

I Easily seen with a graph

I Sharp discontinuity that determines the treatment

For example, people who earn $50,000 or less (or more)

are (not) eligible

30/ 43
Regression Discontinuity Design

Eligibility respected?

See blackboard
(ask for notes of a classmate if you missed this lecture)

31/ 43
Regression Discontinuity Design

Main results:
I Easily seen with a graph

32/ 43
33/ 43
Regression Discontinuity Design
Advantages:
I Many programs have eligibility rules

I Relatively weak assumptions (compared to other

non-experimental methods)

Disadvantages:
I Other programs might use the same (or related)
eligibility rules
I Households can respond to eligibility rules

I Thus there are threats to internal validity

I Only get causal effect for people near cut-off

I Thus maybe not a lot of external validity

34/ 43
Difference-in-Differences

When to use DID

I Both with experiments and with “natural” experiments

I Need a group that is affected by the

policy/intervention (treatment) and a group that is not
affected (control)
I Need baseline and at least one round of follow-up
data

The quality of the control group, and the likelihood that

both groups have similar trends will determine the
reliability of the results

35/ 43
Definitions

Dependent variable, independent variable, omitted

variable bias, reverse causality, measurement error, ID
strategy

Experiments
I RCT is an experiment

I Other three methods are “natural” experiments

I Thus they are not really experiments

36/ 43
Difference-in-Differences

t t u u
Impact = (Ȳpost − Ȳpre ) − (Ȳpost − Ȳpre )

Examples:
I Policy change in one province/state/region and not
the others

37/ 43
38/ 43
Difference-in-Differences

Advantages:
I Easily applicable to many policies (no need for a
cut-off)
I DID studies often have more external validity than
RCT studies

Disadvantages:
I Key assumption is strong: no other policy/shock
affected differently the control and treatment groups

39/ 43
Instrumental Variables

Imagine we have a variable that provides “random”

variation in X

And this variable affects Y only through X

This variable, Z, is an instrumental variable

40/ 43
Instrumental Variables

Examples:
I See next lectures

Hard to find good instruments (Z )

I Z is correlated with X but does not determine Y

I Z affects Y only through X

41/ 43
Instrumental Variables

Advantages:
I Some research questions cannot be answered using
the other methods

Disadvantages:
I Key assumption is very (very) strong

I And cannot be tested empirically

42/ 43
Exam

Multiple choices:
I The Hawthorne effect is a) people react and change
the way they behave...

Short or long answers:

I What is the omitted variable bias? Discuss how RCT
helps to overcome the omitted variable bias.

43/ 43