Documente Academic
Documente Profesional
Documente Cultură
Larry V. Hedges
Northwestern University
Prepared for the IES Summer Research
Training Institute July 8, 2008
What is Experimental Design?
Experimental design includes both
• Strategies for organizing data collection
• Data analysis procedures matched to those data
collection strategies
Because of variability
Lindquist (1953)
Conclusion
• Analysis of variance
Studies in Crop Variation IV and VI
Studies in Crop variation IV (1927)
1. Control by matching
2. Control by randomization
3. Control by statistical adjustment
But
Example
The Tennessee class size experiment assigned students to
one of three class size conditions. All three treatment
conditions occurred within each of the participating
schools
Example
The Tennessee class size experiment actually assigned
classrooms to one of three class size conditions. Each
classroom occurred in only one treatment condition
T1 T2 … T1
T1 T2 T1
…
T2 T1 T2
Blocks
1 2 … n
T1 T2 T1
…
T2 T1 T2
Treatment 1
…
Treatment 2
Treatment Control
Individual 1 Individual 1
Individual 2 Individual 2
…
…
Individual nT Individual nC
The Randomized Block Design
Block 1 … Block m
Individual 1 Individual 1
…
…
…
Treatment 1
Individual n1 Individual nm
Individual n1 +1 Individual nm + 1
…
…
Treatment 2 …
Treatment Control
…
Individual n1 Individual nm Individual nm+1 Individual n2m
Randomization Procedures
Randomization has to be done as an explicit process
devised by the experimenter
But
• Hierarchical designs
Experimental Designs
For each design we will look at
• Structural Model for data (and what it means)
• Two inference models
– What does ‘treatment effect’ mean in principle
– What is the estimate of treatment effect
– How do we deal with context effects
• Two statistical analysis procedures
– How do we estimate and test treatment effects
– How do we estimate and test context effects
– What is the sensitivity of the tests
The Randomized Block Design
The population (the sampling frame)
Treatment 1 2 … m
1 …
2 …
The Randomized Block Design
School 1
Schools
Treatment 1 2 … m
1 …
2 …
The Conceptual Model
The statistical model for the observation on the kth person
in the jth school in the ith treatment is
Yijk = μ +αi + βj + αβij + εijk
where
μ is the grand mean,
αi is the average effect of being in treatment i,
βj is the average effect of being in school j,
αβij is the difference between the average effect of
treatment i and the effect of that treatment in school j,
εijk is a residual
Effect of Context
Context Effect
Two-level Randomized Block Design
With No Covariates (HLM Notation)
Level 1 (individual level)
Yijk = β0j + β1jTijk+ εijk ε ~ N(0, σW2)
α1 + αβ1j
The population mean of treatment 2 in school j is
α2 + αβ2j
The estimate of the mean of treatment 1 in school j is
α1 + αβ1j + ε1j●
The estimate of the mean of treatment 2 in school j is
α2 + αβ2j + ε2j●
Effects and Estimates
The comparative treatment effect in any given school j is
(α1 – α2) + (αβ1j – αβ2j)
The estimate of comparative treatment effect in school j is
(α1 – α2) + (αβ1j – αβ2j) + (ε1j● – ε2j●)
The mean treatment effect in the experiment is
(α1 – α2) + (αβ1● – αβ2●)
The estimate of the mean treatment effect in the experiment is
(α1 – α2) + (αβ 1● – αβ2●) + (ε1●● – ε2●●)
Inference Models
Two different kinds of inferences about effects
Source df E{MS}
Source df E{MS}
2σW2 /mn
Fixed Effects Procedures
The test for treatment effects:
α1 α2 α1 α2 n
with m(n – 1) df
Comparing Fixed and Mixed Effects
Statistical Procedures
(Randomized Block Design)
Fixed Mixed
Inference
Model Conditional Unconditional
df 2m(n – 1) (m – 1)
Schools
2
Treatment 1 2 … m m +1 m +2 …
m
2
The Hierarchical Design
Treatment 1 schools
Schools
Treatment 1 2 … m m +1 m+2 … 2m
2
The Hierarchical Design
Treatment 2 schools
Schools
2
The Conceptual Model
The statistical model for the observation on the kth person in the jth
school in the ith treatment is
Yijk = μ + αi + βi + αβij + εjk(i) = μ + αi + βj(i) + εjk(i)
μ is the grand mean,
αi is the average effect of being in treatment i,
βj is the average effect if being in school j,
αβij is the difference between the average effect of treatment i and the
effect of that treatment in school j,
εijk is a residual
(α1 – α2)
Source df E{MS}
FT = MST/MSBS with (m – 2) df
There is no omnibus test for context effects
Power is determined by the operational effect size
α1 α2 n
1 (n 1) ρ
where ρ = σS2/(σS2 + σW2) = σS2/σ2
Expected Mean Squares
Hierarchical Design
(Two Levels, Schools Fixed)
Source df E{MS}
Note that the school effects are confounded with treatment effects
Fixed Mixed
Inference
Conditional Unconditional
Model
Estimand (α1 – α2) + (β●(1) – β●(2)) (α1 – α2)
The precision is
SE 2 N
Precision of the Estimated Treatment Effect
S2 S2
ρ
S2 W
2
T2
Precision in Two-level Hierarchical Design
With No Covariates
The standard error of the treatment effect is
2 1 (n 1) ρ
SE T
m n
SE increases as ρ increases
Statistical Power
Power in simple (simple random sample) designs depends
on:
• Significance level
• Effect size
• Sample size
d
n 0.10 0.20 … 0.80 1.00 1.20 1.40
8 05 07 … 31 46 60 73
9 06 07 … 35 51 65 79
10 06 07 … 39 56 71 84
11 06 07 … 43 63 76 87
Computing Statistical Power
Power in complex (clustered sample) designs depends on:
• Significance level
• Effect size δ
This is the same table that you use for simple designs
Power in Two-level Hierarchical Design
With No Covariates
Basic Idea:
Operational Effect Size = (Effect Size) x (Design Effect)
n
T
1 n 1 ρ
ΔT = δ x (Design Effect)
1 n 1 ρ
Operational sample size is number of schools (clusters)
Power in Two-level Hierarchical Design
With No Covariates
As m (number of schools) increases, power increases
As effect size increases, power increases
Other influences occur through the design effect
n 1
1
1 n 1 ρ n (1 n )
1
2 1 n 1 ρ
R 2
nR 2
R 2
SE T
W S W
m n
SE decreases as m increases
SE deceases as n increases, but only up to point
SE increases as ρ increases
SE decreases as RW2 and RS2 increase
Power in Two-level Hierarchical Design
With Covariates
Basic Idea:
Operational Effect Size = (Effect Size) x (Design Effect)
ΔT = δ x (Design Effect)
1 (1 RS2 ) ρ
As the covariate-outcome correlations RW2 and RS2
increase the design effect (and power) increases
Three-level Hierarchical Design
where
μ is the grand mean,
αi is the average effect of being in treatment i,
βj(i) is the average effect of being in school j, in treatment i
γk(ij) is the average effect of being in class k in treatment i, in
school j,
εijkl is a residual
Three-level Hierarchical Design
With No Covariates (HLM Notation)
Level 1 (individual level)
Yijkl = β0jk + εijkl ε ~ N(0, σW2)
2 1 pn 1 ρS (n 1) C
SE T
m pn
SE decreases as m increases
SE deceases as p and n increase, but only up to point
SE increases as ρS and ρC increase
Power in Three-level Hierarchical Design
With No Covariates
Basic Idea:
Operational Effect Size = (Effect Size) x (Design Effect)
ΔT = δ x (Design Effect)
1 ( pn 1) S n 1 ρC
1 S 1p C
Thus power only increases up to some limit as n increases
Three-level Hierarchical Design
With Covariates (HLM Notation)
Level 1 (individual level)
Yijkl = β0jk + β1jkXijkl + εijkl ε ~ N(0, σAW2)
Level 2 (classroom level)
β0jk = γ00j + γ01jZjk + η0jk η ~ N(0, σAC2)
β1jk = γ10j
Level 3 (school Level)
γ00j = π00 + π01Tj + π02Wj + ξ0j ξ ~ N(0, σAS2)
γ01j = π01
γ10j = π10
The covariate effects β1jk = γ10j = π10 and γ01j = π01 are fixed
Precision in Three-level Hierarchical Design
With Covariates
2
SE T
m
1 ( pn 1) S n 1 ρ RW2 pnRS2 RW2 S nRC2 RW2 C
pn
SE decreases as m increases
SE deceases as p and n increase, but only up to point
SE increases as ρ increases
SE decreases as RW2, RC2, and RS2 increase
Power in Three-level Hierarchical Design
With Covariates
Basic Idea:
Operational Effect Size = (Effect Size) x (Design Effect)
ΔT = δ x (Design Effect)
pn
TA
1 ( pn 1) S n 1 ρ RW2 pnRS2 RW2 S nRC2 RW2 C
1 1 RS2 S 1
p C C
1 R 2
2 1 (nS 1) ρ
SE T
m n
Basic Idea:
Operational Effect Size = (Effect Size) x (Design Effect)
n
T
1 n 1 ρ
ΔT = δ x (Design Effect)
1 nS 1 ρ
Operational sample size is number of schools (clusters)
Precision in Two-level Randomized Block Design
With Covariates
Basic Idea:
Operational Effect Size = (Effect Size) x (Design Effect)
ΔT = δ x (Design Effect)
S2 S2
ρS
S2 C2 W
2
T2
At the classroom (subcluster) level
C2 C2
ρC
S2 C2 W
2
T2
Three-level Randomized Block Design
Heterogeneity Parameters
In three-level designs, as in two-level randomized block
designs, there is also a parameter reflecting the degree
of heterogeneity of treatment effects across schools
Thus
ωS = σTxS2/σS2
Precision in Three-level Randomized Block Design
With No Covariates
2 1 pnS 1 ρS (n 1) C
SE T
m pn
SE decreases as m increases
SE deceases as p and n increase, but only up to point
SE increases as ωS increases
SE increases as ρS and ρC increase
Power in Three-level Randomized Block Design
With No Covariates
Basic Idea:
Operational Effect Size = (Effect Size) x (Design Effect)
ΔT = δ x (Design Effect)
1 ( pnS 1) S n 1 ρC
1 2 S S 1p C
Thus power only increases up to some limit as n increases
Power in Three-level Randomized Block Design
With Covariates
2
SE T
m
1 ( pnS 1) S n 1 ρ RW2 pnS RS2 RW2 S nRC2 RW2 C
pn
SE decreases as m increases
SE deceases as p and n increases, but only up to point
SE increases as ρ and ωS increase
SE decreases as RW2, RC2, and RS2 increase
Power in Three-level Randomized Block Design
With Covariates
Basic Idea:
Operational Effect Size = (Effect Size) x (Design Effect)
ΔT = δ x (Design Effect)
1 2 1 RS2 S S 1
p C C
1 R 2
Assign Schools 1 pn 1 ρS (n 1) C
(Hierarchical Design)
pn