2008 AGI-Information Management Consultants May be used for personal purporses only or by libraries associated to dandelon.com network.

EXPERIMENTAL AND QUASI-EXPERIMENTAL DESIGNS FOR GENERALIZED CAUSAL INFERENCE

William R. Shadish
THE UNIVERSITY OF MEMPHIS

Thomas D. Cook
NORTHWESTERN UNIVERSITY

Donald T. Campbell

HOUGHTON MIFFLIN COMPANY

Boston

New York

Contents
Preface 1. EXPERIMENTS AND GENERALIZED CAUSAL INFERENCE Experiments and Causation
Defining Cause, Effect, and Causal Relationships Causation, Correlation, and Confounds Manipulable and Nonmanipulable Causes Causal Description and Causal Explanation Modern Descriptions of Experiments Randomized Experiment Quasi-Experiment Natural Experiment Nonexperimental Designs Experiments and the Generalization of Causal Connections Most Experiments Are Highly Local But Have General Aspirations Construct Validity: Causal Generalization as Representation External Validity: Causal Generalization as Extrapolation Approaches to Making Causal Generalizations Experiments and Metascience The Kuhnian Critique Modern Social Psychological Critiques Science and Trust Implications for Experiments A World Without Experiments or Causes?

xv

1 3
3 7 7 9 12 13 13 17 18 18 18 20 21 22 26 27 28 28 13 31

CONTENTS

2. STATISTICAL CONCLUSION VALIDITY AND INTERNAL VALIDITY

Validity A Validity Typology Threats to Validity Statistical Conclusion Validity Reporting Results of Statistical Tests of Covariation Threats to Statistical Conclusion Validity The Problem of Accepting the Null Hypothesis Internal Validity Threats to Internal Validity Estimating Internal Validity in Randomized Experiments and Quasi-Experiments The Relationship Between Internal Validity and Statistical Conclusion Validity

33
34 37 39 42 42 45 52 53 54 61 63

3. CONSTRUCT VALIDITY AND EXTERNAL VALIDITY

Construct Validity
Why Construct Inferences Are a Problem Assessment of Sampling Particulars Threats to Construct Validity Construct Validity, Preexperimental Tailoring, and Postexperimental Specification External Validity Threats to External Validity Constancy of Effect Size Versus Constancy of Causal Direction Random Sampling and External Validity Purposive Sampling and External Validity More on Relationships, Tradeoffs, and Priorities The Relationship Between Construct Validity and External Validity The Relationship Between Internal Validity and Construct Validity Tradeoffs and Priorities Summary

64
64
66 69 72 81 83 86 90 91 92 93 93 95 96 102

CONTENTS

VII

4. QUASI-EXPERIMENTAL DESIGNS THAT EITHER LACK A CONTROL GROUP OR LACK PRETEST OBSERVATIONS ON THE OUTCOME 103
The Logic of Quasi-Experimentation in Brief Designs Without Control Groups
The One-Group Posttest-Only Design The One-Group Pretest-Posttest Design The Removed-Treatment Design The Repeated-Treatment Design Designs That Use a Control Group But No Pretest Posttest-Only Design With Nonequivalent Groups Improving Designs Without Control Groups by Constructing Contrasts Other Than With Independent Control Groups The Case-Control Design

104 106
106 108 111 113 115 115 125 128

Conclusion

134

5. QUASI-EXPERIMENTAL DESIGNS THAT USE BOTH CONTROL GROUPS AND PRETESTS

Designs That Use Both Control Groups and Pretests
The Untreated Control Group Design With Dependent Pretest and Posttest Samples Matching Through Cohort Controls Designs That Combine Many Design Elements Untreated Matched Controls With Multiple Pretests and Posttests, Nonequivalent Dependent Variables, and Removed and Repeated Treatments Combining Switching Replications With a Nonequivalent Control Group Design An Untreated Control Group With a Double Pretest and Both Independent and Dependent Samples The Elements of Design Assignment Measurement Comparison Groups Treatment Design Elements and Ideal Quasi-Experimentation

135
136
136 148 153

153 154 154 156 156 158 159 160 160

viii

CONTENTS

Conclusion Appendix 5.1: Important Developments in Analyzing Data From Designs With Nonequivalent Groups Propensity Scores and Hidden Bias Selection Bias Modeling Latent Variable Structural Equation Modeling 6. QUASI-EXPERIMENTS: INTERRUPTED TIME-SERIES DESIGNS What Is a Time Series? Describing Types of Effects Brief Comments on Analysis Simple Interrupted Time Series A Change in Intercept A Change in Slope Weak and Delayed Effects The Usual Threats to Validity Adding Other Design Features to the Basic Interrupted Time Series Adding a Nonequivalent No-Treatment Control Group Time Series Adding Nonequivalent Dependent Variables Removing the Treatment at a Known Time Adding Multiple Replications Adding Switching Replications Some Frequent Problems with Interrupted Time-Series Designs Gradual Rather Than Abrupt Interventions Delayed Causation Short Time Series Limitations of Much Archival Data A Comment on Concomitant Time Series Conclusion 7. REGRESSION DISCONTINUITY DESIGNS The Basics of Regression Discontinuity The Basic Structure

161

161 162 166 169

171 172 172 174 175 175 176 178 179 181 182 184 188 190 192 195 196 197 198 203 205 206 207 208 208

CONTENTS

IX

Examples of Regression Discontinuity Designs Structural Requirements of the Design Variations on the Basic Design Theory of the Regression Discontinuity Design Regression Discontinuities as Treatment Effects in the Randomized Experiment Regression Discontinuity as a Complete Model of the Selection Process Adherence to the Cutoff Overrides of the Cutoff Crossovers and Attrition Fuzzy Regression Discontinuity Threats to Validity Regression Discontinuity and the Interrupted Time Series Statistical Conclusion Validity and Misspecification of Functional Form Internal Validity Combining Regression Discontinuity and Randomized Experiments Combining Regression Discontinuity and Quasi-Experiments Regression DiscontinuityExperiment or Quasi-Experiment? Appendix 7.1: The Logic of Statistical Proofs about Regression Discontinuity 8. RANDOMIZED EXPERIMENTS: RATIONALE, DESIGNS, AND CONDITIONS CONDUCIVE TO DOING THEM The Theory of Random Assignment What Is Random Assignment? Why Randomization Works Random Assignment and Units of Randomization The Limited Reach of Random Assignment Some Designs Used with Random Assignment The Basic Design The Pretest-Posttest Control Group Design Alternative-Treatments Design with Pretest

212 216 219 220 221

224

227
227 228 229

229
229 230 237

238 241 242 243

246 247 248 248 253 256 257 257 261 261

CONTENTS

Multiple Treatments and Controls with Pretest Factorial Designs Longitudinal Designs Crossover Designs Conditions Most Conducive to Random Assignment When Demand Outstrips Supply When an Innovation Cannot Be Delivered to All Units at Once When Experimental Units Can Be Temporally Isolated: The Equivalent-Time-Samples Design When Experimental Units Are Spatially Separated or Interunit Communication Is Low When Change Is Mandated and Solutions Are Acknowledged to Be Unknown When a Tie Can Be Broken or Ambiguity About Need Can Be Resolved When Some Persons Express No Preference Among Alternatives When You Can Create Your Own Organization When You Have Control over Experimental Units When Lotteries Are Expected When Random Assignment Is Not Feasible or Desirable Discussion

262 263 266 268 269 269 270 270 271 272 273 273 274 274 275 276 277

9. PRACTICAL PROBLEMS 1: ETHICS, PARTICIPANT RECRUITMENT, AND RANDOM ASSIGNMENT

Ethical and Legal Issues with Experiments The Ethics of Experimentation Withholding a Potentially Effective Treatment The Ethics of Random Assignment Discontinuing Experiments for Ethical Reasons Legal Problems in Experiments Recruiting Participants to Be in the Experiment Improving the Random Assignment Process Methods of Randomization What to Do If Pretest Means Differ Matching and Stratifying Matching and Analysis of Covariance The Human Side of Random Assignment

279
280 281 283 286 289 290 292 294 294 303 304 306 307

Conclusion

311

CONTENTS

xi

Appendix 9.1: Random Assignment by Computer SPSS and SAS World Wide Web Excel 10. PRACTICAL PROBLEMS 2: TREATMENT IMPLEMENTATION AND ATTRITION Problems Related to Treatment Implementation Inducing and Measuring Implementation Analyses Taking Implementation into Account Post-Assignment Attrition Defining the Attrition Problem Preventing Attrition Analyses of Attrition Discussion 11. GENERALIZED CAUSAL INFERENCE: A GROUNDED THEORY The Received View of Generalized Causal Inference: Formal Sampling Formal Sampling of Causes and Effects Formal Sampling of Persons and Settings Summary A Grounded Theory of Generalized Causal Inference Exemplars of How Scientists Make Generalizations Five Principles of Generalized Causal Inferences The Use of Purposive Sampling Strategies Applying the Five Principles to Construct and External Validity Should Experimenters Apply These Principles to All Studies? Prospective and Retrospective Uses of These Principles Discussion

311 311 313 313

314 315 315 320 323 323 324 334 340

341 342 344 346 348 348 349 353 354 356 371 372 373

12. GENERALIZED CAUSAL INFERENCE: METHODS FOR SINGLE STUDIES 374 Purposive Sampling and Generalized Causal Inference Purposive Sampling of Typical Instances Purposive Sampling of Heterogeneous Instances 374 375 376

xii

CONTENTS

Purposive Sampling and the First Four Principles Statistical Methods for Generalizing from Purposive Samples Methods for Studying Causal Explanation Qualitative Methods Statistical Models of Causal Explanation Experiments That Manipulate Explanations

378 386 389 389 392 414

Conclusion

415

13. GENERALIZED CAUSAL INFERENCE: METHODS FOR MULTIPLE STUDIES 417

Generalizing from Single Versus Multiple Studies Multistudy Programs of Research Phased Models of Increasingly Generalizable Studies Directed Programs of Experiments Narrative Reviews of Existing Research Narrative Reviews of Experiments Narrative Reviews Combining Experimental and Nonexperimental Research Problems with Narrative Reviews Quantitative Reviews of Existing Research The Basics of Meta-Analysis Meta-Analysis and the Five Principles of Generalized Causal Inference Discussion of Meta-Analysis Appendix 13.1: Threats to the Validity of Meta-Analyses Threats to Inferences About the Existence of a Relationship Between Treatment and Outcome Threats to Inferences About the Causal Relationship Between Treatment and Outcome Threats to Inferences About the Constructs Represented in Meta-Analyses Threats to Inferences About External Validity in Meta-Analyses 418 418 419 420 421 422 423 423 425 426 435 445 446 447 450 453 454

14. A CRITICAL ASSESSMENT OF OUR ASSUMPTIONS Causation and Experimentation

Causal Arrows and Pretzels Epistemological Criticisms of Experiments Neglected Ancillary Questions

456 453
453 459 461

CONTENTS

xiii

Validity
Objections to Internal Validity Objections Concerning the Discrimination Between Construct Validity and External Validity Objections About the Completeness of the Typology Objections Concerning the Nature of Validity Quasi-Experimentation Criteria for Ruling Out Threats: The Centrality of Fuzzy Plausibility Pattern Matching as a Problematic Criterion The Excuse Not to Do a Randomized Experiment Randomized Experiments Experiments Cannot Be Successfully Implemented Experimentation Needs Strong Theory and Standardized Treatment Implementation Experiments Entail Tradeoffs Not Worth Making Experiments Assume an Invalid Model of Research Utilization The Conditions of Experimentation Differ from the Conditions of Policy Implementation Imposing Treatments Is Fundamentally Flawed Compared with Encouraging the Growth of Local Solutions to Problems Causal Generalization: An Overly Complicated Theory? Nonexperimental Alternatives Intensive Qualitative Case Studies Theory-Based Evaluations Weaker Quasi-Experiments Statistical Controls

462
462 466 473 475 484 484 485 486 488 488 489 490 493 495 497 498 499 500 501 502 503

Conclusion Glossary References Name Index Subject Index

504 505 514 593 609