I

1

I

Experimental and Quasi-Experimental Designs for Generalized Causal Inference

I I

,

I

i

I

II

i

I

' I

I I

II

To Donald T. Campbell and Lee J. Cronbach, who helped us to understand science, and to our wives, Betty, Fay, and Barbara, who helped us to understand life.

Editor-in-Chief: Kathi Prancan Senior Sponsoring Editor: Kerry Baruth Associate Editor: Sara Wise Associate Project Editor: Jane Lee Editorial Assistant: Martha Rogers Production Design Coordinator: Lisa Jelly Senior Manufacturing Coordinator: Marie Barnes Executive Marketing Manager: Ros Kane Marketing Associate: Caroline Guy Copyright 2002 Houghton Mifflin Company. All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, or by any information storage or retrieval system without the prior written permission of the copyright owner unless such copying is expressly permitted by federal copyright law. With the exception of nonprofit transcription in Braille, Houghton Mifflin is not authorized to grant permission for further uses of copyrighted selections reprinted in this text without the permission of their owners. Permission must be obtained from the individual copyright owners as identified herein. Address requests for permission to make copies of Houghton Mifflin material to College Permissions, Houghton Mifflin Company, 222 Berkeley Street, Boston, MA 02116-3764. Printed in the U.S.A. Library of Congress Catalog Card Number: 2001131551 ISBN: 0-395-61556-9 9-MV-08

ContentsPrefaceXV

1. EXPERIMENTS AND GENERALIZED CAUSAL INFERENCEExperiments and CausationDefining Cause, Effect, and Causal Relationships Causation, Correlation, and Confounds Manipulable and Nonmanipulable Causes Causal Description and Causal Explanation

!

133

7 7 9

Modern Descriptions of ExperimentsRandomized Experiment Quasi-Experiment Natural Experiment Nonexperimental Designs

1213 13

17 18

Experiments and the Generalization of Causal ConnectionsMost Experiments Aie Highly Local But Have General Aspirations Construct Validity: Causal Generalization as Representation External Validity: Causal Generalization as Extrapolation Approaches to Making Causal Generalizations

1818 20 21 22

Experiments and Metascience ..The Kuhnian Critique Modern Social Psychological Critiques Science and Trust Implications for Experiments

26 27 28 28 29

A World Without Experiments or Causes?

31

v

vi

I

CONTENTS

2. STATISTICAL CONCLUSION VALIDITY AND INTERNAL VALIDITYValidityA Validity Typology Threats to Validity

33 3437 39

Statistical Conclusion ValidityReporting Results of Statistical Tests of Covariation Threats to Statistical Conclusion Validity The Problem of Accepting the Null Hypothesis

4242 45 52

Internal ValidityThreats to Internal Validity Estimating Internal Validity in Randomized Experiments and Quasi-Experiments

5354 61

The Relationship Between Internal Validity and Statistical Conclusion Validity

63

3. CONSTRUCT VALIDITY AND EXTERNAL VALIDITYConstruct ValidityWhy Construct Inferences Are a Problem Assessment of Sampling Particulars Threats to Construct Validity Construct Validity, Preexperimental Tailoring, and Postexperimental Specification

64 6466 69 72 81

External ValidityThreats to External Validity Constancy of Effect Size Versus Constancy of Causal Direction Random Sampling and External Validity Purposive Sampling and External Validity

8386 90 91 92

More on Relationships, Tradeoffs, and PrioritiesThe Relationship Between Construct Validity and External Validity The Relationship Between Internal Validity and Construct Validity Tradeoffs and Priorities

9393 95 96

Summary

102

CONTENTS

I vii

4. QUASI-EXPERIMENTAL DESIGNS THAT EITHER LACK A CONTROL GROUP OR LACK PRETEST OBSERVATIONS ON THE OUTCOME 103The Logic of Quasi-Experimentation in Brief Designs Without Control GroupsThe One-Group Posttest-Only Design The One-Group Pretest-Posttest Design The Removed-Treatment Design The Repeated-Treatment Design

104 106106 108 111 113

Designs That Use a Control Group But No PretestPosttest-Only Design With Nonequivalent Groups Improving Designs Without Control Groups by Constructing Contrasts Other Than With Independent Control Groups The Case-Control Design

115115 125 128

Conclusion

134

5. QUASI-EXPERIMENTAL DESIGNS THAT USE BOTH CONTROL GROUPS AND PRETESTSDesigns That Use Both Control Groups and :PretestsThe Untreated Control Group Design With Dependent Pretest and Posttest Samples Matching Through Cohort Controls

135 136136 148

Designs That Combine Many Design ElementsUntreated Matched Controls With Multiple Pretests and Posttests, Nonequivalent Dependent Variables, and Removed and Repeated Treatments Combining Switching Replications With a Nonequivalent Control Group Design An Untreated Control Group With a Double Pretest and Both Independent and Dependent Samples

153

153 154 154

The Elements of DesignAssignment Measurement Comparison Groups Treatment Design Elements and Ideal Quasi-Experimentation

15 6156 158 159 160 160

..I I I ' I

viii

J

CONTENTS

Conclusion Appendix5.1: important Developments in Analyzing Data From Designs With Nonequivalent GroupsPropensity Scores and Hidden Bias Selection Bias Modeling Latent Variable Structural Equation Modeling

161

161162 166 169

6. QUASI-EXPERIMENTS: INTERRUPTED TIME-SERIES DESIGNSWhat Is a Time Series?Describing Types of Effects Brief Comments on Analysis

171 172172 174

Simple Interrupted Time SeriesA Change in Intercept A Change in Slope Weak and Delayed Effects The Usual Threats to Validity

175175 176 178 179

Adding Other Design Features to the Basic Interrupted Time SeriesAdding a Nonequivalent No-Treatment Control Group Time Series Adding Nonequivalent Dependent Variables Removing the Treatment at a Known Time Adding Multiple Replications Adding Switching Replications

181182 184 188 190 192

Some Frequent Problems with Interrupted Time-Series DesignsGradual Rather Than Abrupt Interventions Delayed Causation Short Time Series Limitations of Much Archival Data

195196 197 198 203

A Comment on Concomitant Time Series Conclusion

205 206 207 208208

7. REGRESSION DISCONTINUITY DESIGNSThe Basics of Regression DiscontinuityThe Basic Structure

CONTENTS

I ix

Examples of Regression Discontinuity Designs Structural Requirements of the Design Variations on the Basic Design

212 216 219

Theory of the Regression Discontinuity DesignRegression Discontinuities as Treatment Effects in the Randomized Experiment Regression Discontinuity as a Complete Model of the Selection Process

220221 224

Adherence to the CutoffOverrides of the Cutoff Crossovers and Attrition Fuzzy Regression Discontinuity

22 7227 228 229

Threats to ValidityRegression Discontinuity and the Interrupted Time Series Statistical Conclusion Validity and Misspecification of Functional Form Internal Validity

229229 230 ?37

Combining Regression Disconfinuity and Randomized Experiments Combining Regression Discontinuity and I. Quasi-Experiments Regression Discontinuity-E:J'periment or Quasi-Experiment? Appendix 7.1: The Logic of Statistical Proofs alwut Regression Discontinuity

238 241 242 243

8. RANDOMIZED EXPERIMENTS: RATIONALE, DESIGNS, AND CONDITIONS CONDUCIVE TO DOING THEMThe Theory of Random AssignmentWhat Is Random Assignment? Why Randomization Works Random Assignment and Units of Randomization The Limited Reach of Random Assignment

246 24 7248 248 253 256

Some Designs Used with Random AssignmentThe Basic Design The Pretest-Posttest Control Group Design Alternative-Treatments Design with Pretest

257257 261 261

~.. I'

X

I CONTENTSMultiple Treatments and Controls with Pretest Factorial Designs Longitudinal Designs Crossover Designs 262 263 266 268

Conditions Most Conducive to Random AssignmentWhen Demand Ou!strips Supply When an Innovation Cannot Be Delivered to All Units at Once When Experimental Units Can Be Temporally Isolated: The Equivalent-Time-Samples Design When Experimental Units Are Spatially Sep;lrated or I11terunit Communication Is Low When Change Is Mandated and Solutions Are Acknowledged to Be Unknown When a Tie Can Be Broken or Ambiguity About Need Can Be Resolved When Some fersons Express No Preference Among Alternatives When You Can Create Your Own Organization When You Have Control over Experimental Units When Lotteries Are Expected

269269 270 270 271 272 273 273 274 274 2 75

When Random Assignment Is Not Feasible or Desirable Discussion

276 277

9. PRACTICAL PROBLEMS 1: ETHICS, PARTICIPANT RECRUITMENT, ~ND RANDOM ASSIGNMENTEthical and Legal Issues with ExperimentsThe Ethics of Experimentation Withholding a Potentially Effective Treatment The Ethics of Random Assignment Discontinuing Experiments for Ethical Reasons Legal Problems in Experiments

279 280281 283 286 289 290

Recruiting Participants to Be in t}le Experiment Improving the Random Assignmer.t ProcessMethods of Randomization What to Do If Pretest Means Differ Matching and Stratifying Matching and Analysis of Covariance The Human Side of Random As~ignment

292 294294 303 304 306 307

Conclusion

311

CONTENTS

I xi

Appendix 9.1: Random Assignment by ComputerSPSS and SAS World Wide Web Excel

311311 313 313

10. PRACTICAL PROBLEMS 2: TREATMENT IMPLEMENTATION AND ATTRITIONProblems Related to Treatment ImplementationInducing and Measuring Implementation Analyses Taking Implementation into Account

314 315315 320

Post-Assignment AttritionDefining the Attrition Problem Preventing Attrition Analyses of Attrition

323323 324 334

Discussion

340

11. GENERALIZED CAUSAL INFERENCE: A GROUNDED THEORYThe Received View of Generalized Causal Inference: Formal SamplingFormal Sampling of Causes and Effects Formal Sampling of Persons and Settings Summary

341 342344 346 348

A Grounded Theory of Generalized Causal. InferenceExemplars of How Scientists Make Generalizations Five Principles of Generalized Causal Inferences The Use of Purposive Sampling Strategies Applying the Five Principles to Construct and External Validity Shoulg Experimenters Apply These Principles to All Studies? Prospective and Retrospective Uses of These Principles

348349 353 354 356 371 372

Discussion

373

12. GENERALIZED CAUSALJNFERENCE: METHODS FOR SINGLE STUDIES 374Purposive Sampling and Generalized Causal InferencePurposive Sampling of Typical Instances Purposive Sampling of Heterogeneous Instances

374375 376

I I II' ,,

xii

I

CONTENTS

Purposive Sampling and the First Four Principles Statistical Methods for Generalizing from Purposive Samples

378 386

Methods for Studying Causal ExplanationQualitative Methods Statistical Models of Causal Explanation Experiments That Manipulate Explanations

389389 392 414

Conclusion

415

13. GENERALIZED CAUSAL INFERENCE: METHODS 417 FOR MULTIPLE STUDIESGeneralizing from Single Versus Multiple Studies Multistudy Programs of ResearchPhased Models of Increasingly Generalizable Studies Directed Programs of Experiments

418 418419 420

Narrative Reviews of Existing ResearchNarrative Reviews of Experiments Narrative Reviews Combining Experimental and Nonexperimental Research Problems with Narrative Reviews

421422 423 423

Quantitative Reviews of Existing ResearchThe Basics of Meta-Analysis Meta-Analysis and the Five Principles of Generalized Causal Inference Discussion of Meta-Analysis

425426 435 445

Appendix 13.1: Threats to the Validity of Meta-AnalysesThreats to Inferences About the Existence of a Relationship Between Treatment and Outcome Threats to Inferences About the Causal Relationship Between Treatment and Outcome Threats to Inferences About the Constructs Represented in Meta-Analyses Threats to Inferences About External Validity in Meta-Analyses

44644 7 450 453 454

14. A CRITICAL ASSESSMENT OF OUR ASSUMPTIONSCausation and ExperimentationCausal Arrows and Pretzels Epistemological Criticisms of Experiments Neglected Ancillary Questions

456 453453 459 461

CONTENTS

/ xiii

ValidityObjections to Internal Validity Objections Concerning the Discrimination Between Construct Validity and External Validity Objections About the Completeness of the Typology Objections Concerning the Nature of Validity

462462 466 473 475

Quasi-ExperimentationCriteria for Ruling Out Threats: The Centrality of Fuzzy Plausibility Pattern Matching as a Problematic Criterion The Excuse Not to Do a Randomized Experiment

484484 485 486

Randomized ExperimentsExperiments Cannot Be Successfully Implemented Experimentation Needs Strong Theory and Standardized Treatment Implementation Experiments Entail Tradeoffs Not Worth Making Experiments Assume an Invalid Model of Research Utilization The Conditions of Experimentation Differ from the Conditions of Policy Implementation Imposing Treatments Is Fundamentally Flawed Compared with Encouraging the Growth of Local Solutions to Problems

488488 489 490 493 495 497

Causal Generalization: An Overly Complicated Theory? Nonexperimental AlternativesIntensive Qualitative Case Studies Theory-Based Evaluations Weaker Quasi-Experiments Statistical Controls

498 499500 501 502 503

Conclusion Glossary References Name Index Subject Index

504 505 514 593 609

Preface

a book for those who have already decided that identifying a dependable relationship between a cause and its effects is a high priority ~nd who wish to consider experimental methods for doing so. Such causal relationships are of great importance in human affairs. The rewards associated with being correct in identifying causal relationships can be high, and the costs of misidentification can be tremendous. To know whether increased schooling pays,off in later life happiness or in increased lifetime earnings is a boon to individuals facing the decision about whether to spend more time in school, and it also helps policymakers determine how much financiaJ support to give educational ipstitutions. In health, from the earliest years of hmnan existence, causation has heiped to identify which strategies are effective in dealing with disease. In pharmaq:>logy, divinations and reflections on experience in the remote past sometimes led to 'the development of many useful treatments, but other judgments about effective plants and ways of placating gods were certainly more incorrect than correct and presumably contributed to many unnecessary deaths. The utility of finding such causal connections is so widely understood that much effort goes to locating them in both human affairs in general and in science in particular. However, history also teaches us that it is rare for those causes to be so universally tme that they hold under all conditions with all types of people and at all historical time periods. All causal statements are inevitably contingent. Thus, although threat from an out-group often causes in-group cohe~ion, this is not always the case. For instance, in 1492 the king of Granada had to watch as his Moorish subjects left the city to go to their ancestral homes in North Africa, being unwilling to fight against the numerically superior troops of the Catholic kings of Spain who were lodgeq in Santa Fe de la Frontera nearby. Here, the external threat from the out-group of Christian Spaniards led not to increased social cohesion among the Moslem Spaniards but rather to the latter's disintegration as a defensive force. Still, some causal hypotheses are more contingent than others. It is of obvious utility to learn as much as one can about those contingencies and to identify those relationships that hold more consistently. For instance, aspirin is such a wonderful drug because it reduces the symptoms associated with many different kinds of illness, including head colds, colon cancer, eliei th:tt design eleJucnts .'I:Ui.:h a.'ll coutror hrrou ps or pret{:!'.ts are mtdcsirahle or unnec:es.'llaly-c\'Cn when desc.ripthc causal infen:t1ce is the highest priority. We wanl' to ca~t duuht ut1

such beliefs by showing th~ co.rs ro iuteru:tl v:tlidit y that such dtsigns can incur so that re.earchers c:1n choose whether to incur these costs given their other resca rch. pri()ritics, Second, :'"'e usc these clcsih.,,s to illu.'lltrat(: how various ..,a lidity d1rcais operate ~:ith :tctuaJ c.xarnp1c.'ll, for it is rnon: jmportant to learn how LO tnink ~ritically :tbout these threats than it is to learn a list of desigr