research 8 quasiexpdesign

7/30/2019 Research 8 QuasiExpDesign

1/10

N O X O

N O O

N O O X O

N O O O

Quasi-Experimental Design

A quasi-experiment is one where the treatment variable is manipulated but the

groups are not equated prior to manipulation of the independent variable. I shalldiscuss a few such designs here.

Pretest-Posttest Nonequivalent Groups Design

This design looks a lot like the randomized pretest-posttestdesign that I discussed earlier, but in this case the two groups havenot been equated prior to treatment. Since one has not randomlyassigned subjects to groups, one cannot assume that the populations being comparedare equivalent on all things prior to the treatment, and accordingly internal validity isthreatened. When a post-treatment difference between groups is observed, one cannot

with great confidence attribute that effect to the treatment, since the groups may have

had pre-existing differences that caused the observed post-treatment difference.In the language of Campbell and Stanley, the threat here is selection and all of

the various interactions involving selection. One can try to mitigate the problem byassigning subjects to groups (or selecting intact groups) in ways that make it likely thatthe groups do not differ greatly prior to the treatment, but one always worries about theunknown variables on which the groups might differ and which might affect the criterionvariable.

Trochim has written several pages in which he presents hypothetical examples ofthis design and then critiques them with respect to which threats to internal validityseem of most concern and which seem unlikely. You should read those pages carefullyto get a feel for the sort of thinking that is involved when trying to determine howsuspicious one should be of conclusions drawn from research gathered through apretest-posttest nonequivalent groups design.

So, why would one ever choose to employ this design? Frankly, one should not,if a better design (such as a randomized pretest-posttest design) is feasible, butsometimes you cannot accomplish random assignment, especially when dealing withhuman subjects (who dont like to be told what treatments they are or are not going toget) in field settings.

Double-Pretest Nonequivalent Groups Design

This modification of the pretest-posttestnonequivalent groups design helps to control for aSelection x Maturation interaction by including a second pretest. If the groups arematuring at different rates, that difference may appear in the comparison between thefirst pretest and the second pretest.

Regression-Discontinuity Design

Copyright 2003, Karl L. Wuensch - All rights reserved.

Research-8-QuasiExpDesign.doc


2/10

C O X O

C O O

This design looks a lot like the pretest-posttestnonequivalent groups design, but the groups are nonequivalentby choice. The C the first column indicates that the subjects areassigned to groups based on their score on the covariate (thepretest).

One usually starts by deciding what the treatment and criterion variables will be.For example, I may decide that my treatment variable will involve an online tutorialprogram in basic statistics and my criterion variable will be students performance in anundergraduate statistics class. I shall offer the program to some students (and enticethem to use it) but not to other students. At the end of the semester I shall use scoreson the comprehensive final examination as the criterion variable.

So, what do I use as the pretest? I could administer the comprehensive finalexamination twice, at the beginning of the semester (pretest) and the end of thesemester (posttest), but that might just reveal that none of my students know anystatistics before they take a statistics class. I might decide to use an alternative sort ofpretest, such as a test of statistics aptitude (based largely on verbal and logicalreasoning but with a little math thrown in too). I would want scores on this test to behighly predictive of final examination performance -- that is students who do well on thisaptitude test (those with good verbal and logical skills, even if they dont know anystatistics yet) will be highly likely to do well in the course, while those who door poorlyon the aptitude test will likely have great difficulty with the course.

Now, how am I going to assign subjects to groups? I could just announce theavailability of the tutorial program and let anybody use it. I would keep track of whoused it and who did not, and at the end of the semester I would compare those twogroups in terms of how they performed on the final examination. I would use the pretestscores as a covariate in an ANCOV. Conducted in this fashion, this would be an

example of a pretest-posttest nonequivalent groups design. There almost certainlywould be pre-treatment differences between those who elected to use the tutorialprogram and those who elected not to. While some researchers think that the use ofANCOV enables one statistically to remove such a confound, that is not exactly true.Interpretation of the results of an ANCOV where the experimental groups differ on thecovariate is very tricky. Even if the ANCOV could remove the confound involving pre-treatment differences on the covariate, the groups probably differ on other importantcharacteristics that could affect the results.

So, I abandon the idea of using a pretest-posttest nonequivalent groups design.I decide instead to use a randomized pretest-posttest design. I use a random numbergenerator to decide who gets to use the tutorial program and who does not. Not long

after the start of the semester I start to get angry phone calls from concerned parentswho are worried that their children will be at a disadvantage in the class because theyare in the deprived (control) group. They explain that being so deprived could lead tothem failing the class, or getting a grade that would keep them out of graduate school orcost them a scholarship and so on. I am risking those students futures for my stinkingresearch and they are not going to stand for it. After a few of these parents (and/or thestudents) call up my chair, my dean, the chancellor, the Board of Governors, theirlegislative representatives, the local media, and so on, and their attorneys phone the

2


3/10


4/10


5/10


6/10


7/10


8/10

N O1 X O2

N O1 O2

N ON X O

N O

N O

N O X O ON O O X O

Proxy-Pretest Design

In this design one gathers the pretest information after theexperimental treatment has started. In other words, one finds anarchival proxy for the pretest. For example, suppose I ask the following question:

Does completion of PSYC 2210 (experimental psychology) have an effect on astudents knowledge of statistics? Ideally I would measure the students statisticalknowledge at the beginning of the semester, but suppose that the question did notoccur to me until the middle of the semester. I might decide to use as a proxy-preteststudents performance in their PSYC 2101 (statistics) class. My control group mightconsist of a group of students taking some other class (not 2210). For each student I

would obtain a continuous measurement of the students performance in PSYC 2101and, at the end of the semester, a continuous measurement of the students knowledgeof statistics. ANCOV would be used, with the proxy-pretest serving as a covariate.

Separate Pre-Post Samples Design

In this design the sample of subjects that you use for thepretest is different from the sample of subjects that you use forthe posttest. There are several variations of this design.Suppose that I want to evaluate the effect of a tutorial programin my statistics class. My colleague Suzie Q and I each taughtstatistics both this semester and the previous semester and weeach gave our students a standardized test of statistics achievement at the end of thesemester (as part of a departmental evaluation of the course). This semester I shallmake the tutorial program available to all of the students in my class, but my colleague

will not. Again, both teachers administer the statistics achievement test at the end ofthe semester.

Look at the design notation in the box above. Note that there are fournonequivalent groups. The first line represents my students last semester, when I didnot make the tutorial program available. The second line represents my students thissemester, when the tutorial program was made available. The third line represents mycolleagues students last semester, and the fourth line represents my colleaguesstudents this semester. The potential for selection problems is clearly large with thisdesign.

Nonequivalent Groups Switching Replications Design

Recall our earlier (Chapter 7) discussion of

the randomized groups switching replicationsdesign. The quasi-experimental version of thisdesign differs in that the comparison groups are notequated by randomization. For example, when evaluating the effect of my experimentalstatistics tutorial, I could make it available to one class of students during only the firsthalf the semester and to the other class during only the second half of the semester.This might reduce, somewhat, complaints about not getting the special treatment rightaway, unless the one class learns what is going on in the other class.

8


9/10


10/10

research 8 quasiexpdesign

Documents