chapter 22 quasi-experimental and n=1 designs of research

Chapter 22

Quasi-Experimental And N=1 Designs OF Research

• In earlier chapters we stated and emphasized that one of the major goals of science is to find causal relations. In the behavioral sciences, the true experiment is the strongest approach used to meet this goal.

• However, there are research problems in the behavioral sciences and especially educational research that cannot be studied using a true experimental design. We will examine two research designs where one or more of the components of the true experiment have been compromised. The first is called quasi experimental designs and the second is called single subject or N=1 designs.

Compromise Designs a.k.a. Quasi-Experimental Designs

• Recall that true experimentation requires at least two groups, one receiving an experimental treatment and one not receiving the treatment, or receiving it in different form. The true experiment requires the manipulation of at least one independent variable, the random assignment of participants to groups, and the random assignment of treatments to groups.

Compromise Designs a.k.a. Quasi-Experimental Designs

• Cook and Campbell (1979) present two major classifications of quasi-experimental design.

• The first is called the ”nonequivalent control group designs,” the second is the “interrupted time series designs.”

Nonequivalent Control Group Designs

• No-treatment control group designs

• Nonequivalent dependent variables designs

• Removed treatment group designs

• Repeated treatment designs

• Reversed treatment nonequivalent control group designs

• Posttest only designs

• Regression continuity designs

No-Treatment Control Group Designs

• Design 22.1• An effort should be made to at least use

samples from the same population, or samples that are as alike as possible. The experimental treatments should be assigned at random. Then the similarity of the groups should be checked using any information available (sex, age, social class, and so on). The equivalence of the groups could be verified using the means and standard deviations of the pretests: t-test and F-test will do.


• There are still difficulties, all of which are subordinate to one main difficulty—selection. When participants are selected into groups on bases extraneous to the research purposes, we call this “selection” or alternatively, “self-selection”

• Note that if we had used only volunteers and had assigned them to experimental and control groups at random, the selection difficulty is lessened. External validity or representativeness, however, would be decreased.


• Without the benefit of random assignment, attempts should be made through other means to eliminate rival hypotheses. We consider only the design that uses the pretest because the pretest could provide useful information concerning the effectiveness of the independent variable on the dependent variable.


• Another more frequent example in educational research is to take some school, classes for the experimental group and others for the control group. If a fairly large number of classes are selected and assigned at random to experimental and control groups, there is no great problem.

• But if they are not assigned at random, certain ones may select themselves into the experimental groups, and these classes may have characteristics that predispose them to have higher mean Y scores than the other classes.


• In other words, something that influences the selection process (e.g., volunteer participants), also influences the dependent variable measures. This occurs even though the pretest may show the groups to be the same (alike) on the dependent variable. The X manipulation is “effective” because of selection, or self-selection, but it is not effective in and of itself.


• Possible outcomes from this design are given in Figure 22.1. There is the possibility of a different interpretation on causality depending on which outcome the researcher obtains. In almost all of the cases the most likely threat to internal validity would be the selection-maturation interaction.


• You might recall that this interaction occurs when (1) two groups are different to begin with as measured by the pretest; then (2) one of the groups experience greater differential changes, such as getting more experienced, more accurate, more tired, and so on, than the other group. The after-treatment difference, as observed in the posttest, can not exactly be attributed to the treatment itself.


• There are four alternative explanations to the outcome in Figure 22.1(a)

• The first is selection-maturation interaction. Group E’s increase may be due to their higher level of intelligence. With a higher level of intelligence, these participants can process more, or grow faster, than Group C.


• A second explanation is one of instrumentation. The scale used to measure the dependent variable may be more sensitive at certain levels than others. In a normal distribution, changes in raw scores near the center of the distribution reflect bigger percentile changes than at the tails.


• The third explanation is statistical regression. The increase in scores by Group E would be due to their selection on the basis of extreme scores. On the posttest, their scores would go up because they would be approaching the population baseline.

• The fourth explanation centers on the interaction between history and selection.


• All of the threats mentioned for Figure 22.1(a) are also true for Figure 22.1(b). To determine if selection-maturation is playing a main role in the results, Cook and Campbell (1979) recommend two methods. The first method involves looking only at the data for the experimental group (Group E). If the within-group variance for the posttest is considerably greater than the within-group variance of the pretest, then there is evidence of a selection-maturation explanation.


• The second method is to develop two plots and the regression line associated with each plot. Figure 22.2

• If the regression line slopes for each plot differ from each other, then there is evidence of a differential average growth rate, meaning that there is the likelihood of a selection-maturation interaction.


• The outcome shown in Figure 22.1(c) is more commonly found in clinical psychology studies. The treatment is intended to lead to a decline of an undesired behavior.

• This outcome is also susceptible to selection-maturation interaction and three others.


• The fourth outcome is shown in Figure 22.1(d). The selection-maturation threat can be ruled out since this effect usually results in a slower growth rate for low scores and a faster growth rate for high scores.

• The final outcome is shown in Figure 22.1(e). The four threats can be ruled out. Hence, the outcome in Figure 22.1(e) seems to be the strongest one and should enable the researcher to make a causal statement concerning treatment.

Research Examples

• Nelson, Hall, and Walsh-Bowers (1997): Nonequivalent Control Group Design.

• They were unable to assign participants to different housing settings randomly.

• They state that the difference they found between these three groups on posttest measures could have been due to the selection problem, and not the type of care facility.

Research Examples

• Chapman and McCauley (1993): Quasi-Experiment

• Although one can perhaps think of this study as a nonexperimental one, Chapman and McCauley felt that it came under the classification of quasi-experimental.

• Awards were given to approximately half of a homogeneous group of applicants in a procedure that Chapman and McCauley say approximates random assignment to either fellowship or honorable mention.

Time Designs

• Design 22.2: A longitudinal Time Design (a.k.a. Interrupted Time Series Design)

• The reactive effect should show itself by comparing Y3 to Y4; this can be contrasted with Y5. If there is an increase at Y5 over and above the increase at Y4 from Y3, it can be attributed to X. A similar argument applies for maturation and history.

Time Designs

• One difficulty with longitudinal or time studies, especially with children, is the growth or learning that occurs naturally over time: Children do not stop growing and learning for research convenience. The longer the time period, the greater the problem. In other words, time itself is a variable.

Time Designs• The most widely used statistical test is ARIMA

(autoregressive, integrated, moving average) developed by Box and Jenkins (1970). The use of such a statistical analysis requires the availability of many data points.

• The usual tests of significance applied to time measures can yield spurious results. One reason is that such data tend to be highly variable, and it is as easy to misinterpret changes not due to X as due to X.

Multiple Time Series Design

• Design 22.3• This design has the advantage of

eliminating the history effect by including a control group comprised of an equivalent—or at least comparable—group of participants who do not receives the treatment condition. Consequently, the design offers a greater degree of control over sources of alternative explanations or rival hypotheses.

Single Subject Experimental Designs

• The majority of today’s behavioral research involves using groups of participants. However, there are other approaches.

• The single-subject designs are sometimes referred to as the N=1 design. They are an extension of the interrupted time series design. Where the interrupted time series generally looks at a group of individuals over time.


• Common characteristics:

• Only one or a few participants are used in the study.

• Each subject participants in a number of trials (repeated measures).

• Randomization procedures are hardly ever used.


• These design observe the organism’s behavior before the experimental treatment and use the observations as a baseline measure. Observations taken after the treatment are then compared to the baseline observations. The participant serves as his or her own control.


• Behavioral scientists doing research before the development of modern statistics attempted to solve the problem of reliability and validity by making extensive observations and frequent replication of results. This is a traditional procedure used by researchers doing single-subject experiments.

• The assumption is that individual participants are essentially equivalent and that one should study additional participants only to make certain that the original subject was within the norm.


• The single-subject approach assumes that the variance in the subject’s behavior is dictated by the situation. As a result, this variance can be removed through careful experimental control.

• The group difference research attitude assumes that the bulk of the variability is inherent and can be controlled and analyzed statistically.

Some Advantages of Doing Single-Subject Studies

• In Figure 22.3, if group-oriented research is employed, two groups have the same means and measures of variability. But visual inspection for the data shows a trend pattern vs. a random pattern. The single-subject approach does not have this problem, because a participant is studied extensively over time. The cumulative record for that participant shows the actual performance of the participant.


• Statistical significance and practical significance are two different things. The experiment may have little practical significance even if it has plenty of statistical significance.

• Simon (1987) advocates using well-constructed designs with the number of participants necessary to find the strongest effects. Single-subject researcher, on the other hand, favor increasing the size of the effect rather than attempting to lower error variance. They feel that this can be done through tighter control over the experiment.


• With single-subject studies, the researcher can avoid some of the ethical problems that face group-oriented researchers. One such ethical problem concerns the control group, which does not receive any real treatment.

• If there are not enough participants of a certain characteristic available for study, the researcher can consider single-subject designs instead of abandoning the study.

Some Disadvantages of Doing Single-Subject Studies

• One of the more general problems with the single-subject paradigm is external validity. Some find it difficult to believe that the findings from one study using one subject can be generalized to an entire population.

• With repeated trials on one participant, one can question whether the treatment would be equally effective for a participant who has not experienced previous treatments.

Some Disadvantages of Doing Single-Subject Studies

• Single-subject studies are perhaps even more sensitive to aberrations on the part of the experimenter and participant. These studies are effective only if the researcher can avoid biases and the participant is motivated and cooperative.

• A researcher doing single-subject research could be affected more so than the group-oriented researcher and needs to develop a system of checks and balances to avoid this pitfall.

Some Single-Subject Research Paradigms

• The Stable Baseline: An Important Goal• The behavior before the treatment

intervention must be measured over a long enough time period so that a stable baseline can be obtained. This baseline, or operant level, is important because it is compared to later behavior.

• If the baseline varies considerably, it could be more difficult to assess any reliable change in behavior following intervention.

Designs that Use the Withdrawal of Treatment

• The ABA Design• The ABA design involves three major steps.

The first step is to establish a stable baseline (A). The experimental intervention is applied to the participant in the second step (B). If the treatment is effective, there will be a response difference from the baseline. In order to determine if the treatment intervention caused the change in behavior, the researcher exercises step three: a return to baseline (A).


• There are also some ethical concerns about reverting the organism back to the original state if that state was an undesirable behavior. Experiments in behavior modification seldom return the participant back to baseline. To benefit the participant, the treatment is reintroduced. The ABAB design does this.


• Repeating Treatments (ABAB Designs)• There are two versions of the ABAB

design. The first was briefly described in the above section. Repeating the treatment also provides the experimenter with additional information about the strength of the treatment intervention.

• The ABAB design essentially produces the experimental effect twice.


• The second variation of the ABAB design is called the alternating treatments design. In this variation there is no baseline taken. The A and B in this design are two different treatments that are alternated at random. The goal of this design is to evaluate the relative effectiveness of the two treatment interventions.

• The advantage this design has over the first ABAB design is that there is no baseline to be taken, and the participant is not subjected to withdrawal procedures. Since this method involves comparing two sets of series of data, some have called it the between-series design.


• There are some other interesting variations of the ABAB design where withdrawal of the treatment is not done. McGuigan (1996) calls it the ABCB design where in the third phase, the organism is given a “placebo” condition. This placebo condition is essentially a different method.

A Research Example

• Powell and Nelson (1997): Example of an ABAB Design

• The treatment intervention was letting Evan choose the class assignment he wanted to work on. There are two conditions: choice and no-choice. Baseline data were collected during the no-choice phase.

Using Multiple Baselines

• There is a form of single-subject research that uses more than one baseline. Several different baselines are established before treatment is given to the participant. These types of studies are called multiple baseline studies.

• There are three classes of multiple baseline research designs: across behaviors, across participants, and across environments.


• There is a common pattern for implementing all three classes of this design. That pattern is shown in Figure 22.4.


• With the multiple baselines across behaviors, the treatment intervention for each different behavior is introduced at different times. If one of the behavior changes, while the other behaviors remain constant or stable at the baseline, the researcher could state that the treatment was effective for specific behavior.

• After a certain period of time has passed, the same treatment is applied to the second undesirable behavior (Baseline 2).


• An important consideration with this particular class of multiple baseline design is that one assumes the responses for each behavior are independent of the responses for other behaviors.

• The intervention can be considered effective if this independence exists. If the responses are in some way correlated, then the interpretation of the results becomes more difficult.


• In the multiple baseline design across participants, the same treatment is applied in series to the same behavior of different individuals in the same environment.

• In the multiple baseline design across environments, the same treatment is given to different participants who are in different environment.

chapter 22 quasi-experimental and n=1 designs of research

Documents

research designs

designs of research

compromise designs

quasi experimental designs

repeated treatment designs

quasi experimental designs

treatment group

experimental group