experiments © pine forge press, an imprint of sage publications, 2004

EXPERIMENTS

© Pine Forge Press, an imprint of Sage Publications, 2004

Different Types ofExperimental Design

true experimentsquasi-experimentsevaluation researchnonexperimental designs


Example of non-experimental design [before-after/pretest-posttest]

The Imaginary Seattle Bike Patrol Study

Pretest Posttest

O X O

Look at change in crime ratesSubtract pretest score from posttest

O X O

Could have been;

Change in Economy

Change in weather

Change in media coverage

etc.

Instead of X, or combined with X

“Confounding”

Combining a second variable with the

independent variable is Confounding

External events

Such additional variables in the research situation which provide an alternative explanation to the one that X is changing Y, in this case are classified as “external events” because they are occurring outside the study at the same time as the independent variable is occurring.

HISTORY

Eternal eventsHISTORY - ENVIRONMENTAL CHANGES …all events that occurred during the time of the study that might affect the individuals studied and provide a rival explanation for the change in the dependent variable…all events between the pretest [if given] and the posttest…Changes in the economyChanges in the weatherChanges in information provided by the mediaWarNatural disasterEtc.

Exogenous events

Note: “history” variables are also known as “exogenous” events. This means that the events are “outside” the study.

How can we ensure that the change was due to X and not one or more of the external events?... “History”

Classical experimental design:

Pretest Bike patrolPosttest

RA

• Pretest No bike patrol Posttest

We have randomly selected a bunch of cities for the E group and the C group…then…

We compare changes in each group.

Simultaneous variation

Pretest Bike patrol PosttestChange in economyChange in weatherMedia coverage

Etc.

RA

Pretest no bike patrol Posttestchange in EconomyChange in weatherMedia coverageEtc

Simultaneous variation

Simultaneous variation allows for equal changes in all other variables in the experimental group and control group.

Since randomization would equalize [out] the effects in both groups, the only reason for different results in the experimental and control group should be due to X.

S.V. allows for changes to occur in each group yet it “CONTROLS” for effects of other variables.

N.B.

Be able to explain how simultaneous variation and random assignment do their work to “control.”

Review

History is controlled by random assignment and allowing the History Variables to make changes in Y equally in both the experimental group and the control group. Hence the only difference in changes in the two groups can be attributed to X.

If this doesn’t make immediate sense ask now, work on it, and get help!

True experiments must have at least one experimental group (subjects who receive some treatment) and at least one comparison group (subjects to whom the experimental group can be compared).

True Experiments

01

01


True Experiments

True experiments must have at least three things:•Two comparison groups (in the simplest case, an experimental and a control group)

• Variation in the independent variable before assessment of change in the dependent variable

•Random assignment to the two (or more) comparison groups


True experiments must have at least one experimental group (subjects who receive some treatment) and at least one comparison group (subjects to whom the experimental group can be compared).

True Experiments

01

01


All true experiments have a posttest—that is, measurement of the outcome in both groups after the experimental group has received the treatment. Many true experiments also have pretests that measure the dependent variable prior to the experimental intervention. A pretest is exactly the same as a posttest, just administered at a different time.

True Experiments

01

01


INTERNAL VALIDITYThe issue of X being responsible for the changes to Y is an issue of INTERNAL VALIDITY.

Since we ruled out all the outside factors, history factors, external events which could alternatively explain why the bike patrols worked, we may tentatively assume that the study indicates internal validity.

Why tentatively? RA & INTERACTION

Tentativeness

RA…random assignment doesn’t always truly equalize the E group and the C group.

Interaction allows for the possibility that bike patrols work sometimes and not others and we may not “uncover” a hidden relationship that work in different directions.

RA doesn’t always equalize E & C group in values of Y.

Or, Interaction may mask actual relationship:

Z1 O1 O2

Z2 O1 O2

X1 X2

Internal Validity

The 5 subtypes of internal validity examined here are:1. External [exogenous outside] events / History

2. Endogenous inside events / Maturation, Instrumentation, Testing, Regression

3. Selection bias / Selection Mortality

4. Treatment misidentification / Reactivity, Experimenter Bias, Demoralization, Compensation, Placebo effect

5. Contamination / Experimental Diffusion, Contamination

Causal (Internal) Validity

When characteristics of the experimental and comparison group subjects differ SELECTION BIASWhen the subjects develop or change during the experiment as part of an ongoing process independent of the experimental treatment ENDOGENOUS EVENTS.When something occurs during the experiment, other than the treatment, which influences outcome scores HISTORY.When either the experimental group or the comparison group is aware of the other group and is influenced in the posttest as a result (Mohr, 1992)CONTAMINATION.

There are four basic sources of noncomparability

(other than the treatment) between a comparison group and an experimental group. They produce four of the five sources of internal invalidity:

Causal (Internal) Validity

The fifth source of internal invalidity can be termed TREATMENT MISIDENTIFICATION: Variation in the independent variable (the treatment) is associated with variation in the observed outcome, but the change occurs through a process that the researcher has not identified.


Review

We have done history and found that it is controlled by

E

RA

C

SIMULTANEOUS VARIATION

Endogenous Change

The next set of variables which may interfere with internal validity are exogenous variables. “When the subjects develop or change during the experiment as part of an ongoing process independent of the experimental treatment.” These include: Maturation, Instrumentation, Testing, & Regression artifact.

Maturation

The bike patrol period was the beginning of a baby bustlet a small baby bust for folks growing into their teen age years. Fewer teenagers…probably fewer crimes. Folks grow out of lots of crimes when they get older, marry, get decent jobs etc.

Maturation

MATURATION – GROWING OLDER, WISER, TIREDER, DEVELOPMENTAL CHANGES…biological, psychological, or social processes that produce changes in the individuals or units studied with the passage of time that are not produced by the independent variable(s) under study in the experiment.Changes in the age distribution of the populationChanges in the interest in participating in the dependent variable [getting tired of it]

How is it controlled?

E O X ORA Bike patrol

Age changes

C O O Age changes


INSTRUMENTATION

Police could have learned to be more vigilant…thus may have caught more crooks.

Alternatively, police may have grown tired and thus been less vigilant.

INSTRUMENTATION

INSTRUMENTATION – INSTRUMENT “DECAY” OR IMPROVEMENT…changes in the measuring instrument between the pretest and posttest.

Note changing springs on a scale…

Note changing experience of interviewers…

Note changing ability of coders…

How to control?

O X O Bike patrol

Police Reporting changes

RA

O O Police Reporting changes


TESTING

The study of crime may have made the potential victims more careful or may have made the criminals more careful and thus less likely to be caught.

TESTING

TESTING – PRACTICE OR AWARENESS CHANGES THE SUBJECT…the possible reactivity of measurement …testing itself may change the phenomenon being measured…through awareness & reflection, through experience & practice…

How to control?

E X & changes in testing

RA

C changes in testing


REGRESSION ATRIFACT

Crime scores were unreliably measured at the top & the only way they could go was down as they varied on second test.

Example using unreliable scale“Extreme” scores can only go down.

300 [297]

300 [400]

300 [295]

300 [298]

300 [296]

____

1500

1500/5=300

292

300

293

294

300

____

1479

1479/5=295.8

300-295.8=4.2

REGRESSION ATRIFACT

REGRESSION ATRIFACT – UNRELIABLE MEASURES OF GROPUS WITH EXTREME SCORES; THERE’S ONLY ONE WAY TO GO…if only extreme cases are selected, and if the measurement instrument is unreliable, when remeasuring the only way unreliably low scores can go is up and …high scores…down

How to control for “regression artifact”

RA and simultaneous variation

Sam Ting w.r.t. changes due to unreliable measures with only one way to go will happen in the E group and the C group

Review

All endogenous changes involve changes within the experiment other than X. They are controlled in the same way. SAM TING!They are controlled by SIMULTANEOUS VARIATION – changing equally in the E group and the C group. RA should place the same kinds of cases in both groups so that change [aside from X] is “equalized/”Hence SIMULTANEOUS VARIATION. O X O Other stuffRA

O Other stuff O

Selection Bias

e.g.1 Seattle may have already had a “thing” about crime such that the community was going to reduce it and the police on bikes was just a temporal coincidence.

Seattle was already hell bent on changing crime and so the selection of bikes coincided with an already established tendency.

So Seattle has “selected” to make the change apart from the bike patrol.

e.g.2 the film may not be the cause of change when a survey is taken on film effects. Filmgoers may have been more liberal before they saw the film.

Confounding by selection

No X = not going to film

Non liberal folks avoid film

It isn’t film exposure .

Film exposure doesn’t change people.

X = film

Liberal folks go to such a film

It’s the initial liberalism of the audience

How to control for selection

See above

Sam Ting

RA will place the same kinds of cases in the E group and the C group. Folks won’t select themselves.

INTERACTIONS WITH SELECTION – SUBJECT SELECTION - FOLKS CHOOSE WHICH GROUP THEY’RE IN SO GROUPS AREN’T REALLY EQUAL subject may be selected with a predilection to change or may appear to change because of how they have selected themselves…

EXPERIMENTAL MORTALITY

Perhaps all the crooks were jailed …thus there was a smaller population of criminals committing crime at time 2.

EXPERIMENTAL MORTALITY - “DYING” / DIFERENTIAL ATTRITION / DROPPING OUT OF THE EXPERIMENTAL OR CONTROL GROUP…dropout problems that prevent the researcher from obtaining information on all cases.

How to control for mortality?

Same as above…assuming that the experimental treatment or the control treatment doesn't “kill off” subjects at a different rate.

Treatment misidentification

some process of which the researcher may not be aware is responsible for the apparent effect of treatment


N.B. Controlling for these sources of internal invalidity involves procedures different from employing a “true” experimental design.

REACTIVITY

Realizing that the bike “experiment” was in place, criminals may have been on guard. Potential victims may have protected each other.

Reactivity

REACTIVITY – KNOWLEDGE ONE IS IN AN EXPERIMENT CHANGES HOW ONE ACTS [“REACTS”]…sometimes inappropriately known as the “Hawthorne effect” …subjects sometimes become aware that they are in an experiment and, because of this awareness, “react” differently…Subjects may “feel special” and work harder. Alternatively they may suffer from an “audience effect.”

How to control?

Have both the E group and the C group think they’re the “experimental” group.

E.g. they’re “blind” to what group they’re in.

Experimenter bias

Maybe there was not less crime but the cops liked the bikes so they cheated.

Maybe the cops misperceived.

EXPERIMENTER BIAS – INADVERTANTLY GUIDING SUBJECTS TOWARDS RESULTS TO “CONFIRM” THE HYPOTHESIS ; OUTRIGHT CHEATING

How to control?

The experimenter must be “blind” to what condition the subject is in…during initial instructions…conducting the experiment and coding / analyzing the data.

Demoralization

DEMORALIZATION – SUBJECTS IN THE CONTROL GROUP GIVING UP

Note: not relevant in the Seattle study, because no control group

How to control?

Have subjects “blind” to which group they’re in

EXPECTANCIES OF THE EXPERIMENTAL STAFF/ COMPENSATION

EXPECTANCIES OF THE EXPERIMENTAL STAFF/ COMPENSATION – STAFF PROVIDING “EXTRAS” TO THE CONTROL GROUP

How to control?

Have staff “blind to which group subjects are in.

When not possible train and monitor the staff.

PLACEBO EFFECT

Anticipating cop’s bike patrols’ effectiveness, criminals did less crime.

PLACEBO EFFECT

Placebo effect: Subjects change because of expectations of change, not because of treatment itself

How to control

o bike patrol o

RA

o faux bike patrol? o

Sugar-pill analogyS’s [and experimenters] must be BLIND w.r.t. nature of treatment. Both must see the subject as in the experimental group.

Random Assignment is insufficient here. For all these issues, either the subjects or

the experimenters must be “blind.”

You may have heard of medical experiments wherein the procedures are “double blind.” This is to ensure that neither knowledge of the experimenter or the subject will alter the results.

Review: Treatment Misidentification

Contamination

Think of stuff from the experimental group analogous to leeching through the soil to the control group or vice versa.

EXPERIMENTAL DIFFUSION

EXPERIMENTAL DIFFUSION – SUBJECTS ON ONE EXPERIMENTAL CONDITION INFLUENCING ANOTHER…by sharing information.

How to control?

Isolate…sepArate groups.

CONTAMINATION

CONTAMINATION – COMPARISON [CONTROL] GROUP IS AFFECTED BY OR AFFECTS THE EXPERIMENTAL GROUPCompensatory rivalry or “John Henry effect.” …or demoralization…is different from # 13 in that in #13 information is passed on whereas in 14 only the knowledge of what group one is in is passed on such that group members behave differently. Wienir sees it as analogous to # 8 and doesn’t feel it is the same sort of contamination as in # 13.

How to control?

Separate groups and keep members form communication about which group they’re in.

Comparison of two contamination situations:

Diffusion: sharing content form experiment.

Contamination: One set of subjects letting the other know who the “experimental subjects” are.

Review

Type of invalidity:

History

Endogenous variables

Selection bias


Contamination

How to control:RA & simultaneous variation

RA & simultaneous variation

RA & simultaneous variation

Blind

Separate

N.B.

Be able to explain fully how controlling works in the situations in the slide above!

Richard Price, Michelle Van Ryn, and Amiram Vinokur (1992) hypothesized that a job-search

program to help newly unemployed persons could reduce the risk of depression among this group.

True Experiments

An Example:

The researchers tested this hypothesis with a sample of unem-ployed persons who volunteered

for job-search help at Michigan Employment Security Commission offices.


True ExperimentsThe unemployed volunteers were randomly assigned either to participate in eight 3-hour group seminars over a 2-week period (the treatment) or to receive

self-help information in the mail on how to conduct a job search (the comparison condition).

Those in the seminars were more likely to obtain jobs, which would naturally decrease their risk of

depression.© Pine Forge Press, an imprint of Sage Publications, 2004

Quasi-experimentsA quasi-experimental design is one in which the

comparison group is predetermined to be comparable to the treatment group in critical ways,

such as being eligible for the same services or being in the same school cohort (Rossi & Freeman,

1989:313).•Nonequivalent control group designs have experimental and comparison groups that are designated before the treatment occurs and are not created by random assignment.

•Before-and-after designs have a pretest and posttest but no comparison group. In other words, the subjects exposed to the treatment serve, at an earlier time, as their own controls.© Pine Forge Press, an imprint of Sage Publications, 2004

Quasi-experiments

Ruth Wageman (1995) used a quasi-experimental design to investigate how the way tasks were designed and rewards allotted affected work team functioning.

Her research question was whether it was preferable to organize work tasks and work rewards in a way that stressed team interdependence or individual autonomy.


Quasi-experiments

David P. Phillips’s (1982) study of the effect of TV soap-opera suicides on the number of actual suicides in the United States illustrates a more powerful multiple group before-and-after design.


Nonexperimental DesignsThe ex post facto control group design has experimental and comparison groups that are not created by random assignment. Unlike the groups in nonequivalent control group designs, the groups in ex post facto (after the fact) designs are designated after the treatment has occurred.


Other Nonexperimental Designs

Cross-sectional designs, termed “one-shot case studies” in the experimental design literature, are easily able to establish whether an association exists between two variables, but we cannot be anywhere near as confident in their conclusions about appropriate time order or nonspuriousness as with true experiments or even quasi-experiments.Longitudinal designs improve greatly our ability to test the time order of effects, but they are unable to rule out all extraneous influences.


External ValidityResearchers are often interested in determining

whether treatment effects identified in an experiment hold true for subgroups of subjects

and across different populations, times, or settings.

There is always an implicit tradeoff in experimental design between maximizing

causal validity and generalizability. The more that assignment to treatments is randomized

and all experimental conditions are controlled, the less likely it is that the research subjects and set-ting will be

representative of the larger population.© Pine Forge Press, an imprint of Sage Publications, 2004

Although it may be possible to test a hypothesis with an experiment, it may not always bedesirable to do so.

The intersectingcomplexity of societies, social

relationships, and social beings—of people and the groups to

which they belong—is so great that it often defies reduction to the simplicity of a

laboratory orrestriction to the requirements of

experimental design.© Pine Forge Press, an imprint of Sage Publications, 2004

True ExperimentsRandomization, or random assignment, is what makes the comparison group in a true experiment such a powerful tool for identifying the effects of the treatment. A randomized comparison group can provide a good estimate of the counterfactual—the outcome that would have occurred if the subjects who were exposed to the treatment actually had not been exposed but otherwise had had the same experiences


experiments © pine forge press, an imprint of sage publications, 2004

Documents