Experimental Design

All experiments have independent variables, dependent variables, and experimental units.

Independent variable. An independent variable (also called a factor) is an explanatory variable manipulated by the experimenter.

Each factor has two or more levels, (IE, different values of the factor). Combinations of factor levels are called treatments. The table below shows independent variables, factors, levels, and treatments for a hypothetical experiment.

Vitamin C

0 mg 250 mg 500 mg

Vitamin E0 mg Treatment 1 Treatment 2 Treatment 3

400 mg Treatment 4 Treatment 5 Treatment 6

In this hypothetical experiment, the researcher is studying the possible effects of Vitamin C and Vitamin E on health. There are two factors - dosage of Vitamin C and dosage of Vitamin E. The Vitamin C factor has three levels - 0 mg per day, 250 mg per day, and 500 mg per day. The Vitamin E factor has 2 levels - 0 mg per day and 400 mg per day. The experiment has six treatments. Treatment 1 is 0 mg of E and 0 mg of C, Treatment 2 is 0 mg of E and 250 mg of C, and so on.

Vitamin C

0 mg 250 mg 500 mg

Vitamin E0 mg Treatment 1 Treatment 2 Treatment 3

400 mg Treatment 4 Treatment 5 Treatment 6

Dependent variable. In the hypothetical experiment above, the researcher is looking at the effect of vitamins on health. The dependent variable in this experiment would be some measure of health (annual doctor bills, number of colds caught in a year, number of days hospitalized, etc.).

Experimental units. The recipients of experimental treatments are called experimental units or subjects. The experimental units in an experiment could be anything - people, plants, animals, or even inanimate objects.

In the hypothetical experiment above, the experimental units would probably be people (or lab animals). But in an experiment to measure the tensile strength of string, the experimental units might be pieces of string.

Vitamin C

0 mg 250 mg 500 mg

Vitamin E0 mg Treatment 1 Treatment 2 Treatment 3

400 mg Treatment 4 Treatment 5 Treatment 6

Characteristics of a Well-Designed Experiment

A well-designed experiment includes design features that allow researchers to eliminate extraneous variables as an explanation for the observed relationship between the independent variable(s) and the dependent variable. The three basic principles of experimental design are Control, Randomization, and Replication

Control. Control refers to steps taken to reduce the effects of extraneous variables (i.e., variables other than the independent variable and the dependent variable). These extraneous variables are called lurking variables.

Control involves making the experiment as similar as possible for subjects in each treatment condition. Three control strategies are control groups, placebos, and blinding.

Control group. A control group is a baseline group that receives no treatment or a neutral treatment. To assess treatment effects, the experimenter compares results in the treatment group to results in the control group.

Placebo. Often, subjects respond differently after they receive a treatment, even if the treatment is neutral. A neutral treatment that has no "real" effect on the dependent variable is called a placebo, and a subject's positive response to a placebo is called the placebo effect.

To control for the placebo effect, researchers often administer a neutral treatment (i.e., a placebo) to the control group. The classic example is using a sugar pill in drug research. The drug is effective only if subjects who receive the drug have better outcomes than subjects who receive the sugar pill.

Blinding. Of course, if subjects in the control group know that they are receiving a placebo, the placebo effect will be reduced or eliminated; and the placebo will not serve its intended control purpose.

Blinding is the practice of not telling subjects whether they are receiving a placebo. In this way, subjects in the control and treatment groups experience the placebo effect equally. Often, knowledge of which groups receive placebos is also kept from people who administer or evaluate the experiment. This practice is called double blinding. It prevents the experimenter from "spilling the beans" to subjects through subtle cues; and it assures that the analyst's evaluation is not tainted by awareness of actual treatment conditions.

Randomization. Randomization refers to the practice of using chance methods (random number tables, flipping a coin, etc.) to assign subjects to treatments. In this way, the potential effects of lurking variables are distributed at chance levels (hopefully roughly evenly) across treatment conditions.

Replication. Replication refers to the practice of assigning each treatment to many experimental subjects. In general, the more subjects in each treatment condition, the lower the variability of the dependent measures.

Confounding

Confounding occurs when the experimental controls do not allow the experimenter to reasonably eliminate plausible alternative explanations for an observed relationship between independent and dependent variables.

Consider this example. A drug manufacturer tests a new cold medicine with 200 volunteer subjects - 100 men and 100 women. The men receive the drug, and the women do not. At the end of the test period, the men report fewer colds.

This experiment implements no controls at all! As a result, many variables are confounded, and it is impossible to say whether the drug was effective. For example, gender is confounded with drug use. Perhaps, men are less vulnerable to the particular cold virus circulating during the experiment, and the new medicine had no effect at all. Or perhaps the men experienced a placebo effect.

This experiment could be strengthened with a few controls. Women and men could be randomly assigned to treatments. One treatment could receive a placebo, with blinding. Then, if the treatment group (i.e., the group getting the medicine) had sufficiently fewer colds than the control group, it would be reasonable to conclude that the medicine was effective in preventing colds.

The term experimental design refers to a plan for assigning subjects to treatment conditions. A good experimental design serves three purposes.

Causation. It allows the experimenter to make causal inferences about the relationship between independent variables and a dependent variable.

Control. It allows the experimenter to rule out alternative explanations due to the confounding effects of extraneous variables (i.e., variables other than the independent variables).

Variability. It reduces variability within treatment conditions, which makes it easier to detect differences in treatment outcomes.

An Experimental Design Example

Consider the following hypothetical experiment. Acme Medicine is conducting an experiment to test a new vaccine, developed to immunize people against the common cold. To test the vaccine, Acme has 1000 volunteer subjects - 500 men and 500 women. The subjects range in age from 21 to 70.

In this lesson, we describe three experimental designs - a completely randomized design, a randomized block design, and a matched pairs design. And we show how each design might be applied by Acme Medicine to understand the effect of the vaccine, while ruling out confounding effects of other factors.

The completely randomized design is probably the simplest experimental design, in terms of data analysis and convenience. With this design, subjects are randomly assigned to treatments.

A completely randomized design layout for the Acme Experiment is shown in the table to the right. In this design, the experimenter randomly assigned subjects to one of two treatment conditions. They received a placebo or they received the vaccine. The same number of subjects (500) were assigned to each treatment condition (although this is not required). The dependent variable is the number of colds reported in each treatment condition. If the vaccine is effective, subjects in the "vaccine" condition should report significantly fewer colds than subjects in the "placebo" condition.

A completely randomized design relies on randomization to control for the effects of extraneous variables. The experimenter assumes that, on average, extraneous factors will affect treatment conditions equally; so any significant differences between conditions can fairly be attributed to the independent variable.

Treatment

Placebo Vaccine

500 500

Randomized Block Design

Gender

Treatment

Placebo Vaccine

Male 250 250

Female 250 250

With a randomized block design, the experimenter divides subjects into subgroups called blocks, such that the variability within blocks is less than the variability between blocks. Then, subjects within each block are randomly assigned to treatment conditions. Because this design reduces variability and potential confounding, it produces a better estimate of treatment effects.

It is known that men and women are physiologically different and react differently to medication. This design ensures that each treatment condition has an equal proportion of men and women. As a result, differences between treatment conditions cannot be attributed to gender. This randomized block design removes gender as a potential source of variability and as a potential confounding variable.

In this Acme example, the randomized block design is an improvement over the completely randomized design. Both designs use randomization to implicitly guard against confounding. But only the randomized block design explicitly controls for gender.

Note 1: In some blocking designs, individual subjects may receive multiple treatments. This is called using the subject as his own control. Using the subject as his own control is desirable in some experiments (e.g., research on learning or fatigue). But it can also be a problem (e.g., medical studies where the medicine used in one treatment might interact with the medicine used in another treatment).

Note 2: Blocks perform a similar function in experimental design as strata perform in sampling. Both divide observations into subgroups. However, they are not the same. Blocking is associated with experimental design, and stratification is associated with survey sampling.

Matched Pairs Design

Pair

Treatment

Placebo Vaccine

1 1 1

2 1 1

... ... ...

499 1 1

500 1 1

A matched pairs design is a special case of the randomized block design. It is used when the experiment has only two treatment conditions; and subjects can be grouped into pairs, based on some blocking variable. Then, within each pair, subjects are randomly assigned to different treatments.