Scientific Method and Experimental Design

A student discovers that a sewage treatment plant will be built soon in his hometown and its wastes will be dumped in a stream near his favorite fishing hole. At school the student learns that pollution lowers populations of certain stream insects (mayflies, stoneflies and caddisflies) and that scientists often sample these insects to determine stream health.

mayfly

stonefly

caddisfly

Principles of Biology II, Dr. David Dussourd, Fall 2009

After the sewage treatment plant is built, the student samples populations of these insects and discovers that they are indeed rare. He concludes that the treatment plant is harming the stream and files a complaint. Is the student correct with his conclusion?

Answer - We don’t know because the student has no control. Maybe this stream always had low levels of mayflies, stoneflies and caddisflies.

Don’t forget the control!

But what control is needed?

Another student in the same town measured levels of mayflies, stoneflies and caddisflies the year before the sewage treatment plant was built and the year after. She found that levels were much lower the year after the plant was built.

She concludes that the treatment plant is harming the stream and files a complaint. Is the student correct?

Answer - The student now has a control (samples from the year before) so her experimental design is greatly improved. However, it is possible that the year after the plant was built, insect populations dropped for some other reason. Maybe there was a chemical spill upstream or a drought.

Can you think of a better experimental design?

A third student sampled aquatic insects upstream and downstream of the sewage plant before it was built (year 1) and at the same sites the year after it was built (year 3). The student found that upstream and downstream sites had similar insect populations in year 1, but the downstream site had much lower levels than the upstream site in year 3.

The student concludes that the treatment plant is harming the stream and files a complaint. Is the student correct?

The student now verifies that the downstream and upstream sites initially have similar insect numbers, then documents that the downstream site has lower numbers after the plant is built .

This is strong evidence. When the student reports her data, government agencies and plant managers will listen!

The better the experimental design, the stronger the conclusion.

A student has difficulty sleeping in his loud dorm. He decides to take melatonin before going to bed; melatonin is a hormone that regulates the sleep/wake cycle. He also buys a more comfortable bed and ear plugs. Sure enough, he sleeps much better.

He concludes that melatonin helped him sleep better.

Does it?

Only test 1 variable at a time. In other words their should only be 1 difference between your control group and your experimental group!

You want to figure out how to decrease muscle soreness quicker after your workouts. You think icing your muscles may help, so after a strenuous arm workout you ice one arm and leave the other arm as a control. The next day the arm that you iced is less sore than the arm that you did not ice. You conclude that icing your arm decreases muscle soreness quicker. Is your conclusion scientifically valid?

It is possible that the arm that you iced would have been less sore than the other arm even if you didn’t ice it. A rigorous scientific approach requires that we replicate this experiment with more people.

Replicate or repeat the experiment to show that the results are consistent!

You rethink your experiment and decide to enlist 100 friends. Where should they apply ice?

a. 50 should put ice on both arms and 50 should not put ice on either arm

b. All 100 should put ice on the same randomly chosen arm

c. Randomly chosen arm for each person

d. All 100 should put ice on both arms

If the side receiving ice is less sore the next day in most of the students, then we can conclude that the ice helps.

(Biologists use statistics to determine if the number of students that improved with the ice is so large that a change this substantial is unlikely to have happened by chance)

Problems with the other answers:a. multiple variablesb. Random assignment is good but even more

randomization is possible with c.c. Correct answerd. No control. How will we know if the ice made a

difference?

Randomly assign subjects to eliminate conscious or subconscious bias!

A student interested in the health consequences of cell phone use sends a survey to 10,000 adults in Springdale. He discovers that adults that use cell phones the most have the lowest levels of cancer. Adults that rarely or never use cell phones have high rates of cancer. The student concludes that cell phone use reduces the incidence of cancer. Is the student correct?

Answer: Most older people use cell phones less than do younger people and older folks suffer from cancer more often. Thus, cell phone use and cancer rates are both affected by age. It is highly unlikely that cell phone use decreases cancer rates.

A correlation between two variables (cell phone use and cancer) does not necessarilyindicate that one variable causes the other to change.

(Correlation does not mean causation)

How could the student test if cell phone use reduces cancer?

Answer: One could randomly assign adults of the same age to use or not use cell phones for a long time (10 years?)

Monarch butterflies migrate each spring from Mexico to the U.S., then back to Mexico in the fall. How do they find their way over this immense journey? In a recent paper, scientists altered the magnetic field around monarchs in the lab and found that they changed their flight direction and that they maintained the correct orientation relative to the magnetic field.

Did the scientists prove that monarchs can not only detect the magnetic field (resulting in the change in flight direction), but can also use the magnetic field to orient (because of the appropriate change in direction)?

Scientists rarely use the term “prove”. We acquire evidence that supports or refutes a hypothesis.

OK. Then did the scientists show convincingly that monarchs can detect the magnetic field and use the magnetic field to orient?

Would you change your mind if you knew that the scientists were unable to obtain the same answer when they repeated the experiment?

In this case, the scientists had to retract their published paper. It turned out that the butterflies were not orienting to the magnetic field, but to the light colored t-shirt worn by the technician, who caught each butterfly after it was tested. The technician knew which direction the butterflies should fly if they were using the magnetic field. He stood there waiting to catch them and sure enough they came straight to him – oops!

Scientists don’t prove something is true. They perform experiments that support or refute their hypotheses.

Scientific papers should describe methods in sufficient detail that others can replicate the experiment. If the same results are not obtained, then the conclusions from the first experiment are suspect.

A student decides to test if watching TV is detrimental to health. He moves his pet rat next to the TV and leaves the TV on 12 hours a day. Two days later the rat dies. The student concludes that watching TV is harmful for human health. Is the student’s conclusion justified? Describe 2 flaws of the experimental design.

The student’s conclusion is not justified due to the following flaws:

1.No control (rats not exposed to TV)2.No replication (e.g. 20 rats that watch TV and 20 rats that do not)3.Inappropriate extrapolation to humans (was the rat actually “watching” the TV?)

A well-designed experiment in the sciences should have:

a. One or more control groupsb. Only one variable altered at a timec. Large sample sized. Replicatione. Subjects randomly assigned to treatment

And don’t forget that correlation does not equal causation.And that we don’t prove anything true in science. We only support or refute our hypotheses.