Designing Scientific Designing Scientific ExperimentsExperiments

Dr. Gail P. Dr. Gail P. TaylorTaylor

MBRS-RISE CoordinatorMBRS-RISE Coordinator

UT San AntonioUT San Antonio

08/2006

References

• CRITICAL THINKING, THE SCIENTIFIC METHOD, AND PAGE 25* OF GILBERT Dany S. Adams, Department of Biology, Smith College, Northampton, MA 01063 http://www.sdbonline.org/SDBEduca/dany_adams/critical_thinking.html#blurb

• Validity: http://carbon.cudenver.edu/~lsherry/rem/validity.html

• At the Bench, A Laboratory Navigator; updated edition. Kathy Barker, Cold Spring Harbor Press, 2005

Scientific Method

• Observe phenomenon & conceive ideas

• Make predictions/develop a hypothesis

• Devise a test/formulate experiment

• Carry out experiments

• Draw conclusions from results

• Reject or support hypothesis

Types of Experiments

• Science does not generally deal with facts, but rather with evidence

• Each experiment weakens or strengthens a hypothesis

• All evidence is not equal

• Try to discern cause and effect!

Planning Experiments I

• What ideas have you come up with?

• Why is your idea important?

• Have other people tested this idea before?– http://www.pubmed.org

• What type of background information is available?

• Define question/Develop hypothesis (and null hypothesis)

Determining Causality

• Causality can be difficult to prove• Different experimental designs impact differently

• Correlative Evidence (weak evidence)Found together in time or space

• Loss of Function (stronger evidence)– Blocked a phenomenon

• Gain of Function (strongest evidence)– Initiation of event leads to second event (additional function)

Example – Protein X may be involved in Cellular Aggregation

• “Show it”• Correlative evidence (time and space):

– Antibody used to detect:• Found in particular microorganism when aggregating

(and not when free living)• Found in area where cells are contacting one another

during aggregation

– No causality; nothing beyond inference about function

• Clumping could cause the protein expression• Clumping and protein expression could be induced by

same causative agent• Could be completely coincidental

Example – Protein X may be Necessary for Cell Aggregation

• “Block it”• Loss of Function - What does its loss do to

clumping?– Antibody to protein used to block it from

functioning. – Or knock out gene– Clumping no longer takes place

• Need controls- – Clumping specifically and only being inhibited– cells not dying– May support “real” clumping agent to function

– Therefore it is necessary for clumping

Example – Protein X may be Sufficient for Cellular Aggregation

• “Move it”• Gain of Function

– In organism that does not normally clump…• Artificially introduce required protein• Or artificially turn it on at all times (constituitively

express)

– Aggregation now takes place

• Therefore is sufficient to induce clumping

Progression to Necessary and Sufficient

• Often you will see this progression through Biological scientific papers– What is it?– Yes, it’s there– Yes, it’s in the right place– It’s loss produces this response– It’s addition produces this response…

Planning Experiments II

• Consider statistical methodologies during planning stages

• Look in prior papers for ideas about statistics.• Statistical analysis will generally discern that

likelihood that a result occurred by chance• Consult mentor or statistician for confirmation

– Compare 1 treatment and control: t-test• Decide on p (Probability value) p < 0.05 or 0.01

– Compare many treatment groups: ANOVA– Many more…

Planning Experiments III

• Variables– Independent (manipulated)– Dependent (outcome)

• # of samples (minimum 2, 3 better)

• # repetitions (minimum 2x)

Internal Validity• Cause and Effect- Did the experimental treatment,

and only the experimental treatment, cause the effect!– Controls (Be Careful!!!)– Prevent additional variables from sneaking into your

experiment– Must control for:

• Selection: Anything that makes treatment and control groups different at beginning (random assignment)

• History: What different things may happen between expt. And control groups between initial treatment and measurement

• Maturation: Natural changes in subjects (aging)• Instrumentation: All tests/equipment/reagents must stay

same throughout experiment• Testing: “incoming” may “teach” the subject• Mortality: Subjects may leave or die (contamination)• Regression: If initial test scores were high, on average, will

naturally move towards mean

External Validity• The extent to which the findings of the study can be applied,

reproduced, or generalized to another setting or systems. i.e., techniques to ensure that these groups correspond to general population

• Unrepresentative Sample: Sample members not representative of general population.

• Clear Description of the Treatment or Protocol (replicability)

• Hawthorne Effect: Subjects know that they are being studied and it influences behavior

• Novelty Effect: Particularly in humans…enjoy experiment, then possibly don’t.

• Pretest Sensitization: If the pretest is part of the treatment, it will obviously affect the results or findings.

• History and Treatment Interaction: something else happened which influenced results, for all participants

• Measurement of the Dependent Variable: Treatment and data collection must be the same every time!

Types of Controls• Experimental

– Standards/calibration– Animal/Cell selection/care– Positive controls

• See what a positive response looks like and that it can be obtained. (positively expressing cells…)

– Negative controls• Shows what a zero response looks like

• Treatment controls– All groups treated identically except for indep.

Variable– If two treatments combined, show individual– All time points must be covered– Multiple samples

Keeping it SimpleKeeping it Simple

• Your mentor wants to look at the time course effects of a possible cancer suppressor on proliferation and mRNA expression in six breast cancer cell lines. Wants to look at 0, 12, 24, 36, 48, 72, 96h

The beauty of Small experiments….

• Mega Experiment– Ex: 6 types of cells, 7 time points, treated

and untreated (2), in triplicate (3).– 6x7x2x3 = 252 plates

• Plan Strategically and Break it down…– 1 cell line, treated and untreated, duplicate,

7 time points = 28. Or postpone duplicates.

Results from Small Results from Small ExperimentsExperiments

• Low possibility for confusion• Reasonable workload• Reasonable use of resources• Ability to “assess as progress”

– Easy to interpret– Can change directions on fly

• Easy to create discrete graphs

Chasing the Big Chasing the Big ProblemProblem

• For a Publication…– Need a Big Picture of what you are

pursuing; tell a good story– Start with correlation– Get additional information– Knock it out/Add it back/Overexpress

• Slight modifications, depending on field

How to do Experiment – Obtain Protocol

• Instructions for carrying out a particular technique• If followed, will produce desired results• Best if it’s a proven protocol

– Designing your own is time-consuming– Obtain from another investigator

• In lab, best

– A book of protocols, from web, from kit• Will need fine-tuning for your local circumstances

– Methods section from published papers (least reliable)

Review Protocol

• Read and do dry run-through

• May find logic gaps

• May find references to “common” procedures you do not know

Personalize Protocol

• Rewrite (keeping same steps, etc) to make more sense to you.

• Add notes about own equipment required

Fully Prepare before Experiment

• Buy all required materials– Radioisotopes

• Make all solutions and buffers

• Reserve machine time if needed

Follow Protocol exactly, first time through

• If it doesn’t work, you can assume it’s you.

• Do again. Not work?

• Can get help from person who provided

Modify Protocol

• Once protocol is working, modify.

• Make notations of changes

• Rewrite for next run though

• Good if type into computer…can record changes and re-print.

During Experiment

• Record which media, temperature or date-sensitive agents you used

• Record any procedural deviation– Dropped something– Delayed– Calibration questions

• Put in lab notebook in a timely fashion

Interpreting Results I

• Did the Experiment work?– Examine procedural (markers, cells lived)– Examine positive control (yes, antibody

working)– Examine negative control (No, did not have

everything come up positive)

Interpreting Results II

• What were the results?– Compared to controls, did you see effect?– Graph your data– How big was effect?– Did effect vary over time?

Interpreting Results III

• What does the experiment mean?– Do the results make sense?– Was the result what you expected?– Can you explain spurious results?– What additional controls may you need?

Interpreting Results IV

• What do other investigators think?– Talk to lab members– Discuss results with someone versed in

technique– Run through background papers again– Repeat results

Interpreting Results V

• Are the results repeatable?– Do experiment again– Add any additional controls

Agh! It didn’t work!

1. Check notes. 2. Redo the experiment3. Focus on individual parts of expt.

1. + and – controls…

4. Do partial expt. to insure it’s fixed5. When you’ve done all…try again

several times6. If external protocol, may want to switch

Switching Projects

• Never can reproduce data

• Project has little support from PI

• Direction of project has changed

• Not technically possible to do experiments well

• Project too difficult or involved