outline% the%scienefic%method% - cog scicreel/cogs14/weekly_schedule_files/lecture… · 4/1/12 4...
TRANSCRIPT
4/1/12
1
COGS 14: Design and Analysis of Experiments
Fun with numbers Course details
The scienEfic method
Outline
• What you’re going to learn • Why this may be the most useful class you ever take
• Syllabus • Empirical loop
• Research designs • Variables, and how to keep them under control
The ScienEfic Method
• Get knowledge via objec&ve observa&on – Unlike math, religion
• Create situaEons to observe – Unlike history, which just happens
4/1/12
2
Why intuiEon can steer you the wrong way
• Placebo effect • Observer bias • Natural course of an illness • Regression to the mean
• The “study effect” (Hawthorne effect)
Design and Analysis
• First, we’ll talk about design – What’s an experiment vs. an observaEon?
– Should your experiment be between subjects or within subjects?
– How best to keep extraneous factors from messing up your experiment?
Design and Analysis
• Then, we’ll talk about analysis—tools for making sense of your data. – 51 vs. 48…??? – Graphs (and how not to graph) – Summary staEsEcs/descripEve staEsEcs
– InferenEal staEsEcs (t-‐tests; ANOVAs; chi-‐squared)
4/1/12
3
People who need to understand staEsEcs
People who need to understand staEsEcs
Diets! AuEsm: cured! Lying poliEcians vs. lying poliEcians!
Syllabus
hfp://www.cogsci.ucsd.edu/~creel/COGS14 Quizzes and grades: on TED
4/1/12
4
Empirical loop
Hypothesis
Research Design
Collect Data
DescripEve StaEsEcs
InferenEal StaEsEcs
Does caffeine help
memorizaEon?
Empirical loop
Hypothesis
Research Design
Collect Data
DescripEve StaEsEcs
InferenEal StaEsEcs
Give some students caffeine, some no caffeine.
Empirical loop
Hypothesis
Research Design
Collect Data
DescripEve StaEsEcs
InferenEal StaEsEcs
Assess students’ performance on a memorizaEon
test.
Empirical loop
Hypothesis
Research Design
Collect Data
DescripEve StaEsEcs
InferenEal StaEsEcs
0%
50%
100%
Caf
Decaf
4/1/12
5
Empirical loop
Hypothesis
Research Design
Collect Data
DescripEve StaEsEcs
InferenEal StaEsEcs
Caffeinated students were more accurate
than decaf, p<.05.
Empirical loop
Hypothesis
Research Design
Collect Data
DescripEve StaEsEcs
InferenEal StaEsEcs
Caffeine does help
memorizaEon!
Research Designs
• Experiments (controlled studies) • ObservaEonal studies*
*Cau0on—these can look very experiment-‐like!
Experiments
• Have an independent variable • Have a dependent variable
• The scienEst manipulates the independent variable and looks to see if the dependent variable changes.
4/1/12
6
Experiments
• Random selecEon (from the populaEon) • Random assignment (to condiEons)
Example Research ques&on: Does chocolate help you study? Give some people Cadbury Eggs and give other people Peeps. Randomly assign who gets what candy. Let them study a list. Then measure their memorizaEon performance.
Experiments in Cog Sci Independent variables Dependent variables Type of speech Listening Eme
Type of picture viewed Change in BOLD signal Word similarity Speed of eye movements
Word expectedness Size of N400 (ERP waveform) Task difficulty ReacEon Eme, accuracy
ObservaEonal study
• Only dependent variables • (You don’t manipulate anything) – SomeEmes you can’t (gender, culture, climate change, diet [low compliance], takes too long)
– SomeEmes you shouldn’t (brain damage, disease, language deprivaEon)
ObservaEonal study
• Only dependent variables • (You don’t manipulate anything) – SomeEmes you can’t (gender, culture)
– SomeEmes you shouldn’t (brain damage, disease)
Example Research ques&on: Does chocolate help you study? Find some people who have just eaten Cadbury Eggs or Peeps. Let them study a list. Then measure their memorizaEon performance.
4/1/12
7
ObservaEonal study
• Only dependent variables • (You don’t manipulate anything) – SomeEmes you can’t (gender, culture)
– SomeEmes you shouldn’t (brain damage, disease)
• Tricky: SomeEmes talked about as if one variable was manipulated • “Quasi-‐experiment” • But can’t really assign causality
ObservaEonal studies in Cog Sci
• What sorts of infant-‐parent interacEons lead up to joint afenEon?
• How do airline personnel communicate?
• What kinds of social groups do elephants have? • How do people use smart pen technology?
ObservaEonal studies in Cog Sci
• Quasi-‐experiments (sEll observaEonal!)
• Does neonatal leu hemisphere damage result in worse language outcomes than right hemisphere damage?
• Does music make you befer at learning tone languages? • Does eaEng chocolate make you thinner?*
You can make a reasonable argument for causality, but can’t actually establish it
defini0vely.
CorrelaEon is not causaEon!
4/1/12
8
CorrelaEon is not causaEon!
Though someEmes there is a plausible explanaEon.
Confounding variables
• An uncontrolled variable that compromises interpretaEon of the study
• ObservaEonal studies: – Big potenEal for confounds – Example: What % of UCSD undergrads are Cog Sci majors?
• Do an experiment if you can.
Confounding variables: Experiments
• Random sampling • Random assignment – Time of day; day of week; Eme during quarter
– Experiment length—signup bias? – Order effects
• Double-‐blind studies – ParEcipant doesn’t know what predicEon is – Experimenter doesn’t know what predicEon is
4/1/12
9
Confounding variables: Experiments
• Double-‐blind studies – ParEcipant doesn’t know what predicEon is – Experimenter doesn’t know what predicEon is
• Why? Beliefs affect behavior. – Placebo effect: ParEcipant may get befer (or report feeling befer) just because they think they’re supposed to
– Observer bias: experimenter may treat you differently/report data differently depending on what condiEon they think you’re in
Between vs. within subjects
Between
Individual variaEon in memorizaEon abiliEes will make it harder to measure
effect of caffeine. How to fix this?
Between vs. within subjects
Within
TWO measurements per person, so that your groups are balanced in terms of memorizaEon abiliEes.
Of course, it may not be possible (effects of drug on rat mortality) or ethical (effect of drug on cancer treament).
Extraneous variables
• We’ve randomly assigned parEcipants to double-‐blind condiEons
• SEll have to deal with – Measurement noise
• Really tough if you’re looking for a small effect*
– Individual variability • Can offset this using a within-‐subjects design
4/1/12
10
Empirical loop
Hypothesis
Research Design
Collect Data
DescripEve StaEsEcs
InferenEal StaEsEcs
Types of data
• Favorite foods • Lefer grades • Heights
Types of data
Level Proper&es Observa&ons
reflect Example Type of data
Nominal ClassificaEon Differences in
kind Favorite food QualitaEve
Ordinal Order
classificaEon Differences in
degree Lefer grade Ranked
Interval/RaEo*
• Equal intervals • Order
classificaEon
Differences in total amount
Height QuanEtaEve
*Some people disEnguish these from each other—raEo has true zero point.
Nominal
• What’s your favorite food?
4/1/12
11
Ordinal
A B C D F 10 23 15 6 2
Lefer grades
Interval and RaEo
0’-‐1’ 1’-‐2’ 2’-‐3’ 3’-‐4’ 4’-‐5’ 5’-‐6’ 6’-‐7’ 1 3 2 16 24 26 7
Heights
Interval vs. RaEo
• Interval (no true zero) – IQ – GPA – Fahrenheit degrees
• RaEo (true zero) – Kelvin degrees – Income – Family size
Examples • Ethnic Group • Age • Family Size • Academic Major • PoliEcal Preference • Cooking Time for Pasta • Parole ViolaEons by Convicts • Freshman, Sophomore, Junior, Senior • SAT Score • Net Worth ($) • Favorite Sport • Gender • PosiEon in line at a cashier • A Town’s PopulaEon • Car speed
4/1/12
12
ConEnuous vs. Discrete
• Interval/raEo can be described as conEnuous or discrete
• ConEnuous – Numbers always separated by another number
• Fahrenheit • GPA
• Discrete – Numbers separated by gaps
• Vocabulary size • Number of correct answers • Number of textbooks
Examples
• Age • Family Size • Cooking Time for Pasta • Parole ViolaEons by Convicts • Number of votes • Money in a bank • A Town’s PopulaEon • Car speed
ConEnuous measurements
• Always approximate. • She’s 5 feet tall really means
• She’s between 4.95 and 5.05 feet tall
Empirical loop
Hypothesis
Research Design
Collect Data
DescripEve StaEsEcs
InferenEal StaEsEcs
4/1/12
13
DescripEve vs. InferenEal StaEsEcs
• Descrip&ve Sta&s&cs-‐The area of staEsEcs concerned with organizing and summarizing informaEon about a collecEon of actual observaEons (i.e., your sample). – Your current GPA is 3.7. – The favorite food of students in this class is ______. – Just the facts, ma’am!
• Inferen&al Sta&s&cs-‐The area of staEsEcs concerned with generalizing beyond actual observaEons (i.e., making inferences about the populaEon). – Your GPA this quarter will be around 3.7. – College students’ favorite food is ______.
DescripEves
• Summary staEsEcs – Percentages – Averages – Standard deviaEons
• Graphs!
InferenEal stats
• Generalize past your own data (too expensive or impossible to measure Everything Everywhere)
• Has only really existed for ~100 years • Based on probability theory • StaEsEcal significance • Confidence intervals
Examples
Describing Inferring The average temperature in my house is 5.1 degrees Fahrenheit hofer than it was 30 days ago.
90% of people polled do not use public transportaEon.
Sally completed the marathon in 2.26 hours.
The earth’s average near surface atmospheric temperature rose 1.1 degrees Fahrenheit in the 20th century.
If these trends conEnue, the average global temperature will rise between 2.5 and 10.4 degrees Fahrenheit by 2100.
Based on a poll of 5000 people, 90% of Americans do not use public transportaEon.
Regular exercise significantly (p<.05) reduces one’s risk of heart disease.
4/1/12
14
Empirical loop
Hypothesis
Research Design
Collect Data
DescripEve StaEsEcs
InferenEal StaEsEcs
QuesEons to ask when reading (about) research
• Who produced and who interpreted the data? • Did they measure what they should have measured?
• Did they use clear and reasonable definiEons? • Is the research published in a peer-‐reviewed journal?
• How did they get the data? Largely based on material from:
Huff, D. & Geis, I. (1954) How to Lie with StaEsEcs Best, P. (2001) Damned Lies and StaEsEcs
Who produced, interpreted data?
• Academics (Researchers at non-‐profit universiEes or research centers)
• Government Agencies (e.g., the census, FBI)
• For-‐profit corporaEons (e.g., drug companies, pollsters)
• Non-‐profit acEvists (e.g., Amnesty InternaEonal, NaEonal Center for Public Policy Research)
Who produced, interpreted data?
• What bias might the researchers/interpreters have?
• What are the consequences for the researchers/interpreters if they’re wrong?
• If the people interpreEng the data are different than the people who produced the data, is it possible they’re misrepresenEng the data?
Conflict of interest
4/1/12
15
Did they measure what they should have measured?
• If you were tesEng their hypothesis, what would you have measured?
• (If you think their measure is a lifle funny, it probably is.)
Clear and reasonable definiEons?
• Good surgeons were defined as those with less than an 80% mortality rate.
• (???) • Good surgeons were defined as those with less than a 5% mortality rate.
Is it in a peer-‐reviewed journal?
• Peer review – ScienEsts submit a manuscript of some research to a peer-‐reviewed journal.
– Editors of the journal select other scienEsts to evaluate the research to ensure that it is quality work. The idenEty of the scienEsts who evaluate the work is usually kept secret from the people who produced the research.
– The evaluators approve the manuscript, request improvements, or reject it.
(Reviewer 2 must be stopped!!)
Is it in a peer-‐reviewed journal?
• Peer reviewed journals: – Science, Nature, PNAS (“Glamour mags”)
– NEJM, JAMA
• I n Cog Sci: – CogniEon – Journal of Memory and Language
– Journal of Neuroscience – Journal of CogniEve Neuroscience
Impact Factors
4/1/12
16
Finding peer-‐reviewed journals
• Google Scholar (scholar.google.com) – Plus: Free! Links to library resources! – Minuses: Lots of junk
• Web of Science
• PubMed
• PsycINFO Via library > databases
How did they get the data?
• Experiment – P-‐values or confidence intervals?
• Margin of error is ± 3 percentage points • P < .002
• ObservaEon – Is a causal relaEonship reasonable?
• Guess – (90% of all staEsEcs are made up.) – Confidence ≠ Accuracy
• Sample size? • Sampling bias?