a thousand zeros is still nothing science & statistics in psychology lecture 3 a thousand zeros...
TRANSCRIPT
Science & Statistics in PsychologyLecture 3
A thousand zeros is still nothingA thousand zeros is still nothing
Dr Caleb Owens
Lecture Plan
1. Theory & Evidence : Science and pseudoscience; the importance of a rationale
2. The power of a name : Measurement and constructs3. A thousand zeros : Types of research design; internal
and external validity4. Predictions : Hypotheses in science; the null hypothesis;
the importance of a disprovable hypothesis5. The coin toss : Understanding variability in sampling
and measurement; probability and the appeal to ignorance
6. Too much of a good thing : Statistical power; Practical significance
Lecture Outline• Research designs
– Anecdotes and case studies– Correlational studies– Quasi-experiments– True experiments
• External validity– Random sampling
• Internal validity– Random allocation
• Blinding
Have you heard ?
• The plant Echinacea was marketed throughout the 1990’s as an ancient herbal remedy which could cure the common cold (not unlike Vitamin C in the 1960’s)
– But in 2003 when the evidence from a large double blind study was released, the alternative-remedy lobby group backed down, saying that it was unwilling to promote questionable cures which may cause serious side-effects
• Or did it ?
What actually happened
• Some alternative health lobby groups strongly disputed that the tests were valid
– “Children have different immune systems!”
Excuses
• The answer is coming
• The tests were flawed
• “This topic is beyond the scope of western science”
• We are convinced by overwhelming anecdotal evidence
Anecdotal evidence
• “The plural of anecdote is not evidence”• Anecdotes are usually worse than useless
– Because they generally support what we believe– The anecdote contains the (proposed) explanation
which cannot be untangled once it is passed on– It cannot be replicated or documented by non-
prejudiced observers– It changes with every retelling and depends on fallible
memory• At most anecdotes provide clues and leads
Case studies
• Significantly more systematic than anecdotes– All details taken and recorded in a scientific manner
• May simply consist of the combination of previously collected information
• Can consist of the systematic collection of longitudinal data– E.g. The famous television series: “7-up”, “49-up”
• Provide important details which can lead to later systematic research and breakthroughs
• Should not be over-interpreted since the sample size is often small and rarely randomly selected– Ideal for investigating rare or highly specific topics
• E.g. prosopagnosia
• Anecdote
• My Aunty Liz got Cholera from the terrible smells and gases coming from the river near her house. She moved house. Whenever she smells something bad from now on she stays away and hasn’t got Cholera since!
• Case Study
• Elizabeth Cromwell contracted Cholera on 12th July 1723 during a particularly hot summer.
• When winds were blowing from the East, her house became filled with noxious gases causing a terrible smell.
• She was an otherwise healthy women whose only other affliction was mild Dysentery some years earlier.
• Her house was located next to the river Thames which at the time contained a large amount of raw sewage.
• She drew her drinking water from a well which was exposed to the river.
Correlational studies• Measure two naturally occurring variables
• Find a relationship between them
• Only a very weak causal inference could be drawn (a suggestion, a lead for more research)
• No independent variables are controlled
e.g. Simply look through lots of data and find relationships
Correlation Coefficients
XY = – 0.7
The sign (+ or – ) indicates the direction of the
relationship (+ : as values on one variable get bigger,
values on the other get bigger; - : as values on one variable get bigger, values on the
other get smaller)
The number indicates the strength of the
relationship(0.1, 0.2, 0.3 = weak ; 0.7, 0.8, 0.9 = strong)
Scatterplots
X
100806040200
Y
600
400
200
0
-200
-400
-600
X
100806040200
Y
120
100
80
60
40
20
0
-20
XY = 0.25
XY = 0.88Positive Correlations
The score on two IQ tests taken on the same day
The relationship between High School grades and University grades
Each person has two scores and these are plotted on separate axes
Scatterplots
X
100806040200
Y
20
0
-20
-40
-60
-80
-100
XY = -0.93
X
100806040200
Y
400
300
200
100
0
-100
-200
-300
-400
XY = -0.16
Negative Correlations
The number of hours of exercise people do and their excess fat in a pinch test
The amount of time people spend on the internet and their university grades
Correlation and Causation
• Just because two variables are correlated with each other does not mean that one causes the other.
• Example: The number of ice-creams consumed on a certain day correlates positively with deaths by drowning (0.8)
Correlation and Causation
• It could be that:– Eating ice-cream causes drowning– Drowning causes ice-cream to be eaten– Amount of ice-cream eaten and number of
drownings are both caused by the temperature on a particular day
• Does the cold of winter cause colds and flu ?
Some plausible correlational findings
• Living close to the beach is correlated with being rich– Causation in the other direction ?
• Religiousness is correlated with being on death row – Causation in the other direction ? Third variable ?
• Being aboriginal is correlated with being in prison– Third variable ?
• Being at University is correlated with being homosexual– Measurement bias ?
• Height is correlated with IQ– Third variable ?
Correlation and Causation
• In general, if A & B are correlated it is possible that:– A causes B– B causes A– Some other variable C causes both A & B
• Correlation Causation!
Independent & Dependent Variables
• Presumed Cause• Antecedent Cause• Stimulus Variable
• Behaviour of Interest• Response Variable
IndependentVariable
DependentVariable
Manipulated byexperimenter
Measured byexperimenter
Correlational studies• Measure two naturally occurring variables
• Find a relationship between them
• Only a very weak causal inference could be drawn (a suggestion, a lead for more research)
• No independent variables are controlled
e.g. Simply look through lots of data and find relationships
True Experiments• Randomly allocate subjects to different levels of the
independent variable• So all independent variables of interest are ‘controlled’• By not letting subjects choose, most confounds
cancel each other out because you have a random amount of each in each group
– E.g. When subjects turn up you toss a coin to see which condition they go into. If the experiment is based on speed, then with enough subjects, naturally ‘fast’ and ‘slow’ people in each condition will cancel each other out
Quasi-experiments• Some attempt at controlling variables
– Matching– Manipulating some IV’s but not others
• Always a study of ‘pre-existing groups’ in some sense• Whenever you have pre-existing groups you have
obvious confounds• Causal inference is weak• Example
– When studying the psychological impact of domestic violence, you find a ‘control’ group which has not experienced violence (and you want to ensure the groups are as similar as possible), and you may give both groups different kinds of therapy
Random allocation of independent variables
• Correlational study : NONE
• Quasi – experiment : At least one (but not others); or none, but some attempt to control one (e.g. matching)
• True Experiment : ALL
Research Designs: Drug use example
• Correlational: simply record the amount of alcohol consumed and University Grades– Weak causal inference– Numerous alternative explanations
• Quasi-experimental: Select a group of people who drink regularly and a group who do not, match them on diet, exercise, age and lifestyle and compare their grades– Eliminates some confounds– Does not control for all variables (e.g. attitude, personality)
• True experiment: Randomly allocate 250 people to drink heavily and 250 to drink a placebo control– Controls for most extraneous variables– Puts 50% at risk !
Research Designs: Self-Esteem Example
• Correlational: Record self-esteem and University Grades and look for a relationship– Weak causal inference– Numerous alternative explanations
• Quasi-experiment: take people who have high self-esteem and low self-esteem; compare their grades and try to improve these with extra classes or peer support – Eliminates some alternative explanations– Self-esteem is still pre-existing and may have been caused
by a third variable• True experiment: Randomly allocate 250 people to
high self-esteem and 250 people to low self-esteem and measure the impact on Uni grades– Controls for most extraneous variables– Impractical, unethical
Controlling variables
• Hard to randomly allocate:– Treatments– Lifestyles – Education– Experience– Diet
• Impossible to randomly allocate– Sex – Brain damage (in humans)*– Psychological disorders– Extreme stress / trauma
/anxiety– Sexuality– Family– Location– Religion– Race– Income
*TMS, transcranial magnetic stimulation, now makes this possible
Sex differences• Males are better than females at spatial tasks
(e.g. mental rotation)• Hypothesize: Maybe this is caused by males
having more experience with ‘spatial’ toys• Match males & females on “Lego experience”
– This is called ‘controlling’ the variable (holding the variable constant: it’s not the same as the control you have in a true experiment)
– ‘Males are better than females at spatial tasks but when we controlled for spatial task experience the difference disappeared’
External validity
• Based on sample size, and the representativeness of the sample and where and how the experiment is conducted
• Depends on random selection of subjects and how closely the setting matches the real world
• How applicable is the data to the real world?– How applicable to the real world are laboratory based
findings?
• How well can you generalize from your results to the real world?– NB: This depends heavily on the topic
Internal validity
• The ability to draw a causal inference
• The extent to which changes in the DV are likely to be due to changes in the IV
• Depends on random allocation of subjects to different levels of the IV and the elimination of alternative explanations and confounding variables
Type of study Internal validity* External validity**
Case studies None Extremely high ?
Survey / Questonnaire
Very poor Good
Quasi-experiment Poor-good Good
True (field) experiment
Good Okay
True (lab) experiment
Excellent Poor
*Decreasing internal validity certainly limits the causal inference you can draw but does not make the study ‘bad’ in any overall sense- there are plenty of well designed and very useful survey studies, and some awful, badly designed studies using true experiments
**External validity depends on many factors unrelated to design (especially sampling), but in general tends to decrease as the experiment becomes more controlled
Blindness• Being ‘blind’ means you do not know what
treatment/condition is being administered
• If the researcher is not blind– May not randomly allocate correctly– May unconsciously change data– May unconsciously affect outcome
• If the subject is not blind– The effect arises because they expect it to (placebo effect)
• If both subject and experimenter don’t know what condition is being administered it is called ‘double-blind’
• Could anecdotal evidence ever be of a randomized double blind nature?
• Unblinded– Participant takes a pill knowing that it is intended to
relieve pain
• Single-blind– Participant is given a pill by a researcher (who knows
what it does) but is not told whether or not it will relieve pain
• Double-blind– Researcher 1 makes the pills Yellow and Blue. She
gives them to Researcher 2 who does not know which one has an active ingredient. Researcher 2 gives the participant one of the pills and records how effective it is. Researcher 1 takes the data which records how effective the Yellow and Blue pills were and makes a conclusion (only they knew that the Blue pill contained an active ingredient)
Important points • You can have a ‘good’ correlational study and a
‘bad’ true experiment – based on how well they are designed
• Beyond the design, replication is everything in science– Replication means the same findings can be found
again by repeating the study– Beware of people and websites who pick out just a
handful of (often older) articles to support their view, and ignore the fact such things were never found again, or are found very rarely (indicating a result due to chance or bias)
– Beware of groups who insist their results can only be replicated by people who are ‘trained’ in doing what they do