PSY 1950Meta-analysis

December 3, 2008

Definition“the analysis of analyses . . . the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature.”– Glass (1976)

“Mega-silliness”– Eysenck (1977)

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

History• Pre-history

– Pearson (1904), Fisher (1948), Cochran’s (1955)

• The Great Debate– 1952: Eysenck concluded that psychotherapy was bunk

– 20 years of research did not settle debate

– 1978: Glass & Smith statistically aggregated findings from 375 studies, concluding that psychotherapy works

• Necessity is the mother of invention– Psychology abounds!

Rationale• Meta-analyses avoid the limitations of qualitative/narrative/traditional reviews:– Weak effects overloooked

• Meta-analyses are more powerful

– Differences between studies over-interpreted• In meta-analysis, heterogeneity assessed statistically

– Moderating variables overestimated or overlooked• In meta-analysis moderators assessed statistically

– Limited, subjective sampling of studies• In meta-analysis, exhaustive search and defined inclusion/exclusion criteria

– Overwhelmed by large database• Meta-analyses can summarize hundreds of effects

– Subjective assessment• In meta-analysis, any subjectivity is discernible

Example: Finding Weak Effects

• Cooper, H. M., & Rosenthal, R. (1980). Statistical versus traditional procedures for summarizing research findings. Psychological Bulletin, 87, 442-449.– 32 grad students, 9 faculty members– Randomly assigned to statistical and traditional review technique

– Given 7 studies that examined sex differences in persistence•For 2 studies, females more persistent than males (ps = .005, .001)

•For other studies, no significant difference

– Does evidence presented support the conclusions that females are more persistent?

actual p = .016

Example: Assessing Moderators Statistically

Criticisms• Weak

– Apples and oranges– Flat Earth society– Garbage in, garbage out– File-drawer problem

• Strong– Post-hoc

Apples and Oranges• Critique

– Meta-analyses add together apples and oranges

• Response– Glass: “in the study of fruit, nothing else is sensible”

– Analogy with single experiments– Empirical question resolved through examination of moderating variables

Flat Earth Society• Critique

– Cronbach: "...some of our colleagues are beginning to sound like a kind of Flat Earth Society. They tell us that the world is essentially simple: most social phenomena are adequately described by linear relations; one-parameter scaling can discover coherent variables independent of culture and population; and inconsistences among studies of the same kind will vanish if we but amalgamate a sufficient number of studies.... The Flat Earth folk seek to bury any complex hypothesis with an empirical bulldozer.”

• Response– Code and analyze moderating variables

QuickTime™ and a decompressor

are needed to see this picture.

Garbage In, Garbage Out• Critique

– The inclusion of flawed studies “dirties” the database, obscures the truth, and invalidates meta-analytic conclusions

• Response– Glass: “I remain staunchly committed to the idea that meta-analyses must deal with all studies, good bad and indifferent, and that their results are only properly understood in the context of each other, not after having been censored by some a priori set of prejudices.”

– Empirical question: Study quality (or better yet, related variables) can be coded and analyzed as moderators

File-drawer Problem• Critique

– Meta-analytic database is biased sampling of studies

– Significant findings are more likely to be published than nonsignificant findings

• Response– Less bias than narrative reviews– File-drawer analyses (e.g., funnel plots) can empirically address the presence and influence of missing studies

Post-hoc• Criticism

– By definition, meta-analysis is a post-hoc endeavor, i.e., an observational study•Moderating variables may be confounded, sometimes extremely so

•Effects may be correlational

• Response– Confounding may be interested in its own right

– Statistical control– Hypothesis generation versus hypothesis testing

Type of publication

Type of meditation

JournalDissertationOtherType of meditation

Type of meditation

TM-0.15<. 010.19Yoga0.10-0.140.03Other0.090.12-0.25

Author affiliation-0.020.03<. 010.160.15-0.31Control

group/ cond-0.040.07-0.03-0.09-0.080.17-0.20State0.06-0.190.14-0.050.010.05-0.100.22Trait-0.080.20-0.110.03-0.01-0.030.15-0.22Both0.07-0.05-0.030.030.001-0.04-0.120.05 p < .001

General intelligence0.06-0.090.030.09-0.09-0.030.11-0.04-0.050.07-0.05 p < .0001

Fluid intelligence-0.070.020.060.08-0.07-0.04-0.120.02-0.130.100.03Crystallized intelligence-0.060.10-0.040.10-0.08-0.050.030.03-0.080.040.08

Memory and learning-0.100.060.07-0.060.030.05-0.10<. 010.01-0.010.001Visual

perception0.110.02-0.15-0.060.050.030.11-0.06-0.040.09-0.10Auditory

perception-0.08-0.040.140.17-0.09-0.12-0.150.120.25-0.20-0.05Retrieval/ creativity0.08-0.04-0.050.02-0.110.06-0.020.11-0.020.05-0.06Cognitive speediness-0.020.11-0.08-0.080.17-0.050.00-0.020.04-0.01-0.05Processing

speed0.04-0.080.030.09-0.06-0.050.15-0.070.01-0.040.07Psychomotor-0.05-0.100.17-0.050.17-0.08-0.01-0.060.10-0.180.19

Attention0.050.06-0.13-0.230.010.26-0.040.04-0.060.08-0.05Between

participants-0.080.12-0.030.13-0.220.02-0.10-0.27-0.140.110.04-0.210.01-0.070.030.030.050.01-0.070.030.070.04Within

participants-0.06-0.110.200.000.13-0.110.080.110.31-0.26-0.040.05-0.06-0.04-0.08-0.050.25-0.060.03-0.010.09-0.06Mixed0.10-0.08-0.05-0.130.170.020.060.220.02<. 01-0.020.180.010.09<. 01-0.01-0.150.020.06-0.03-0.10-0.01

Random assignment0.030.10-0.15-0.16-0.050.220.170.25-0.030.04-0.030.150.050.100.12-0.09-0.140.050.07-0.13-0.130.01-0.38n/a0.38

Age-0.240.41-0.14-0.02-0.080.090.14-0.04-0.070.040.04-0.11-0.09-0.02-0.03-0.010.02-0.040.040.030.080.130.230.01-0.23-0.08Meditative experience-0.060.08-0.010.50-0.27-0.360.18-0.29-0.020.05-0.08-0.050.03-0.02-0.07-0.020.14-0.08-0.010.090.01-0.010.52-0.02-0.51-0.360.28

Impact factorn/an/an/a-0.09-0.210.290.140.24-0.140.120.020.040.020.09-0.01-0.11-0.070.04-0.08-0.04-0.170.36-0.27-0.110.30-0.360.11-0.21

Broad cognitive ability

Type of meditation

Type of effect

AgeRandom assignment

StateTraitBoth

Type of design

Between participantsWithin participantsMixed

Meditative experience

Cognitive speedinessProcessing speedPsychomotorAttentionMemory and learningVisual perceptionAuditory perceptionRetrieval/ creativityGeneral intelligenceFluid intelligenceCrystallized intelligence

Control group/ condAuthor affiliation

OtherYogaTMType of meditation

Type of effect

Broad cognitive ability

Type of design

Steps of a Meta-analysis• Define question• Search literature• Determine inclusion/exclusion criteria

• Code moderating variables• Analyze data

• This is an iterative process!

Defining Meta-analytic Question

• Interestingness– Establish presence of effect– Determine magnitude of effect– Resolve differences in literature– Test competing theories

• e.g., psychotherapy, imagery v1

Inclusion/Exclusion Criteria

• Theoretical considerations– Scope/generalizability– Quality

• Practical considerations– Power– Missing data– Time

• Studies were included if they– had written published or unpublished reports in English available by March 1, 2008

– presented original data from between-participants, within-participants (i.e., single-group pretest-posttest, or PP), or mixed design (i.e., independent-groups pretest-posttest, or IG-PP) experiments or quasi-experiments

– objectively, quantitatively evaluated performance on at least one cognitive task as a function of meditative experience or state

• Studies were excluded if they– used a psychopathologically or neurologically disordered population

– confounded meditation with other mental training (e.g., education), maturation, or practice and used measures susceptible to such confounding (e.g., academic achievement test)

– did not report data on or contained data that allow estimation of participants’ age or meditative experience

– did not contain basic methodological information (e.g. type of task administered)

Literature Search• Types of searches

– Keyword– Ancestor– Descendent

• Available Resources– Electronic

• e.g., PsychInfo, SCI, Google scholar

– Physical• Conference proceedings• Bibliographies• Key journals

– Mental• Experts in the field

Harvard’s Electronic Resources

• SSCI/SCI (Social/Science Citations Index)– http://nrs.harvard.edu/urn-3:hul.eresource:scicitin

• PsychInfo– http://nrs.harvard.edu/urn-3:hul.eresource:psycinfo

• Google Scholar– http://scholar.google.com/

• E-Journals @ Harvard– http://sfx.hul.harvard.edu/sfx_local/az/

• HOLLIS– http://holliscatalog.harvard.edu/

• Interlibrary Loan– https://illiad.hcl.harvard.edu/

• Dissertations (Proquest)– http://nrs.harvard.edu/urn-3:hul.eresource:psycinfo

0

0.0005

0.001

0.0015

0.002

0.0025

0.003

1968 1973 1978 1983 1988 1993 1998 2003

Year

Meditation publications (%)

TM Buddhist Yoga

Coding• What to code

– Anything possibly interesting•e.g., control group/condition, participant variables

– Anything possibly confounding•e.g., publication year, journal impact factor

– How you coded effect sizes

• How to code– Using explicit coding scheme– Set measurement scale– Multiple coders– Calculate reliability

Coded VariableType of data Categories (if applicable)

Potential Moderator?

First author Nominal no

Publication year Interval no

Type of publication Nominal peer-reviewed journal, dissertation/thesis, other yes

Journal impact factor Ratio yes

Type of meditation Nominal TM, yoga, other yes

Author affiliation Nominal yes, no yes

Control group/conditionOrdinal 1, 2, 3 yes

Type of effect Nominal state, trait, both yes

Task Nominal no

Narrow cognitive abilityNominal see Carroll (1993) no

Broad cognitive abilityNominal

general intelligence (g), fluid intelligence (2F), crystallized

intelligence (2C), memory and learning (2Y), visual perception (2V),

auditory perception (2U), retrieval/creativity (2R), cognitive speediness (2S), processing speed (2T), psychomotor ability (2P),

attention

yes

Experimental design Nominal between-participants, within-participants, mixed yes

Random assignment Nominal yes, no yes

# participants (meditation)Ratio no

# participants (control)Ratio no

Mean age (meditators)Interval 0-18, 18-25, 25-40, 40+ (years old)yes

Meditative experienceInterval 0-3 months, 3-12 months, 1-3 years, 3+ years yes

Analysis• Calculate effect size• Weight effect size• Describe• Infer

– Univariate analyses– Multivariate analyses

Calculating Effect Size• Only one ES per construct per study– Balance between dependency and thoroughness

• Typically d or r

• Can be calculated in lots of way (from raw data to graphs)

• Effect size calculator

Weighting Effect Size• Why weight?

– Studies vary significantly in size– Studies with large n have more reliable effect sizes than studies with small n

• How weight?– Simple approach: weight by sample size– Better approach: weight by precision

• What is precision weighting?– Each effect size has associated SE– Hedges showed that best meta-analytic estimate of precision is weight by inverse sampling variance

Describing Distribution• Central tendency• Spread• Shape

0

5

10

15

20

25

30

-1.6-1.4-1.2-1

-0.8-0.6-0.4-0.20

0.2 0.4 0.6 0.81

1.2 1.4 1.6 1.82

2.2 2.4

Effect size

Proportion total (%)

NonpartisanPartisan

Inferencial Statistics• Select a model

– Fixed effects– Random effects

• Univariate analyses– Analogous to one-way ANOVA– Examine how much variation in effect sizes is explained by one (categorical) variable

• Multivariate analyses– Analogous to multivariate regression– Examine how much variation in effect sizes is explained by set of (categorical or continuous) variables

– Examine how much unique variation in effect sizes is explained by one (categorical or continuous) variable

k d lowerupperQb Qw p k d lowerupperQbQw p k d lowerupperQb Qw pOverall2180.390.330.44 <.00011290.200.140.26 <.0001940.630.560.71 <.0001

Type of publication 44.91 <.0001 1.63 0.20 12.78 0.002Journal1260.340.270.42 8.78<.0001930.230.150.30 5.78<.0001330.610.480.74 8.96<.0001

Dissertation/Thesis510.150.030.28 2.410.02360.130.020.25 2.250.02150.23-0.040.49 1.690.09Other410.770.630.90 11.12<.0001 380.760.620.90 10.93<.0001

Type of meditation 9.26 0.01 1.18 0.55 5.94 0.01TM1460.390.310.46 9.69<.0001690.170.080.25 3.690.0002750.590.490.68 11.81<.0001

Yoga270.580.410.75 6.52<.0001150.250.080.41 2.910.004110.930.671.19 7.02<.0001Other450.230.080.37 3.080.002450.230.120.34 3.990.0001

Author affiliation 20.34 <.0001No1590.290.220.37 7.93<.0001Yes590.610.490.72 10.26<.0001

Control group/cond 6.02 0.05 3.38 0.18 5.64 0.0611260.400.320.49 9.28<.0001650.210.110.30 4.42<.0001580.600.490.71 10.46<.00012 380.510.350.67 6.15<.0001260.310.160.47 3.950.0001120.920.661.17 7.00<.00013 540.270.140.39 4.26<.0001380.140.030.24 2.450.01160.550.350.75 5.43<.0001

Type of effect 4.10 0.13 2.66 0.27 5.00 0.03State310.440.270.62 5.07<.0001190.16-0.010.33 1.800.07110.900.641.16 6.86<.0001Trait1790.380.310.46 10.60<.00011030.220.150.29 6.05<.0001740.590.490.69 11.48<.0001Both8 0.06-0.270.39 0.350.72 7 0.01-0.240.27 0.110.911 0.89

Broad cognitive ability 12.48 0.25 11.46 0.32 12.49 0.25General intelligence130.450.210.70 3.630.00035 0.12-0.180.43 0.780.448 0.620.360.88 4.63<.0001

Fluid intelligence190.350.140.57 3.210.001120.240.040.44 2.380.027 0.520.200.84 3.210.001Crystallized intelligence9 0.13-0.170.43 0.860.39 5 0.05-0.230.33 0.370.714 0.27-0.160.69 1.230.22

Memory and learning270.550.360.73 5.84<.0001170.320.150.49 3.650.00039 0.760.471.06 5.12<.0001Visual perception440.410.270.55 5.67<.0001260.180.040.32 2.450.01160.770.580.97 7.69<.0001

Auditory perception120.630.330.93 4.14<.00017 0.480.140.83 2.730.015 0.740.361.13 3.840.0001Retrieval/creativity170.320.100.54 2.830.005110.11-0.110.32 0.970.336 0.650.350.95 4.26<.0001

Cognitive speediness150.20-0.040.43 1.650.10100.14-0.060.35 1.360.175 0.34-0.030.71 1.800.07Processing speed270.400.220.58 4.36<.0001120.330.130.54 3.220.001150.480.270.70 4.41<.0001

Psychomotor200.310.090.53 2.780.01120.02-0.190.23 0.190.858 0.850.531.17 5.14<.0001Attention150.300.050.55 2.380.02120.210.0030.42 1.990.053 0.720.191.26 2.670.01

Full sample Nonpartisan sample Partisan sample95% CI 95% CI 95% CI

B SEB Q p B SEB Q p B SEB Q pType of publicationa 58.15<.0001 4.44 0.04 13.200.001

Dissertation/Thesis-0.240.0520.56<.0001-0.180.084.44 0.04-0.270.115.71 0.02

Other0.320.0537.59<.0001 0.230.087.49 0.01

Type of meditationb 2.78 0.25 0.04 0.98 3.83 0.05

Yoga0.100.062.39 0.12 0.010.070.02 0.89 0.360.193.83 0.05

Other-0.040.060.39 0.53-0.010.060.02 0.89

Author affiliation0.340.0724.35<.0001

Control Group/Cond-0.020.040.20 0.65 0.010.040.05 0.82-0.150.074.55 0.03

Type of effectc 1.40 0.50 3.66 0.16 0.33 0.85

State0.040.070.29 0.59 0.040.080.25 0.62 0.090.220.18 0.67

Both-0.110.101.11 0.29-0.180.103.41 0.06 0.160.400.16 0.69

Broad cognitive abilityd 14.690.14 14.000.17 23.830.008

General intelligence-0.010.100.01 0.92-0.120.140.71 0.40 0.120.130.94 0.33

Crystallized intelligence-0.130.121.19 0.28-0.070.130.28 0.60-0.240.191.58 0.21

Memory and learning0.190.085.79 0.02 0.140.092.51 0.11 0.170.131.65 0.20

Visual perception0.020.060.16 0.69-0.050.070.39 0.53 0.170.112.59 0.11

Auditory perception0.220.142.60 0.11 0.300.163.45 0.06-0.140.230.36 0.55

Retrieval/creativity-0.020.090.04 0.84-0.140.101.88 0.17 0.210.132.36 0.12

Cognitive speediness-0.130.101.79 0.18-0.030.100.06 0.80-0.420.175.98 0.01

Processing speed-0.030.080.20 0.65 0.140.101.84 0.17-0.040.100.17 0.68

Psychomotor-0.160.102.84 0.09-0.160.102.59 0.11-0.350.193.37 0.07

Attention0.030.110.07 0.79-0.060.110.28 0.60 0.530.244.83 0.03

Experimental designe 0.56 0.76 1.11 0.57 1.04 0.59

Between participants0.010.060.04 0.84 0.090.100.80 0.37 0.060.090.39 0.53

Within participants0.070.090.51 0.47-0.090.160.31 0.58 0.090.110.65 0.42

Random assignmentf -0.070.070.95 0.33-0.170.093.63 0.06 0.160.131.47 0.22

Age 0.000.030.0010.98 0.060.051.58 0.21 0.020.060.10 0.76

Meditative experience-0.050.041.81 0.18-0.060.042.04 0.15-0.100.081.81 0.18

Full sampleNonpartisan samplePartisan sample

Notes. aReference = Journals, bReference = TM, cReference = Trait, dReference = Fluid intelligence, eReference = Mixed, fFor tolerance reasons, the regression model that included random assignment did not include the "within-participants" effect-coded variable.