metaanalysis4a032011 2

8/7/2019 metaanalysis4a032011 2

1/16

Research Synthesis and Meta-Analysis 4A03 2011

'We need not confine ourselves to mere speculationabout the socio-economic processes that comprise

empirical economic research'

Stanley and Jarrell(1989):'Meta-Regression analysis: A Quantitative Method ofLiterature Surveys'. Journal of Economic Surveys


2/16

Intro

How do/should we proceed when we havemultiple studies on the same topic?

Not choose arbritrarily!

A number of approaches are possible.


3/16

Literature Reviews 1

Characteristics:

Focus(Findings, Methods, Theories,interventions)?

Goals-1) Integration: Formulating generalstatements that characterize multiple specificinstances related to the focal points

Goals-2) Critical Evaluation: Does notnecessarily compare studies to each other butto a gold standard.

Evaluate methods in empirical context andevaluate interpretation of findings


4/16

Literature Reviews 2

Your case: Focus on findings and methods withgoals of integration and critical evaluation.

Other characteristics:

Perspective: You should aim at neutrality

Coverage: May vary depending on the approachtaken and the available literature. Be as

exhaustive as possible and justify inclusion andexclusion decisions


5/16

Research Synthesis 1 A research synthesis is a particular kind of

literature review: Handbook of ResearchSynthesis and Meta-Analysis p6.

'Research syntheses attempt to integrateempirical research for the purpose of creatinggeneralizations. Implicit in this definition is thenotion that seeking generalizations alsoinvolves seeking the limits of generalizations.

Also research syntheses almost always payattention to relevant theories, critically analyzethe research they cover, try to resolve conflictsin the literature, and attempt to identify central

issues for future research'


6/16

Research Synthesis 2 See Table 1.2 from Handbook of Research

Synthesis and Meta-Analysis

Stage 1: Question definition

Stage 2:Based on systematic literature search

Report search terms, where(e.g Econlit),when(1993-) etc

Stage 3: Inclusion and exclusion criteria aremade explicit


7/16

Meta-Analysis 1

Meta-Analysis is a type of research synthesisdealing with stages 4-6.

Glass: The statistical analysis of a large

collection of results from individual studies forthe purpose integrating the findings'

Involves describing and explaining the centraltendency and variability in the outcomes ofsimilar studies using a statistical approach.


8/16

Meta-Analysis 2 Stanley. T.D(2001) :'Wheat from Chaff: Meta-

Analysis as Quantitative Literature review'Journal of Economic Perspectives 15, 3, 131-150.

Combining results can furnish more insight andexplanatory power than listing individual resultsor vote-counting

E.g by taking weighted average of summarystatistic(e.g treatment effects) to constructmean effect will be more precise.

Meta-Regression goes further and uses

summary statistic as dependent variable in


9/16

Meta-Regression 1

Independent variables are characteristics of themethod, design, data used etc

Can identify extent to which particular choice of

methods, design, data affect reported results Can help explain variability and offer reasons

why evidence on a certain question iscontradictory or very variable.

Biggest strength-moves away from subjectivechoice of good and poor studies by analyst andprovides replicable framework for objective

analysis


10/16

Meta-Regression 2

What are the steps in a Meta-Regression?

Step 1) Identify relevant studies.

Computer search of standard databases. Bestpractice gives details of computer search toallow replication.

Difference in quality, data, or methods do not

provide justification for exclusion. Criteria for inclusion and exclusion standards

applied consistently independent of findings ofstudy.


11/16

Meta-Regression 3 Step 2) Choose summary statistic and reduce

evidence to a common metric(e.g co-efficientestimates, elasticities, t-stats)

Be careful with coefficients if outcomes or

regressors are on different scales. May need toconvert to common scale

Usual first overall estimate-combining these toestimate average effect(Stanley p135-136)

Technical issues relate to heteroskedasticity dueto different degrees of freedom and relation ofsummary statistic to degrees of freedom.


12/16

Meta-Regression 4

Step 3) Choose explanatory variables (Stanleyp137-138)

Usually there will be much more variability in

the summary statistics of interest than would beproduced by chance for the same underlyingparameter

Want to explain this variability using studycharacteristics

# of variables chosen depends on number ofstudies and variability in study characteristics-

Why?


13/16

Meta-Regression 5

Suggested regressors: Inclusion/Omission of particular regressors from

original model

Specification variables Data variables

Sample size

Author characteristics(particular problem comesabout with multiple studies by same author orestimates from same study)

Quality Measures(Method variables)


14/16

Meta-Regression 6

Step 4) Conduct the Meta-Regression(standard software forregression analysis) Stanley p139-141

Step 5) Subject Meta-Regression to specification testing andSensitivity analysis. Stanley p141-143

Use standard tests for specification testing and out of sampleforecasting

Example of normalized test-statistics(test-statistics that have p-valuesmatched to standard normal): Purely random variation should have avariance equal to one. Can test this and if not rejected provides

general specification test that additional regressors will not addexplanatory power.

Example of t-statistics: Including degrees of freedom(or sample size)there should be a positive relationship if the null of zero effect is falseand zero if true(in absence of publication bias)

Note that discussion in Stanley p142-3 should be discussing absolute-


15/16

Meta-Regression Problems

1) Disagreement over regressors(can be dealt with-sensitivityanalysis, testing)

2) Weighting of each study equally(overweights those who publishlots-can be dealt with by sensitivity analysis and author effects andexplicit weights)

3)Invariant errors-Cannot deal with this! Critical evaluation required.

4) Publication Bias-test exist(see Stanley 2005. Journal of EconomicSurveys 19, 3, 309-345)

Similar issues also exist with narrative reviews with less explicit

criteria and sensitivity analysis


16/16

Recent Examples

Gallet and List(2003) Health Economics 12,821-835 :Price elasticities for Tobacco

Gemmill et al(2007) Health Economics 16: 27-

643: Prescription Drug elasticities Gallet (2007) Economics Bulletin 9, 7, 1-4:Price

and Income elasticities for Higher education

Card et al(2010) Economic Journal 120, F452-477: Active Labour Market Policies

metaanalysis4a032011 2

Documents