introduction to robust standard errors
DESCRIPTION
Introduction to Robust Standard Errors. Emily E. Tanner-Smith Associate Editor, Methods Coordinating Group Research Assistant Professor, Vanderbilt University Campbell Collaboration Colloquium Copenhagen, Denmark May 30 th , 2012. Outline. Types of dependencies Dealing with dependencies - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/1.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Introduction to Robust Standard Errors
Emily E. Tanner-SmithAssociate Editor, Methods Coordinating Group
Research Assistant Professor, Vanderbilt University
Campbell Collaboration ColloquiumCopenhagen, Denmark
May 30th, 2012
![Page 2: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/2.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Outline• Types of dependencies• Dealing with dependencies• Robust variance estimation• Practical considerations
– Choosing weights– Handling covariates
• Robust variance estimation in Stata
![Page 3: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/3.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Types of Dependencies• Most meta-analysis techniques assume effect sizes are
statistically independent• But there are many instances when you might have
dependent effect sizes– Multiple measures of the same underlying outcome construct– Multiple measures across different follow-up periods– Multiple treatment groups with a common control group– Multiple studies from the same research laboratory
![Page 4: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/4.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Types of Dependencies• Assume Ti = θi + εi, where
– Ti is the effect size estimate– θi is the effect size parameter– εi is the estimation error
• Statistical dependence can arise because– εi are correlated– θi are correlated– or both
![Page 5: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/5.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Correlated Effects Model (εi correlated)
kj correlated estimates of the study specific ES
τ2 = between study variation
in ESMeta-
regression
Dependent effect size
meta-regression
Study 1, θ1
Estimate 1.1 of θ1
Estimate 1.2 of θ1
Estimate 1.3 of θ1Study 2, θ2
Estimate 2.1 of θ2
Study 3, θ3Estimate 3.1 of θ3
Estimate 3.2 of θ3
![Page 6: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/6.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Hierarchical Model (θi correlated)
ω2= between-study, within-cluster variation in ES
τ2 = between cluster variation
in average ESMeta-
regression
Dependent effect size
meta-regression
Cluster 1, θ1
Study 1.1 estimate of θ1.1
Study 1.2 estimate of θ1.2
Study 1.3 estimate of θ1.3
Cluster 2, θ2 Study 2.1 estimate of θ2.1
Cluster 3, θ3Study 3.1 estimate of θ3.1
Study 3.2 estimate of θ3.2
![Page 7: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/7.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Dealing with Dependencies• Ignore it and analyze the effect sizes as if they are
independent (not recommended)• Select a set of independent effect sizes
– Create a synthetic mean effect size– Randomly select one effect size– Choose the “best” effect size
• Model the dependence with full multivariate analysis– This requires information on the covariance structure
• Use robust variance estimation (Hedges, Tipton, & Johnson, 2010)
![Page 8: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/8.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Variance Estimation• Assume T = Xβ + ε, where
T = (T1,…,Tm)’ Tj is a kj x 1 vector of effect sizes for study jX = (X1,…,Xm)’ Xj is a kj x p design matrix for study jβ = (β1,…,βp)’ β is a 1 x p vector of unknown regression coefficientsε = (ε1,…,εm)’ εj is a kj x 1 vector of residuals for study j
E(εj) = 0, V(εj) = Σj
![Page 9: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/9.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Variance Estimation• We can estimate β by:
• And the covariance matrix for this estimate is
• The problem is that although the variances in Σj are known, the covariances are UNKNOWN
![Page 10: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/10.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimator (RVE)• The RVE of b is
where ej = Tj – Xjb is the kj x 1 estimated residual vector for study j
![Page 11: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/11.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimator (RVE)• A robust test of H0: βa = 0 uses the statistic where vR
aa is the ath diagonal of the VR matrix
Note: the t-distribution with df = m - p should be used for critical values
![Page 12: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/12.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimator (RVE)• Under regularity conditions and as m -> ∞, Vm
R is a consistent estimator of the true covariance matrix
• RVE theorem is asymptotic in the number of studies m, not the number of effect sizes
• Results apply to any type of dependency• No distributional assumptions needed for the effect sizes• Correlations do not need to be known or specified, though
may impact the standard errors• RVE theorem applies for any set of weights
![Page 13: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/13.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Practical Issues: Choosing Weights• Although the RVE works for any weights, the most efficient
weights are inverse-variance weights
– In the hierarchical model:
W 𝒊𝒋=1/ (𝑉 j+𝜏2+ω2)
![Page 14: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/14.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Practical Issues: Choosing Weights• In the correlated effects model, we can estimate
approximately efficient weights by assuming a simplified correlation structure:– Within each study j, the correlation between all pairs of effect
sizes is a constant ρ– ρ is the same in all studies– kj sampling variances within the study are approximately equal
with average Vj
W 𝒊𝒋=1
{(𝑉 j+𝜏2 ) [1+(k 𝑗−1 ) 𝜌 ] }
![Page 15: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/15.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Practical Issues: Choosing Weights• In the correlated effects model, we can take a conservative
approach when calculating weights by also assuming ρ = 1, and weights become:
• Conservative approach, because studies do not receive additional weight for contributing multiple effect sizes
W 𝒊𝒋=1
{𝑘 𝑗 (𝑉 j+𝜏2 )}
![Page 16: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/16.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Practical Issues: Choosing Weights• In the correlated effects model, ρ also occurs in the
estimator of τ2:
• Use external information about ρ if available (test reliabilities, correlations reported in studies, etc.)
• Take a sensitivity approach when estimating τ2 by estimating the model with various values of ρ in (0, 1)
![Page 17: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/17.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Practical Issues: Choosing Weights• For the correlated effects model, RVE software is currently
programmed to default to (per Hedges, Tipton, & Johnson recommendation):– Conservative approach to estimate weights (assume ρ = 1)– User must specify ρ for estimation of τ2 ; sensitivity tests
recommended
![Page 18: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/18.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Practical Issues: Handling Covariates• Some covariates may vary within groups (i.e., studies or
clusters) and between groups, e.g.,– Length of follow-up after intervention– Time frame of outcome measure– Outcome reporter (self-report vs. parent-report)– Type of outcome construct (frequency vs. quantity of alcohol use)
• When modeling the effects of a covariate, ask if the effect of interest is between- or within-groups
![Page 19: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/19.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Practical Issues: Handling Covariates• In a standard meta-regression with independent effect sizes,
Tj = β0 + Xjβ1 + …
where Xj is length to follow-up, β0 and β1 can be interpreted as:– β0 = the average effect size when Xj = 0
• e.g. the average effect size in studies in which the intervention just occurred
– β1 = the effect of a 1-unit increase in Xj on Tj
• e.g. the effect size change associated with moving from a study in which the intervention just occurred to a study in which the effect size was measured at a 1 month posttest follow-up
![Page 20: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/20.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Practical Issues: Handling Covariates• In the correlated effects model, for a fixed study (j = 1), now
assume there are multiple outcomes. This study has its own regression equation:
Ti1 = β01 + Xi1β11 +…• The coefficients β01 and β11 can be interpreted as:
– β01 = the average effect size when Xi1 = 0 • e.g. the average effect size for units in the study (j = 1) when the
intervention just occurred– β11 = the effect of a 1-unit increase in Xi1 on Ti1
• e.g. the effect size change for units in the study (j = 1) at the time of intervention and at follow-up 1 month later
![Page 21: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/21.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Practical Issues: Handling Covariates• When using the RVE, these two types of regression occur in
one analysis:Within Group Tij = β0j + Xijβ2 + … Between Group
β0j = β0 + Xjβ1 + …
• These two regressions are combined into one analysis and model:
Tij = β0 + Xijβ2 + Xjβ1 + …
![Page 22: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/22.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Practical Issues: Handling Covariates• To properly separate estimation of within- and between-
group effects of covariates, use group mean centering:Xcij = Xij – Xj
where Xj is the mean value of Xij in group j (and where group is either study or cluster). So now,
Tij = β0 + Xcijβ2 + Xjβ1 + …
• If you don’t center Xij you are actually modeling a weighted combination of the within- and between-study effect, which is difficult to interpret
![Page 23: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/23.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Practical Issues: Handling Covariates• When using a covariate, ask if the effect of interest is
between- or within-groups• Make sure to group-center your within-group variables• Acknowledge that if only a few groups have variability in X ij
– Within-group estimate of β2 (associated with Xcij) will be imprecise (i.e. have a large standard error)
– The types of groups in which Xij varies may be different than (i.e. not representative of) groups in which Xij does not vary
![Page 24: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/24.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Calculating Robust Variance Estimates• Variables you will need in your dataset
– Group identifier (e.g., study/cluster identification number)– Effect size estimate– Variance estimate of the effect size– Any moderator variables or covariates of interest
• Additional pieces of information you will need to specify– In a correlated effects model: assumed correlation between all
pairs of effect sizes (ρ)– Fixed, random, hierarchical, or user-specified weights
![Page 25: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/25.jpg)
The Campbell Collaboration www.campbellcollaboration.org
• Stata ado file available at SSC archive: type ssc install robumeta
• SPSS macro available at:http://peabody.vanderbilt.edu/peabody_research_institute/methods_resources.xml
• R functions available at:http://www.northwestern.edu/ipr/qcenter/RVE-meta-analysis.html
Calculating Robust Variance Estimates
![Page 26: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/26.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata• Install robumeta.ado file from the Statistical Software
Components (SSC) archive
![Page 27: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/27.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata• Access example datasets and syntax/output files here:https://my.vanderbilt.edu/emilytannersmith/training-materials/
![Page 28: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/28.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata• robumeta.ado command structure
robumeta depvar [indepvars] [if] [in], study(studyid) variance(variancevar) weighttype(weightingscheme) rho(rhoval) [options]
![Page 29: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/29.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata• Example of a correlated effects model (correlated ε) • Fictional meta-analysis on the effectiveness of alcohol abuse
treatment for adolescents• Effect sizes represent post-treatment differences between
treatment and comparison groups on some measure of alcohol use (positive effect sizes represent beneficial treatment effects)– Number of effect sizes k = 172– Number of studies m = 39– Average number of effect sizes per study = 4.41
![Page 30: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/30.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata
![Page 31: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/31.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Intercept only model to estimate random-effects mean effect size with robust standard error, assuming ρ = .80
Robust Variance Estimation in Stata
![Page 32: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/32.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata
![Page 33: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/33.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata
![Page 34: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/34.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata• 4 moderators of interest: 2 vary within and between studies,
2 vary between studies only
![Page 35: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/35.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata• To model both the within- (Xcij) and between- effects (Xj) of
the type of alcohol outcome and follow-up time frame, create group mean and group mean centered variables
![Page 36: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/36.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata
![Page 37: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/37.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata• Let’s say we have a similar meta-analysis, but now need to
estimate a hierarchical model (correlated θ) • Effect sizes represent post-treatment differences between
treatment and comparison groups on some measure of alcohol use (positive effect sizes represent beneficial treatment effects)– Number of effect sizes k = 68– Number of research labs m = 15– Average number of effect sizes per research lab = 4.5
![Page 38: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/38.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata
τ2 – between lab variance component; ω2 between-study within-lab variance component𝑑=.25 , p=.001 ;95% CI (.12 ,.38)
![Page 39: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/39.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata• 5 moderators of interest: all vary within and between clusters
(research labs)
![Page 40: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/40.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Robust Variance Estimation in Stata• To model both the within- (Xcij) and between- effects (Xj) of
the covariates of interest, create group mean and group mean centered variables
![Page 41: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/41.jpg)
The Campbell Collaboration www.campbellcollaboration.org
![Page 42: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/42.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Conclusions & Recommendations• Robust variance estimation is one way to handle
dependencies in effect size estimates, and allows estimation of within- and between-study effects of covariates – Method performs well when there are 20 or more studies with
an average of 2 or more effect size estimates per study
• Choose the proper model for the type of dependencies in your data (correlated ε or correlated θ)
![Page 43: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/43.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Conclusions & Recommendations• When using the correlated effects model (correlated ε), with
efficient weights, if you have no information on ρ:– Use a sensitivity approach for estimating τ2
– Assume ρ = 1 in your weights, i.e.,
• For each covariate Xij in your model, remember that you can estimate:– Between-group effect: group mean (Xj)– Within-group effect: group mean centered variable (Xcij = Xij – Xj)
W 𝒊𝒋=1
{𝑘 𝑗 (𝑉 j+𝜏2 )}
![Page 44: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/44.jpg)
The Campbell Collaboration www.campbellcollaboration.org
Recommended Reading
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39-65.
![Page 45: Introduction to Robust Standard Errors](https://reader036.vdocuments.us/reader036/viewer/2022081420/5681623b550346895dd26e17/html5/thumbnails/45.jpg)
The Campbell Collaboration www.campbellcollaboration.org
P.O. Box 7004 St. Olavs plass0130 Oslo, Norway
E-mail: [email protected]://www.campbellcollaboration.org