Meta-Analysis of Correlated Data

Meta-Analysis of Correlated Data

Common Forms of Dependence

• Multiple effects per study– Or per research group!

• Multiple effect sizes using same control

• Phylogenetic non-independence

• Measurements of multiple responses to a common treatment

• Unknown correlations…

Multiple Sample Points per Study!

Study Experiment in Study Hedges D V Hedges D

Ramos & Pinto 2010 1 4.32 7.23

Ramos & Pinto 2010 2 2.34 6.24

Ramos & Pinto 2010 3 3.89 5.54

Ellner & Vadas 2003 1 -0.54 2.66

Ellner & Vadas 2003 2 -4.54 8.34

Moria & Melian 2008 1 3.44 9.23

Hierarchical Models

• Study-level random effect

• Study-level variation in coefficients

• Covariates at experiment and study level

Hierarchical Models• Random variation within study (j)

and between studies (i)

Tijij,ij2)

ijj,j2

j,2

Study Level Clustering

Hierarchical Partitioning of One Study

Grand Mean

Study Mean

Variation due to

Variation due to

Example: Data Set 1

Group Effect Variance

1 A 0.2 0.10

2 A 0.6 0.15

3 A 0.5 0.05

4 A 0.1 0.06

5 B 0.8 0.08

6 B 0.4 0.05

7 B 0.9 0.04

8 C 0.2 0.09

...

A Two-Step SolutionTijij,ij

2)

ijj,j2

j,2library(plyr)

data1_study <- ddply(data1, .(Group), function(adf){

mod <- rma(Effect, Variance, data=adf)

cbind(theta_j = coef(mod),

se_theta_j = coef(summary(mod))[1,2],

omega2 = mod$tau2)

})

A Two-Step Solution

Tijij,ij2)

ijj,j2

j,2

> data1_study

Group theta_j se_theta_j omega2

1 A 0.3312500 0.1369306 0.00000000

2 B 0.7005364 0.1654476 0.02854676

3 C 0.6788453 0.1987595 0.17151248

4 D 0.7836646 0.2677693 0.26470540

5 E 0.8552760 0.1556476 0.14561528j j

A Two-Step Solution

Tijij,ij2)

ijj,j2

j,2

> rma(theta_j, I(se_theta_j^2), data=data1_study)

Random-Effects Model (k = 5; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0.0272 (SE = 0.0414)

...

estimate se zval pval ci.lb ci.ub

0.6472 0.1087 5.9545 <.0001 0.4342 0.8603 ***

2

Multiple Effects per Research Group

Solutions to Multiple Hierarchies

• Multiple-Step Meta-analyses

• Multi-level hierarchical model fits– Better estimator– Accommodates more complex data

structures–May need to go Bayesian (don't be scared!)

• Model correlation…

Common Forms of Dependence

• Multiple effects per study– Or per research group!

• Multiple effect sizes using same control

• Phylogenetic non-independence

• Measurements of multiple responses to a common treatment

• Unknown correlations…

Multiple Effect Sizes with Common Control

Effect of each treatment calculated using same control!

The Control Keeps Showing Up!

• nc and sdc are going to be the same for all treatments

• Effect sizes will covary

Calculating Covariance

Formulae available or derivable for all effect sizes

A Mixed Effect Group Model

• Group means, random study effect, and then everything else is error

Tiim,i2)

where

imm,2

A Mixed Effect Group Model

• Group means, random study effect, and then everything else is error

TiMVNi,i)

where

iMVN Xi, 2

What are i and i?

i =

i=

TiMVNi,i)

What about the treatment effects?

Xi =

i= iMVN Xi, 2

What if treatments are correlated?

i =

TiMVNi,i)

Why does covariance matter?

x-y =

x + y + 2

x,y

• In asking if two treatments differ, cov helps tighten confidence intervals

• High cov more weight for a study as treatments share information

Multiple Treatments

study trt m1i m2i sdpi n1i n2i

1 1 1 7.87 -1.36 4.2593 25 25

2 1 2 4.35 -1.36 4.2593 22 25

3 2 1 9.32 0.98 2.8831 38 40

4 3 1 8.08 1.17 3.1764 50 50

5 4 1 7.44 0.45 2.9344 30 30

6 4 2 5.34 0.45 2.9344 30 30CommonControl!

http://www.metafor-project.org/doku.php/analyses:gleser2009

Calculating the Variance/Covariance Matrix

[,1] [,2] [,3] [,4] [,5] [,6]

[1,] 0.113 0.060 0.000 0.000 0.000 0.000

[2,] 0.060 0.098 0.000 0.000 0.000 0.000

[3,] 0.000 0.000 0.105 0.000 0.000 0.000

[4,] 0.000 0.000 0.000 0.064 0.000 0.000

[5,] 0.000 0.000 0.000 0.000 0.098 0.055

[6,] 0.000 0.000 0.000 0.000 0.055 0.082

http://www.metafor-project.org/doku.php/analyses:gleser2009

Fitting a Model with a VCOV Matrix

> rma.mv(yi ~ factor(trt)-1,

V,

random =~ 1|study,

data=dat)

Comparison to No Correlation Model

With correlation estimate se zval pval ci.lb ci.ub

factor(trt)1 2.3796 0.1641 14.4984 <.0001 2.0579 2.7013

factor(trt)2 1.5784 0.2007 7.8662 <.0001 1.1851 1.9716

Without correlation estimate se zval pval ci.lb ci.ub

factor(trt)1 2.3759 0.1511 15.7196 <.0001 2.0797 2.6722

factor(trt)2 1.5177 0.2125 7.1405 <.0001 1.1011 1.9343

Common Forms of Dependence

• Multiple effects per study– Or per research group!

• Multiple effect sizes using same control

• Phylogenetic non-independence

• Measurements of multiple responses to a common treatment

• Unknown correlations…

Effect Size on Related Organisms Not Independent

Warming onLitterfall

Pine TreesRedwoodsFir Trees

Oak Trees{

Phylogenetic Distances Determines Covariances for

Weights

What about Multiple Studies of Some Species?

Common Forms of Dependence

• Multiple effects per study– Or per research group!

• Multiple effect sizes using same control

• Phylogenetic non-independence

• Measurements of multiple responses to a common treatment

• Unknown correlations…

Common Treatments

Treatment

Response 1 Response 2 Response 3

Common Treatments

CO2

CO2 Assimilation

GS

Stomatal Conductance

PN

Correlation Between Responses

What does Correlation between effects mean?

Xi =

i= iMVN Xi, 2

What Do We Do?1. Create a 'composite' measure

– Average

– Weighted Average

2. Estimate different coefficients directly

3. Robust Variance Estimation (RVE)

The CO2 Effect Data

experiment Paper Measurement Hedges Var

1 1 121 GS -0.4862 0.3432

2 1 121 PN 0.9817 0.3735

3 2 121 GS 0.1535 0.3343

4 2 121 PN 2.0668 0.5113

5 3 121 GS 0.0965 0.3337

6 3 121 PN 2.6101 0.6172

7 4 121 GS 0.0000 0.2857

8 4 121 PN 3.6586 0.7638

9 5 168 GS -1.5271 0.4305

10 5 168 PN 1.8355 0.4737

Direct Estimation

rma.mv(Hedges ~ Measurement,

Var,

random =~ Measurement|Paper,

data=co2data,

struct="HCS")

and Different Correlation Structures

• Different structures for different data

• We do not always know which one is correct!

Estimates of Variance, Covariance

Multivariate Meta-Analysis Model (k = 68; method: REML)

Variance Components:

outer factor: Paper (nlvls = 18)

inner factor: Measurement (nlvls = 2)

estim sqrt k.lvl fixed level

tau^2.1 4.5098 2.1236 34 no GS

tau^2.2 3.5799 1.8921 34 no PN

rho 0.4751 no

Disadvantages to Multivariate Meta-Analysis

1. Difficult to estimate with few studies

2. Additional assumptions of covariance structure

3. Often little improvement over univariate meta-analysis

4. Publication bias exacerbated if data not missing at random

Jackson et al. 2011 Satist. Med.

Robust Variance Estimation

• Essentially, bound weights within a group j to 1/mean varj and assume a value of

– Test sensitivity to choice of

– Correct DF for small sample sizes

• Methods developed by Hedges, Tipton, and others

• robumeta package in R

robumeta & RVE

library(robumeta)

robu(Hedges ~ Measurement, data=co2data,

studynum=Paper,

var.eff.size=Var)

RVE: Correlated Effects Model with Small-Sample Corrections

Model: Hedges ~ Measurement

Number of studies = 18

Number of outcomes = 68 (min = 2 , mean = 3.78 , median = 4 , max = 10 )

Rho = 0.8

I2 = 85.59992

Tau.Sq = 2.561661

Struct="CS" only so far

Often, Choice of Matters Little

> sensitivity(co2modRVE)

Type Variable rho=0 rho=0.2 rho=0.4 rho=0.6 rho=0.8 rho=1

1 Estimate intercept 0.00454 0.00457 0.00459 0.00462 0.00464 0.00467

2 - MeasurementPN 1.03149 1.03139 1.03128 1.03118 1.03107 1.03097

3 Std. Err. intercept 0.51173 0.51179 0.51185 0.51192 0.51198 0.51204

4 - MeasurementPN 0.61984 0.61990 0.61996 0.62003 0.62009 0.62015

5 Tau.Sq - 2.55334 2.55542 2.55750 2.55958 2.56166 2.56374

Results May Differ…

Multivariate Meta-AnalysisModel Results:

estimate se zval pval ci.lb ci.ub

intrcpt -0.0503 0.5221 -0.0963 0.9233 -1.0735 0.9730

MeasurementPN 1.0579 0.5359 1.9742 0.0484 0.0076 2.1082 *

Robust Variance EstimationModel Results:

Estimate StdErr t-value df P(|t|>) 95% CI.L 95% CI.U Sig

1 intercept 0.00464 0.512 0.00907 16.7 0.993 -1.077 1.09

2 MeasurementPN 1.03107 0.620 1.66278 16.7 0.115 -0.279 2.34

Other Sources of Unknown Correlations

• Shared system types

• Shared environmental events

• Labs or investigators

• Re-sampling experiments

• Experiments repeated in a region

• More…

Why Model Correlation instead of Hierarchy?

• Depends on question

• Analytical difficulty

• Leveraging correlation to aid with missing data