the effect size
DESCRIPTION
The Effect Size. The effect size (ES) makes meta-analysis possible The ES encodes the selected research findings on a numeric scale There are many different types of ES measures, each suited to different research situations Each ES type may also have multiple methods of computation. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/1.jpg)
The Effect Size• The effect size (ES) makes meta-
analysis possible• The ES encodes the selected research
findings on a numeric scale• There are many different types of ES
measures, each suited to different research situations
• Each ES type may also have multiple methods of computation
Practical Meta-Analysis -- D. B. Wilson
![Page 2: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/2.jpg)
Examples of Different Types of Effect Sizes (ES)
• Standardized mean difference– Group contrast research
• Treatment groups• Naturally occurring groups
– Inherently continuous construct• Odds-ratio (OR)
– Group contrast research• Treatment groups• Naturally occurring groups
– Inherently dichotomous construct• Correlation coefficient
– Association between variables researchPractical Meta-Analysis -- D. B. Wilson
![Page 3: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/3.jpg)
Examples of Different Types of Effect Sizes
• Risk-ratio or Relative Risk (RR)– Group differences research
(naturally occurring groups)– Commonly used by epidemiologist and
medical meta-analyses– Inherently dichotomous construct– Easier to interpret than the odds-ratio
(OR)– Does not overstate ES like OR does.
Practical Meta-Analysis -- D. B. Wilson
![Page 4: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/4.jpg)
Examples of Different Types of Effect Sizes
• Proportion– Central tendency research
• HIV/AIDS prevalence rates• Proportion of homeless persons found to be
alcohol abusers• Standardized gain score
– Gain or change between two measurement points on the same variable
• Cholesterol level before and after completing a therapy
• Others?Practical Meta-Analysis -- D. B. Wilson
![Page 5: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/5.jpg)
What Makes Something an Effect Size
for Meta-analytic Purposes• The type of ES must be comparable across
the collection of studies of interest• This is generally accomplished through
standardization• Must be able to calculate a standard error
for that type of ES– The standard error is needed to calculate the
ES weights, called inverse variance weights (more on this later)
– All meta-analytic analyses are weighted “averages” (simple example on next slide)
Practical Meta-Analysis -- D. B. Wilson
![Page 6: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/6.jpg)
Weighted “Averages”
Practical Meta-Analysis -- D. B. Wilson
Example: Calculating GPA’s
![Page 7: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/7.jpg)
Weighted “Averages”
Practical Meta-Analysis -- D. B. Wilson
Example: Calculating GPA’s
![Page 8: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/8.jpg)
Weighted “Averages”
Practical Meta-Analysis -- D. B. Wilson
Example: Calculating GPA’s
![Page 9: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/9.jpg)
Weighted “Averages”
Practical Meta-Analysis -- D. B. Wilson
Example: Calculating GPA’s
![Page 10: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/10.jpg)
The Standardized Mean Difference
• Represents a standardized group contrast on an inherently continuous measure
• Uses the pooled standard deviation (some situations use control group standard deviation)
• Commonly called “d” or occasionally “g”• Cohen’s d (see separate short lecture on Cohen’s d)
Practical Meta-Analysis -- D. B. Wilson
pooled
GG
sXXES 21
2
11
21
2221
21
nn
nsnsspooled
![Page 11: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/11.jpg)
The Correlation Coefficient (r)
• Represents the strength of linear association between two inherently continuous measures
• Generally reported directly as “r” (the Pearson product moment correlation)
Practical Meta-Analysis -- D. B. Wilson
rES
![Page 12: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/12.jpg)
The Odds-Ratio (OR)• Recall, the odds-ratio is based on a 2 by 2
contingency table, such as the one below
Practical Meta-Analysis -- D. B. Wilson
Frequencies
Success Failure
Treatment Group a b
Control Group c dbcadES
o The Odds Ratio (OR) is the odds for “success” in the treatment group relative to the odds for “success” in the control group.
o OR’s can also come from results of logistic regression analysis, but these would difficult to use in a meta-analysis due to model differences.
![Page 13: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/13.jpg)
Relative Risk (RR)• The relative risk (RR) is also based on
data from a 2 by 2 contingency table, and is the ratio of the probability of success (or failure) for each group
Practical Meta-Analysis -- D. B. Wilson
ESa a bc c d
/ ( )/ ( )
![Page 14: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/14.jpg)
Unstandardized Effect Size Metric
• If you are synthesizing a research domain that using a common measure across studies, you may wish to use an effect size that is unstandardized, such as a simple mean difference.
• Multi-site evaluations or evaluation contracted by a single granting agency.
Practical Meta-Analysis -- D. B. Wilson
![Page 15: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/15.jpg)
Effect Size Decision Tree for Group Differences Research (from Wilson & Lipsey)
Practical Meta-Analysis
- Wilson & Lipsey
All dependent variables areinherently dichotomous
All dependent variables areinherently continuous
All dependent variables measuredon a continuous scale
All studies involve samemeasure/scale
Studies use differentmeasures/scales
Some dependent variables measuredon a continuous scale, someartificially dichotomized
Some dependent variables areinherently dichotomous, some areinherently continuous
Difference between proportionsas ES [see Note 1]
Odds ratio; Log of the odds ratioas ES
Unstandardized mean differenceES
Standardized mean differenceES
Standardized mean difference ES;those involving dichotomiescomputed using probit [see Note 2]or arcsine [see Note 3]
Do separate meta-analyses fordichotomous and continuousvariables
Gro
up c
ontra
st o
n de
pend
ent v
aria
ble
![Page 16: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/16.jpg)
Methods of Calculating the Standardized Mean Difference
• The standardized mean difference probably has more methods of calculation than any other effect size type.
Practical Meta-Analysis -- D. B. Wilson
![Page 17: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/17.jpg)
Degrees of Approximation to the ES ValueDepending of Method of Computation
– Direct calculation based on means and standard deviations
– Algebraically equivalent formulas (t-test)– Exact probability value for a t-test– Approximations based on continuous data (correlation
coefficient)
– Estimates of the mean difference (adjusted means, regression b weight, gain score means)
– Estimates of the pooled standard deviation (gain score standard deviation, one-way ANOVA with 3 or more groups, ANCOVA)
– Approximations based on dichotomous data
Practical Meta-Analysis -- D. B. Wilson
Gre
atG
ood
Poor
![Page 18: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/18.jpg)
Methods of Calculating the Standardized Mean Difference
Practical Meta-Analysis -- D. B. Wilson
pooledsXX
nnnsns
XXES 21
21
2221
21
21
2)1()1(
Direction Calculation Method
(Independent Samples)
![Page 19: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/19.jpg)
Methods of Calculating the Standardized Mean Difference
Practical Meta-Analysis -- D. B. Wilson
21
21
nnnntES
Algebraically Equivalent Formulas:
21
21 )(nnnnFES
independent samples t-test
two-group one-way ANOVA
Exact p-values from a t-test or F-ratio can be convertedinto t-value and the above formula applied.
(Independent Samples)
![Page 20: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/20.jpg)
Methods of Calculating the Standardized Mean Difference
Practical Meta-Analysis -- D. B. Wilson
A study may report a grouped frequency distribution from which you can calculate means and standard deviations and apply to direct calculation method.
𝑋=∑𝑖=1
𝑘
𝑓 𝑖 𝑋 𝑖
∑𝑖=1
𝑘
𝑓 𝑖
𝑠2=∑𝑖=1
𝑘
𝑓 𝑖 ( 𝑋 𝑖− 𝑋 )2
∑𝑖=1
𝑘
𝑓 𝑖−1
![Page 21: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/21.jpg)
Methods of Calculating the Standardized Mean Difference
Practical Meta-Analysis -- D. B. Wilson
212r
rES
Close Approximation Based on Continuous Data:Point-Biserial Correlation - For example, the correlationbetween treatment/no treatment and outcome measuredon a continuous scale.
Point-Biserial Correlation: Pearson’s Product Moment Correlation (r) between the response (Y) and group indicator (X) coded as: Group 1 = 0, Group 2 = 1 and treated as a numeric variable.
(Independent Samples)
![Page 22: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/22.jpg)
Methods of Calculating the Standardized Mean Difference
Estimates of the Numerator of ES – The Mean Difference
– difference between covariance adjusted means
– unstandardized regression coefficient (b) for group membership
Practical Meta-Analysis -- D. B. Wilson
(Independent Samples)
![Page 23: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/23.jpg)
Methods of Calculating the Standardized Mean Difference
Estimates of the Denominator of ES -Pooled Standard Deviation
error
between
pooledMS
FMSs
1
)( 22
k
nnX
nXMS j
jjjj
between
one-way ANOVA more than 2 groups
should be found in an ANOVA table in the paper
(Independent Samples, more than two groups)
![Page 24: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/24.jpg)
Methods of Calculating the Standardized Mean Difference
Practical Meta-Analysis -- D. B. Wilson
Estimates of the Denominator of ES - Standard Deviation of the Paired Differences (Gain Scores)
1 nSEsd
SE = standard error of the mean the paired differences
Paired difference between scores = gain scoresGain = pre-test – post-test (or vise versa)
![Page 25: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/25.jpg)
Methods of Calculating the Standardized Mean Difference
Practical Meta-Analysis -- D. B. Wilson
Estimates of the Denominator of ES --Pooled Standard Deviation
)1(2 rs
s gainpooled
standard deviation of gain scores, where r is the correlation between pretest and posttest scores
![Page 26: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/26.jpg)
Methods of Calculating the Standardized Mean Difference
Practical Meta-Analysis -- D. B. Wilson
Estimates of the Denominator of ES --Pooled Standard Deviation
21
1 2
error
errorerrorpooled df
dfr
MSsANCOVA, where r is the correlation between the covariate and the dependent variable.
![Page 27: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/27.jpg)
Methods of Calculating the Standardized Mean Difference
Practical Meta-Analysis -- D. B. Wilson
Estimates of the Denominator of ES --Pooled Standard Deviation
WABB
WABBpooled dfdfdf
SSSSSSs
A two-way factorial ANOVAwhere B is the irrelevant factorand AB is the interactionbetween the irrelevant factorand group membership (factor A).
![Page 28: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/28.jpg)
Methods of Calculating the Standardized Difference between Two Proportions
Practical Meta-Analysis -- D. B. Wilson
Approximations Based on Dichotomous Data
)()(21 groupgroup pprobitpprobitES
- the difference between the probits transformation of the proportion successful in each group
- converts proportion into a z-value
![Page 29: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/29.jpg)
Methods of Calculating the Standardized Mean Difference
Practical Meta-Analysis -- D. B. Wilson 29
Approximations Based on Dichotomous Data
ratioOddsES log3
this represents the rescaling of the logged odds-ratio (see Sanchez-Meca et al 2004 Psychological Methods article)
![Page 30: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/30.jpg)
Methods of Calculating the Standardized Mean Difference
Practical Meta-Analysis -- D. B. Wilson
Approximations Based on Dichotomous Data
2
2
2
N
ESchi-square must be based ona 2 by 2 contingency table(i.e., have only 1 df)
212rrES
phi coefficient
![Page 31: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/31.jpg)
Practical Meta-Analysis -- D. B. Wilson LINKED TO LECTURE SECTION OF COURSE WEBSITE
![Page 32: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/32.jpg)
Practical Meta-Analysis -- D. B. Wilson
![Page 33: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/33.jpg)
Formulas for the Correlation Coefficient (r)
• Results typically reported directly as a correlation.
• Any data for which you can calculate a standardized mean difference effect size, you can also calculate a correlation type effect size.
Practical Meta-Analysis -- D. B. Wilson
![Page 34: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/34.jpg)
Formulas for the Odds Ratio
• Results typically reported in one of three forms:– Frequency of successes in each group– Proportion of successes in each group– 2 by 2 contingency table
Practical Meta-Analysis -- D. B. Wilson
![Page 35: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/35.jpg)
Data to Code Along With the ES• The effect size (ES)
– May want to code the statistics from which the ES is calculated– Confidence in ES calculation– Method of calculation– Any additional data needed for calculation of the inverse
variance weight
• Sample size• ES specific attrition• Construct measured• Point in time when variable measured• Reliability of measure• Type of statistical test used
Practical Meta-Analysis -- D. B. Wilson
![Page 36: The Effect Size](https://reader036.vdocuments.us/reader036/viewer/2022062411/56816703550346895ddb68ba/html5/thumbnails/36.jpg)
Issues in Coding Effect Sizes
• Which formula to use when summary statistics are available for multiple formulas
• Multiple documents/publications reporting the same data (not always in agreement)
• How much guessing should be allowed– sample size is important but may not be
presented for both groups– some numbers matter more than others
Practical Meta-Analysis -- D. B. Wilson