(3) identifying effect size (es) for each study. overview general information to keep in mind: the...

(3) Identifying Effect Size (ES) for each study

Overview General Information to keep in mind:

The goal is to convert each study to a single effect size (ES) number that you type into the excel file

Sometimes calculating the ES will be straightforward and easy

Sometimes the study will not provide detailed information to calculate the ES so we will have to make decisions (guesstimates) to arrive at best guess

Keep a record of every decision you make (in excel file) because we will be reporting that information (and last few slides of PPT explains some of the decisions

Three Preliminary Steps: (1) convert to “r” or “d” ?View #1 Empirical research can take many forms (e.g., IV can be

dichotomous and/or continuous IV, DV can be dichotomous and/or continuous DV, two

variables relationships, etc) and the form of research you are analyzing helps determine which metric may be best:

r – Correlation Coefficient - Johnson & Eagly, 2000 suggest using r when the studies composing the meta- analysis primarily report the correlation between variables,

d – Standardized Difference - Johnson & Eagly, 2000 suggest using d when the studies composing the meta-analysis primarily report ANOVAs, t-tests between groups.

Three Preliminary Steps: (1) convert to “r” or “d” ?View #2 Rosenthal & DiMatteo, 2001 discussion the

advantages of using r over d: Primary studies contain both “correlation”-based and

“between-group”-based studies, so how to choose? Converting ds to rs sometimes loses information but

converting rs to ds does not. r is more easily interpreted. (FYI – flip side is that d is

harder to interpret and always a larger number than r)

Three Preliminary Steps: (2) which software? You need to decide:

Free versus $? Does it have the functions you want/need? How long to learn how to use it?

If have money/time: “Comprehensive Meta Analysis”

If don’t have money/time: Wilson (Practical Meta Analysis) website

See for links to 11 different types of statistical software - http://www.psychwiki.com/wiki/Meta-analysis

Three Preliminary Steps: (3) direction of effect? You must decide ahead of time what

constitutes a positive ES and negative ES based upon direction of effect

e.g., imagine a two-group study that manipulates happiness and sadness; if ES is positive, does that mean more happiness (?) or more sadness (?).

Identifying ES for each study

(1) Download “ES Calculator” (see our website)

Identifying ES for each study

(2) For each study, identify data from “Results”

Identifying ES for each study

(3) Find corresponding data from “Calculator”

Identifying ES for each study

(4) Input information to get ES

Identifying ES for each study

(5) Put ES into excel file

Two independent raters of ES Double-checking

Two raters of each effect size Each rates independently of the other Compare and resolve discrepancies You will find that rating ES is an iterative process in

which you will double-check your own findings over and over again as you learn more information and become more adept at the process

Issues about identifying ES

Overarching goal: “A conservative estimate of effect size was used

whenever studies did not report specific statistics.”

Issues about identifying ES

What if… p-value is reported only as “p < .05”

Solution: treat as if p = .05

Issues about identifying ES

What if… only reports that it was “non-significant”

Solution: treat as ES = 0

Issues about identifying ES

What if… only reports grand N, not n for conditions?

Solution: treat as equal sample size in each condition(s)

Issues about identifying ES

What if… does not provide enough info to calculate ES?

Solution: Could try contacting author, but unlikely to work Keep a record of which studies don’t contain

sufficient information because will report that in Method section such as: “Studies were excluded if they did not contain enough statistical information to calculate an effect size (citation, citation, citation, citation, etc)”