defining and evaluating ‘study quality’
Post on 22-Feb-2016
56 Views
Preview:
DESCRIPTION
TRANSCRIPT
Defining and Evaluating ‘Study Quality’
Luke PlonskyCurrent Developments in
Quantitative Research MethodsLOT Winter School, 2014
Study Quality Matters?Building theory (or a house)
Studies = 2x4s, bricks, etc.Self-evident?
Rarely discussed in linguistics researchBut lack of attention to quality ≠ low quality
Implication: Study quality needs to be examined, not assumed
YES!
Defining ‘Study Quality’How was SQ defined in Plonsky & Gass (2011) and
Plonsky (2013)?How was SQ operationalized?Do you agree with this definition &
operationalization?Now consider your (sub-)domain of interest
How would you operationalize SQ?How would you weight or prioritize different
features?
Missing data
Data Type Primary Secondary / Meta-AnalyticSDs - Sample variability - Calculate ESs (d)
exclusion?Reliability - Small effects due to
treatment or dependent measure?- Inform instrument design
- Adjust ESs for attenuation
Effect sizes
- Interpret magnitude of effects- Future power analyses
- Compare/combine results- Power for moderator analysis
LIMITED INFLUENCE ON L2 THEORY, PRACTICE, AND
FUTURE RESEARCH INEFFICIENT
Sources/Considerations for an Instrument to Measure Study Quality?
1. (Over 400) Existing measures of study quality from the meta-analysis literature (usually for weighting Ess) (e.g., sample: Valentine & Cooper, 2008—Table 2)
2. Societal guidelines (e.g., APA, APS, sample: JARS Working Group—Table 1, 2008, AERA 2006 reporting standards, LSA??, AAAL/AILA??)
3. Journal guidelines (e.g., Chapelle & Duff, 2003)
4. Methodological syntheses from other social sciences (e.g., Skidmore & Thompson, 2010)
5. Previous reviews / meta-analyses (e.g., Chaudron, 2001; Norris & Ortega, 2000; Plonsky, 2011)
6. Methods/stats textbooks (Larson-Hall, 2010; Porte, 2010)
7. Others?
(Only) Two studies in this area to address study quality empirically
Plonsky & Gass (2011); Plonsky (2013, in press)
Rationale & MotivationsStudy quality needs to be measured, not assumedConcerns expressed about research and reporting
practices“Respect for the field of SLA can come only
through sound scientific progress” (Gass, Fleck, Leder, & Svetics, 1998)
No previous reviews of this nature
Plonsky & Gass (2011) & Plonsky (2013)
Two common goals: 1. Describe and evaluate quantitative research
practices2. Inform future research practices
Methods(very meta-analytic but focus on methods rather than substance/effects/outcomes)
Plonsky & Gass (2011) Domain: Interactionist L2
research; quantitative onlyAcross 16 journals & 2 books
(all published, 1980-2009)K = 174Coded for: designs, analyses,
reporting practicesAnalyses: frequencies/%s
Plonsky (2013)Domain: all areas of L2
research; quantitative onlyTwo journals: LL & SSLA (all
published, 1990-2010)K = 606Coded for: designs, analyses,
reporting practices (sample scheme)
Analyses: frequencies/%sHow would you define your domain? Where would
you search for primary studies?
RESULTS
Results: Designs
Major Designs across Research SettingsPlonsky (2013) P&G
(2011)Design Class Lab (all)Observational 20% 80% 65Experimental 45% 55% 35
Results: Designs
SamplesStudy Average
nTotal N Groups
P&G (2011) 22 7,951 365Plonsky (2013)
19 181,255 1,732
Results: Designs
Plonsky (2013) P&GFeature Class Lab AllRandom assign.
23% 48% 32%
Ctrl/Comp group
90% 84% 55%
Pretest 78% 59% 39%Delayed posttest
50% 29% 79%
Results: AnalysesAnalysis P&G (2011)
%P (2013)
%ANOVA 54 56t test 69 43Correlation 18 31Chi-square 50 19Regression 8 15MANOVA 7 7ANCOVA 7 5Factor analysis 2 5SEM - 2Other - 7Nonparametrics
- 5
Results: Analyses
P&G (2011) %
P (2013) %
Zero 6 12One 32 28Multiple 62 60
Number of Unique Statistical Analyses Used in L2 Research
M35
SD64
95% CIs
30-40
Median18
Tests of Statistical Significance in L2 ResearchPlonsky (2013)
Results: Descriptive Statistics
Item P&G (2011) %
P (2013) %
Percentage 62 68Frequency 71 48Correlation - 30Mean 64 77Standard deviation
52 60
Mean without SD - 31Effect size 18 26Confidence interval
3 5
Plonsky (2013)
Results: Inferential StatisticsItem P&G (2011)
%P (2013)
%F 26 61t 32 36x2 - 17p = 44 49p < or > 61 80p either = or </> - 44p = and p < or > - 42ANOVA / t test without M - 20ANOVA / t test without SD - 35ANOVA / t test without f or t - 24
Results: Other Reporting Practices
Item P&G (2011) %
P (2013) %
RQs or hypotheses - 80Visual displays of data
- 53
Reliability 64 45Pre-set alpha 25 22Assumptions checked 3 17Power analysis 2 1
Plonsky (2013)
?
Studies excluded due to missing data (usually SDs) (as % of meta-analyzed sample)
Li (2010)Pennock-Roman & Rivera (2011)
Bowles (2010)Kieffer et al. (2009)
Abraham (2008)Goldschnieder & DeKeyser (2003)
Norris & Ortega (2000)Plonsky (2011)
Wa-Mbaleka (2006)Lin et al. (2013)
Grgurović et al. (2013)Biber et al. (2011)
Russell & Spada (2006)Nekrasova & Becker (2009)
Dinsmore (2006)Wu (1991)
0 40 80 120 160 200 240 280
6212527
364249
595960
81104110
119194
300
MEDIAN
Median K = 16 (Plonsky & Oswald, under review)
Data missing in meta-analyzed studies (as % of total sample)
Nekrasovia & Becker (2009)
Norris & Ortega (2000)
Plonsky & Gass (2011)
Plonsky (in press)
Keck et al. (2006)
Wang (2010)
Lee & Huang (2008)
0 10 20 30 40 50 60 70 80 90100
14
18
20
29
24
12
35
29
6
17
24
49
19
Missing test statMissing SDMissing M
Reporting of reliability coefficients (as % of meta-analyzed sample)
Nekrasova & Becker (2009)
Mackey & Goo (2007)
Norris & Ortega (2000)
Russell & Spada (2006)
Jeon & Kaya (2006)
Plonsky (2011)
Ziegler (2013)
Plonsky (in press)
Adesope et al. (2010)
Adesope et al. (2011)
Plonsky & Gass (2011)
0 25 50 75 100
67
1620
3841434546
5064
Keck et al. (2006)
Norris & Ortega (2000)
Wang (2010)
Mackey & Goo (2007)
Plonsky & Gass (2011)
Plonsky (in press)
Ziegler (2013)
0 10 20 30 40 50 60 70 80 90 100
0
6
11
14
18
26
36
0
3
0
3
5
0
CIES
Reporting of effect sizes & CIs (as % of meta-analyzed sample)
Other data associated with quality/transparency and recommended or required by APA (as % of meta-analyzed sample)
Plonsky & Gass (2011)
Plonsky (in press)
Mackey & Goo (2007)
Nekrasovia & Becker (2009)
Norris & Ortega (2000)
Ziegler (2013)
Keck et al. (2006)
0 10 20 30 40 50 60 70 80 90 100
25
22
25
20
29
38
3
17
36
2
1
7
8053
34
56
PowerAssumptionsPre-set alphaVisualsRQs
Elsewhere in the social sciences…Kesselman et al. (1998-a)
Kesselman et al. (1998-c)
Kesselman et al. (1998-b)
Kieffer et al. (2001-a)
Kesselman et al. (1998-d)Bangert & Baumberger
(2005)Thompson & Snyder (1997)
Kieffer et al. (2001-b)
Vacha-Haase et al. (1999)
Willson (1980)Sedlmeier & Gigerenzer
(1989)Cashen & Geiger (2004)
0 10 20 30 40 50 60 70 80 90 100
7
7
10
18
24
43
64
72
61
15
9
33
3
0
0
1
0.003
0
7
0
0.003
11
39
36
37
PowerAssumptionsReliabilityCIsESs
Results: Changes over timeMeara (1995): “[When I was in graduate school], anyone who could explain the difference between a one-tailed and two-tailed test of significance was regarded as a dangerous intellectual; admitting to a knowledge of one-way analyses of variance was practically the same as admitting to witchcraft in 18th century Massachusetts” (p. 341).
Changes Over Time: DesignsPlonsky & Gass (2011) Plonsky (in press)
Changes Over Time: DesignsPlonsky (in press)
Changes Over Time: AnalysesPlonsky & Gass (2011)
Changes Over Time: AnalysesPlonsky (in press)
Changes Over Time: Reporting Practices
Plonsky & Gass (2011)
Changes Over Time: Reporting Practices
Plonsky (in press)
Relationship between quality and outcomes?
Plonsky (2011)
Plonsky & Gass (2011): larger effects for studies that include delayed posttests
Discussion (Or: So what?)General:Few strengths and numerous methodological weaknesses are
present—common even—in quantitative L2 researchQuality (and certainly methodological features) vary across
subdomains AND over time.Possible relationship between methodological practices and the
outcomes they produce.Three common themes:
Means-based analyses Missing data, NHST, and the ‘Power Problem’ Design Preferences
Discussion: Means-based analyses
ANOVAs, t tests dominate, increasinglyNot problematic as long as
Assumptions checked (17% of Plonsky, 2013)Data are reported thoroughlyTest are most appropriate for RQs (i.e., not default)
Benefits to increased regression analyses (see Cohen, 1968)Less categorization of continuous variables (e.g., proficiency,
working memory) to use ANOVA loss of variance!More precise results (R2s +informative than an overall p or
eta2)Fewer tests preservation of experiment-wise power
Discussion: Missing data, NHST, & PowerIn general: lots of missing and inconsistently reported data!
BUT We’re getting better!The “Power Problem”
Small samples Heavy reliance on NHSTEffects not generally very largeOmission of non-statistical results inflated summary resultsRarely check assumptionsRarely use multivariate statisticsRarely analyze power
Discussion: Design PreferencesSigns of domain maturity?
+classroom-based studies +experimental studies+delayed posttests
Discussion-SummaryCauses/explanations
- Inconsistencies among reviewers- Lack of standards- Lack of familiarity with design and appropriate data analysis and reporting- Inadequate training (Lazaraton et al., 1987)- Non-synthetic-mindedness- Publication bias
Effects• Limited
interpretability• Limited meta-
analyzability• Overestimation of
effects• Overreliance on p
values
S l o w e r P r o g r e s s
Study Quality in Secondary/Meta-analytic Research?
IntroM-As = high visibility and impact on theory and practice
quality is criticalSeveral instruments proposed for assessing M-A quality
Stroup et al. (2000)Shea et al. (2007) JARS/MARS (APA, 2008)Plonsky (2012)
Plonsky’s (2012) Instrument for Assessing M-A Quality
Goal 1: Assess transparency and thoroughness as a means to Clearly delineate the domain under investigation Enable replication Evaluate the appropriateness of the methods in
addressing/answering the study’s RQs
Goal 2: Set a tentative, field-specific standard Inform meta-analysts and reviewers/editors of M-As
Organization: Lit review/intro Methods Discussion
What items would you include?
Plonsky’s (2012) Instrument for Assessing M-A Quality—Section I
Com
bine
?
Plonsky’s (2012) Instrument for Assessing M-A Quality—Section II
Plonsky’s (2012) Instrument for Assessing M-A Quality—Section III
Looking FORWARDRecommendations for:- Individual researchers- Journal editors- Meta-researchers- Researcher trainers- Learned societies
Consider power before AND after a study (but especially before)p is overrated (meaningless?) especially when working with (a)
small samples, (b) large samples, (c) small effects, (d) large effectsReport and interpret data thoroughly (EFFECT SIZES!)Consider regression and multivariate analysesCalculate and report instrument reliabilityTeam up with an experimental (or observational) researcherDevelop expertise in one or more novel (to you) methods/analyses
Love,Luke
Dear individual researchers,
Dear journal editors,Use your influence to improve rigor, transparency, and
consistency It’s not enough to require reporting (of…ES, SDs, reliability
etc.) – interpretation too! Develop field-wide and field-specific standards Include special methodological reviews (see Magnan, 1994)Devote (precious) journal space to methodological discussions
and reports
Love,
Luke
Dear meta-researchers, Use your voice! Guide interpretations of effect sizes in your domains Evaluate and make knows methodological strengths, weaknesses,
and gaps; encourage effective practices and expose weak ones Don’t just summarize Explain variability in effects, not just means (e.g., due to small
samples, heterogeneous samples or treatments) Examine substantive and methodological changes over time and as
they related to outcomes Cast the net wide in searching for primary studies
Love,
Luke
Dear researcher trainers,Lots of emphasis on the basics: descriptive statistics,
sample size+power+effect size+p; synthetic approach, ANOVA
Encourage more specialized courses, in other departments if necessary
Love,
Luke
Dear learned societies (AILA/AAAL, LSA, etc.),
To Learned Societies (AILA, AAAL, LSA, etc.)Designate a task force or committee to establish field-
specific standards for research and reporting practices:(a) at least one member of the executive committee, (b) members from the editorial boards of relevant
journals, (c) a few quantitatively- and qualitatively-minded
researchers, (d) and one or more methodologists in other disciplines
Love,Luke
ClosureContent objectives: conceptual and practical (but
mostly conceptual) Inform participants’ current and future research
effortsMotivate future inquiry with a methodological focus
Happy to consult or collaborate on projects related to these discussions
THANK YOU!
top related