Download - Managing missing data
![Page 1: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/1.jpg)
Session 4: Analysis and reporting
Managing missing dataRob Coe (CEM, Durham)
Developing a statistical analysis planHannah Buckley (York Trials Unit)
Panel on EEF reporting and data archivingJonathan Sharples, Camilla Nevill, Steve Higgins and Andrew Bibby
![Page 2: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/2.jpg)
Managing missing data
Rob CoeEEF Evaluators Conference, York, 2 June 2014
![Page 3: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/3.jpg)
∂
The problem
Only if everyone responds to everything is it still a randomised trial– Any non-response (post-randomisation) → not an RCT
It may not matter (much) if– Response propensity is unrelated to outcome– Non-response is low
Lack of ‘middle ground’ solutions– Mostly people either ignore or use very complex stats
3
![Page 4: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/4.jpg)
∂
What problem are we trying to solve?
We want to estimate the distribution of likely effects of [an intervention] in [a population]– Typically represented by an effect size and CI
Missing data may introduce bias and uncertainty– Point estimate effect size different from observed– Probability distribution for ES (CI) widens
4
![Page 5: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/5.jpg)
What kinds of analysis are feasible to reduce the risk of bias from missing data?
5
![Page 6: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/6.jpg)
∂
Vocabulary
Missing Completely at Random (MCAR)– Response propensity is unrelated to
outcomeMissing at Random (MAR)
– Missing responses can be perfectly predicted from observed data
Missing Not at Random (MNAR)– We can’t be sure that either of the
above apply
6
Ignore missingness
Statistics:IWP, MI
??
![Page 7: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/7.jpg)
∂
“When data are missing not at random, no method of obtaining unbiased estimates exists that does not incorporate the mechanism of non-random missingness, which is nearly always unknown. Some evidence, however, shows that the use of a method that is valid under missing at random can provide some reduction in bias.”
Bell et al, BMJ 2013
7
![Page 8: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/8.jpg)
∂
Recommendations1. Plan for dealing with missing data should be in
protocol before trial starts2. Where attrition likely, use randomly allocated
differential effort to get outcomes3. Report should clearly state the proportion of
outcomes lost to follow up in each arm4. Report should explore (with evidence) the reasons
for missing data5. Conduct simple sensitivity analyses for strength of
relationship betweenOutcome score and missingnessTreatment/Outcome interaction and missingness
8
![Page 9: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/9.jpg)
∂
If attrition is not low (>5%?)
6. Model outcome response propensity from observed variables
7. Conduct MAR analyses• Inverse weighted probabilities• Multiple imputation
8. Explicitly evaluate plausibility of MAR assumptions (with evidence)
9
![Page 10: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/10.jpg)
∂
10
![Page 11: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/11.jpg)
∂
11
![Page 12: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/12.jpg)
∂
Useful references Bell, M. L., Kenward, M. G., Fairclough, D. L., & Horton, N. J. (2013).
Differential dropout and bias in randomised controlled trials: when it matters and when it may not. BMJ: British Medical Journal, 346:e8668. http://www.bmj.com/content/346/bmj.e8668
Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual review of psychology, 60, 549-576.
National Research Council. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: The National Academies Press, 2010. http://www.nap.edu/catalog.php?record_id=12955
Shadish, W. R., Hu, X., Glaser, R. R., Kownacki, R., & Wong, S. (1998). A method for exploring the effects of attrition in randomized experiments with dichotomous outcomes. Psychological Methods, 3(1), 3.
www.missingdata.org.uk
12
![Page 14: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/14.jpg)
Overview
• What is a SAP?
• When is a SAP developed?
• Why is a SAP needed?
• What should be included in a SAP?
![Page 15: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/15.jpg)
What is a SAP?
• Pre-specifies analyses
• Expands on the analysis section of a
protocol
• Provides technical information
![Page 16: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/16.jpg)
When is a SAP developed?
• After protocol finalised
• Before final data received
• Written in the future tense
![Page 17: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/17.jpg)
Why create a SAP
• Pre-specify analyses
• Think through potential pitfalls
• Benefit to other analysts
![Page 18: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/18.jpg)
ACTIVITY• What do you think should be covered
in a SAP?
• Sort the cards into two piles
What should be in a SAP?
![Page 19: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/19.jpg)
ACTIVITY DISCUSSION• Which topics do you think do not
need to be covered in a SAP?
• Are there any topics which you were
unsure about?
What should be in a SAP?
![Page 20: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/20.jpg)
What should be in a SAP?
ACTIVITY1. Which of the cards cover key
background information and which are related to analysis?
2. Which order would you deal with the topics in?
![Page 21: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/21.jpg)
Setting the scene
• Restate study objectives
• Study design
• Sample size
• Randomisation methods
The structure of a SAP
![Page 22: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/22.jpg)
Description of outcomes
• Primary outcome
• Secondary outcome(s)
• When outcomes will be measured
• Why outcomes chosen
The structure of a SAP
![Page 23: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/23.jpg)
Analysis - overview• Analysis set (ITT)• Software package• Significance levels • Blankets statements on confidence
intervals, effect sizes or similar• Methods for handling missing data
The structure of a SAP
![Page 24: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/24.jpg)
Analysis methods• Baseline data• Primary analysis• Secondary analyses• Subgroup analyses• Sensitivity analyses
The structure of a SAP
![Page 25: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/25.jpg)
Conclusions
• Producing a SAP is good practice
• Can help avoid problems in analysis
• Finalised before final data received
• Fairly detailed
• Flexible but should cover key points
![Page 26: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/26.jpg)
References and resources
References
• ICH E9 ‘Statistical principles for clinical trials’
http://www.ich.org/products/guidelines/efficacy/article/effica
cy-guidelines.html
Resources
• PSI ‘Guidelines for standard operating procedures for good
statistical practice in clinical research’
www.psiweb.org/docs/gsop.pdf
![Page 27: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/27.jpg)
Thank you!
Any questions or discussion points?
![Page 28: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/28.jpg)
EEF reporting and data archivingJonathan Sharples (EEF)Camilla Nevill (EEF)Steve Higgins (Durham) - ChairAndrew Bibby (FFT)
![Page 29: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/29.jpg)
The reporting process and publication of results on EEF’s websiteJonathan Sharples (EEF)
![Page 30: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/30.jpg)
Classifying the security of findings from EEF evaluationsCamilla Nevill (EEF)
Group Number of pupils Effect size
Estimated months’ progress Evidence strength
Literacy intervention 550 0.10 (0.03, 0.18) +2
www.educationendowmentfoundation.org.uk/evaluation
![Page 31: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/31.jpg)
Example Appendix: Chatterbooks
Rating 1. Design 2. Power (MDES)
3. Attrition 4. Balance 5. Threats to validity
5 Fair and clear experimental design (RCT) < 0.2 < 10% Well-balanced on
observables No threats to validity
4 Fair and clear experimental design (RCT, RDD) < 0.3 < 20%
3 Well-matched comparison (quasi-experiment) < 0.4 < 30% Some
threats
2 Matched comparison (quasi-experiment) < 0.5 < 40%
1 Comparison group with poor or no matching < 0.6 < 50%
0 No comparator > 0.6 > 50% Imbalanced on observables
Significant threats
![Page 32: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/32.jpg)
Combining the results of evaluations with the meta-analysis in the Teaching and Learning ToolkitSteve Higgins (Durham)
![Page 33: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/33.jpg)
Andrew Bibby
Archiving EEF project data
![Page 34: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/34.jpg)
![Page 35: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/35.jpg)
![Page 36: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/36.jpg)
![Page 37: Managing missing data](https://reader036.vdocuments.us/reader036/viewer/2022062323/56816151550346895dd0da3d/html5/thumbnails/37.jpg)
1. Include permission for linking and archiving in consent forms
2. Retain pupil identifiers
3. Label values and variables
4. Save Syntax or Do files
Prior to archiving…