12630 statistics for authors canadian journal of anesthesia journal canadien d’anesthésie...
Post on 18-Jan-2016
216 Views
Preview:
TRANSCRIPT
12630
Statistics for AuthorsCanadian Journal of Anesthesia Journal canadien d’anesthésie
Winterlude Anesthesia SymposiumFebruary 1, 2015Ottawa, Ontario
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Short version of this session• Hire an experienced PhD biostatistician• … before you submit to ethics• … understand your data, but don’t touch it• … statistician writes statistical methods (results).• Thank you, let’s go watch the SuperBowl.
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Disclosure• No financial COI• Not, definitely not, a statistician• Not particularly good at math• MSc Epidemiology• Deputy Editor-in-Chief, Canadian Journal of Anesthesia• I do a lot of peer review
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Objectives• I will not tell you how to do statistics• I will not show you formulae• Discuss common statistical problems identified at
peer review• Design• Analysis• Reporting
• Links and take home points on Twitter• @glbryson• #wntrane15
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
• Studies are small• Effect sizes are small• Greater number of less predetermined statistical tests• Flexibility in design, outcomes, and analysis• Conflict of interest• Multiple teams competing
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
What is the research question?
P = PopulationI = InterventionC = ComparatorO = Outcome
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Q1. What percent of trials change their primary outcome between registration and publication?
A. NoneB. 3%C. 10%D. 30%
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Trials N (%)Identified 323Registered 147 (46)Different primary outcome 46 (31)Change statistically significant 19 (41)
Things change …
JAMA. 2009;302(9):977-984
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Q1. What percent of trials change their primary outcome between registration and publication?
A. NoneB. 3%C. 10%D. 30%
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Q2. What is a meaningful change in pain score?
A. 1 pointB. 10%C. 2 pointsD. 30%
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Does it feel better?
• 2,700 patients in 10 pregabalin-neuropathy trials• 5 to 12 weeks of therapy• NRS pain cw 7 point “improved” scale
• -1.74 points or -27.9% decrease • 134 patients, fentanyl cancer-pain breakthrough
• Measurements every 15 minutes for 1 hour• Success = not requiring additional medication at 30 minutes
• - 2 points or <33% decrease
Farrar JT Pain 2001;94;149–158
Farrar JT. J Pain Sympt Manage 2003;25:406-11
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Q2. What is a meaningful change in pain score?
A. 1 pointB. 10%C. 2 pointsD. 30%
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Q3. What’s wrong with this sample size estimate?
In this three-group trial the authors state“… therapy would reduce [pain] by 30%; power analysis with α = 0.05 and β = 0.80 revealed that we would need to enrol 24 patients in each group”
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Required Elements for Sample Size Estimate
Mean ProportionAlpha (type 1) error Alpha (type 1) errorBeta (1-power) error Beta (1-power) errorMean intervention Rate interventionMean control Rate controlStandard deviation
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Q3. Identify errors in this sample size estimate
In this three-group trial the authors state“… therapy would reduce [pain] by 30%; power analysis with α = 0.05 and β = 0.80 revealed that we would need to enrol 24 patients in each group”
Category ErrorCentral tendency Mean pain in target populationDispersion SD pain in target population
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Q4. How many comparisons in 3 group trial?
A
CB
0.05
0.01670.0167
0.0167
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Q5. Are we any good at sample size?• Two years worth of top medicine journals
• NEJM, JAMA, PLoS Med, Lancet, BMJ, Ann Int Med
• 215 citations with median sample size of 425• 113 (53%) reported all elements required• 146 (68%) assumptions were <30% off observed result• 73 (34%) estimates were complete and accurate• 96 (45%) were registered with a sample size estimate• 46 (21%) registration matched report
BMJ 2009;338:b1732
NOT SO MUCH
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
TreatmentN=30
ControlN=30
P Value
48 hr morphine 27 (15) 32 (16) 0.12
NRS pain 2 (3) 3(3) 0.23
BPI function 4 (5) 4 (6) 0.78
SF-MPQ -2 3.2 (3.6) 3.6 (3.1) 0.78
Continuous3.2 (2.5) 4.0 (2.0) 0.04*
Intermittent3.3 (2.6) 2.8 (3.0) 0.23
Neuropathic2.2 (2.9) 2.1 (2.4) 0.87
Affective 2.6 (2.8) 2.4 (2.4) 0.75
Q6. What’s wrong with this table?
Data presented as mean (SD). P for unpaired T-test indicated
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
TreatmentN=30
ControlN=30
P Value
PACU 7 (8) 8 (10) 0.45
6 hr 6 (6) 8 (10) 0.23
12 hr 2 (5) 6 (8) 0.04*
24 hr 6 (8) 6 (9) 0.78
48 hr 6 (10) 4 (10) 0.32
Total 27 (15) 32 (16) 0.12
Q7. Or this one?
Data presented as mean (SD). P for unpaired T-test indicated
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Answer Qs 6 and 7 = Multiplicity• Alpha errors accumulate• FWER = 1 – (1-alpha)number of tests
• FWER = 1 – (1-0.05)8 = 1-0.663 = 33.7%• Multiple primary outcomes• Multiple related outcomes• Repeated measures• Multiple treatment groups• Interim analyses
• The more cuts at the data the more like you are to make a type 1 error.
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
P-hacking• Coined by Joseph Simmons, Leif Nelson, and Uri Simonsohn• Generally refers to repeated analysis of data until P < 0.05
appears• Complicated debate, Bayes’ theorem, false discovery rates• Bottom line.
• Analyze only what you need• In the way your protocol described it
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Q8. Do we favor positive results?
Kühberger, Fritz, and Schendl. PLoSOne 2014;9(9):e105825.
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Q8. Do we favor positive results?
RT @mc_hankins
You bet we do
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
P values by XKCD
http://xkcd.com/1478/
Or if all else fails, use “significant at P>0.05” and hope no one notices.
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
Q9. Which of the following is “better?”
A. Morphine consumption reduced cw placebo (P 0.017)B. 33% reduction in morphine consumption cw placebo (P 0.017)C. Morphine consumption 18mg (7) v 12mg (6) (P 0.017)D. Mean difference morphine consumption 6mg (95% CI 2-9)
• P values conflate statistical and clinical significance• Must report measure of effect• Absolute measure preferred to relative.• Mean difference with 95% confidence limits• Absolute risk reduction or NNT
Canadian Journal of Anesthesia Journal canadien d‘anesthésie
springer.com/12630
• Registration• Replication• Open data• Standardized outcomes• More stringent statistical approach
top related