clinical trial statstics 2016
TRANSCRIPT
EBM: Clinical Trial Statistics
Stefan Tigges MD MSCRDepartment of Radiology, Emory
1
2
External Industry External Industry
Relationships Relationships **Company Name(s) Company Name(s) Role Role
Equity, stock, or options in Equity, stock, or options in biomedical industry companies biomedical industry companies or publishersor publishers****
General Electric and Microsoft Stockholder
Board of Directors or officerBoard of Directors or officer None
Royalties from Emory or from Royalties from Emory or from external entityexternal entity
None
Industry funds to Emory for my Industry funds to Emory for my research research
None
OtherOther None
*Consulting, scientific advisory board, industry-sponsored CME, expert witness for company, FDA representative for company, publishing contract, etc.
**Does not include stock in publicly-traded companies in retirement funds and other pooled investment accounts managed by others.
Stefan Tigges, Personal/Professional Financial Relationships with Industry within the past year
Lecture/Reading Goals and Objectives• Define: probability, distribution, variability and
central tendency.• Explain how a sample may be biased and the
difference between bias and random sampling error.• Explain how and why hypothesis testing and
statistical inference are used in clinical trial analysis. • Define: confidence intervals, statistical significance,
type I error, type II error, power, p-value, alpha and beta.
• Describe the effect of increasing sample size on type I and type II error.
• Describe the effect of sample size, effect variability, level of alpha, and effect size on power.
3
Learning Approach
1) Readings, 2 Comix2) Lecture3) Homework
1) Optional2) Based on student ?s3) Hard
4) E-mail me
4
Big Question: Approach to Claims
5
Three Explanations• Truth
• Dumb luck
• Fishy
6
Antihypertensive Trial: Result is Positive
7No Effect−10−20−30 +10 +20 +30
Explanations:1)Real Effect: HA true2)FP: Random Error (α)3)FP: Bias
NewDrug
OldDrug
FP Alpha (random) error
8
Antihypertensive Trial: Result is Negative
9No Effect−10−20−30 +10 +20 +30
Explanations:1)Real Effect: H0 true2)FN: Random Error (β)3)FN: Bias
NewDrug
OldDrug
FN Beta error, effect exists, not detected
10
Diagnostic Tests, 2x2 Table
11
TestFinding
Disease Positive
Disease Negativ
ePositive TP FP → PPVNegativ
eFN TN → NPV
↓Sensitiv
ity
↓Specific
ity
Total
Clinical Research, 2x2 Table
12
TrialResult
HA True
HA
FalsePositive TP FP (α)
Type I→ PPV
Negative
FN (β) Type II
TN → NPV
↓Power
↓*p value
Total
Randomized Clinical Trial Steps
13
Populationof interest
Sample
DrugA
DrugB Time
Drug A
∆ BPDrug
B∆ BP
Time Compare,publish,acceptNobelprize
H0: A=BHA: A≠B
Bias vs.random
error
Bias: Systematic
errors in data collection &
interpretation
14
15
16
Voters
Sample
17
$ $
$
$
$
$
$
$
$
$
Types of Statistics• Descriptive– Summarize/display data– Mean, median, mode, σ etc.
• Inferential– Use sample to make
conclusions about population – Example: Hypertension• Test population: all w/ ↑ BP
– Definitive, descriptive stats only
• Test sample– Hypothesis testing– P(Observed results given H0)
13%
17%
57%
13%
1st Qtr2nd Qtr3rd Qtr4th Qtr
18
Population:all w/↑ BP
Sample
Hypothesis Testing: Is H0 plausible? EUSM vs. NBA Mean Height
19
H 0: Expecte
d
Trial: Observed
When you stare into the abyss [of statistics], the abyss stares back into you.
Statistics: P(O given H0)
p<.05reject H0
p≥.05, cannotreject H0190
H0:μEUSM=μNBA(190)HA:μEUSM≠μNBA (190)
170
Determining p value: Normal Distribution
20
0 1 2−2 −1
Central Tendency:Mean, median and mode
Dispersion:Standard deviation
√Σ(x-μ)2/N68%
95%
Normal Distribution: EUSM M1 Height
21
170
σEUSM=10 cm
Population: EUSM M1 Heights (cm)
22
170 180150 160 190
EUSMEUSMClass of ‘18Class of ‘18
Number of σs from mean is probability
23
3σ from mean, p=.0027
140 cm
Example: Heights
24
170 cm 190 cm160 cm
μ= 160, 170,190σ=10, α=.05
ie, 2σ
Example 1: EUSM M-1s vs. NBA Heights• Is mean height of EUSM M-1s different than mean
height of NBA players? • H0:μEUSM=μNBA (190 cm) with σ=10 cm • HA:μEUSM≠μNBA(190 cm) with σ=10 cm• 25 M-1 heights, mean=170 cm, ∆=20 cm• SEM= σ/√n=10/√ 25=2• 20/2= 10 σ, p<.0001• Reject H0 at α of .05• α predetermined for H0 rejection
25
Observe: 170
Expec
t: 190
M-1 vs. NBA Heights: H0 is False (TP)
26
160
190
170
190
170
190
150
160
170
150
180
170
170
180
160
160
180
180
160
150
150
170
160
170
180
180
150
160
180
190
170
190
180
150
190
180
150
170
<150
170
160
170
170
160
170
160
160
170
150
180
150
180
150
>190
190
170
170
170
170
170
<150
170
180
170
180
180
170
170
150
170
180
160
170
160
170
190
160
170
190
160
160
190
180
180
160
170
170
160
150
>190
190
170
>190
190
150
180
180
160
190
150
> Mean170 cmMean < Mean
170↓20
0 1 2−2 −1
Example 2: Heights
27
170 cm 190 cm160 cm
Example 2: EUSM M-1s vs. Brand X M-1s• Is mean height of EUSM M-1s different than mean
height of M-1s at Brand X medical school? • H0:μEUSM=μBrand X (170 cm) with σ=10 cm • HA:μEUSM≠μBrand X(170 cm) with σ=10 cm• 25 M-1 heights, mean=170 cm, ∆=0 cm• SEM= σ/√n=10/√ 25=2• 0/2= 0 σ, p=1• Don’t reject H0
28
Observe: 170
Expec
t: 170
M-1 vs. Brand X Heights: H0 is True (TN)
29
160
190
170
190
170
190
150
160
170
150
180
170
170
180
160
160
180
180
160
150
150
170
160
170
180
180
150
160
180
190
170
190
180
150
190
180
150
170
<150
170
160
170
170
160
170
160
160
170
150
180
150
180
150
>190
190
170
170
170
170
170
<150
170
180
170
180
180
170
170
150
170
180
160
170
160
170
190
160
170
190
160
160
190
180
180
160
170
170
160
150
>190
190
170
>190
190
150
180
180
160
190
150
> Mean170 cmMean < Mean
170
0 1 2−2 −1
M-1 vs. Brand X Heights: Type I error (FP)
30
160
190
170
190
170
190
150
160
170
150
180
170
170
180
160
160
180
180
160
150
150
170
160
170
180
180
150
160
180
190
170
190
180
150
190
180
150
170
<150
170
160
170
170
160
170
160
160
170
150
180
150
180
150
>190
190
170
170
170
170
170
<150
170
180
170
180
180
170
170
150
170
180
160
170
160
170
190
160
170
190
160
160
190
180
180
160
170
170
160
150
>190
190
170
>190
190
150
180
180
160
190
150
> Mean170 cmMean < Mean
180↑10
0 1 2−2 −1
Example 2: Heights
31
Bran
d X
Example 3: Heights
32
170 cm 190 cm160 cm
Example 3: EUSM M-1s vs. Jockeys• Is mean height of EUSM M-1s different than mean
height of Jockeys? • H0:μEUSM=μJockey (160 cm) with σ=10 cm • HA:μEUSM≠μJockey(160 cm) with σ=10 cm• 25 M-1 heights, mean=170 cm, ∆=10 cm• SEM= σ/√n=10/√ 25=2• 10/2= 5 σ, p=.0062• Reject H0 at α of .05
33
Obs
erve
: 170Exp
ect:
160
M-1 vs. Jockey Heights: HA is True (TP)
34
160
190
170
190
170
190
150
160
170
150
180
170
170
180
160
160
180
180
160
150
150
170
160
170
180
180
150
160
180
190
170
190
180
150
190
180
150
170
<150
170
160
170
170
160
170
160
160
170
150
180
150
180
150
>190
190
170
170
170
170
170
<150
170
180
170
180
180
170
170
150
170
180
160
170
160
170
190
160
170
190
160
160
190
180
180
160
170
170
160
150
>190
190
170
>190
190
150
180
180
160
190
150
> Mean170 cmMean < Mean
170↑10
0 1 2−2 −1
M-1 vs. Jockey Heights: Type II Error (FN)
35
160
190
170
190
170
190
150
160
170
150
180
170
170
180
160
160
180
180
160
150
150
170
160
170
180
180
150
160
180
190
170
190
180
150
190
180
150
170
<150
170
160
170
170
160
170
160
160
170
150
180
150
180
150
>190
190
170
170
170
170
170
<150
170
180
170
180
180
170
170
150
170
180
160
170
160
170
190
160
170
190
160
160
190
180
180
160
170
170
160
150
>190
190
170
>190
190
150
180
180
160
190
150
> Mean170 cmMean < Mean
160
0 1 2−2 −1
Putting random errors and p-
values in context36
Meaning of P Value• P value tells us about plausibility of H0, (A=B)
– Assumes H0 is true, what is probability of observed given expected
– Example: Hypertension trial, Drug A>Drug B, p=.031, reject H0
– Example: Coin toss, 5 heads in a row chance
37
.500.500 .250.250 .125.125 .063.063 .031.031
Multiple p values
3899.4%10092.3%5072.3%2540.1%1022.6%518.5%414.3%39.8%25%1
P(≤1 Test Sig) Test #
Statistical Significance≠ Clinical Significance
39
Drug A ↓ BP 11 mm HgDrug A ↓ BP 11 mm HgDrug B ↓ BP 10 mm HgDrug B ↓ BP 10 mm Hg
∆∆=1 mm Hg, p=.01, n=100k=1 mm Hg, p=.01, n=100k
P value: Effect Size & SNR (variability)• Example: Weight loss pills vs. placebo:
• Precise pill: 2 lb loss w/ sem of .9 lbs, p value < .05, reject H0 • Noisy pill: 10 lb loss w/ sem of 6 lbs, p value > .05, don’t
reject H0
• Which pill is more effective?
40
0 lb
s
2 lbs 10 lbs
Confidence limits vs. p values• P value says nothing about effect size or variability• 95% confidence limits: sample mean±2(sem)• Estimate of effect size and precision (variability)• 95%CI≠95% chance μ is w/in CI, more complex• CI does not include bias• Can be used for significance testing
4112 lbs10 lbs8 lbs0 lbs
95% CI
What effects β Error/Power?• Power is P(Detecting real effect) Sensitivity• β is P(Missing a real effect) FN, random effects• Power=1-β• Power effects:– Level of α– Effect size– Sample variability– Sample size
42
Clinical Research, 2x2 Table
43
TrialResult
HA True
HA
FalsePositive TP FP (α)
Type I→ PPV
Negative
FN (β) Type II
TN → NPV
↓Power
↓*p value
Total
44
α=.20, .05, .01
45
TP
FP
α=.20, Big Hole to reject H0
46
TP
FP
α=.05, Just Right Hole to reject H0
47
TP
FP
α=.01, Small Hole to reject H0
What effects Power?
48
Use SNR AnalogyWaldo=effect (signal),
Others=variability/σ (noise) Waldo
Power and sample size: Rachel’s coin
49
Clinical Research, 2x2 Table
50
TrialResult
HA True
HA
FalsePositive TP FP (α)
Type I→ PPV
Negative
FN (β) Type II
TN → NPV
↓Power
↓*p value
Total
Prior Probability and Trial PPV/NPV
51
Eye of newt
Rest of newt
Placebo vs. Emesis for Plague
52
Summary: Clinical Trial Results
53
True? Random? Bias?
