august, 2002mcp a general multiplicity adjustment approach and its application to evaluating several...
TRANSCRIPT
![Page 1: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/1.jpg)
August, 2002 MCP
A General Multiplicity Adjustment Approach and
Its Application to Evaluating Several Independently Conducted Studies
Qian Li and Mohammad HuqueCDER/FDA
![Page 2: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/2.jpg)
August, 2002 MCP
Disclaimer
• The views expressed in this talk are those of the authors and do not necessarily represent those of the Food and Drug Administration
![Page 3: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/3.jpg)
August, 2002 MCP
Outlines
• Motivation• Extending the union-intersection method• Issues in application
– relationship of the decision errors and decision rules– choosing a decision error– power characteristics
• An example and closing remarks
![Page 4: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/4.jpg)
August, 2002 MCP
Motivation
• More than one independently conducted Phase III study in NDA submissions to support efficacy evaluation
• Current practice– count the number of studies that are significant– combine studies
• Interpretation of regulatory requirement– two successful studies – no consistent interpretation when more than two studies
are conducted
![Page 5: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/5.jpg)
August, 2002 MCP
Union-Intersection method
• Roy (1953) first proposed a method of constructing a hypothesis test:H0: K
i=1H0i, for K hypothesis tests.
PH0(Ti>, i=1,2,…,K)=
where is a critical cut point.
![Page 6: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/6.jpg)
August, 2002 MCP
Extending the UI Approach
• PH0(p(1)1 p(2)2... p(K)K) ’
– where 1 2 ... K 1 are p-value cut points
– p(1), p(2), …, p(K) are ordered p-values of p1, p2, …, pK
’ is an overall type I error
![Page 7: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/7.jpg)
August, 2002 MCP
Definitions of overall hypotheses
• Possible choices of overall hypotheses:– at least one alternative is true
H01/K: i=1
KH0i vs. HA1/K: i=1
KHAi
– at least two alternatives are trueH0
2/K: j=1K (i=1 to K,ijH0i ) vs. HA
2/K : j=1K (i=1 to K,ijH0i )
…
– all the alternatives are trueH0
K/K: j=1K H0i vs. HA
K/K : j=1K HAi
![Page 8: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/8.jpg)
August, 2002 MCP
Overall type I error and p-value cut points
• For H01/k, the extended approach can be rewritten
as follows when p-values are independent
K
KK ppK
pppdpdppdK
1
2
12
1
1
210
!
![Page 9: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/9.jpg)
August, 2002 MCP
Overall type I error and p-value cut points
• For HAm+1/k (m>1) ,
– max overall type I error occur when m studies have power 1 to reject individual null
I’s satisfy the following
and 1= 2= … = m m+1.
K
KK
m
mm
m
mpp
Kpp mm
pdpdppdmK
1
2
12
1
1
210
)!(
![Page 10: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/10.jpg)
August, 2002 MCP
A special case of two hypotheses H0
1/2: H01 H02
• H01: 10, H02: 2 0
• The null space is the third quadrant
• max decision error occur at (1=0, 2=0)
• 1 and 2 satisfy 21 2 -12’
• More than one set of 1 and 2
• For ’=0.05, • if 1 = 0.025, 2 =1
• if 1 = 0.05, 2 =0.525
2
1(0,0)
![Page 11: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/11.jpg)
August, 2002 MCP
Rejection regions
0.525
0.525
1=0.050
2=0.525
1=0.025
2=1.000
1
p1 p1
p2p20.05
0.05
1
0.0250.025
![Page 12: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/12.jpg)
August, 2002 MCP
Possible decision rules for two studieswhen ’= 0.0252
1
2
0.025 0.0250.020 0.0260.010 0.0360.004 0.075
![Page 13: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/13.jpg)
August, 2002 MCP
Special case of two studiesH0
2/2: H01 H02
• The null space is all the area except first quadrant
• max decision error occurs at (1= , 2=0) & (1=0, 2=)
• max overall type I error is controlled when 1 =2’
1
2
(0,0)
![Page 14: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/14.jpg)
August, 2002 MCP
Issues in application
• Choice of overall hypothesis
• Choice of decision error
• Power characteristics
![Page 15: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/15.jpg)
August, 2002 MCP
Relationship of decision errors among different overall hypotheses
• For a set of K independent p-values, there exists a common rejection region p1 = p2 =…=pk .
• The corresponding decision errors for the overall hypotheses are: H0
1/k: ’= k
H02/k : ’= k-1
…
H0k-1/k: ’= 2
H0k/k : ’=
![Page 16: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/16.jpg)
August, 2002 MCP
Relationship of decision rules
• It can be shown that the decision rules derived from a stringent hypothesis can also be derived from a less stringent hypothesis, given the relationship of the decision errors among different overall hypotheses.
![Page 17: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/17.jpg)
August, 2002 MCP
Relationship of decision errors
• Exist a common rejection region in (p1, p2)
That is to require two significant studies,
p1 = p2
Decision errors : – H0
1/2: H01 H02 ’= 2
– H02/2 : H01 H02 ’= p1
p2
0
![Page 18: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/18.jpg)
August, 2002 MCP
Strategies for choosing decision errors
• Considering 4 strategies– Use the same decision error for all HA
k/k
– Use the same decision error for all HA1/k
– Find the decision errors that keep a constant power for HA
k/k
– Control the power increase for HAk/k
![Page 19: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/19.jpg)
August, 2002 MCP
Same decision error for HAk/k
• Require all the K studies significant at level ’• Similar to UI decision rules • Power is low
Each study has 90% power at level 0.025 K: 12 3 4 5 6
Error: 0.025 0.025 0.025 0.025 0.025 0.025
Power: 90.0 80.8 72.9 65.6 59.0 53.1
• Not fair to use the same decision error for HAk/k when
K is large
![Page 20: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/20.jpg)
August, 2002 MCP
Same decision error for HA1/K
• Decision error and power for HAK/K
Each study has 90% power at level 0.025 K: 1 2 3 4 5 6
Error: 0.0006 0.025 0.085 0.158 0.2290.292
Power: 50.0 81.0 96.9 98.7 99.4 99.6
• When K increases, there is a large increase in decision error,
• therefore large increase in power
![Page 21: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/21.jpg)
August, 2002 MCP
Keeping consistent power for HAK/K
• This strategy is in between the first two strategies
• Decision error and power for HAK/K
Each study has 90% power at level 0.025 K: 1 2 3 4 5 6
Error: 0.009 0.025 0.040 0.054 0.0660.077
Power: 81.0 81.0 81.0 81.0 81.0 81.0
• In case not satisfied ...
![Page 22: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/22.jpg)
August, 2002 MCP
Increasing power for HAK/K
• Decide a reasonable power for HAK/K, then figure
out the error rate
• Decision error and power for HAK/K
Each study has 90% power at level 0.025 K: 1 2 3 4 5 6
Error: 0.007 0.025 0.046 0.068 0.0930.121
Power: 78.6 81.0 83.0 85.0 87.0 89.0
• Less conservative than the previous strategy
![Page 23: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/23.jpg)
August, 2002 MCP
Power characteristics
• Power function
yK=PHA(p(1)1 p(2)2... p(K)K)
• Tedious to write when K>3
• Can be evaluated numerically
• Search the optimal power numerically
![Page 24: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/24.jpg)
August, 2002 MCP
Two studies - H01/2: H01 H02 Power curve
for 1= 2=1.5, ’=0.05
![Page 25: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/25.jpg)
August, 2002 MCP
An example
• Three studies are conducted
• Use HA1/3 , ’=0.0463
• p-value cut points are– 0.025, 0.025, 0.067
• Observed p-values were – 0.0065, 0.0125, 0.06
![Page 26: August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad](https://reader036.vdocuments.us/reader036/viewer/2022062407/56649f575503460f94c7ca2b/html5/thumbnails/26.jpg)
August, 2002 MCP
Remarks
• having the flexibility to choose p-value cut point • allows us to control decision error for multiple
studies• possible to balance variations among p-values