![Page 1: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/1.jpg)
1
Multiple testing in large-scale gene expression experiments
Lecture 22, Statistics 246, April 13, 2004
![Page 2: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/2.jpg)
2
OutlineOutline
• Motivation & examples
• Univariate hypothesis testing
• Multiple hypothesis testing
• Results for the two examples
• Discussion
![Page 3: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/3.jpg)
3
SCIENTIFIC: To determine which genes are differentially expressed between two sources of mRNA (trt, ctl).
STATISTICAL: To assign appropriately adjusted p-values to thousands of genes, and/or make statements about false discovery rates.
I will discuss the issues in the context of two experiments, one which fits the aims above, and one which doesn’t, but helps make a number of points.
MotivationMotivation
![Page 4: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/4.jpg)
4
• 8 treatment mice and 8 control mice
• 16 hybridizations: liver mRNA from each of the 16 mice (Ti , Ci ) is labelled with Cy5, while pooled liver mRNA from the control mice (C*) is labelled with Cy3.
• Probes: ~ 6,000 cDNAs (genes), including 200 related to lipid metabolism.
Goal. To identify genes with altered expression in the livers of Apo AI knock-out mice (T) compared to inbred C57Bl/6 control mice (C).
Apo AI experiment Apo AI experiment (Matt Callow, LBNL)(Matt Callow, LBNL)
![Page 5: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/5.jpg)
5
![Page 6: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/6.jpg)
6
Golub Golub et alet al (1999) experiments (1999) experiments
Goal. To identify genes which are differentially expressed in acute lymphoblastic leukemia (ALL) tumours in comparison with acute myeloid leukemia (AML) tumours.
• 38 tumour samples: 27 ALL, 11 AML.• Data from Affymetrix chips, some pre-processing.• Originally 6,817 genes; 3,051 after reduction.
Data therefore a 3,051 38 array of expression values.
Comment: this wasn’t really the goal of Golub et al.
![Page 7: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/7.jpg)
7
DataDataThe gene expression data can be summarized as follows
treatment control
X =
Here xi,j is the (relative) expression value of gene i in sample j. The first n1 columns are from the treatment (T); the remaining n2 = n - n1 columns are from the control (C).
![Page 8: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/8.jpg)
8
Steps to find diff. expressed genesSteps to find diff. expressed genes
1. Formulate a single hypothesis testing strategy
2. Construct a statistic for each gene
3. Compute the raw p-values for each gene by permutation procedures or from distribution models
4. Consider the multiple testing problem
a. Find the maximum # of genes of interest
b. Assign a significance level for each gene
![Page 9: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/9.jpg)
9
Univariate hypothesis testingUnivariate hypothesis testing
Initially, focus on one gene only.
We wish to test the null hypothesis H that the gene is not differentially expressed.
In order to do so, we use a two sample t-statistic:
€
t= averofn1 trtx − averofn2ctlx
[ 1n1
(SDofn1trtx)2 +1n1
(SDofn1ctlx)2]
![Page 10: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/10.jpg)
10
The p-values for two sample t-statThe p-values for two sample t-stat
p-value
![Page 11: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/11.jpg)
11
pp-values-values
The p-value or observed significance level p is the chance of getting a test statistic as or more extreme than the observed one, under the null hypothesis H of no differential expression.
Although the previous test statistic is denoted by t, it would be unwise to assume that its null distribution is that of Student’s t. We have another way to assign p-values which is more or less valid: using permutations.
![Page 12: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/12.jpg)
12
Computing Computing pp-values by permutations-values by permutationsWe focus on one gene only. For the bth iteration, b = 1, ,
B;
1. Permute the n data points for the gene (x). The first n1 are referred to as “treatments”, the second n2 as “controls”.
2. For each gene, calculate the corresponding two sample t-statistic, tb.
After all the B permutations are done;
3. Put p = #{b: |tb| ≥ |t|}/B (plower if we use >).With all permutations in the Apo AI data, B = n!/n1! n2! = 12,870;
for the leukemia data, B = 1.2109 .
![Page 13: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/13.jpg)
13
Many tests: a simulation studyMany tests: a simulation study
Simulation of this process for 6,000 genes with 8 treatments and 8 controls.
All the gene expression values were simulated i.i.d from a N (0,1) distribution, i.e. NOTHING is differentially expressed in our simulation.
We now present the 10 smallest raw (unadjusted) permutation p-values.
![Page 14: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/14.jpg)
14
DiscussionDiscussionAt this point in the lecture we discussed the following question: what assumptions on the null distributions of the gene expression values xi = (xi,1 , xi,2 , …xi,n) are necessary or sufficient for the permutation-based p-values just described to be valid? And, are they applicable in our examples?First, we figured out that p-values are valid if their distribution is uniform(0,1) under the null hypothesis. Secondly, we concluded that if the null distribution of xi is exchangeable, i.e. invariant under permutations of 1,…,n, then, we could reasonably hope (and actually prove) that the distribution of the permutation-based p-values is indeed uniform on 1,…,n. We also noted that having the joint distribution i.i.d. would be sufficient, as this implied exchangeability.Next, we considered the ApoAI experiment. Because the 16 log-ratios for each gene involved a term from the pooled control mRNA, called C* above, it seems clear that an i.i.d. assumption is unreasonable. Had the experiment been carried out by using pooled control mRNA from mice other than the controls in the experiment, an exchangeability assumption under the null hypothesis would have been quite reasonable. Unfortunately, C* did come from the same mice as the Ci, so exchangeability is violated, and the assumption is at best an approximation.
![Page 15: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/15.jpg)
15
gene t p-valueindex value (unadj.)2271 4.93 210-4
5709 4.82 310-4
5622 -4.62 410-4
4521 4.34 710-4
3156 -4.31 710-4
5898 -4.29 710-4
2164 -3.98 1.410-3
5930 3.91 1.610-3
2427 -3.90 1.610-3
5694 -3.88 1.710-3
Unadjusted p-values
Clearly we can’t just use standard p-value thresholds of 05 or .01.
![Page 16: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/16.jpg)
16
Multiple testing: Counting errorsMultiple testing: Counting errors Assume we are testing H1, H2, , Hm .
m0 = # of true hypotheses R = # of rejected hypotheses
# true # false
null hypo. null hypo.# accepted U T m - R# rejected V S R
m0 m-m0
V = # Type I errors [false positives]
T = # Type II errors [false negatives]
![Page 17: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/17.jpg)
17
Type I error ratesType I error rates• Per comparison error rate (PCER): the expected value of the number of Type I errors over the number of hypotheses,
PCER = E(V)/m.
•Per-family error rate (PFER): the expected number of Type I errors,
PFER = E(V).
•Family-wise error rate: the probability of at least one type I error
FEWR = pr(V ≥ 1)
•False discovery rate (FDR) is the expected proportion of Type I errors among the rejected hypotheses
FDR = E(V/R; R>0) = E(V/R | R>0)pr(R>0).
• Positive false discovery rate (pFDR): the rate that discoveries are false
pFDR = E(V/R | R>0).
![Page 18: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/18.jpg)
18
Multiple testingMultiple testingFamily-wise error rates
FWER = Pr(# of false discoveries >0)
= Pr(V>0)
Bonferroni (1936)Tukey (1949)Westfall and Young (1993) discussed resampling……
![Page 19: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/19.jpg)
19
FWER and microarraysFWER and microarrays
Two approaches for computing FWER
maxT step-down procedure
Dudoit et al (2002) Westfall et al (2001)
minP step-down procedure
Ge et al (2003), a novel fast algorithm
![Page 20: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/20.jpg)
20
Multiple testingMultiple testingFalse discovery ratesFalse discovery rates
Q is set to be 0 when R=0
FDR = expectation of Q = E(V/R; R>0)
Seeger (1968)
Benjamini and Hochberg (1995)
![Page 21: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/21.jpg)
21
Caution with FDR
• Cheating:
Adding known diff. expressed genes reduces FDR
• Interpreting:
FDR applies to a set of genes in a global sense, not to individual genes
![Page 22: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/22.jpg)
22
Some previous work on FDRSome previous work on FDR
Benjamini and Hochberg (1995)
Benjamini and Yekutieli (2001)
Storey (2002)
Storey and Tibshirani (2001)
……
![Page 23: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/23.jpg)
23
Types of control of Type I errorTypes of control of Type I error
• strong control: control of the Type I error whatever the true and false null hypotheses. For FWER, strong control means controlling
max pr(V ≥ 1 | M0)
M0H0C
where M0 = the set of true hypotheses (note |M0| = m0);
• exact control: under M0 , even though this is usually unknown.
• weak control: control of the Type I error only under the complete null hypothesis H0
C = iHi . For FWER, this is control of pr( V ≥ 1 | H0
C ).
![Page 24: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/24.jpg)
24
Adjustments to Adjustments to pp-values-values For strong control of the FWER at some level , there are
procedures which will take m unadjusted p-values and modify them separately, so-called single step procedures, the Bonferroni adjustment or correction being the simplest and most well known. Another is due to Sidák.
Other, more powerful procedures, adjust sequentially, from the smallest to the largest, or vice versa. These are the step-up and step-down methods, and we’ll meet a number of these, usually variations on single-step procedures.
In all cases, we’ll denote adjusted p-values by , usually
with subscripts, and let the context define what type of adjustment has been made. Unadjusted p-values are denoted by p.
![Page 25: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/25.jpg)
25
What should one look for in aWhat should one look for in a multiple testing procedure? multiple testing procedure?
As we will see, there is a bewildering variety of multiple testing procedures. How can we choose which to use? There is no simple answer here, but each can be judged according to a number of criteria:
Interpretation: does the procedure answer a relevant question for you?
Type of control: strong, exact or weak?
Validity: are the assumptions under which the procedure applies clear and definitely or plausibly true, or are they unclear and most probably not true?
Computability: are the procedure’s calculations straightforward to calculate accurately, or is there possibly numerical or simulation uncertainty, or discreteness?
![Page 26: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/26.jpg)
26
pp-value adjustments: single-step-value adjustments: single-step
Define adjusted p-values π such that the FWER is controlled at level where Hi is rejected when πi ≤ .
• Bonferroni: πi = min (mpi, 1)
• Sidák: πi = 1 - (1 - pi)m
Bonferroni always gives strong control, proof next page.
Sidák is less conservative than Bonferroni. When the genes are independent, it gives strong control exactly (FWER= ), proof later. It controls FWER in many other cases, but is still conservative.
![Page 27: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/27.jpg)
27
Proof for BonferroniProof for Bonferroni((single-step adjustment)single-step adjustment)
pr (reject at least one Hi at level | H0C)
= pr (at least one i ≤ | H0C)
≤ 1m pr (i ≤ | H0
C) , by Boole’s inequality
= 1m pr (Pi ≤ /m | H0
C), by definiton of i
= m /m , assuming Pi ~ U[0,1])= .
Notes: 1. We are testing m genes, H0
C is the complete null hypothesis, Pi is the unadjusted p-value for gene i , while i here is the Bonferroni adjusted p-value.
2. We use lower case letters for observed p-values, and upper case for the corresponding random variables.
![Page 28: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/28.jpg)
28
Proof for Sidák’s methodProof for Sidák’s method(single-step adjustment)(single-step adjustment)
pr(reject at least one Hi | H0C)
= pr(at least one i ≤ | H0C)
= 1 - pr(all i > | H0C)
= 1-∏i=1m pr(i > | H0
C) assuming independence
Here i is the Sidák adjusted p-value, and so i > if and only if Pi > 1-(1- )1/m (check), giving
1-∏i=1m pr(i > | H0
C)
= 1-∏i=1m pr(Pi > 1-(1- )1/m | H0
C)
= 1- { (1- )1/m }m since all Pi ~ U[0,1],=
![Page 29: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/29.jpg)
29
Single-step adjustments (ctd)Single-step adjustments (ctd)
The minP method of Westfall and Young:
i = pr( min Pl ≤ pi | H)
1≤l≤m
Based on the joint distribution of the p-values {Pl }. This is the most powerful of the three single-step adjustments.
If Pi U [0,1], it gives a FWER exactly = (see next page).
It always confers weak control, and gives strong control under subset pivotality (definition next but one slide).
![Page 30: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/30.jpg)
30
Proof for (single-step) minP adjustmentProof for (single-step) minP adjustment
Given level let c be such that pr(min1 ≤ i ≤ m Pi ≤ c| H0
C) = . Note that {i ≤ } {Pi ≤ c} for any i. pr(reject at least one Hi at level | H0
C) = pr (at least one i ≤ | H0
C) = pr (min1 ≤ i ≤ m i ≤ | H0
C) = pr (min1 ≤ i ≤ m Pi ≤ | H0
C) =
![Page 31: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/31.jpg)
31
Strong control and subset pivotalityStrong control and subset pivotality Note the above proofs are under H0
C, which is what we term weak control.
In order to get strong control, we need the condition of subset pivotality. The distribution of the unadjusted p-values (P1, P2, …Pm) is said to have the subset pivotality
property if for all subsets L {1,…,m} the distribution of the subvector {P i: i L} is identical under the restrictions {Hi: i L} and H0
C . Using the property, we can prove that for each adjustment under their conditions, we have pr (reject at least one Hi at level i M0 = pr (reject at least one H i at level i M00
C ≤ pr (reject at least one H i at level for all i 0
C ≤
herefore, we have proved strong control for the previous three adjustments, assuming subset pivotality.
![Page 32: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/32.jpg)
32
Permutation-based single-step minP Permutation-based single-step minP adjustment of p-valuesadjustment of p-values
For the bth iteration, b = 1, , B;
1. Permute the n columns of the data matrix X, obtaining a matrix Xb. The first n1 columns are referred to as “treatments”, the second n2 columns as “controls”.
2. For each gene, calculate the corresponding unadjusted p-values, pi,b , i= 1,2, m, (e.g. by further permutations) based on the permuted matrix Xb.
After all the B permutations are done.
3. Compute the adjusted p-values πi = #{b: minl pl,b ≤ pi}/B.
![Page 33: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/33.jpg)
33
The computing challenge: The computing challenge: iterated permutationsiterated permutations
The procedure is quite computationally intensive if B is very large (typically at least 10,000) and we estimate all unadjusted p-values by further permutations.
Typical numbers:
• To compute one unadjusted p-value B = 10,000
• # unadjusted p-values needed B = 10,000
• # of genes m = 6,000. In general run time is O(mB2).
![Page 34: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/34.jpg)
34
Avoiding the computational difficulty of Avoiding the computational difficulty of single-step minP adjustmentsingle-step minP adjustment
• maxT method: (Chapter 4 of Westfall and Young)
πi = Pr( max |Tl | ≥ | ti | | H0C )
1≤l≤m
needs B = 10,000 permutations only.
However, if the distributions of the test statistics are not identical, it will give more weight to genes with heavy tailed distributions (which tend to have larger t-values)
•There is a fast algorithm which does the minP adjustment in O(mBlogB+mlogm) time.
![Page 35: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/35.jpg)
35
Proof for the single-step maxT adjustmentProof for the single-step maxT adjustment
Given level , let c such that pr(max1 ≤ i ≤ m |Ti |≤ c | H0C) = .
Note the { Pi ≤ } { |Ti | ≤ c} for any i. Then we have (cf. min P) pr(reject at least one Hi at level | H0
C)
=pr (at least one Pi ≤ | H0C)
=pr ( min1 ≤ i ≤ m Pi ≤ | H0C)
=pr (max1 ≤ i ≤ m |Ti | ≤ ca | H0C)
= . To simplify the notation we assumed a two sided test by using the
statistic Ti .We also assume Pi ~ U[0,1].
![Page 36: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/36.jpg)
36
More powerful methods: More powerful methods: step-down adjustmentsstep-down adjustments
The idea: S Holm’s modification of Bonferroni.
Also applies to Sidák, maxT, and minP.
![Page 37: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/37.jpg)
37
S Holm’s modification of BonferroniS Holm’s modification of Bonferroni
Order the unadjusted p-values such that pr1 ≤ pr2 ≤ ≤ prm. . The indices r1, r2, r3,.. are fixed for given data.
For control of the FWER at level , the step-down Holm adjusted p-values are
πrj = maxk {1,…,j} {min((m-k+1)prk, 1).
The point here is that we don’t multiply every prk by the same factor m, but only the smallest. The others are multiplied by successively smaller factors: m-1, m-2, ..,. down to multiplying prm by 1.
By taking successive maxima of the first terms in the brackets, we can get monotonicity of these adjusted p-values.
Exercise: Prove that Holm’s adjusted p-values deliver strong control. Exercise: Define step-down adjusted Sidák p-values analogously.
![Page 38: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/38.jpg)
38
Step-down adjustment of minPStep-down adjustment of minP
Order the unadjusted p-values such that pr1 ≤ pr2 ≤ ≤ prm.
Step-down adjustment: it has a complicated formula, see below, but in effect is
1. Compare min{Pr1, , Prm} with pr1 ;
2. Compare min{Pr2, , Prm} with pr2 ;
3 Compare min{Pr3 , Prm} with pri3 …….
m. Compare Prm with prm . Enforce monotonicity on the adjusted pri . The formula is
πrj = maxk{1,,…,j} {pr(minl{rk,…rm} Pl ≤ prk | H0C )}.
![Page 39: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/39.jpg)
39
False discovery rateFalse discovery rate(Benjamini and Hochberg 1995)(Benjamini and Hochberg 1995)
Definition: FDR = E(V/R |R>0) P(R >0).
Rank the p-values pr1 ≤ pr2 ≤ …≤ prm.
The adjusted p-values are to control FDR when Pi are independently distributed are given by the step-up formula:
ri= mink {i…m} { min (mprk/k ,1) }. We use this as follows: reject pr1 ,pr2 ,…, ,prk* where k* is the largest k such that
prk ≤ k/m. . This keeps the FDR ≤ under independence, proof not given.
Compare the above with Holm’s adjustment to control FWE, the step-down version of Bonferroni, which is i = maxk {1,…i} { min (kprk ,1) }.
![Page 40: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/40.jpg)
40
Positive false discovery rate Positive false discovery rate (Storey, 2001, independent case)(Storey, 2001, independent case)
A new definition of FDR, called positive false discovery rate (pFDR) pFDR= E(V/R | R >0)
The logic behind this is that in practice, at lease one gene should be expected to be differentially expressed.
The adjusted p-value (called q-value in Storey’s paper) are to control pFDR.
i= mink {1..,i} {m/k pk 0}
Note 0 = m0 /m can be estimated by the following formula for suitable 0= #{pi>}/ {(1-) m}.
One choice for is 1/2; another is the median of the observed (unadjusted) p-values.
![Page 41: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/41.jpg)
41
Positive false discovery rate Positive false discovery rate ( Storey, 2001, dependent case)( Storey, 2001, dependent case)
In order to incorporate dependence, we need to assume identical distributions.Specify 0 to be a small number, say 0.2, where most t-statistics will fall
between (-0, 0) for a null hypothesis, and to be a large number, say 3, where we reject the hypotheses whose t-statistics exceeds .
For the original data, find the W = #{i: |ti| 0} and R= #{i: |ti| }.We can do B permutations, for each one, we can compute Wb and Rb simply
by: Wb = #{i: |ti| 0} and Rb= #{i: |ti| }, b=1,…, B.The we can compute the proportion of genes expected to be null 0=W/{(W1+W2+…+Wb)/B)
An estimate of the pFDR at the point will be 0{(R1+R2+…+RB)/B}/R.Further details can be found in the references.
![Page 42: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/42.jpg)
42
ResultsResults
Random data
![Page 43: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/43.jpg)
43
![Page 44: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/44.jpg)
44
ResultsResults
Apo AI data
![Page 45: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/45.jpg)
45
Histogram & normal q-q plot of t-statisticsHistogram & normal q-q plot of t-statistics
ApoA1
![Page 46: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/46.jpg)
46
![Page 47: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/47.jpg)
47Callow data with some FDR values included
![Page 48: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/48.jpg)
48
![Page 49: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/49.jpg)
49
gene t unadj. p minP plower maxTindex statistic (104) adjust. adjust.2139 -22 1.5 .53 8 10-5 2 10-4
4117 -13 1.5 .53 8 10-5 5 10-4
5330 -12 1.5 .53 8 10-5 5 10-4
1731 -11 1.5 .53 8 10-5 5 10-4
538 -11 1.5 .53 8 10-5 5 10-4
1489 -9.1 1.5 .53 8 10-5 1 10-3
2526 -8.3 1.5 .53 8 10-5 3 10-3
4916 -7.7 1.5 .53 8 10-5 8 10-3
941 -4.7 1.5 .53 8 10-5 0.652000 +3.1 1.5 .53 8 10-5 1.005867 -4.2 3.1 .76 0.54 0.904608 +4.8 6.2 .93 0.87 0.61948 -4.7 7.8 .96 0.93 0.66
5577 -4.5 12 .99 0.93 0.74
![Page 50: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/50.jpg)
50
The gene namesThe gene names
Index Name
2139 Apo AI
4117 EST, weakly sim. to STEROL DESATURASE
5330 CATECHOL O-METHYLTRANSFERASE
1731 Apo CIII
538 EST, highly sim. to Apo AI
1489 EST
2526 Highly sim. to Apo CIII precursor
4916 similar to yeast sterol desaturase
![Page 51: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/51.jpg)
51
ResultsResults
Golub data
![Page 52: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/52.jpg)
52
![Page 53: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/53.jpg)
53
![Page 54: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/54.jpg)
54
![Page 55: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/55.jpg)
55
Golub data with minP
![Page 56: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/56.jpg)
56
Golub data with maxT
![Page 57: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/57.jpg)
57
![Page 58: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/58.jpg)
58
Discussion
The minP adjustment seems more conservative than the maxT adjustment, but is essentially model-free.
With the Callow data, we see that the adjusted minP values are very discrete; it seems that 12,870 permutations are not enough for 6,000 tests.
With the Golub data, we see that the number of permutations matters. Discreteness is a real issue here to, but we do have enough permutations.
The same ideas extend to other statistics: Wilcoxon, paired
t, F, blocked F.Same speed-up works with the bootstrap.
![Page 59: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/59.jpg)
59
Leukemia data: run timesLeukemia data: run times
Constants computed from timings for B = 5,000 permutations
![Page 60: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/60.jpg)
60
Discussion, ctd.
Major question in practice: Control of FWER or some form of FDR?
In the first case, use minP, maxT or something else? In the second case, FDR, pFDR or something else. If minP is to be used, we need guidelines for its use in terms of
sample sizes and number of genes.
Another approach: Empirical Bayes. There are links with pFDR.
![Page 61: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/61.jpg)
61
Selected referencesSelected referencesWestfall, PH and SS Young (1993) Resampling-based multiple testing: Examples and methods for p-value adjustment, John Wiley & Sons, Inc
Benjamini, Y & Y Hochberg (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing JRSS B 57: 289-300
J Storey (2001): 3 papers (some with other authors), www-stat.stanford.edu/~jstorey/
The positive false discovery rate: a Bayesian interpretation and the q-value.A direct approach to false discovery ratesEstimating false discovery rates under dependence, with applications to microarraysY Ge et al (2001) Resampling-based multiple testing for microarray data analysis, Test (to appear), see #633 in http://www.stat.Berkeley.EDU/tech-reports/index.html
![Page 62: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/62.jpg)
62
SoftwareSoftware
C and R code available for different tests:multtest in
http://www.bioconductor.org
![Page 63: Multiple testing in large-scale gene expression experiments](https://reader035.vdocuments.us/reader035/viewer/2022062501/56815bde550346895dc9cf06/html5/thumbnails/63.jpg)
63
AcknowledgementsAcknowledgements
Yongchao GeYee Hwa (Jean) Yang
Sandrine DudoitBen Bolstad