1

Risk Assessment Tests

Marina Kondratovich, Ph.D.

OIVD/CDRH/FDA

March 9, 2011

Molecular and Clinical Genetics Panel for Direct-to-Consumer (DTC) Genetic Tests

2

1. Introduction2. Basic concepts: risks, relative risks, likelihood ratios, odds ratios3. Description of a typical DTC risk assessment test4. Clinical validation (discrimination and calibration)

Overview

3

1. Introduction Susceptibility/Pre-dispositional tests (DTC Risk Assessment tests): tests that estimate the risk (relative or absolute) that an individual will develop a condition during the lifetime Examples: test for Alzheimer’s disease, test for prostate cancer, test for type 2 diabetes

Possible Intended Use Claim:“…to estimate the likelihood that an individual will develop <target condition> during the lifetime…”

4

Typical DTC Risk Assessment Test

Individual

Covariates Markers

Race Gender SNP1 SNP2 SNP3 SNP4

Pre-test risk (average risk)

Race and gender specific

Relative risk Absolute risk Risk category(as “Low”, “Average”,

“High”)

20%Race=European

Gender=Male

1.5 30%

(1.5 x 20%)

“High”

5

2. Basic Concepts Absolute risks, relative risks

Likelihood ratios, odds ratios

Test with more than two outcomes

6

Consider Test with Two Outcomes (Pos., Neg.)

D+ D- Total

T + 70 160 230

- 30 240 270

Total 100 400 500

Clinical Performance of the Test

Sensitivity 70.0% (70/100)

Specificity 60.0% (240/400)

Let us have 500 subjects who are representative subjects from intended use population (target population). Each subject has results of the Test (Pos., Neg.) and a Clinical Reference Standard (“Gold Standard”) (D+, D-). Prevalence of 20% (100/500) reflects prevalence in the IU population.

7

Risks (Absolute Risks)

Clinical Performance of the Test

R1=Risk of D+ for T+ (PPV)* 30.4% (70/230)

R0=Risk of D+ for T- (1-NPV)* 11.1% (30/270)

π = Pre-test risk of D+ (baseline risk, prevalence, average risk (averaged over risk factors))

20.0% (100/500)

D+ D- Total

T + 70 160 230

- 30 240 270

Total 100 400 500

*Post-test risk for T Pos, post-test risk for T Neg.

8

Relative Risks

R1/π = 1.52 : For a subject with T+, the risk increases by

1.52 times with regard to pre-test risk (=30.4/20.0); R0/π = 0.56 : For a subject with T-, the risk increases by 0.56

times (decreased by 1.80 (1/0.56) times) with regard to pre-test risk (=11.1/20.0); R1/R0 = 2.74 : For a subject with T+, the risk increases by 2.74

times with regard to the subjects with T- (=30.4/11.1)

Clinical Performance of the Test

R1=Risk of D+ for T+ (PPV) 30.4% (70/230)

R0=Risk of D+ for T- (1-NPV) 11.1% (30/270)

π = Pre-test risk of D+ 20.0% (100/500)

9

Absolute risks and relative risk depend on the sensitivity, specificity and also on the pre-test risk.

D+ D- Total

T + 70 160 230

- 30 240 270Total 100 400 500

Se Sp

R1

R0

D+ D- Total

T + 140 120 260

- 60 180 240Total 200 300 500

Se = 70.0% (70/100)Sp = 60.0% (240/400)Pre-test risk =20% (100/500)R1=30.4% (70/230)R0=11.1% (30/270)R1/π=1.52; R0/π=0.56; R1/R0= 2.74

Se = 70.0% (140/200)Sp = 60.0% (180/300)Pre-test risk =40% (200/500)R1=53.8% (140/260)R0=25.0% (60/240)R1/π=1.35; R0/π=0.63; R1/R0= 2.15

10

“Odds” are the ratio of the probability of one outcome to the probability of its opposite outcome.

Example: Single fair coin with outcomes {Head, Tail}: odds =1 because Pr (Head)=0.5 and Pr (Tail)=1-0.5=0.5 => odds=1 (0.5/0.5=1).

Likelihood Ratios (LR) are another way to describe the performance of a test.

Likelihood Ratios, Odds Ratios

11

Subject from the IU population with pre-test risk π, two outcomes (D+, D-); Pr(D+) = π.

-1

oddstest -Pre

Likelihood Ratios, Odds Ratios (Continued)

After the test is performed (with knowledge of the test results):

1

1

1)T|Pr(D-1

)|Pr( )odds(Ttest -Post

R

RTD

0

0

1T-)|Pr(D-1

)|Pr( odds(T-)test -Post

R

RTD

Is there a relationship between post-test odds and pre-test odds?

12

1

)(1 1

1 TLRR

R

1

)(1 0

0 TLRR

R

Post-test odds = Likelihood Ratio x Pre-test odds

Likelihood Ratios, Odds Ratios (Continued)

Sp

SeTLR

1)(

Sp

SeTLR

1)(

13

Post-test odds = Likelihood Ratio x Pre-test odds

Likelihood Ratios, Odds Ratios (Continued)

Sp

SeTLR

1)(

Sp

SeTLR

1)(

)(

)((OR) Ratio Odds

TLR

TLR

Likelihood Ratios do not depend on the pre-test risk.

Odds Ratio does not depend on the pre-test risk.

14

Consider Test with More than Two Outcomes.

Let us have 500 subjects who are representative subjects from the intended use population (target population). Each subject has results of the Test and a Clinical Reference Standard (D+, D-). Prevalence of 20% (100/500) reflects prevalence in the IU population.

In the hypothetical example, the test measures four markers: each marker has three possible results (aa, Aa, AA) Then the test has 81 possible results (=3 x 3 x 3 x 3).

For the sake of simplicity, consider test with three outcomes: as (Result1, Result2 and Result3).

15

Test with Three Outcomes: as (Result1), (Result2) and (Result3).

D+ D- Total Risk

TResult3 24 72 96 25.0%Result2 56 216 272 20.6%Result1 20 112 132 15.2%

100 400 500 20.0%

Pre-test odds: 0.200/(1-0.200) = 0.250 Post-test odds(Result3): 0.250/(1-0.250) = 0.333 Post-test odds(Result2): 0.206/(1-0.206) = 0.259 Post-test odds(Result1): 0.152/(1-0.152) = 0.179

Is there a relationship between post-test odds and pre-test odds?

16

Post-test odds (Resulti) = LR(Resulti) x Pre-test odds

Likelihood Ratios, Odds Ratios

D-) |Pr(Result

)D|Pr(Result)LR(Result

i

ii

LR is a way of quantifying how much given test result changes the pre-test (baseline) risk of the target condition.

D+ D- Total Risk D+ D- LR

TResult3 24 72 96 25.0% 24.0% 18.0% 1.33

Result2 56 216 272 20.6% 56.0% 54.0% 1.04

Result1 20 112 132 15.2% 20.0% 33.0% 0.61

100 400 500 20.0% 100% 100%

17

Likelihood Ratios, Odds Ratios (Continued)

ORs are usually considered with regard to the Result with the lowest risk (normalized to the lowest risk): ORi=LRi/LR1. LRs are related to the pre-test risk (average risk).

D+ D- D+ D- LR OR

TResult3 24 72 96 24.0% 18.0% 1.33 2.18

Result2 56 216 272 56.0% 54.0% 1.04 1.70

Result1 20 112 132 20.0% 33.0% 0.61 1.00

100 400 500 100% 100%

18

Summary

Risks and relative risks Risks and relative risks depend on corresponding likelihood ratios and pre-test (baseline) risk.

Because risks (and relative risks) depend on pre-test risk, in some study designs, they cannot be estimated (as in case-control studies).

Risks and relative risks measure probabilities of events in a way that is interpretable and consistent with how people think.

19

Likelihood Ratios (LR) and Odds Ratio (OR) LRs and ORs do not depend on the pre-test risk.

Because they do not depend on the pre-test risk, LRs and ORs can be calculated even in the case-control studies.

It is easy to adjust an OR for other variables (logistic regression)

LRs and ORs are more difficult for interpretation because they are related to pre-test and post-test odds, which are not intuitive.

Summary (Continued)

20

3. Description of Typical DTC Risk Assessment Test

For the sake of simplicity, consider a test which measures four markers: each marker has three possible results: (aa, Aa, AA). Then the test has 81 possible results (=3 x 3 x 3 x 3).

Post-test Odds = Likelihood Ratio x Pre-test Odds

1

),,,(1 ,,,

,,,lkji

lkji

lkji DCBALRR

R

Consider that an individual has test result (Ai, Bj, Ck, Dl).Basic idea of calculation of the risk for this individual is

21

),,,( lkji DCBALRNote 1

For a given race/ethnicity, information from case-control studies in published literature is used (independent confirmations of GWAS)

1) Even for the same set of published papers related to the target condition (disease), different markers (SNPs) can be included in the test (different approaches for selection of SNPs are used).

2) Even for the same set of published papers and for the same SNP included in the test, different OR estimates can be used in the calculation of the LR for the test result (different approaches are used). For example, estimates of OR are: 1.2 in paper 1; 1.4 in paper 2; 1.1 in paper 3

(study with largest sample size? meta-analysis? …)

3) Information about OR in the case-control studies is used for calculation of LR. Different assumptions are considered.

For example, an assumption that “Controls” are not subjects without disease but a random

sample from population.

22

),,,( lkji DCBALRNote 2

Consider the test for the target condition with 4 markers (SNPs);

ORs for individual markers are obtained from literature.

SNP1 SNP2 SNP3 SNP4

LRResult3 1.27

BB1.55

DD

Result21.05

Aa

Result1 0.77cc

Multiplicative Model: an assumption that all four SNPs are independent (no interactions). This assumption may be not correct.

59.155.177.027.105.1),,,( lkji DCBALR

23

Note 3 Pre-Test Risk, π

Absolute risk Ri,j,k,l is calculated based on corresponding LR and the pre-test risk (average risk).

Pre-test risk is provided based on the publicly available information about race- and gender- specific lifetime risks (for example, Surveillance Epidemiology and End Results (SEER) Cancer Statistics Review).

Pre-test risk (average risk) is gender- and race- specific (very limited number of factors). The average risks present risks averaged over other important risk factors (such as family history, smoking, environmental factors and so on). => An individual pre-test risk taking into account other important factors can be very different from the average risk.

24

Note 3 Pre-Test Risk, π (Continued)

Race and gender specific risk averaged over other

important factors

5% 20% 35%

Pre-Test Risk 5% 20% 35%

LR 1.5

Post-Test Risk* 7.3% 27.3% 44.7%

Increase in Risk 2.3% 7.3% 9.7%

Relative risk 1.46 1.36 1.28

Subjects of the same gender and race and with low

risk factors

Subjects of the same gender and race and with high

risk factors

1

5.11 R

R*

Hypothetical Example

R

risk Relative

25

Absolute values of the post-test risks are considerably affected by the pre-test risk. In hypothetical example of LR=1.5 and pre-test average risk =20% (range 5%-35%), post-test risks were from 7.3% to 44.7%.

Relative risks are also affected by the pre-test risks but to much lesser degree. In hypothetical example of LR=1.5 and pre-test average risk =20% (range 5%-35%), relative risks were from 1.28 to 1.46.

If the pre-test risk is very low, then relative risk ≈ LR In hypothetical example of LR=1.5 and pre-test average risk =3% (range 1%-5%), relative risks were from 1.46 to 1.49.

If there is an assumption that “Controls” in the case-control study are not subjects without disease but a random sample from the intended use population (this assumption may be not correct), then relative risk = LR.

Note 3 Pre-Test Risk, π (Continued)

26

Note 4 Risk Categories “Low”, “Average”, “High”

Various approaches (based on either relative risks, or likelihood ratios, or absolute risks) and different cutoffs can be used for defining these three categories.

Pre-test risk 20%

LR Low Average High 0.80 1.00 1.20

RR Low Average High

Risk Low Average High

0.833 1.00 1.154

16.67% 20.0% 23.08%

Hypothetical example:

The same person can be classified into different risk categories.

27

DTC risk assessment tests report an absolute risk for an individual.

Note that, with regard to the study designs, the absolute risks cannot be evaluated in the case-control studies.

For the absolute risks, a clinical validation includes two aspects: discrimination and calibration.

4. Clinical Validation

28

4. Clinical Validation (Continued)DiscriminationUnder discrimination, we understand the ability of the test to discriminate between subjects who have target condition and subjects who do not. We would like that the subjects with target condition have higher values of the absolute risk compare to the absolute risks of the subjects without target condition. For assessing discrimination, a receiver operating characteristic (ROC) analysis is used (ROC curve and area under ROC curve along with 95% confidence interval).

As a general rule, the larger is the area under ROC curve, the better is the discriminatory capability of the test.

29

ROC AnalysisTest Variable: absolute risk values for Diseased group (Y);

absolute risk values for Non-Diseased group (X)

AUC = P{Y>X}, probability that an absolute risk of a randomly selected Diseased subject is larger than an

absolute risk of a randomly selected Non-Diseased subject.

X Y

Consider, for example, a wrong pre-test risk and correct pre-test risk (all other calculations are the same): ROC curves are the same.

30

CalibrationAbsolute risks should be well-calibrated. If one has 100 subjects and the test is telling that that their risk is 12%, then one can anticipate that among these 100 subjects, approximately 12 subjects will have the target condition in reality.

Calibration evaluates the degree of correspondence between the risk of the target condition provided by the test (Expected according to the absolute risk by the test) and the actual risk of the target condition (Observed). Calibration of the test which provides absolute risks cannot be evaluated in the case-control studies.

4. Clinical Validation (Continued)

31

Summary

1. Absolute and relative risks provided by the DTC risk assessment tests are calculated based on different approaches that can lead to inconsistencies in the results.

2. Absolute risks depend considerably on the pre-test risks.

3. Absolute risks in the DTC risk assessment tests do not include important risk factors other than markers measured by the DTC risk assessment tests and some limited number of factors (as race, gender, sometimes age).

32

Thank You!

Marina.Kondratovich@fda.hhs.gov