Recommendation Methods

Advisory Committee on Heritable Disorders and Genetic Diseases

of Newborns and Children

Ned Calonge, M.D., M.P.H.

Workgroup Members

Denise Dougherty, Piero Rinaldo, Coleen Boyle, Michael Watson, Tracy Trotter, Sharon Terry

Liaison from External Workgroup: Nancy Green, MD, Columbia University

Liaison/Committee Staff: Michele Puryear

Process for creating recommendations based on Systematic Evidence Review

Anticipate not having direct evidence of screening efficacy

Create chain of evidence, evaluating» Analytic validity» Clinical validity» Clinical utility

Base recommendation on certainty of net benefit

Steps in process

Define the question regarding testing for the specific question within an analytic framework that includes an overarching key question (direct evidence) and a chain of related key questions (indirect evidence)

Figure 1. Analytic Framework

General population

of newborns

Testing for condition

3Mortality, morbidity,and other outcomes

1

4

6 7

Harms of testing/identification

Harms of treatment/other

interventions

2 Identificationof condition

Treatment ofCondition

6

Key question 1

Is there direct evidence that screening for the condition at birth leads to improved health outcomes? (overarching question)

Best evidence would be randomized trials involving screen-detected infants

For many conditions considered by the Advisory Committee, it is unlikely that there will be direct evidence

Key question 2

What is known about the condition? » Is the condition well-defined and important? » What is the incidence of the condition in the U.S.

population? » What is the spectrum of disease for the

condition? » What is the natural history of the condition,

including the impact of recognition and treatment?

Key question 3

Is there a test for the condition with sufficient analytic utility and validity?

Refers to the laboratory performance of the test

» analytic reliability

» assay robustness

Key question 4

Does the test accurately and reliably detect the condition and clinical disease? Clinical validity:» Sensitivity» Specificity» Positive predictive value» False positive rate

Measures must relate to clinical/ symptomatic disease (phenotype)

Key question 5

Are there available treatments for the condition that improve important health outcomes?

Does treatment of the condition detected through NBS improve important health outcomes when compared with waiting until clinical detection?

Are there subsets of affected children more likely to benefit from treatment that can be identified through testing or clinical findings?

Are the treatments for affected children standardized, widely available, and if appropriate, FDA approved?

Key question 5

The Advisory Committee will need to determine which outcomes should be considered as important health outcomes» Patient outcome impacts (mortality/morbidity) » Therapeutic/management decisions» Diagnostic thinking/health information impact» Familial and Societal impact

Not necessarily of equal weight

Key questions 6 & 7

Are there harms or risks identified for the identification? Harms or risks for treatment of affected children?» Harms of screening, including ELSI» Harms of diagnostic workup for screen positives» Harms of treatment (especially if no benefit, or if

provided to false positives)

Key question 8

What is the estimated cost-effectiveness of testing for the condition?» It is unlikely there will be empiric data» May be addressed through decision

modeling, which can provide estimates that the Advisory Committee will take into consideration when considering a recommendation

Translating evidence into recommendations

Judgment regarding the magnitude of net benefit (benefits minus harms)

Judgment of the adequacy of evidence in answering the key questions

Judgment of the certainty of net benefit

Magnitude of net benefit

Significant: benefits clearly outweigh harms Zero/net harm (more harm than benefit) Small net benefit

» Must carefully consider level of certainty, other issues such as cost effectiveness

Adequacy of evidence

Evidence should be classified as adequate or inadequate

Inadequate evidence for a key question represents a “break” in the evidence chain that would lead to a finding of insufficient certainty of net benefit

Adequacy should be determined by applying a set of critical appraisal questions to each key question

Critical appraisal questions

1. Do the studies have the appropriate research design to answer the key question?

2. To what extent are the studies of high quality (internal validity)?

3. To what extent are the studies generalizable to the U.S. population (external validity)?

4. How many studies and how large have been done to answer the key question (precision of the evidence)?

5. How consistent are the studies?

6. Are there additional factors supporting conclusions?

Judge the certainty of net benefit

Based on the evidence, estimate the magnitude of benefit or potential benefit

Based on the evidence, estimate the magnitude of harm or potential harm

Estimate net benefit (benefits minus harms) Base judgment of certainty of net benefit

through applying critical appraisal questions across the chain of evidence

Certainty Sufficient: evidence is sufficient to determine

effects on health outcomes with an acceptable risk or level of comfort of “being wrong” and thus a low susceptibility to being overturned or otherwise altered by additional research

Insufficient: evidence is insufficient to assess effects on health outcomes; additional information from future studies may allow for assessment

Insufficient certainty, but compelling contextual issues

There may be conditions where the evidence is inadequate to reach a conclusion, but contextual issues support a recommendation to add the condition, with a commitment to fill in the gaps in evidence going forward

Contextual issues might include: » Known benefits associated with testing (and

intervention) for similar conditions» High incidence that would translate to potential

substantial net benefit» Availability of promising but yet unproven new therapies» Indirect evidence of perhaps less important health

outcomes, but with evidence of low potential harms

Recommendation matrix

RECOMMENDATION LEVEL OF CERTAINTY MAGNITUDE OF NET BENEFIT

Recommend adding the condition to the core set

Sufficient Significant (note—special considerations in net benefit if small)

Recommend not adding the condition to the core set

Sufficient Zero or net harm (more harm than benefit)

Recommend adding the condition with “provisional status”

Insufficient, but potential net benefit is compelling, add and re-evaluate

Potentially significant, supported by contextual issues

Recommend not adding the condition now, but pilot studies

Insufficient, additional information is needed to support a recommendation

Potentially significant or unknown

Acknowledgements

Michele Puryear, Nancy Green, Piero Rinaldo and the other members of the decision process work group

The U.S. Preventive Services Task Force, especially former member Russ Harris

The Evaluating Genomic Applications in Practice and Prevention Work Group (EGAPP), especially Steve Teutsch and Glen Palomaki