Optimizing the Format for Presenting Updates of Evidence Reviews:

Seeking User Preferences

Sydne Newberry, PhDSouthern California

Evidence-Based Practice Center September 11, 2012

A9379-05/2011 2

Agenda

· Introduction

· Examples of executive summaries we created

· Results of focus group testing 

· Conclusions

· Next steps

A9379-05/2011 3

No one knows the optimal format for updates

· Do users prefer to see the original and the updated findings side by side?

- Numerical or narrative?

· Do users care about seeing the original findingsat all?

· Does the answer depend on the users? - And who are they???

We tested reactions to a variety of formats among various categories of users

A9379-05/2011 4

The Goals for Optimizing the Formats for Update Reports

· To achieve some degree of consistency across EPC CER updates

· But be flexible enough to allow for different situations

- e.g., the number and kind of new studies

· No single format may be appropriate for every report- or every user…SO

· Accessibility to the various stakeholders is key

A9379-05/2011 5

What we did…Chose a sample report to modify

Identified formats in use

(and users)

Created 5 test executive

summaries

Assessed reactions of a focus group

(clinical decision makers) to the summaries online and

via conference call

Created 2 new versions of the summaries and

discarded 1 based on feedback

Assessed reactions of 2nd focus group

(research decision makers) to the summaries

Assessed reactions of 3rd group (community physicians)

to the summaries

Summarized results

A9379-05/2011 6

We transformed an update report using the formats of 3 other update reports

· A typical update report of a large consortium

· An update report for a government agency

· The EPC update review on acute otitis media itself- Executive summary only - Meta-analyses and quantitative results

· An EPC update review on osteoporosis treatment - Executive summary- Qualitative findings only

A9379-05/2011 7

· Health plan managers

· Other payers (private insurers, CMS)

· Clinical guideline developers (professional practice societies, USPSTF, …)

· Research planners

· Congress

· Clinicians at the point of care?

- EPC’s Eisenberg Center focuses on this group

Who Uses Evidence Reports?

A9379-05/2011 8

What we did…Chose a sample report to modify

Identified formats in use

(and users)

Created 5 test executive

summaries

Assessed reactions of a focus group

(clinical decision makers) to the summaries online and

via conference call

Created 2 new versions of the summaries and

discarded 1 based on feedback

Assessed reactions of 2nd focus group

(research decision makers) to the summaries

Assessed reactions of 3rd group (community physicians)

to the summaries

Summarized results

A9379-05/2011 9

Version 1…

· Shows new publication date

· Shows new search end date

· Lists new included studies, if any, in evidence table

· Provides a table of changes to staff, key questions, methods at the end of the report

In other words, previous and updated pooled results are not explicitly compared

A9379-05/2011 10

Version 1 replaces old with new

OverviewThe objective of this report was to analyze the evidence on the initial management of uncomplicated acute otitis media (AOM)…

The Technical Expert Panel and project staff developed a literature search strategy. The initial strategy was developed for MEDLINE and was customized for other databases. Project staff searched MEDLINE (January 1998-July 2010), the Cochrane Library ( January 1998-July 2010),and the Web of Science . Additional articles were identified by review of reference lists in proceedings, published articles, reports, and guidelines.

A9379-05/2011 11

Focus Group 1 Included 7 Participants· Health plan pediatrician who serves on guidelines

committees and has been on 2 TEPs

· Academic family practitioner who has been on 2 TEPS

· Insurance company research director, former EPC project leader

· Academic physician/health services researcher/ guidelines developer

· Health plan physician/manager and guideline developer

· MS-level coordinator for guidelines committees

· External affairs officer (former congressional staffer)

A9379-05/2011 12

Focus group 1’s reactions to Version 1 wereuniformly negative

· BUT thought this format might be preferred by the average clinician

- i.e., one who is less interested in what is old vs. new, and simply needs to know what is up-to-date

· The health plan guidelines writer shared that his organization’s guidelines for practitioners are formatted exactly this way

A9379-05/2011 13

Version 2 “Highlights” Changes

The objective of this report was to analyze the evidence on the initial management of uncomplicated acute otitis media (AOM) in children. AOM represents the most common childhood infection for which antibiotics are prescribed in the United States. Data from the National Ambulatory Medical Care Surveys (NAMCS), which did not differentiate between AOM and otitis media with effusion, indicated that the number of office visits for AOM increased more than two-fold from 1975 to 1990. Gates (1996a) believed that the majority of these cases represented AOM. A diagnosis of AOM requires 1) a history of acute onset of signs and symptoms, 2) the presence of middle ear effusion (MEE), and 3) signs and symptoms of middle-ear inflammation. (Marcy, Takata, Shekelle, et al., 2001).  

for which antibiotics are prescribed in the United States.

represents the most common childhood infection

A9379-05/2011 14

· Enthusiasm increased if the update report had identical key questions and only quantitative findings changed

Several participants slightly preferred Version 2

A9379-05/2011 15

Version 3 Used Revision Marks to Show Changes to Text

• In general we could not make definitive conclusions regarding differences in adverse event rates among antibiotics when taking into account a MCID of 5%. Adverse events were generally more frequent for amoxicillin-clavulanate than for cefdinir, ceftriaxone, or azithromycin.

• Meta-analysis demonstrated that children treated with 7 to 10 day amoxicillin-clavulanate had an 12.9 18 percent (95 percent confidence intervals,4.58 percent and 21.228 percent) greater rate of gastrointestinal adverse effects than children treated with 5-day azithromycin (although not reported in the studies, the clavulanate concentration was most likely 31.25 mg per 125 mg of amoxicillin, i.e., original formulation.)

A9379-05/2011 16

No one in Focus Group 1liked this version!!!

A9379-05/2011 17

Version 4 represented entirely new text but featured comparison tables

Comparison # Trials

Success rate diff % (95% CI)

# New Trials

Total # Trials

Success rate diff (95% CI) Conclusion

Drug vs. placebo, wait-and-see, and/or Rx to hold

2001 Report 2010 Update

Ampicillin/amoxicillin vs. placebo

5 12 (3,22) 2 7 12 (5,18) Ampicillin/amoxicillin more successful than placebo

Amoxicillin vs. placebo/wait-and-see/Rx to hold

0 n/a 9 9 12 (6, 17) Amoxicillin more successful than placebo/wait-and-see/Rx to hold

A9379-05/2011 18

Key Question Strength of Evidence

Conclusion

What is the natural history of AOM

About 85 percent of children with AOM who are not initial treated with antibiotics have gotten better – had resolution of pain and fever - on their own within 7 days

Diagnosis: What are the operating characteristics of clinical symptoms and otoscopic findings (such as bulging tympanic membrane), both individual and composite, to diagnose uncomplicated AOM and to distinguish it from otitis media with effusion (OME)?

Three clinical criteria are necessary to diagnose AOM: 1. acute symptoms of infection, 2. evidence of acute tympanic membrane (TM) inflammation, and 3. presence of middle ear effusion (MEE)

Only otalgia (ear pain) and ear rubbing seemed to predict a clinical diagnosis of AOM. AOM diagnosis was not associated with the occurrence, duration, or severity of parent-reported symptoms (e.g., ear pain, ear rubbing, fever)

Version 5 Features a Narrative Summary Table with Bolding, and Shading

Some SRs have only narrative

findings

A9379-05/2011 19

Focus group 1 participants infinitely preferred

Versions 4 and 5!!They shared that

· Version 4 would be preferable for a report with quantitative findings (i.e., MA and pooled effect sizes), whereas

· Version 5 would be preferable for a report with qualitative findings

· Tables like these might even make version 1 more useful to a broad range of users…

A9379-05/2011 20

More Suggestions to Emphasize Important Changes in an Update Report

· Show updated findings in context of old, - most readers won’t have the earlier report on hand- compare results in text

· Include both qualitative AND quantitative tables - or at least include strength of evidence in table- Show changes to key findings in a bulleted list

· For decisionmakers, provide the figures and tables with the actual data in addition to the summary tables

- One of our focus group participants shared the format used by his health plan…

Based on the feedback, we tossed out Version 3, combined Versions 4 and 5, and created a new version

A9379-05/2011 21

What we did…Chose a sample report to modify

Identified formats in use

Created 5 test executive

summaries

Assessed reactions of a focus group

(clinical decision makers) to the summaries online and

via conference call

Created 2 new versions of the summaries and

discarded 1 based on feedback

Assessed reactions of 2nd focus group

(research decision makers) to the summaries

Assessed reactions of 3rd group (community physicians)

to the summaries

Summarized results

A9379-05/2011 22

The new Version 3 simply included both quantitative and narrative tables…

A9379-05/2011 23

The new version 4 contained no narrative text

Table S-1. Scope of the Report and Definitions

Disease Entity

Uncomplicated AOM

Patient Population

Age 4 weeks to 18 yearsExclude: patients with immunodeficiencies and craniofacial deficiencies including cleft palate, etc.

Table S-2. Search CriteriaKey Question

Databases Article Types Timeframe # Studies Reviewed

# Studies Included

Results:• Quantitative tables• Narrative tables

• Conclusions• Limitations• Future Research Needs

A9379-05/2011 24

What we did…Chose a sample report to modify

Identified formats in use

(and users)

Created 5 test executive

summaries

Assessed reactions of a focus group

(clinical decision makers) to the summaries online and

via conference call

Created 2 new versions of the summaries and

discarded 1 based on feedback

Assessed reactions of 2nd focus group

(research decision makers) to the summaries

Assessed reactions of 3rd group (community physicians)

to the summaries

Summarized results

A9379-05/2011 25

Focus Group 2 Included 5 Participants· A director of an office at FDA

· A director of a federally funded academic research center

· An FDA microbiologist

· An administrator at CMS

· A former congressional staffer with health policy experience

A9379-05/2011 26

Focus group 2 participants preferred versions 3 and 4

· Versions 1 and 2 were not helpful- Want to see what changed and why

· New version 3 (both sets of tables) was better, but- show how the new findings contributed to changing the

conclusions;- use color coding or different fonts to increase the salience

of what changed; and - include some wording that would put the changes in

context. - Show why conclusions changed

· New version 4 was even better but not a substitute for full report

A9379-05/2011 27

What we did…Chose a sample report to modify

Identified formats in use

(and users)

Created 5 test executive

summaries

Assessed reactions of a focus group

(clinical decision makers) to the summaries online and

via conference call

Created 2 new versions of the summaries and

discarded 1 based on feedback

Assessed reactions of 2nd focus group

(research decision makers) to the summaries

Assessed reactions of 3rd group (community physicians)

to the summaries

Summarized results

A9379-05/2011 28

Community physicians…

· May be the busiest of all groups we included!

· Disliked all but version 4! - Preferred graphic display and lack of narrative

text

A9379-05/2011 29

· Tested only 1 report

· Included only the executive summary, not the full report, but· Equivalent to format used by large HMO

· Original report included (almost) only quantitative results

· Small number of focus groups and participants· Possible sample bias

Our study had a number of limitations…

A9379-05/2011 30

Conclusions

• Update reports need to show at least what changed

• A stand-alone executive summary without narrative text may suffice for point-of-care clinicians

• Policy makers need full reports

− But stand-alone summary serves as useful map to full report

• Optimum method of presenting conclusions depends on whether meta-analytic or narrative…

A9379-05/2011 31

Next Steps

· We have submitted draft report to AHRQ

· Followup testing with more reports and focus groups would be helpful

− We propose submitting an update report with a “Version 4” style executive summary for peer review and public comment