optimizing the format for presenting updates of evidence reviews: seeking user preferences sydne...
Post on 27-Dec-2015
212 Views
Preview:
TRANSCRIPT
Optimizing the Format for Presenting Updates of Evidence Reviews:
Seeking User Preferences
Sydne Newberry, PhDSouthern California
Evidence-Based Practice Center September 11, 2012
A9379-05/2011 2
Agenda
· Introduction
· Examples of executive summaries we created
· Results of focus group testing
· Conclusions
· Next steps
A9379-05/2011 3
No one knows the optimal format for updates
· Do users prefer to see the original and the updated findings side by side?
- Numerical or narrative?
· Do users care about seeing the original findingsat all?
· Does the answer depend on the users? - And who are they???
We tested reactions to a variety of formats among various categories of users
A9379-05/2011 4
The Goals for Optimizing the Formats for Update Reports
· To achieve some degree of consistency across EPC CER updates
· But be flexible enough to allow for different situations
- e.g., the number and kind of new studies
· No single format may be appropriate for every report- or every user…SO
· Accessibility to the various stakeholders is key
A9379-05/2011 5
What we did…Chose a sample report to modify
Identified formats in use
(and users)
Created 5 test executive
summaries
Assessed reactions of a focus group
(clinical decision makers) to the summaries online and
via conference call
Created 2 new versions of the summaries and
discarded 1 based on feedback
Assessed reactions of 2nd focus group
(research decision makers) to the summaries
Assessed reactions of 3rd group (community physicians)
to the summaries
Summarized results
A9379-05/2011 6
We transformed an update report using the formats of 3 other update reports
· A typical update report of a large consortium
· An update report for a government agency
· The EPC update review on acute otitis media itself- Executive summary only - Meta-analyses and quantitative results
· An EPC update review on osteoporosis treatment - Executive summary- Qualitative findings only
A9379-05/2011 7
· Health plan managers
· Other payers (private insurers, CMS)
· Clinical guideline developers (professional practice societies, USPSTF, …)
· Research planners
· Congress
· Clinicians at the point of care?
- EPC’s Eisenberg Center focuses on this group
Who Uses Evidence Reports?
A9379-05/2011 8
What we did…Chose a sample report to modify
Identified formats in use
(and users)
Created 5 test executive
summaries
Assessed reactions of a focus group
(clinical decision makers) to the summaries online and
via conference call
Created 2 new versions of the summaries and
discarded 1 based on feedback
Assessed reactions of 2nd focus group
(research decision makers) to the summaries
Assessed reactions of 3rd group (community physicians)
to the summaries
Summarized results
A9379-05/2011 9
Version 1…
· Shows new publication date
· Shows new search end date
· Lists new included studies, if any, in evidence table
· Provides a table of changes to staff, key questions, methods at the end of the report
In other words, previous and updated pooled results are not explicitly compared
A9379-05/2011 10
Version 1 replaces old with new
OverviewThe objective of this report was to analyze the evidence on the initial management of uncomplicated acute otitis media (AOM)…
The Technical Expert Panel and project staff developed a literature search strategy. The initial strategy was developed for MEDLINE and was customized for other databases. Project staff searched MEDLINE (January 1998-July 2010), the Cochrane Library ( January 1998-July 2010),and the Web of Science . Additional articles were identified by review of reference lists in proceedings, published articles, reports, and guidelines.
A9379-05/2011 11
Focus Group 1 Included 7 Participants· Health plan pediatrician who serves on guidelines
committees and has been on 2 TEPs
· Academic family practitioner who has been on 2 TEPS
· Insurance company research director, former EPC project leader
· Academic physician/health services researcher/ guidelines developer
· Health plan physician/manager and guideline developer
· MS-level coordinator for guidelines committees
· External affairs officer (former congressional staffer)
A9379-05/2011 12
Focus group 1’s reactions to Version 1 wereuniformly negative
· BUT thought this format might be preferred by the average clinician
- i.e., one who is less interested in what is old vs. new, and simply needs to know what is up-to-date
· The health plan guidelines writer shared that his organization’s guidelines for practitioners are formatted exactly this way
A9379-05/2011 13
Version 2 “Highlights” Changes
The objective of this report was to analyze the evidence on the initial management of uncomplicated acute otitis media (AOM) in children. AOM represents the most common childhood infection for which antibiotics are prescribed in the United States. Data from the National Ambulatory Medical Care Surveys (NAMCS), which did not differentiate between AOM and otitis media with effusion, indicated that the number of office visits for AOM increased more than two-fold from 1975 to 1990. Gates (1996a) believed that the majority of these cases represented AOM. A diagnosis of AOM requires 1) a history of acute onset of signs and symptoms, 2) the presence of middle ear effusion (MEE), and 3) signs and symptoms of middle-ear inflammation. (Marcy, Takata, Shekelle, et al., 2001).
for which antibiotics are prescribed in the United States.
represents the most common childhood infection
A9379-05/2011 14
· Enthusiasm increased if the update report had identical key questions and only quantitative findings changed
Several participants slightly preferred Version 2
A9379-05/2011 15
Version 3 Used Revision Marks to Show Changes to Text
• In general we could not make definitive conclusions regarding differences in adverse event rates among antibiotics when taking into account a MCID of 5%. Adverse events were generally more frequent for amoxicillin-clavulanate than for cefdinir, ceftriaxone, or azithromycin.
• Meta-analysis demonstrated that children treated with 7 to 10 day amoxicillin-clavulanate had an 12.9 18 percent (95 percent confidence intervals,4.58 percent and 21.228 percent) greater rate of gastrointestinal adverse effects than children treated with 5-day azithromycin (although not reported in the studies, the clavulanate concentration was most likely 31.25 mg per 125 mg of amoxicillin, i.e., original formulation.)
A9379-05/2011 16
No one in Focus Group 1liked this version!!!
A9379-05/2011 17
Version 4 represented entirely new text but featured comparison tables
Comparison # Trials
Success rate diff % (95% CI)
# New Trials
Total # Trials
Success rate diff (95% CI) Conclusion
Drug vs. placebo, wait-and-see, and/or Rx to hold
2001 Report 2010 Update
Ampicillin/amoxicillin vs. placebo
5 12 (3,22) 2 7 12 (5,18) Ampicillin/amoxicillin more successful than placebo
Amoxicillin vs. placebo/wait-and-see/Rx to hold
0 n/a 9 9 12 (6, 17) Amoxicillin more successful than placebo/wait-and-see/Rx to hold
A9379-05/2011 18
Key Question Strength of Evidence
Conclusion
What is the natural history of AOM
About 85 percent of children with AOM who are not initial treated with antibiotics have gotten better – had resolution of pain and fever - on their own within 7 days
Diagnosis: What are the operating characteristics of clinical symptoms and otoscopic findings (such as bulging tympanic membrane), both individual and composite, to diagnose uncomplicated AOM and to distinguish it from otitis media with effusion (OME)?
Three clinical criteria are necessary to diagnose AOM: 1. acute symptoms of infection, 2. evidence of acute tympanic membrane (TM) inflammation, and 3. presence of middle ear effusion (MEE)
Only otalgia (ear pain) and ear rubbing seemed to predict a clinical diagnosis of AOM. AOM diagnosis was not associated with the occurrence, duration, or severity of parent-reported symptoms (e.g., ear pain, ear rubbing, fever)
Version 5 Features a Narrative Summary Table with Bolding, and Shading
Some SRs have only narrative
findings
A9379-05/2011 19
Focus group 1 participants infinitely preferred
Versions 4 and 5!!They shared that
· Version 4 would be preferable for a report with quantitative findings (i.e., MA and pooled effect sizes), whereas
· Version 5 would be preferable for a report with qualitative findings
· Tables like these might even make version 1 more useful to a broad range of users…
A9379-05/2011 20
More Suggestions to Emphasize Important Changes in an Update Report
· Show updated findings in context of old, - most readers won’t have the earlier report on hand- compare results in text
· Include both qualitative AND quantitative tables - or at least include strength of evidence in table- Show changes to key findings in a bulleted list
· For decisionmakers, provide the figures and tables with the actual data in addition to the summary tables
- One of our focus group participants shared the format used by his health plan…
Based on the feedback, we tossed out Version 3, combined Versions 4 and 5, and created a new version
A9379-05/2011 21
What we did…Chose a sample report to modify
Identified formats in use
Created 5 test executive
summaries
Assessed reactions of a focus group
(clinical decision makers) to the summaries online and
via conference call
Created 2 new versions of the summaries and
discarded 1 based on feedback
Assessed reactions of 2nd focus group
(research decision makers) to the summaries
Assessed reactions of 3rd group (community physicians)
to the summaries
Summarized results
A9379-05/2011 22
The new Version 3 simply included both quantitative and narrative tables…
A9379-05/2011 23
The new version 4 contained no narrative text
Table S-1. Scope of the Report and Definitions
Disease Entity
Uncomplicated AOM
Patient Population
Age 4 weeks to 18 yearsExclude: patients with immunodeficiencies and craniofacial deficiencies including cleft palate, etc.
Table S-2. Search CriteriaKey Question
Databases Article Types Timeframe # Studies Reviewed
# Studies Included
Results:• Quantitative tables• Narrative tables
• Conclusions• Limitations• Future Research Needs
A9379-05/2011 24
What we did…Chose a sample report to modify
Identified formats in use
(and users)
Created 5 test executive
summaries
Assessed reactions of a focus group
(clinical decision makers) to the summaries online and
via conference call
Created 2 new versions of the summaries and
discarded 1 based on feedback
Assessed reactions of 2nd focus group
(research decision makers) to the summaries
Assessed reactions of 3rd group (community physicians)
to the summaries
Summarized results
A9379-05/2011 25
Focus Group 2 Included 5 Participants· A director of an office at FDA
· A director of a federally funded academic research center
· An FDA microbiologist
· An administrator at CMS
· A former congressional staffer with health policy experience
A9379-05/2011 26
Focus group 2 participants preferred versions 3 and 4
· Versions 1 and 2 were not helpful- Want to see what changed and why
· New version 3 (both sets of tables) was better, but- show how the new findings contributed to changing the
conclusions;- use color coding or different fonts to increase the salience
of what changed; and - include some wording that would put the changes in
context. - Show why conclusions changed
· New version 4 was even better but not a substitute for full report
A9379-05/2011 27
What we did…Chose a sample report to modify
Identified formats in use
(and users)
Created 5 test executive
summaries
Assessed reactions of a focus group
(clinical decision makers) to the summaries online and
via conference call
Created 2 new versions of the summaries and
discarded 1 based on feedback
Assessed reactions of 2nd focus group
(research decision makers) to the summaries
Assessed reactions of 3rd group (community physicians)
to the summaries
Summarized results
A9379-05/2011 28
Community physicians…
· May be the busiest of all groups we included!
· Disliked all but version 4! - Preferred graphic display and lack of narrative
text
A9379-05/2011 29
· Tested only 1 report
· Included only the executive summary, not the full report, but· Equivalent to format used by large HMO
· Original report included (almost) only quantitative results
· Small number of focus groups and participants· Possible sample bias
Our study had a number of limitations…
A9379-05/2011 30
Conclusions
• Update reports need to show at least what changed
• A stand-alone executive summary without narrative text may suffice for point-of-care clinicians
• Policy makers need full reports
− But stand-alone summary serves as useful map to full report
• Optimum method of presenting conclusions depends on whether meta-analytic or narrative…
A9379-05/2011 31
Next Steps
· We have submitted draft report to AHRQ
· Followup testing with more reports and focus groups would be helpful
− We propose submitting an update report with a “Version 4” style executive summary for peer review and public comment
top related