updating surveillance system: assessing the need for updating comparative effectiveness reviews the...
TRANSCRIPT
Updating Surveillance System:Assessing the Need for Updating
Comparative Effectiveness Reviews
The University of Ottawa Evidence–based Practice Center The RAND Southern California Evidence-based Practice Center
Alexander Tsertsvadze MD, MSc
Date: September 11, 2012
Outline Background
Why update systematic reviews (SRs)? What is an update of a SR?
When/how to update a SR?AHRQ updating surveillance project
MethodsOverview of the processThe Ottawa/RAND EPC methodology
Findings and implicationsBrief overview
Background
Why update SRs?
SRs play a central role in comparative effectiveness research
With the emergence of new evidence over time, conclusion(s) of any given SR may become out of date and sometimes misleading
Changes in the evidence can have significant implications for clinical practice guidelines and for clinical and consumer decision-making
Background
What is an update of SR?
“A discrete event with the aim to search for and identify new evidence to incorporate into a previously completed systematic review”
An update must involve a search for new studies
Moher and Tsertsvadze. Lancet 2006; 367:881-3Cochrane Hand Book section 3.2.2
Background
Shojania 2007Sets of quantitative/qualitative signals indicating need for updating tested on 100 SRs; within 2 yrs, 23% of SRs had signals indicating the need for updating; the median time to the emergence of signal indicating the need for updating was 5.5 years
French 2005Of updated SRs (n=254), 9% (n=23) had changes in conclusions within 4 yrs
Garritty 201053/103 of respondents estimated >50% of their respective SRs were likely out-of-date
Background
One SR found very few methods and strategies used for updating SRs
The authors identified the need for standardized reliable/efficient method(s) for updating
(Moher and Tsertsvadze 2007; Tsertsvadze 2011)
Background
In 2007-09, Ottawa and RAND Evidence-based Practice Centers (EPCs) within the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program developed a methodology for determining the need of updating Comparative Effectiveness Reviews (CERs)
• Abbreviated searches for new evidence• Identification of signals indicating the need of updating by applying a priori set
criteria to the new evidence• Expert opinion on validity of CER conclusions• Topic-specific safety alerts from FDA and Health Canada• Rules/guidance for determining the updating status of conclusion(s) for each
key question (KQ) within CER and updating priority for each CER (low, medium, high)
(Shojania 2007; Shekelle 2009)
Background
Objectives
• Recognizing the importance of keeping CERs up-to-date, the AHRQ commissioned the RAND and Ottawa EPCs in collaboration with ECRI Institute to implement an updating surveillance system project
• This collaborative project would use the previously developed methodology to assess already published CERs (n=42) for the need of updating
___________________________________________________ - CERs unlike SRs contain several key questions/conclusions - Updating vs. assessing the need for updating
Original CER
Abbreviated searches for new evidence using the same strategy
as in the CER
Screening and extracting data from
relevant studies
Identifying qualitative/quantitative signals
Signal detection completedFDA/Health Canadaalerts Expert opinion
Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER
Low priority High priorityMedium priority
Refereed for updating
Goes to 2nd cycle of assessment 6 months later
Contacting expertsECRI surveillance Start of review
End of review
0
3rd month
Updating surveillance process
Methods
Abbreviated search for new evidence
Search strategies – the same as used in the original CER
but limited to
5 general medicine journals(Ann Intern Med,
BMJ , JAMA, Lancet and NEJM)
5 specialty journals (specific to a topic
of CER; most frequently cited journals in CER)
Original CER
Abbreviated searches for new evidence using the same strategy
as in the CER
Screening and extracting data from
relevant studies
Identifying qualitative/quantitative signals
Signal detection completedFDA/Health Canadaalerts Expert opinion
Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER
Low priority High priorityMedium priority
Refereed for updating
Goes to 2nd cycle of assessment 6 months later
Contacting expertsECRI surveillance
Methods Contacting content experts
Conclusions from CER (executive summary)
Is the conclusion(s) in this CER still
valid?(Yes/No/Don’t
know)
Are you aware of any new evidence that is sufficient to invalidate the finding(s) in
CER?(Yes/No/Don’t know)If yes, please provide
references
Comments
Key Question # 1:
Conclusion for key question # 1:
Key question # 2:
Conclusion for key question # 2:
Key question # 3:
Conclusion for key question # 3:
Key Question # 4:
Conclusion for key question # 4:
Abbreviations:
Expert Opinion
CER-specific clinical experts (content experts, technical expert
panel members, expert peer reviewer)
Other clinical experts (external, local)
Feeds into the decision on updating status of KQ specific conclusion(s) in
each CER
Original CER
Abbreviated searches for new evidence using the same strategy
as in the CER
Screening and extracting data from
relevant studies
Identifying qualitative/quantitative signals
Signal detection completedFDA/Health Canadaalerts Expert opinion
Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER
Low priority High priorityMedium priority
Refereed for updating
Goes to 2nd cycle of assessment 6 months later
Contacting expertsECRI surveillance
Methods
Safety surveillance alerts
FDA/Health Canada alerts on CER topics (monthly)
On CERs assigned to RAND EPC
On CERs assigned to Ottawa EPC
Feeds into the decision on updating status of KQ
specific conclusion(s) in each CER
Original CER
Abbreviated searches for new evidence using the same strategy
as in the CER
Screening and extracting data from
relevant studies
Identifying qualitative/quantitative signals
Signal detection completedFDA/Health Canadaalerts Expert opinion
Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER
Low priority High priorityMedium priority
Refereed for updating
Goes to 2nd cycle of assessment 6 months later
Contacting expertsECRI surveillance
Methods
Identifying signals for updating
Qualitative signals
Potentially invalidating change in evidence
[pivotal trial/MA]
- Opposing findings in effectiveness- Substantial harm- A superior new treatment
Major change in evidence
[pivotal trial/MA]
- Important changes in effectiveness short of opposing findings - Clinically important expansion of treatment - Clinically important caveat
[non-pivotal trial/MA]
- Opposing findings in effectiveness/harms
Methods
Identifying signals for updating
Quantitative
Change in statistical significance
significant vs. non-significant non-significant vs. significant
At least 50% change in effect size
Original CER
Abbreviated searches for new evidence using the same strategy
as in the CER
Screening and extracting data from
relevant studies
Identifying qualitative/quantitative signals
Signal detection completedFDA/Health Canadaalerts Expert opinion
Deciding on updating status of conclusion(s) for each KQ within CER and determining updating priority for each CER
Low priority High priorityMedium priority
Refereed for updating
Goes to 2nd cycle of assessment 6 months later
Contacting expertsECRI surveillance
Methods Rules for assessing updating status of conclusion(s) for each KQ
Up to date•No new evidence•Only confirmatory evidence•All responding experts: conclusions as still valid
•Some new evidence•A minority of responding experts: having new evidence that might change the conclusion
Possibly out of date
•Substantial new evidence•Majority of responding experts: having new evidence that might change the conclusion
Probably out of date
•New evidence that rendered the CER conclusion out of date or no longer applicable • E.g. withdrawal of a drug or surgical
device from the market, a black box warning from FDA, etc
Out of date
Methods Rules for assigning the updating priority to a CER
high, medium, or low
How many conclusions? Probably out of
date
Possibly out of date
Out of date
How out of date are
conclusions?
Magnitude/direction of changes in estimates
Potential changes in practice or
therapyDrugs withdrawn
from the market /black box warning
Availability of a new
treatment
Methods
• A brief report summarizing the findings from the surveillance process– Title page lists the final priority for updating (“low,”
“medium,” or “high”)– Description of methods and results– Summary table by conclusion:
• findings of the literature search• findings from FDA and Health Canada• expert opinion• assessment of the degree to which the conclusion may be
out-of-date– Evidence table
Findings
1st year of surveillance
14 CERs were assessed for updating status and were assigned updating priorities as follows: 2 (14.2%) = ‘high priority’
3 (21.4%) = ‘medium priority’
9 (64.4%) = ‘low priority’
Implications
• We have established a Surveillance Program that has evaluated 14 CERs over the course of 12 months
• We have shown that a program for regular and
active surveillance of CERs is feasible
Implications
• To our knowledge, the surveillance assessment project is the first large scale effort that has applied a set of methods to assess the currency of evidence-based reports (or systematic reviews) in a structured and standardized manner
• The application of this methodology proves to be a relatively feasible, efficient, and at the same time, comprehensive approach for assessing the need for updating individual CERs across a wide range of health interventions
References
• Moher D, Tsertsvadze A. Systematic Reviews: When is an Update an Update? Lancet 2006; 367: 881-883
• Garritty C, Tsertsvadze A, Tricco AC, Sampson M, Moher D. Updating systematic reviews: an international survey. PloS one 2010; 5(4):e9914.
• French SD, McDonald S, McKenzie JE, Green S. Investing in updating: how do conclusions change when Cochrane systematic reviews are updated? BMC Med Res Methodol 2005;5:33.
• Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M et al. A systematic review identified few methods and strategies describing when and how to update systematic reviews. J Clin Epidemiol 2007; 60(11):1095-10e1.
• Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 2007;147:224e33.
• Shekelle P, Newberry S, Maglione M et al. Assessment of the need to update comparative effectiveness reviews: Report of an initial rapid program assessment (2005-2009) [Internet]. 2009 Sep 10.
• Shekelle PG, Newberry SJ, Wu H, Suttorp M, Motala A, Lim Y-W, Balk EM, Chung M, Yu WW, Lee J, Gaylor JM, Moher D, Ansari MT, Skidmore R, Garritty C. Identifying Signals for Updating Systematic Reviews: A Comparison of Two Methods. Rockville (MD): Agency for Healthcare Research and Quality. June 2011. Methods Research Report. AHRQ Publication No. 11-EHC042-EF.
• Tsertsvadze A, Maglione M, Chou R, Garritty C, Coleman C, Lux L et al. Updating comparative effectiveness reviews: current efforts in AHRQ's Effective Health Care Program. J Clin Epidemiol 2011; 64(11):1208-1215.