systematic reviews, meta- analysis and critical reading of medical literature: evidence-based...
TRANSCRIPT
Systematic reviews meta-analysis and critical reading of
medical literatureEvidence-based medicine
Phyllis W Speiser MDChief Div Ped Endo CCMC
Professor of PediatricsHofstra-NSLIJ School of Medicine
The need to be evidence-based
1 Wide variations in practice1 Continued use of ineffective treatments 2 Excess use of inappropriate treatments 3 Poor uptake of effective practice
2 Increasing consumerism 1 Unvetted Internet information2 Direct-to-consumer pharma advertising
3 Increasing demand on $ resources1 Need to demonstrate efficacy
4 Exponential growth in research1 Need to compare amp evaluate many
studiesrsquo variable quality amp conflicting results
Information overload
bull gt20000 biomedical periodicalsbull gt17000 biomedical books annuallybull ~30000 recognized diseasesbull ~15000 therapeutic agents (250yr)
bull MEDLINEndash gt4000 journals surveyedndash 11000000 citations
Types of clinical trials
bull Phase Indash Short-term safety amp pharmacokinetics (PK)ndash Small N healthy adults
bull Phase IIndash Intermediate-term safety amp efficacyndash Many healthy + few disease subjects
bull Phase IIIndash Large multicenter efficacy amp longer-term safety
bull Phase IVndash Post-marketingndash Extended safety profile
- Bias in subject selection or reporting- Inappropriate endpoint selection- Chance associations of common outcomesvariables- True findings but unrelated
Pitfalls of small clinical trials
Why bother with evidence-based medicine (EBM)
bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype
bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions
bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients
Types of EBM studies
bull Diagnosisndash Selecting appropriate diagnostic tests
bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit
bull Prognosisndash Outcomes amp complications
bull Associations Causesndash Identify etiologies eg infectious
environmental iatrogenic
What makes a review ldquosystematicrdquo
bull Basis for EBM recommendations
bull Based on a clearly formulated question
bull Identifies relevant studies with pre-set criteria
bull Appraises quality of studies
bull Summarizes evidence by use of explicit methodology
bull Recommendations are based on evidence gathered
Assessing quality
bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias
bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias
bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical
phenotypendash Low inter-assay and intra-assay variability
bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis
bull Appropriate statistical methods used
Quality validity of studies Design
bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control
ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo
Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061
Quality validity of studies Intervention
ndash Nature of interventionbull Placebo-controlled v best current treatment v
uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose
ndash Verification of methodsbull Same or different assays inter- amp intra- assay
variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
The need to be evidence-based
1 Wide variations in practice1 Continued use of ineffective treatments 2 Excess use of inappropriate treatments 3 Poor uptake of effective practice
2 Increasing consumerism 1 Unvetted Internet information2 Direct-to-consumer pharma advertising
3 Increasing demand on $ resources1 Need to demonstrate efficacy
4 Exponential growth in research1 Need to compare amp evaluate many
studiesrsquo variable quality amp conflicting results
Information overload
bull gt20000 biomedical periodicalsbull gt17000 biomedical books annuallybull ~30000 recognized diseasesbull ~15000 therapeutic agents (250yr)
bull MEDLINEndash gt4000 journals surveyedndash 11000000 citations
Types of clinical trials
bull Phase Indash Short-term safety amp pharmacokinetics (PK)ndash Small N healthy adults
bull Phase IIndash Intermediate-term safety amp efficacyndash Many healthy + few disease subjects
bull Phase IIIndash Large multicenter efficacy amp longer-term safety
bull Phase IVndash Post-marketingndash Extended safety profile
- Bias in subject selection or reporting- Inappropriate endpoint selection- Chance associations of common outcomesvariables- True findings but unrelated
Pitfalls of small clinical trials
Why bother with evidence-based medicine (EBM)
bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype
bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions
bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients
Types of EBM studies
bull Diagnosisndash Selecting appropriate diagnostic tests
bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit
bull Prognosisndash Outcomes amp complications
bull Associations Causesndash Identify etiologies eg infectious
environmental iatrogenic
What makes a review ldquosystematicrdquo
bull Basis for EBM recommendations
bull Based on a clearly formulated question
bull Identifies relevant studies with pre-set criteria
bull Appraises quality of studies
bull Summarizes evidence by use of explicit methodology
bull Recommendations are based on evidence gathered
Assessing quality
bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias
bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias
bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical
phenotypendash Low inter-assay and intra-assay variability
bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis
bull Appropriate statistical methods used
Quality validity of studies Design
bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control
ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo
Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061
Quality validity of studies Intervention
ndash Nature of interventionbull Placebo-controlled v best current treatment v
uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose
ndash Verification of methodsbull Same or different assays inter- amp intra- assay
variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Information overload
bull gt20000 biomedical periodicalsbull gt17000 biomedical books annuallybull ~30000 recognized diseasesbull ~15000 therapeutic agents (250yr)
bull MEDLINEndash gt4000 journals surveyedndash 11000000 citations
Types of clinical trials
bull Phase Indash Short-term safety amp pharmacokinetics (PK)ndash Small N healthy adults
bull Phase IIndash Intermediate-term safety amp efficacyndash Many healthy + few disease subjects
bull Phase IIIndash Large multicenter efficacy amp longer-term safety
bull Phase IVndash Post-marketingndash Extended safety profile
- Bias in subject selection or reporting- Inappropriate endpoint selection- Chance associations of common outcomesvariables- True findings but unrelated
Pitfalls of small clinical trials
Why bother with evidence-based medicine (EBM)
bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype
bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions
bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients
Types of EBM studies
bull Diagnosisndash Selecting appropriate diagnostic tests
bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit
bull Prognosisndash Outcomes amp complications
bull Associations Causesndash Identify etiologies eg infectious
environmental iatrogenic
What makes a review ldquosystematicrdquo
bull Basis for EBM recommendations
bull Based on a clearly formulated question
bull Identifies relevant studies with pre-set criteria
bull Appraises quality of studies
bull Summarizes evidence by use of explicit methodology
bull Recommendations are based on evidence gathered
Assessing quality
bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias
bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias
bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical
phenotypendash Low inter-assay and intra-assay variability
bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis
bull Appropriate statistical methods used
Quality validity of studies Design
bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control
ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo
Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061
Quality validity of studies Intervention
ndash Nature of interventionbull Placebo-controlled v best current treatment v
uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose
ndash Verification of methodsbull Same or different assays inter- amp intra- assay
variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Types of clinical trials
bull Phase Indash Short-term safety amp pharmacokinetics (PK)ndash Small N healthy adults
bull Phase IIndash Intermediate-term safety amp efficacyndash Many healthy + few disease subjects
bull Phase IIIndash Large multicenter efficacy amp longer-term safety
bull Phase IVndash Post-marketingndash Extended safety profile
- Bias in subject selection or reporting- Inappropriate endpoint selection- Chance associations of common outcomesvariables- True findings but unrelated
Pitfalls of small clinical trials
Why bother with evidence-based medicine (EBM)
bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype
bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions
bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients
Types of EBM studies
bull Diagnosisndash Selecting appropriate diagnostic tests
bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit
bull Prognosisndash Outcomes amp complications
bull Associations Causesndash Identify etiologies eg infectious
environmental iatrogenic
What makes a review ldquosystematicrdquo
bull Basis for EBM recommendations
bull Based on a clearly formulated question
bull Identifies relevant studies with pre-set criteria
bull Appraises quality of studies
bull Summarizes evidence by use of explicit methodology
bull Recommendations are based on evidence gathered
Assessing quality
bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias
bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias
bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical
phenotypendash Low inter-assay and intra-assay variability
bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis
bull Appropriate statistical methods used
Quality validity of studies Design
bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control
ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo
Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061
Quality validity of studies Intervention
ndash Nature of interventionbull Placebo-controlled v best current treatment v
uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose
ndash Verification of methodsbull Same or different assays inter- amp intra- assay
variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
- Bias in subject selection or reporting- Inappropriate endpoint selection- Chance associations of common outcomesvariables- True findings but unrelated
Pitfalls of small clinical trials
Why bother with evidence-based medicine (EBM)
bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype
bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions
bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients
Types of EBM studies
bull Diagnosisndash Selecting appropriate diagnostic tests
bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit
bull Prognosisndash Outcomes amp complications
bull Associations Causesndash Identify etiologies eg infectious
environmental iatrogenic
What makes a review ldquosystematicrdquo
bull Basis for EBM recommendations
bull Based on a clearly formulated question
bull Identifies relevant studies with pre-set criteria
bull Appraises quality of studies
bull Summarizes evidence by use of explicit methodology
bull Recommendations are based on evidence gathered
Assessing quality
bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias
bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias
bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical
phenotypendash Low inter-assay and intra-assay variability
bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis
bull Appropriate statistical methods used
Quality validity of studies Design
bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control
ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo
Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061
Quality validity of studies Intervention
ndash Nature of interventionbull Placebo-controlled v best current treatment v
uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose
ndash Verification of methodsbull Same or different assays inter- amp intra- assay
variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Why bother with evidence-based medicine (EBM)
bull Without EBM we are helpless in the face ofndash misguided expertsndash overenthusiastic expertsndash failure to report negative studies adverse outcomesndash drug company hype
bull Without EBM our ability is limitedndash to understand difficult tradeoffs ndash to help our patients make difficult decisions
bull With EBM comesndash understanding and power ndash greater effectiveness in helping our patients
Types of EBM studies
bull Diagnosisndash Selecting appropriate diagnostic tests
bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit
bull Prognosisndash Outcomes amp complications
bull Associations Causesndash Identify etiologies eg infectious
environmental iatrogenic
What makes a review ldquosystematicrdquo
bull Basis for EBM recommendations
bull Based on a clearly formulated question
bull Identifies relevant studies with pre-set criteria
bull Appraises quality of studies
bull Summarizes evidence by use of explicit methodology
bull Recommendations are based on evidence gathered
Assessing quality
bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias
bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias
bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical
phenotypendash Low inter-assay and intra-assay variability
bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis
bull Appropriate statistical methods used
Quality validity of studies Design
bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control
ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo
Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061
Quality validity of studies Intervention
ndash Nature of interventionbull Placebo-controlled v best current treatment v
uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose
ndash Verification of methodsbull Same or different assays inter- amp intra- assay
variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Types of EBM studies
bull Diagnosisndash Selecting appropriate diagnostic tests
bull Therapyndash Selecting most effectivesafest treatmentsndash Cost-benefit
bull Prognosisndash Outcomes amp complications
bull Associations Causesndash Identify etiologies eg infectious
environmental iatrogenic
What makes a review ldquosystematicrdquo
bull Basis for EBM recommendations
bull Based on a clearly formulated question
bull Identifies relevant studies with pre-set criteria
bull Appraises quality of studies
bull Summarizes evidence by use of explicit methodology
bull Recommendations are based on evidence gathered
Assessing quality
bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias
bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias
bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical
phenotypendash Low inter-assay and intra-assay variability
bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis
bull Appropriate statistical methods used
Quality validity of studies Design
bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control
ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo
Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061
Quality validity of studies Intervention
ndash Nature of interventionbull Placebo-controlled v best current treatment v
uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose
ndash Verification of methodsbull Same or different assays inter- amp intra- assay
variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
What makes a review ldquosystematicrdquo
bull Basis for EBM recommendations
bull Based on a clearly formulated question
bull Identifies relevant studies with pre-set criteria
bull Appraises quality of studies
bull Summarizes evidence by use of explicit methodology
bull Recommendations are based on evidence gathered
Assessing quality
bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias
bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias
bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical
phenotypendash Low inter-assay and intra-assay variability
bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis
bull Appropriate statistical methods used
Quality validity of studies Design
bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control
ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo
Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061
Quality validity of studies Intervention
ndash Nature of interventionbull Placebo-controlled v best current treatment v
uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose
ndash Verification of methodsbull Same or different assays inter- amp intra- assay
variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Assessing quality
bull Was an appropriate spectrum of patients included ndash Avoids ascertainment bias
bull All patients subjected to a gold standard diagnostic test with blinded interpretationndash Avoids observer bias
bull Methods described amp validatedndash Appropriate diagnostic tools usedndash Uniform definition of clinical amp biochemical
phenotypendash Low inter-assay and intra-assay variability
bull Adequate number of subjects in each groupndash Low drop-out ratendash Study design included power analysis
bull Appropriate statistical methods used
Quality validity of studies Design
bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control
ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo
Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061
Quality validity of studies Intervention
ndash Nature of interventionbull Placebo-controlled v best current treatment v
uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose
ndash Verification of methodsbull Same or different assays inter- amp intra- assay
variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Quality validity of studies Design
bull Study designbull Prospective v retrospectivebull Cross-sectional v longitudinalbull Clinic population only v case-control
ndash Patient selectionbull Consecutive v nonconsecutive v randombull Age racial ethnic amp gender balancebull Power analysis to determine subject numberbull Number of drop-outsbull ldquoIntention to treatrdquo
Lijmer et al Empiric evidence of design-related bias in studies of diagnostic studies JAMA 19992821061
Quality validity of studies Intervention
ndash Nature of interventionbull Placebo-controlled v best current treatment v
uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose
ndash Verification of methodsbull Same or different assays inter- amp intra- assay
variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Quality validity of studies Intervention
ndash Nature of interventionbull Placebo-controlled v best current treatment v
uncontrolledbull Randomized or notbull Blinded or not bull Dose-ranging v single dose
ndash Verification of methodsbull Same or different assays inter- amp intra- assay
variabilitybull Same or different endpointsbull Empiric or historical normal reference databull Appropriateness of controls
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Quality validity of studies Data
ndash Data collectionbull Prospective or retrospectivebull Intention to treatbull Exclusion criteria for outliersbull Compliance assurance (eg weekly phone calls
patient diaries pill counts etc)
ndash Statistical analysisbull Appropriateness of statistical methods
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
What is a meta-analysis
Optional part of a systematic review
Systematic reviews
Meta-analyses
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Meta-analysis Are the studies consistent
bull Are variations in results between studies consistent with chance
bull If NO then WHYndash Variation in populationndash Variation in study methods (biases)ndash Variation in interventionndash Variation in outcome measure (eg timing)
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Hierarchy of evidence for treatment decisions
Meta-analysis of RCTsSystematic review of RCTs
Individual RCT
Observational studiespatient-important outcomes
Basic researchtest tube animal human physiology
Clinical experience
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Pitfalls of meta-analysis
bull Potential bias in inclusion exclusion criteria for study selectionndash Publication bias toward positive resultsndash Keyword search
bull Sizendash Number of studiesndash Sample size total amp individualndash Attrition
bull Length of follow-up
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Pitfalls of meta-analysis cont
bull Methods of meta-analysisndash Sensitivity analysis for robustness
bull Fixed vs random effectsbull Outlier exclusions
ndash Stratification of subject populations
bull Conclusions of meta-analysisndash Weak if studies on opposite sides of forest plot
ldquoHeterogeneityrdquo
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
GRADE system Knowledge translation
bull Transparent process of moving from evidence to recommendations
bull Developed by representative group of international guideline developers
bull Separates quality of evidence amp strength of recommendations
bull Stresses importance of outcomes of alternative management strategies
bull Explicit acknowledgment of values and preferences
bull Clear pragmatic interpretation of strong versus weak recommendations for clinicians patients and policy makers
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Grading evidence
bull High qualitymdash Further research is very unlikely to change our confidence in the estimate of effect
bull Moderate qualitymdash Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate
bull Low qualitymdash Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate
bull Very low qualitymdash Any estimate of effect is very uncertain
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Strength of Recommendations
Factor High Rank Low Rank
Quality evidence
RCT Case series
Balance of risk amp benefit
Low toxicity amp High efficacy
High toxicity amp
High efficacy
Values amp preferences
Life-saving or QOL-enhancing
No major advance
Cost Inexpensive Costly
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Evidence-based clinical decisions Are antibiotics indicated in pediatric
otitis media
bull Typical case A 3 year old child withndash Ear pain low grade fever irritabilityndash Examination shows bilateral otitis media
bull Should antibiotics be prescribedndash Benefitsndash Risks
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Glasziou Cochrane systematic review 2003
bull Systematic review of RCTs
bull Questionndash Patients children with otitis mediandash Intervention antibioticsndash Outcome resolution of symptomsmdashWHENndash Calculate odds ratios amp confidence interval for each study amp
combine comparable data
bull Comprehensive search
bull Only 8 high quality studies (N= 2287 children) These studies hadndash Concealed randomizationndash Double blinding of treatmentsndash Complete follow-up
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Odds ratios allow comparisons of different studies in meta-analysis
TestTx + TestTx -
Disease A TRUE POS B FALSE NEG
Unaffected C FALSE POS D TRUE NEG
Odds ratio for treatment efficacy =
AD BC
or TP x TN FN x FP
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Confidence intervals Definition
Confidence intervals are based on the assumption that a study provides one sample of observations out of many possible samples that would be derived if the study were repeated many times
For a 95 confidence interval if the experiment were
repeated many times 95 of the intervals would contain the true treatment effect
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Endpoint 2 Pain at 2 ndash 7 days improved wtx Timing is important
01 1 10
Burke
van Buchem (a)
Pooled Estimate
Favours Antibiotics Favours Placebo
Appelman
Damoiseaux
Halsted
Kaleida
van Buchem (b)
Mygind
Thalin
Odds Ratio (95 CI)
N = 121
N = 225
N = 240
N = 89
N = 980
N = 149
N = 316
N = 84
N = 83
N = 2287
086 (034 222)
065 (034 122)
055 (032 094)
108 (039 297)
050 (029 085)
045 (022 090)
057 (029 110)
043 (014 127)
057 (021 156)
057 (045 073)
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
lt00001001003009029p-value
027 to 052010 to 064004 to 067-004 to 071-023 to 07995 CI
0404040404Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0808080808Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
Effect of population size Confidence interval and significance improve with larger or pooled samples
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Moral
Any observed difference between two groups no matter how small can be made ldquostatistically
significantrdquo - at any level of significance - by taking a sufficiently large sample
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Balanced decisionsbull Are the differences are both clinically amp statistically significant
bull Is the benefit greater than the risk
bull Failure to resolve pain at 2 to 7 daysndash 20 controlndash 13 antibiotics
bull Absolute difference 7 in beneficial effect of antibioticsndash Number Needed to Treat Effectively = 15
bull Rates of antibioticsrsquo side effectsndash Increase in vomiting rash diarrhea 5ndash Number Needed to Harm = 20
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Razvi S et al J Clin Endocrinol Metab 2008932998-3007
Forest plot of IHD prevalence in SCH and euthyroid controls Age differences
lt65 yo
gt65 yo
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Evidence-based clinical decisions Subclinical hypothyroidism
bull ldquoOnly well-powered prospective randomized studies with age-stratified groups and vascular events as the primary endpoint rather than surrogate markers will give clear answers to this complex questionrdquo of whether amp when to treat subclinical hypothyroidism
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
EBM Beta blockers post MI- needed to
achieve desired outcome
bull Prospective studies suggest that Mr Jones risk of death in the first year after his infarct is 8
bull A meta-analysis of RCTs of beta-blockers after MI
suggests a 25 risk reduction
bull Must treat 50 such pts to prolong a life
bull Given the relatively small expense amp low toxicity of generic beta-blockers a trial of beta-blockers for Mr Jones is clearly warranted
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
21
5
101
1 2
8
7
8
12
4
3
1
1
2
8
7
2
1
1
1
2
8
1
5
15
6
No
t M
en
tio
ne
d
Ro
uti
ne
Ex
pe
rim
en
tal
Ra
reN
ev
er
Sp
ec
ific
M
M
M
M
M
M
TextbookReviewRecommendations
Odds Ratio (Log Scale)
05 10 20
Favors Treatment Favors Control
RCTs Pts
1 23
2 65
3 149
4 316
7 1793
10 254411 265115 331117 392922 5452
Plt01
23 5767
27 612530 634633 657143 21 05954 22 051
67 47 53165 47 185
70 48 154
Plt001
Plt00001
CumulativeYear
1960
1965
1970
1980
1985
1990
Thrombolytic therapy in MI Power in NImportance of current textbooks
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Growth of Cochrane Reviews and Protocols
1995 2003
reviews
protocols
2500 completed mid-2005
2010 gt4000
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Real world clinical decisions
Research evidence
Patient valuesand preferences
Clinical state and circumstances
Expertise
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Knowledge Translation Researchhellip
hellipstudy of the organization retrieval appraisal refinement dissemination and uptake of knowledge (eg important new knowledge from health research)
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Knowledge Translation Research
KT Type 1 KT Type 2
Based on Hulley et al Designing Clinical Research 2007 p 23
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Barrier Solutions
bull too little research addressing ldquoreal worldrdquo problems
bull large simple randomized trials
bull ldquohead to headrdquo comparisons
Step 1 Generating Research Evidence
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Possible comparators
placebo usual carecurrent best care
Most use for clinical decisionsregulatory requirement fordrugs (not devices or services)
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Systems
Summaries
Synopses
Syntheses
Studies
Examples
Computerized decision support
Evidence-based textbooks
Evidence-based journal abstracts
gt57000 EBM articles 2009
Systematic reviews
Original journal articles
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
By the year 2020 90 of clinical decisions will be supported by accurate timely and
up-to-date clinical information and will reflect the best available evidence
IOM Roundtable on Evidence-Based Medicine
This canrsquot happen without a better understanding of the barriers to translating knowledge into practiceand ways to overcome them This can happen if
EBM+KT=090
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Clinical problem
bull Previously healthy patient with flu wants to knowndash Should he take Tamiflundash Will it make him better quicklyndash Is it worth the cost
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Whatrsquos the question
bull Is Tamiflu more effective than fluids rest and anti-pyretics
bull Does Tamiflu reduce flu severity
bull How much does Tamiflu cost
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
How to find answers
bull Search Tamiflu AND efficacy
bull Search influenza AND Tamiflu AND Adults [limit to RCT in core clinical journals]
bull Search Tamiflu AND severity of illness
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
ResultsSAE + SAE- Totals
Rx 17 228 245
Pbo 19 110 129
Absolute risk reduction (ARR) 147 (19129 pbo) - 69(17245 rx)=78
Relative risk reduction 78 (ARR)147 (pbo SAEs) = 53
Number Needed to Treat 1 78 (ARR) = 13 patients with flu need to be treated with tamiflu for 5 days to prevent one complication
JAMA 2000283(8)1016-1024
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Informed decision
bull MD discusses data with patientndash Benefitsndash Risksndash Cost
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-
Other resources
1 Victor Montori MD
Mayo Clinic KER unit
2 McMaster Inst Institute of Medicine
3 Online tutorial
httpwwwhslunceduServicesTutorialsEBMwelcomehtm
4 Walker E Hernandez AV Kattan MW Meta-analysis Its strengths amp limitations Cleveland Clinic Journal of Medicine June 2008 vol 75 6 431-439
- Systematic reviews meta-analysis and critical reading of medical literature Evidence-based medicine
- The need to be evidence-based
- Information overload
- Types of clinical trials
- - Bias in subject selection or reporting - Inappropriate endpoint selection - Chance associations of common outcomesvariables - True findings but unrelated
- Why bother with evidence-based medicine (EBM)
- Types of EBM studies
- What makes a review ldquosystematicrdquo
- Assessing quality
- Quality validity of studies Design
- Quality validity of studies Intervention
- Quality validity of studies Data
- What is a meta-analysis
- Meta-analysis Are the studies consistent
- Hierarchy of evidence for treatment decisions
- Pitfalls of meta-analysis
- Pitfalls of meta-analysis cont
- GRADE system Knowledge translation
- Grading evidence
- Strength of Recommendations
- Evidence-based clinical decisions Are antibiotics indicated in pediatric otitis media
- Glasziou Cochrane systematic review 2003
- Odds ratios allow comparisons of different studies in meta-analysis
- Confidence intervals Definition
- Slide 27
- Slide 28
- Slide 29
- Balanced decisions
- Subclinical hypothyroidism and ischemic heart disease Winnowing publications for meta-analysis
- Slide 32
- Evidence-based clinical decisions Subclinical hypothyroidism
- EBM Beta blockers post MI- needed to achieve desired outcome
- Slide 35
- Slide 36
- Real world clinical decisions
- Knowledge Translation Researchhellip
- Knowledge Translation Research
- Slide 40
- Possible comparators
- Slide 42
- Slide 43
- Clinical problem
- Whatrsquos the question
- How to find answers
- Results
- Informed decision
- Slide 49
-