rapid critical appraisal of diagnostic accuracy studies professor paul glasziou centre for evidence...
TRANSCRIPT
![Page 1: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/1.jpg)
Rapid Critical Appraisalof diagnostic accuracy studies
Professor Paul GlasziouCentre for Evidence Based Medicine
University of Oxford
www.cebm.net
![Page 2: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/2.jpg)
What are tests used for?
Diagnosis – what is the problem?
Log of reasons by several docs:• Monitoring – has it changed?
• Prognosis – risk/stage within Dx
• Treatment planning, e.g., location
• Stalling for time!
![Page 3: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/3.jpg)
Is the test accurate?
To be accurate a test should be:•Reproducible
oWe get the same (wrong?) answer every time
oP I-I question
•ValidoWe get the right answeroP I O question
![Page 4: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/4.jpg)
Reproducibility:Agreement of histopathologists
Organ Feature Agreement Kappa Reference
Rectal Cancer Grading
50% to 69% 0.11 to 0.5 Thomas
Hodgkins Classification
56% 0.44 Holman
Melanoma depth
82%; 64% 0.68; 0.23 Breslow; Clark
Breast cancer classification
73% 0.46 Stenkvist
Ken Fleming, Evidence-based pathology. EBM 1997
![Page 5: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/5.jpg)
Diagnostic Test Accuracymeasurements
Sensitivity is the probability of a positive test in a diseased person
Specificity is the probability of a negative test in a non-diseased person.
![Page 6: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/6.jpg)
Is the test helpful (valid)?The Youden Index
Youden Index = sensitivity+specificity-1•For a test to be useful, then
osensitivity + specificity > 1 (Youden Index > 0)
Examples:•Coin Toss with +ve = "heads"
sensitivity = 0.5 specificity = 0.5
•Youden = 0
![Page 7: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/7.jpg)
Can a test rule-in or rule-out?
SpPln•Specific test, Positive rules In
eg: Rovsing's sign, ST elevation > 2mm
SnNout•Sensitive test, Negative rules Out
eg: Erect abdominal film for obstruction, Elevated WCC in CSF (>5/mm )
![Page 8: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/8.jpg)
Can I trust the accuracy data from the study?
RAMMbo
Recruitment: Was an appropriate spectrum of patients included? • (Spectrum Bias)
Maintainence: All patients subjected to a Gold Standard?• (Verification Bias)
Measurements: Was there an independent, blind or objective comparison with a Gold Standard? • Observer Bias; Differential Reference Bias
Index test (Comparison)
Outcomes(ReferenceStandard)
Presenting Problem
Index test (Comparison)
Outcomes(ReferenceStandard)
Index test (Comparison)
Outcomes(ReferenceStandard)
Presenting Problem
![Page 9: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/9.jpg)
Appraisal of Tests: RAMMbo was the evaluatioin fair?
OutcomeMeasures(Gold Standard)
OutcomeMeasures(Gold Standard)
Index test
Comparator Test
Maintained?
Representation?
Population
Blinded or Objective?
![Page 10: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/10.jpg)
QUADAS
![Page 11: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/11.jpg)
The Literary Digest PollLandon versus Roosevelt, 1936
% for Roosevelt Literary Digest: 2.4 Million reader poll
• Prediction for Roosevelt 43%
Gallup's 50,000 random sample• Prediction of the election result 56%
Gallup's 3,000 Digest readers• Prediction of Digest prediction 44%
Election result 62%
![Page 12: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/12.jpg)
Good sampling: needs a sample frame & unbiased selection
Target Population
Sample Frame
Actual Sample
Complete data
![Page 13: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/13.jpg)
Were reference Measurements blinded or objective?
Index +ve
Index -ve
High Threshold
Low Threshold
Apparent difference
Use standardised measurement strategy across ALL patients
![Page 14: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/14.jpg)
Smith H, et al BMJ, 2000
BNP screen of GP elderly patients
UK GP setting 155 patients
• 70-84 yrs old
Echocardiogram• 12 with CCF
Sens=92%Spec=65%
Sens=50%Spec=90%
![Page 15: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/15.jpg)
Assessment process systematic reviews
Assessment process original studies
Potentially Eligible Systematic Reviews 1999-2002N=191
157 systematic reviewsexcluded
34 systematic reviewswith
39 meta analysescontaining
678 original studies
6 systematic reviews with 8 meta-analyses excluded
28 systematic reviewswith
31 meta-analysescontaining
545 original studies
31 meta-analysescontaining
487 original studies
58 original studiesexcluded
Replication & Extension Study
AWS Rutjes et al. 2005AWS Rutjes et al. 2005
![Page 16: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/16.jpg)
Other case-control designs
Study characteristics
No description population
No description reference
No description index
Retrospective data collection
Non-consecutive
Non blinded studies
Partial verification
Differential verification
Severe cases and healthy controls
487 studies; 31 meta-analyses487 studies; 31 meta-analysesRDOR0 1 2 3 4 5
![Page 17: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/17.jpg)
How well are diagnostic studies reported?
112 studies in 4 major journals (1978-1993)
Standard N (%)Spectrum composition 30 (27)Avoidance of workup bias 51 (46)Avoidance of review bias 43 (38)Test accuracy precision 12 (11)Indeterminate test results 26 (23)Test reproducibility 26 (23)Accuracy in subgroups 9 (8)
Reid MC, Lachs MS, Feinstein AR. Use of Methodological Standardin diagnostic test research. JAMA 1995;274:645-651
![Page 18: Rapid Critical Appraisal of diagnostic accuracy studies Professor Paul Glasziou Centre for Evidence Based Medicine University of Oxford](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d215503460f949f73ce/html5/thumbnails/18.jpg)
Using Evidence about Tests
Appraise the study - PICO
Always ask• Is it useful at all? (Youden Index > 0)
Usually ask• Can it “rule in” or “rule out” a disease?
Often ask• What is the post-test probability in the same situation as
the study? Rarely ask
• Calculation of post-test probabilities in a different situation