precision in quantitative imaging: trial development and ... pdfs... · • all roi protocols show...
TRANSCRIPT
![Page 1: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/1.jpg)
Precision in Quantitative Imaging:
Trial Development and
Quality Assurance
Susanna I Lee MD, PhD
Thanks to:
Mitchell Schnall, Mark Rosen. Dan Sullivan, Patrick Bossuyt
![Page 2: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/2.jpg)
Imaging Chain: Patient Data
Raw data
Image
analysis123……………
2346…………..
65789…………
6578…………..
Data output
Image
reconstruction
Image
processing
Data analysis
![Page 3: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/3.jpg)
Clinical Trials: Imaging is an “Assay”
♦ Disease Detection
• Screening
♦ Characterize Disease
• Diagnosis, eligibility or prognosis
• Anatomic distribution (e.g. tumor staging)
• Higher level features (e.g. heterogeneity, vascularity, etc.)
♦ Monitor Therapeutic Response
• Change in features with therapy
• Anatomic (e.g. RECIST) or functional (e.g. SUV)
![Page 4: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/4.jpg)
REFERENCE STANDARD OF
PATIENT OUTCOME
• OVERALL SURVIVAL (OS)
• PROGRESSION FREE SURVIVAL
(PFS) OR DISEASE FREE
SURVIVAL (DFS)
• TREATMENT RESPONSE
REFERENCE STANDARD OF
DISEASE STATE
• PATHOLOGY
• CONFIRMATORY TEST
• FOLLOWUP
MANAGEMENT
• STANDARD OF CARE
• STUDY DEFINED
INDEX TEST
• IMAGING EXAM
• IMAGE-GUIDED
PROCEDURE
Schema
ENROLLED
PARTICIPANT
Diagnostic accuracy
Biomarker
![Page 5: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/5.jpg)
Cancer Biomarkers
GeneticGenome
Expression (e.g.
RNA or protein)
Serum
(CA125,
PSA, AFP,
CA19-9)
Tissue(e.g. hormone
receptors,
cytokeratins)
PATIENT OUTCOME
1. Overall survival (OS)
2. Progression free
survival (PFS)
3. Clinical response
Imaging
![Page 6: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/6.jpg)
What is a good biomarker?
♦Stable technology
♦Available widely
♦Standardized image acquisition
♦Reproducible
♦Range of normal defined
Sargent DJ, Rubinstein L, Schwartz L et al. Eur J Cancer 2009; 45: 290.
Balance “state of the art” with
“generalizability”
![Page 7: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/7.jpg)
Pre-treatment
Post-treatment
Range of
test-retest
Lankester KJ, Taylor NJ, Stirling JJ et al. Br J Cancer. 2005.93:979.
Variability: Test – Retest
Same patient, day, scanning protocol but separate imaging sessions
Conclusion
Index test variability precludes detecting pre- vs. post-treatment change.
![Page 8: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/8.jpg)
Signal Requires Data Quality
![Page 9: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/9.jpg)
Precision vs. Bias
precise
accurate
not precise
not accurate
precise
not accurate
![Page 10: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/10.jpg)
Multiple Sclerosis MRI
♦ Image acquisition
• T1
• T2
• Post-gadolinium T1
♦ Image analysis
• Number of new or enlarging lesions
• Number of enhancing lesions
![Page 11: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/11.jpg)
• MRI endpoint in MS treatment studies
• 157 publications from 1995 to 2006
![Page 12: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/12.jpg)
Imaging Chain: Patient Data
Raw data
Image
analysis123……………
2346…………..
65789…………
6578…………..
Data output
Image
reconstruction
Image
processing
Data analysis
![Page 13: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/13.jpg)
Imaging Manual
♦ Hardware and software
♦ Scanner calibration
♦ Patient preparation
♦ Scanning protocol
♦ Post-processing
Image acquisition manual with a step by step description is part
of any prospective study design.
![Page 14: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/14.jpg)
What determines resolution?
♦Physics of acquisition (i.e. modality)
♦Sampling (e.g. matrix, detector size)
♦Filtering and other contributions inherent in
the reconstruction
![Page 15: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/15.jpg)
Point Spread Function
Patient Image
![Page 16: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/16.jpg)
Image ““““edges”””” approximate anatomy
Structure
Real Edge
Image Edge
![Page 17: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/17.jpg)
Resolution and Sampling
160 x 160 matrix 320 x 224 matrix
![Page 18: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/18.jpg)
Filtering
4 2
222
2
2 2 2
1
1
1
1
1111
1
1
1
1 1 1 1 1
![Page 19: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/19.jpg)
Filtering: MRI
Vendor 1 Vendor 2
![Page 20: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/20.jpg)
Filtering: X-ray
Vendor 1 Vendor 2
![Page 21: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/21.jpg)
Partial Volume Effects
Completely
in scan planePartially in
scan plane
![Page 22: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/22.jpg)
Partial Volume Effects: CT
Completely in scan plane Partially in scan plane
HU = 0 HU = 30-60
![Page 23: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/23.jpg)
Partial Volume Effects: PET
lesion
lesion
Blurred margins
Lower intensity
![Page 24: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/24.jpg)
![Page 25: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/25.jpg)
Quantitative Imaging
Biomarkers Alliance (QIBA)
♦ Started by RSNA 2007
♦ Mission: Improve the value and practicality of
quantitative imaging biomarkers by reducing
variability across devices, patients and time
♦ “Build imaging devices that are also measuring
devices”
♦ “Industrialize imaging biomarkers”
https://www.rsna.org/QIBA/
![Page 26: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/26.jpg)
QIBA Approach
1. Identify the sources of error and variability
2. Specify potential solutions in the form of profiles
3. Test these solutions
4. Promulgate profiles to vendors and users
• Advise vendors what must be implemented in their product
• Communicate the necessary procedures to users
Purpose of profiles:
![Page 27: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/27.jpg)
QIBA Profile Activity Diagram
Equipment
Assessment
Subject
Preparation
Image
Reconstruction
Image
AnalysisInterpretation
Image
Acquisition
Manufacturer specification (pre-delivery)
Installation specification
Maintenance Quality
Assurance
![Page 28: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/28.jpg)
ACR Core Lab QA
♦Site qualification
• Instrument performance
• Training
♦Monitoring of image acquisition
• Scan header for protocol compliance
• On-line technologist qualitative review
• Periodic radiologist review
♦Centralized image analysis
• Post-processing
• Reader study
![Page 29: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/29.jpg)
![Page 30: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/30.jpg)
Require Site Protocol Compliance
Type T1 weighted GRE
Orientation Sagittal
Pulse Sequence Dynamic 3D
Field Of View (FOV) 16-18 cm
Slice Thickness 64 slices of thickness ≤ 2.5
mm
Skip
Matrix min. 256 x 192
Frequency A/P
NEX 2
Phase Wrap NO
Fat-Saturation YES
TR ≤ 20 ms
Effective TE 4.5 ms
Scan Duration Between 4.5 and 5 minutes
Flip angle <= 45 degrees
Correct
Submitted
ACRIN 6657 (I-SPY 1 trial), Nola Hylton, PI
![Page 31: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/31.jpg)
Imaging Chain: Patient Data
Raw data
Image
analysis123……………
2346…………..
65789…………
6578…………..
Data output
Image
reconstruction
Image
processing
Data analysis
![Page 32: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/32.jpg)
Image analysis: Turning image into data
♦User extracted features
♦Semi automated
♦Automated
Feature 1
Feature 2
Feature 3
.
.
.
![Page 33: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/33.jpg)
Radiomics: Deep Learning
Untrained neural network
![Page 34: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/34.jpg)
Radiomics: Deep Learning
Trained neural network
![Page 35: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/35.jpg)
Radiomics: Automated Image Analysis
Improve diagnostic accuracy
![Page 36: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/36.jpg)
Radiomics: Automated Image Analysis
Triage clinical workflow
![Page 37: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/37.jpg)
Semi-automated: Manual Segmentation
Quantitative
feature
extraction
Large image
datasets
segmented for
tumor
Integrate with
genomic &
clinical data
for machine
learning
Model
predictive
indices
Aerts HJ et al. Nat Commun. 2014 ;5:4006.
![Page 38: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/38.jpg)
Reader Extracted Features
•Density• Fluid, soft tissue, calcified
•Shape• Round, oval, irregular
•Size• Linear, volume
•Margin• Sharp, blurred, spiculated
•Intensityhigh/medium/low/minimal
•Summary assementBIRADS level
![Page 39: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/39.jpg)
0.0 0.2 0.4 0.6 0.8 1.0
FP
0.0
0.2
0.4
0.6
0.8
1.0
TP
Beam, Layde, Sullivan Arch Intern Med 1996; 156:209-213
ROC operating points of 108 radiologists
reading same mammograms
Skill
Value
judgments
![Page 40: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/40.jpg)
• Increases and decreases of <10% can be a result of
inherent variability.
Reader Variability: Size
Oxnard GR et al. J Clin Oncol. 2011;29:3114-9.
progression
![Page 41: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/41.jpg)
SUV=4.0 SUV=5.6
Variability Introduced by ROI Selection
Slice 483 Slice 479 Slice 479
SUV=6.6
![Page 42: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/42.jpg)
Variability Introduced by ROI Selection
• All ROI protocols show excellent inter-observer agreement (ICC 0.94)
• Different ROI protocols yield different ADC values
Priola AM et al. Eur Radiol. 2016 Aug 11.
![Page 43: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/43.jpg)
Effect of Windowing
Soft tissue window Liver window
![Page 44: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/44.jpg)
Effect of Windowing
• Significant measurement differences between window settings (p<0.001).
• No significant differences in measurement variability between the lung and
mediastinal window settings (p>0.05).
Lung window Mediastinal window
Kim H et al. PLoS One. 2016;11:e0148853.
![Page 45: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/45.jpg)
Reader Study
♦ Reader blinded to reference standard
♦ Multiple readers
♦ Independent rather than consensus reads
♦ Rules for image interpretation
• Clinical information available to reader
• Image selection, windowing, order, etc.
• Choosing index lesions
• Selecting region of interest (ROI)
• Definition of positive vs. negative test
♦ Washout period between read sessions of paired imaging exams
♦ Digital data forms and screen to document reader study
A manual defining reader rules and training cases are part of any
prospective study design.
![Page 46: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/46.jpg)
♦Overall trial framework
♦Hypothesis and specific aims
♦ Participants
♦ Index test
♦ Reference standard
♦ Data analysis plan (statistics)
♦ Conclusions and implications
♦ Funding and compliancehttp://www.equator-network.org/reporting-guidelines/stard/
![Page 47: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/47.jpg)
Index Test (Imaging Exam)
♦ STARD 10 - Index test, in sufficient detail to allow replication
♦ STARD 12 - Definition of and rationale for test positivity cut-offs or
result categories of the index test, distinguishing pre-specified from
exploratory
♦ STARD 13 - Whether clinical information and reference standard
results were available to the performers/readers of the index test
♦ STARD 25 - Any adverse events from performing the index test
![Page 48: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/48.jpg)
Steps toward precision:
♦ Define image acquisition
• Equipment, patient preparation, protocol
• Balance “state of the art” with “generalizability”
♦ Define image analysis
• Read rules, training and testing
♦ Validate the system
• Test-retest, reader agreement measurements
♦ Build in procedures for ongoing QA
![Page 49: Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show excellent inter-observer agreement (ICC 0.94) • Different ROI protocols yield](https://reader031.vdocuments.us/reader031/viewer/2022013019/5d672daf88c993a9318b60a7/html5/thumbnails/49.jpg)