May 2012 UpdateMay 10, 2012

Andrew J. Buckler, MSPrincipal Investigator,

QI-Bench

WITH FUNDING SUPPORT

PROVIDED BY NATIONAL

INSTITUTE OF STANDARDS AND

TECHNOLOGY

Resources are needed to address widening gap in imaging capability as practiced vs. capability of modern medicine

2222222

Example: Beyond Anatomy to Palette of Functional Measures

33333333

glucose metabolism

Biologic Target

bone formation

proliferation

hypoxia

amino acid metabolism

angiogenesis

receptor statusapotosis

18F-FDG18F-NaF

18F-FLT18F-

FACBC

18F-FMISO

18F-XXX

DCE-MRI

PET

18F-FES

QI-Bench is a resource that may be used by single sponsors, defined-entity consortia, or true open science programs

4444444

Example: COPD

Either:• To assist individual suppliers

in optimizing their offerings• To assist groups like

COPDgene consortia• To enable open

development such as by QIBA

or flexible mix of these.

55555555

Data warehouse

QI-Bench is composed of building blocks

6666666

Provenance architecture of Iterate

Example: DCE-MRI using Patient, Synthetic, and Phantom Data• Curate, maintain and serve

reference data sets• Execute batch runs over multi-

parameter synthetic data• Characterize performance

77777777

Data warehouse

Provenance architecture of Iterate

MD Andersen receives

phantoms

Parallel Imaging procedure

GE (new) scanner

Phantoms are comparable

Ship phantom A to UPENN

Ship phantom B to U Chicago

Imaging procedureSiemens MR

scanners

Imaging procedurePhilips (new)

scanner

Ship phantom BTo UC Davis

Image procedureGE (old) scanners

Ship phantom A toDuke

Imaging procedurePhilips (old)

Ship phantom A and B back to MD

Anderson

Test bed: CT volumetry method challenge (“3A”)

8888888

1. Median Technologies2. Vital Images, Inc.3. Fraunhofer Mevis4. Siemens5. Moffitt Cancer Center6. Toshiba

Pilot

Pivotal

Investigation 1

Train

Test

Pilot

Pivotal

Investigation

Train

Test

Pilot

Pivotal

Investigation

Train

Test

Pilot

Pivotal

Investigation n

Train

Test

Pr im

ar y

Se co n dar y

• Defined set of data• Defined challenge• Defined test set policy

Some of the Participants7. GE Healthcare8. Icon Medical Imaging9. Columbia University10. INTIO, Inc.11. Vital Images, Inc.

Broader capability: Systematic qualification of CT volumetry

99999999

1. Median Technologies2. Vital Images, Inc.3. Fraunhofer Mevis4. Siemens5. Moffitt Cancer Center6. Toshiba

Some of the Participants7. GE Healthcare8. Icon Medical Imaging9. Columbia University10. INTIO, Inc.11. Vital Images, Inc. PROFILE Authoring and Testing

Inter-analysis technique (algorithm) variability (3A)

Correlation with clinical endpoints and outcomes (3B)

machine view human expert

view

Transformation

Modality Environment

Therapy Decision

Environment

Patient Patient

Transformation Transformation

FeedbackTherapy-Machine Human Observer

0.0

0.2

0.4

0.6

0.8

1.0

% R

each

ing Pa

rtial R

espo

nse

0 42 84 126 168 210 252 294 336 378

Time on Trial (days)

SLDVolumetric

Survival Plot

Intra- and inter-reader variability (1A)

Minimum detectable biological change (1B)

5 readers, 3 reads

each

Expl

ore

figur

es-

of-m

erit

and

QC

proc

edur

es

Update: Iterate progress• Added small workflows to Iterate

demonstration server.• Stored and retrieved data on server

via workflows.• Distinction between workflow

execution on server and local workstation.

1010

Update: Iterate coming attractions• Upgraded base software installation.• New site with workflows for local

workstation.• Server workflows that produce

outputs.

1111

Analyze performs the statistical analyses…Subject Predicate Object

A Is Patient

A isDiagnosedWith DiseaseA

DiseaseA Is NonSmallLCellLunCancer

A hasClinicalObservation

B

B Is TumorShrinkage

C Is Patient

C hasClinicalObservation

B

D hasClinicalObservation

B

Pazopanib Is TyrosoineKinaseInhibitor

A isTreatedWith Pazopanib

A hasOutcome Death

C hasOutcome Survival

Subject Predicate Object

CT images Tumor

Volumetry analyzes CT

<compliant>LongitudinalVolumetry

estimates TumorSizeChange

TumorSizeChange predicts CytotoxicTreatmentResponse

TyrosoineKinaseInhibitor is CytotoxicTreatment

well-controlled Phase II and III efficacy studies

uses CytotoxicTreatmentResponse

CytotoxicTreatment influences NonSmallCellLungCancer

CT images Thorax

Thorax contains NonSmallCellLungCancer

regulatory drug approval dependsOn PrimaryEndpoint

well-controlled Phase II and III efficacy studies

assess PrimaryEndpoint

CT Volumetry is SurrogateEndpoint for CytotoxicTreatment

1

3

2

1212

…and adds the results to the knowledgebase (using W3C “best practices” for “relation strength”).

Subject Predicate Object

45324 biasMethod <r script used>

45324 bias <summary statistic>

45324 variabilityMethod <r script used>

45324 variability <summary statistic>

9956 <correlation>Method <r script used>

9956 correlation <summary statistic>

9956 <ROC>Method <r script used>

9956 ROC <summary statistic>

98234 Effect of treatment on true endpoint <value>

98234 Effect of treatment on surrogate endpoint <value>

98234 Effect of surrogate on true endpoint <value>

98234 Effect of treatment on true endpoint relative to that on surrogate endpoint

<value>

Subject Predicate Object

CT images Tumor

Volumetry analyzes CT

<compliant>LongitudinalVolumetry

estimates TumorSizeChange

TumorSizeChange predicts CytotoxicTreatmentResponse

TyrosoineKinaseInhibitor is CytotoxicTreatment

well-controlled Phase II and III efficacy studies

uses CytotoxicTreatmentResponse

CytotoxicTreatment influences NonSmallCellLungCancer

CT images Thorax

Thorax contains NonSmallCellLungCancer

regulatory drug approval dependsOn PrimaryEndpoint

well-controlled Phase II and III efficacy studies

assess PrimaryEndpoint

CT Volumetry is SurrogateEndpoint for CytotoxicTreatment

1

3

2

URI=45324

URI=9956

URI=98234

1313

1414

Demonstration of statistical analysis using R within Iterate• Obtain data• Produce summary plots and statistics• Perform analyses

Provenance architecture of Iterate

Conceptual Example• Example data based on QIBA 3A layout• Phantom study

– 5 lesions, each read by 3 readers• The study design enables us to answer several questions

– Is there a difference among readers on volume? • If so, which are different?

– Is there a difference among readers on bias? • If so, which are different?

• Appropriate figures flow from the study design and enable us to visually understand the data

151515151515

DatalesionID readerID trueVolume readVolume

2 lstk 524.7 520.24 lstk 527.4 508.3

10 lstk 431.6 435.239 lstk 280.5 299.663 lstk 439.8 457.8

2 reader1 524.7 560.74 reader1 527.4 504.7

10 reader1 431.6 438.139 reader1 280.5 306.963 reader1 439.8 433.5

2 reader2 524.7 507.34 reader2 527.4 495.1

10 reader2 431.6 396.939 reader2 280.5 258.863 reader2 439.8 421.2

171717171717

181818181818

191919191919

202020202020

212121212121

summary1

222222222222

summary2

232323232323

tukey1

242424242424

tukey2

252525252525

262626262626

27272727272727

Up and running now for you to use

2828

Value proposition of QI-Bench• Efficiently collect and exploit evidence establishing

standards for optimized quantitative imaging:– Users want confidence in the read-outs– Pharma wants to use them as endpoints– Device/SW companies want to market products that produce them

without huge costs– Public wants to trust the decisions that they contribute to

• By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data

• Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders

2929

Summary:QI-Bench Contributions• We make it practical to increase the magnitude of data for increased

statistical significance. • We provide practical means to grapple with massive data sets.• We address the problem of efficient use of resources to assess limits of

generalizability. • We make formal specification accessible to diverse groups of experts that are

not skilled or interested in knowledge engineering. • We map both medical as well as technical domain expertise into

representations well suited to emerging capabilities of the semantic web. • We enable a mechanism to assess compliance with standards or

requirements within specific contexts for use.• We take a “toolbox” approach to statistical analysis. • We provide the capability in a manner which is accessible to varying levels of

collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access.

3030

QI-BenchStructure / Acknowledgements• Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette)

• Co-Investigators– Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer)– Stanford (David Paik)

• Financial support as well as technical content: NIST (Mary Brady, Alden Dima, John Lu)

• Collaborators / Colleagues / Idea Contributors– Georgetown (Baris Suzek)– FDA (Nick Petrick, Marios Gavrielides) – UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad, Yelena Yesha)– Northwestern (Pat Mongkolwat)– UCLA (Grace Kim)– VUmc (Otto Hoekstra)

• Industry– Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner)– Device/Software: Definiens, Median, Intio, GE, Siemens, Mevis, Claron Technologies, …

• Coordinating Programs– RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao)– Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien)

3131