Program UpdateOctober 11, 2012

Andrew J. Buckler, MSPrincipal Investigator,

QI-Bench

WITH FUNDING SUPPORT

PROVIDED BY NATIONAL

INSTITUTE OF STANDARDS AND

TECHNOLOGY

Agenda• Summary and close-out of

FY12 development iteration– Covering what’s been accomplished

from multiple points of view

• FY13 development iteration– Deployment progress and support– Continued Progress on ISA files– Architecture– Contour-based analysis

22

Overal

l

Speci

fy

Form

ulate

Execute

Analyze

Package

Iterat

e

Studies

0

5

10

15

20

25

30

35

40

ResolvedOpen

Overall

Speci

fy

Form

ulate

Execute

Analyze

Package

Iterat

e

Studies

0

0.5

1

1.5

2

2.5

3

3.5

ResolvedOpen

Overall

Speci

fy

Form

ulate

Execute

Analyze

Package

Iterat

e

Studies

0

1

2

3

4

5

6

7

8

ResolvedOpen

Overal

l

Speci

fy

Form

ulate

Execute

Analyze

Package

Iterat

e

Studies

0

2

4

6

8

10

12

14

16

ResolvedOpen

Overall

Speci

fy

Form

ulate

Execu

te

Analyze

Package

Iterat

e

Studies

0

5

10

15

20

25

30

35

ResolvedOpen

3

Autumn 2012 (n=54)FY 2012 (n=110)

Winter 2013 (n=15) In Queue (n=6)

333

• Establish overall structure• Initial Specify, Formulate,

and Iterate• Substantial work in Execute

and Analyze• First work on test-beds

• V&V issues in Execute• Substantial work in Analyze

library• Deployment support• Advance test-beds

• Introduce radiologist workstation component, including scripted reader studies

• First real implementation of Biomarker KB triple store

• W3C-compliance in Specify• Formulate using SPARQL• Full realization of QI-Bench

cohesive architecture• Full realization of worked

example test-bed

Lab Protocol• Develop and run queries

based on data requirements

– Use of Formulate

• Load Reference Data into the Reference Data Set Manager

– Example Pilot3A Data Processing Steps

• Server-Side Processing using the Batch Analysis Service

– Package Algorithm or Method using Batch Analysis Service API

– Prepare Data Set • Create Ground Truth or othe

r Reference Annotation and Markup

• Importing location points and other data for use

– Writing Scripts – Initiate a Batch Analysis Ru

n

• Perform statistical analysis

– Analyze Use Instructions

Design Documents• User Needs and Requirements Analysis • Architecture • Application-specific Design

– Specify • "Specify" Scope Description (ASD) • "Specify" Architecture Specification (AAS) • "Quantitative Imaging Biomarker Ontology (QIB

O)" Software Design Document (SDD)

• "Biomarker DB" (a.k.a., the triple store) Software Design Document (SDD)

• AIM Template Builder Design Documentation:– Formulate

• "Formulate" Scope Description (ASD) • "Formulate" Architecture Specification (AAS) • "NBIA Connector" Software Design Document (S

DD)

– Execute • "Execute" Scope Description (ASD) • "Execute" Architecture Specification (AAS) • Reference Data Set Manager (RDSM) Software D

esign Document (SDD)

• Batch Analysis Service Software Design Document (SDD)

– Analyze • "Analyze" Scope Description (ASD) • "Analyze" Architecture Specification (AAS)

– Package • "Package" Scope Description (ASD) • "Package" Architecture Specification (AAS)

4444

V&V• QI-Bench Project Management P

lan (PMP)• Traceability Report• QI-Bench Verification and Valida

tion Plan• QI-Bench Iteration 1 Validation R

eport• QI-Bench Iteration 1 Verification

Protocol• QI-Bench Iteration 1 Verification

Report• Application Test Protocols,

Reports, and Records:– Specify: from AIM– Formulate: from caB2B– Execute

– From MIDAS– RDSM Integration Test Repor

t– Analyze

– From AVT– Library Integration Test Repo

rt– Iterate: from Taverna

Demonstrator 15, 40, 3A Pilot and Pivotal, and Change Analysis …

Investigation 1:Pilot and pivotal study are finished

FDA, M. A Gavrielides et al., “A resource for the Assessment of lung nodule size estimation methods: database of thoracic CT scans of an anthropomorphic phantom”, Optics Express, vol. 18, n.14, pp. 15244-15255, 2010.

Phantom data, FDA, NIST, QI-Bench

Fig. 1: Radial plot showing comparative performance on the selected descriptive statistics as well as mean of absolute percent errors.

Challenge Definition: estimate absolute volumes in phantom data Explicitly indicate descriptive statistics: bias, variance.

Null hypothesis: analysis software model does not have a significant effect on the bias and variance.

PILOT STUDYPIVOTAL STUDY

10 participants who measured 408 nodules12 participants who measured 97 nodules

5555

6666

7777

FY13 Development Iteration• Deployment progress and support• Continued Progress on ISA files• Architecture• Contour analysis (purpose, methods, and file formats)

88888

Continued Progress on ISA Files

99999

• Assay (a_) and Study (s_) levels:– _dcm.csv: SUBJID, TPINDEX, SITE, ACQREP, SERIESTYPE, <dicom fields>– _loc.csv: SUBJID, TPINDEX, ACQREP, LOCRDR, LOCTOOL, LOCREP, TARGET, locX, locY, locZ,

bbX0, bbX1, bbY0, bbY1, bbZ0, bbZ1 – _rdg.csv: SUBJID, TPINDEX, ACQREP, LOCRDR, LOCTOOL, LOCREP, TARGET, SEGRDR,

SEGTOOL, SEGREP, SERIESTYPE, READING, <project specific fields>– _cov.csv: SUBJID, AGE, GENDER, HEIGHT, WEIGHT, RACE, <project specific fields>– _chg.csv: SUBJID, TARGET, TPINDEX1ST, ACQREP1ST, VALUE1ST, TPINDEX2ND, ACQREP2ND,

VALUE2ND, ARITHDIFF, PCTDIFF, PPNDIFF, <project specific fields>– _dx.csv: SUBJID, TARGET, deltaX, X, SOURCE– _mo.csv: TYPE, INSTANCE, VALUE, MODULE

• Investigation (i_) level:– Works in progress, but concept is serialized triple store roll-up including

aggregation analyses such as aggregate uncertainty

data services (e.g., MIDAS, NBIA, etc.)

RDSM for Images and

ISA files

QI-B

ench

Sta

ck

Modified XIP Host Hibernate

Applications (a generic QI-Bench

template, as well as specific

configurations)

Cached objects: AIM/DICOM, etc

Data Access Layer

Web GUI High level features:• GWT (or Tapestry) UI ; both

desktop and web client versions

• RESTful service layer; need to work out details between Hibernate and Jena

Implemented according to open source best practices;

In such a way as to enable the enhancement roadmap; and

Integrated with projects driving advanced semantics and support for regulatory e-submissions

Ontologies and vocabularies

RDF triple store for Patient info,annotations, Collections

Experiments

Jena

1010

XIP LIB R LIB

Taverna

Applications (a generic QI-Bench

template, as well as specific

configurations)

Cached objects: AIM/DICOM, etc

Data Access Layer

Cohort applications:• Specify• Formulate• Execute’s RDSM and BAS• Analyze/Iterate (may

combine)

1111

XIP LIB R LIB

Taverna

Applications (a generic QI-Bench

template, as well as specific

configurations)

Cached objects: AIM/DICOM, etc

Data Access Layer

Individual Subject (Patient) Workstation• Plug-in to <fill in your favorite

workstation>• We provide:

• wrapper for ClearCanvas as example and template

• Data access layer:• Unified worklist

transactions• Support for Q/R to

RDSM• Support for access

to Biomarker KB• Taverna-desktop

level of capability for workflows

1212

XIP LIB R LIB

Taverna

C++ layer, for leverage of components in C++ and support of ClearCanvas?

Java core, for consistency across QI-Bench?

data services (e.g., MIDAS, NBIA, etc.)

RDSM for Images and

ISA files

Modified XIP Host Hibernate

GUI Local Host (“workstation”) configuration (thick client)

Ontologies and vocabularies

RDF triple store for Patient info,annotations, Collections

Experiments

Jena

1313

local

remote

data services (e.g., MIDAS, NBIA, etc.)

RDSM for Images and

ISA files

Modified XIP Host Hibernate

Web GUI Web-based (thin client)

Ontologies and vocabularies

RDF triple store for Patient info,annotations, Collections

Experiments

Jena

1414

local

remote

Early ideas on Technical Approach• What

– Re-architect UI for Analyze application– Interface XIP Host Services through REST API– Retrieve AIM data and pass to Analyze– Present analysis data in new web UI– Interface with all AIM and DICOM data through XIP Host Services

• How– Incorporate Hibernate object-relational mapping (ORM) for DB2, Midas

• Build XIP Host Services instance• Retrieve all data through XIP Host Services and WADO

– Select Web application Framework GWT, Tapestry, Spring, Wicket, HybridJava, etc., and build UI for

• XIP Host Services for Data Retrieval• presenting results from MVT application• interacting with MVT analysis data based on User Stories and Use Cases.

– Using Hibernate, persist results using Jena KB

1515

Contour-based Analysis• Purpose• Methods:

– STAPLE– Meyer’s P-maps– MICCAI indices– DICE

• File formats:– DICOM segmentation objects– AIM 4.0– STL– MHT

1616161616

1717

Value proposition of QI-Bench• Efficiently collect and exploit evidence establishing

standards for optimized quantitative imaging:– Users want confidence in the read-outs– Pharma wants to use them as endpoints– Device/SW companies want to market products that produce them

without huge costs– Public wants to trust the decisions that they contribute to

• By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data

• Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders

1818

Summary:QI-Bench Contributions• We make it practical to increase the magnitude of data for increased

statistical significance. • We provide practical means to grapple with massive data sets.• We address the problem of efficient use of resources to assess limits of

generalizability. • We make formal specification accessible to diverse groups of experts that are

not skilled or interested in knowledge engineering. • We map both medical as well as technical domain expertise into

representations well suited to emerging capabilities of the semantic web. • We enable a mechanism to assess compliance with standards or

requirements within specific contexts for use.• We take a “toolbox” approach to statistical analysis. • We provide the capability in a manner which is accessible to varying levels of

collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access.

1919

QI-BenchStructure / Acknowledgements• Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette, Kjell Johnson, Jovanna

Danagoulian)

• Co-Investigators– Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer)– Stanford (David Paik)

• Financial support as well as technical content: NIST (Mary Brady, Alden Dima, John Lu)

• Collaborators / Colleagues / Idea Contributors– Georgetown (Baris Suzek)– FDA (Nick Petrick, Marios Gavrielides) – UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad, Yelena Yesha)– Northwestern (Pat Mongkolwat)– UCLA (Grace Kim)– VUmc (Otto Hoekstra)

• Industry– Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner)– Device/Software: Definiens, Median, Intio, GE, Siemens, Mevis, Claron Technologies, …

• Coordinating Programs– RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao)– Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien)

2020

Statistical Validation Service for Imaging

Quantitatively characterizing and optimizing performance of imaging accelerates discovery and widens the availability of new treatments to patients with unmet medical needs.• We bring MVT forward as it could be beyond what it currently is; • Available as a web application with thin and thick-client options with

persistent database.• Implemented to be generalizable to needs of RadOnc, QIN, QIBA, FNIH, C-

Path, and other members of the community. Applications include:• Augmenting the current genome based biomarkers with imaging based

biomarkers in Transcend for Breast cancer and/or TCGA for brain cancer• Community Cancer Centers• Project may be synergistically pursued with the FDA imaging submission

project.

2121