3rd Program Face to FaceNovember 15, 2011

Andrew J. Buckler, MSPrincipal Investigator

WITH FUNDING SUPPORT

PROVIDED BY NATIONAL

INSTITUTE OF STANDARDS AND

TECHNOLOGY

Value proposition of QI-Bench• Efficiently collect and exploit evidence establishing

standards for optimized quantitative imaging:– Users want confidence in the read-outs– Pharma wants to use them as endpoints– Device/SW companies want to market products that produce them

without huge costs– Public wants to trust the decisions that they contribute to

• By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data

• Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders

22

QI-BenchStructure / Acknowledgements• Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette)

• Co-Investigators– Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer)– Stanford (David Paik, Tiffany Ting Liu)

• Financial support as well as technical content: NIST (Mary Brady, Alden Dima, Guillaume Radde)

• Collaborators / Colleagues / Idea Contributors– FDA (Nick Petrick, Marios Gavrielides)– UCLA (Grace Kim)– UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad)– VUmc (Otto Hoekstra)– Northwestern (Pat Mongkolwat)– Georgetown (Baris Suzek)

• Industry– Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner)– Device/Software: Definiens (Maria Athelogou), Claron Technologies (Ingmar Bitter)

• Coordinating Programs– RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao)– Under consideration: CTMM TraIT (Henk Huisman, Jeroen Belien)

33

QI-Bench is Use case DrivenCreate and Manage Semantic Infrastructure and Linked Data Archives

Create and Manage Physical and Digital Reference Objects

Core Activities for Biomarker Development

Commercial Sponsor Prepares Device/Test for Market

Consortium Establishes Clinical Utility/Efficacy of Putative Biomarker

Collaborative Activities to Standardize and/or Optimize the Biomarker

44

Create and Manage Semantic Infrastructure

and Linked Data Archives

Create and Manage Physical and Digital

Reference Objects

Core Activities for Biomarker Development

Commercial Sponsor Prepares Device/Test for

MarketConsortium Establishes Clinical Utility/Efficacy

of Putative Biomarker

Collaborative Activities to Standardize and/or

Optimize the Biomarker

QISLQuantitative Imaging

Specification Language

Batch Analysis Service

Reference Data Set Manager

UPICT Protocols, QIBA Profiles, literature papers and other sources

QIBO-

BatchMake Scripts

Reference Data Sets, Annotations, and Analysis Results

(red edgesrepresent

biostatisticalgeneralizability)

Source of clinical study results

Clinical Body of Evidence (formatted to enable SDTM and/or other standardized registrations

4. Output

3. Batch analysis scripts

UPICT Protocols, QIBA Profiles, entered with

Ruby on Rails web service

QIBO

55

• Specify context for use and assay methods.

• Use consensus terms in doing so.

Specify

• Assemble applicable reference data sets.

• Include both imaging and non-imaging clinical data.

Formulate• Compile evidence for regulatory

filings.• Use standards in transfer to regulatory

agencies.

Package

QISLQuantitative Imaging

Specification Language

Batch Analysis Service

Reference Data Set Manager

UPICT Protocols, QIBA Profiles, literature papers and other sources

QIBO-

BatchMake Scripts

Reference Data Sets, Annotations, and Analysis Results

(red edges represent biostatistical generalizability)

Source of clinical study results

Clinical Body of Evidence (formatted to enable SDTM and/or other standardized registrations

4. Output

3. Batch analysis scripts

UPICT Protocols, QIBA Profiles, entered with

Ruby on Rails web service

QIBO

66

STDM standard of CDISC into repositories like FDA’s

Janus.

MVT portion of AVT, re-useable

library of R scripts.

MIDAS, BatchMake, Condor Grid;

built using CakePHP.

caB2B, NBIA,

PODS data elements, DICOM

query tools.

QIBO, AIM,RadLex/ Snomed/ NCIt; built

using Ruby on Rails.

• Specify context for use and assay methods.

• Use consensus terms in doing so.

Specify

• Assemble applicable reference data sets.

• Include both imaging and non-imaging clinical data.

Formulate

• Compile evidence for regulatory filings.

• Use standards in transfer to regulatory agencies.

Package

77

8

BSD-2 licenseDomain is www.qi-bench.org.

Landing page provides • Access to

prototypes, • Repositories for

download and development,

• Acknowledgements,

• Jira issue tracking, and

• Documentation

8

9

Project wiki includes sections for • Project

management plan,

• User needs analysis (including use cases),

• Lab Protocol, • Developer’s helps

(including use of Git),

• Meeting minutes, and

• Discussion of investigators/ collaborators.

9

Specify:Specify is presently a composite of QISL and the AIM template builder.

The Quantitative Imaging Specification Language (QISL) uses the Quantitative Imaging Biomarker Ontology (QIBO) and other linked ontologies to develop a triple store based on user Q/A.

10

11

• Initial curation to collect terms: reviewed 126 articles across 6 therapeutic areas elaborating 225 imaging markers

• Reusing other publicly available ontologies: MeSH, NCI thesaurus, GO, FMA, and BIRNLex

• Current sate: 490 classes and relationship properties for clinical context for use and assay methods.

• Next steps: Basic Formal Ontology (BFO) as an upper ontology that provides a formal structure of upper level abstract classes that has been adapted by the Open Biological and Biomedical Ontologies (OBO) foundry, a large collaborative effort for the goal of creating orthogonal and interoperable ontologies in biomedical research.

Quantitative Imaging Biomarker Ontology (QIBO)

Specify (cont)The idea is that AIM templates would be constructed and linked to the other specification information from the ontologies.

Presently it just co-exists in the prototype app, it is not yet functionally integrated as ultimately intended.

12

FormulateWeb-enabled service for aggregating reference data based on endpoints

13

Formulate (continued)• One small part of Formulate that we have done is to create a CQL “connecter” to

import data from NBIA. The reason we do this is to optimize storage for grid computing and to include metadata storage needed to run experiments.

14

Endp

oint

s for

For

mul

ate

class Complete Information Model

TRIAL_DATA_PROVENANCE

*PK TRIAL_DP_PK_ID: INTEGER DP_SITE_NAME: VARCHAR(40) DP_SITE_ID: VARCHAR(64) PROJECT: VARCHAR(50)

+ PK_TRIAL_DP_PK_ID(INTEGER)

PATIENT

*PK PATIENT_PK_ID: NUMBER(15) PATIENT_ID: VARCHAR2(64) PATIENT_NAME: VARCHAR2(250) PATIENT_BIRTH_DATE: DATE PATIENT_SEX: VARCHAR2(16) ETHNIC_GROUP: VARCHAR2(16) FK TRIAL_DP_PK_ID: NUMBER(15) TRIAL_SUBJECT_ID: VARCHAR2(64) TRIAL_SUBJECT_READING_ID: VARCHAR2(64) FK TRIAL_SITE_PK_ID: NUMBER(15) VISIBILITY: VARCHAR2(20) VERSION: NUMBER(15) = 0

+ PK_PATIENT_PK_ID(NUMBER)

+ PATIENT_VISIBILITY_IND(VARCHAR2)

+ FK_TRIAL_DP_PK_ID(NUMBER)+ FK_TRIAL_SITE_PK_ID(NUMBER)

STUDY

*PK STUDY_PK_ID: INTEGER STUDY_INSTANCE_UID: VARCHAR(500) STUDY_DATE: DATETIME STUDY_TIME: VARCHAR(16) STUDY_DESC: VARCHAR(64) ADMITTING_DIAGNOSES_DESC: VARCHAR(64) ADMITTING_DIAGNOSES_CODE_SEQ: VARCHAR(500) FK PATIENT_PK_ID: INTEGER STUDY_ID: VARCHAR(16) TRIAL_TIME_POINT_ID: VARCHAR(64) TRIAL_TIME_POINT_DESC: VARCHAR(1024) PATIENT_AGE: VARCHAR(4) AGE_GROUP: VARCHAR(10) PATIENT_SIZE: INTEGER PATIENT_WEIGHT: INTEGER OCCUPATION: VARCHAR(16) ADDITIONAL_PATIENT_HISTORY: VARCHAR(4000) VISIBILITY: VARCHAR(20) VERSION: INTEGER = 0

+ PK_STUDY_PK_ID(INTEGER)

+ STUDY_DATE_IDX(DATE)+ STUDY_DESC_IDX(VARCHAR)+ STUDY_VISIBILITY_IND(VARCHAR)

+ FK_PATIENT_PK_ID(INTEGER)

GENERAL_SERIES

*PK GENERAL_SERIES_PK_ID: NUMBER(15) MODALITY: VARCHAR2(16) SERIES_INSTANCE_UID: VARCHAR2(64) SERIES_LATERALITY: VARCHAR2(16) SERIES_DATE: DATE PROTOCOL_NAME: VARCHAR2(64) SERIES_DESC: VARCHAR2(64) BODY_PART_EXAMINED: VARCHAR2(16) FK STUDY_PK_ID: NUMBER(15) FK GENERAL_EQUIPMENT_PK_ID: NUMBER(15) TRIAL_PROTOCOL_ID: VARCHAR2(64) TRIAL_PROTOCOL_NAME: VARCHAR2(64) TRIAL_SITE_NAME: VARCHAR2(64) STUDY_DATE: DATE STUDY_DESC: VARCHAR2(64) ADMITTING_DIAGNOSES_DESC: VARCHAR2(64) PATIENT_AGE: VARCHAR2(4) PATIENT_SEX: VARCHAR2(16) PATIENT_WEIGHT: NUMBER(15) AGE_GROUP: VARCHAR2(10) PATIENT_PK_ID: NUMBER(15) SERIES_NUMBER: NUMBER(15) SYNC_FRAME_OF_REF_UID: VARCHAR2(64) PATIENT_ID: VARCHAR2(64) FRAME_OF_REFERENCE_UID: VARCHAR2(64) VISIBILITY: VARCHAR2(20) SECURITY_GROUP: VARCHAR2(300) ANNOTATIONS_FLAG: VARCHAR2(5) VERSION: NUMBER(15) = 0

+ PK_G_SERIES_PK_ID(NUMBER)

+ BODY_PART_EXAMINED_IDX(VARCHAR2)+ GENERAL_SERIES_SEC_GRP_IDX(VARCHAR2)+ GENERAL_SERIES_SITE_IDX(VARCHAR2)+ MODALITY_IDX(VARCHAR2)+ SERIES_DATE_IDX(DATE)+ SERIES_DESC_IDX(VARCHAR2)+ SERIES_VISIBILITY_IND(VARCHAR2)

+ FK_G_EQUIPMENT_PK_ID(NUMBER)+ FK_GS_STUDY_PK_ID(NUMBER)

GENERAL_IMAGE

«column»*PK IMAGE_PK_ID: NUMBER(15) INSTANCE_NUMBER: NUMBER(15) CONTENT_DATE: DATE CONTENT_TIME: VARCHAR2(16) IMAGE_TYPE: VARCHAR2(16) ACQUISITION_DATE: DATE ACQUISITION_TIME: VARCHAR2(16) ACQUISITION_NUMBER: NUMBER(15) LOSSY_IMAGE_COMPRESSION: VARCHAR2(16) PIXEL_SPACING: NUMBER(15) IMAGE_ORIENTATION_PATIENT: VARCHAR2(200) IMAGE_POSITION_PATIENT: VARCHAR2(200) SLICE_THICKNESS: NUMBER(15) SLICE_LOCATION: NUMBER(15) I_ROWS: NUMBER(15) I_COLUMNS: NUMBER(15) CONTRAST_BOLUS_AGENT: VARCHAR2(64) CONTRAST_BOLUS_ROUTE: VARCHAR2(64) SOP_CLASS_UID: VARCHAR2(64) SOP_INSTANCE_UID: VARCHAR2(64) FK GENERAL_SERIES_PK_ID: NUMBER(15) PATIENT_POSITION: VARCHAR2(16) SOURCE_TO_DETECTOR_DISTANCE: NUMBER(15) SOURCE_SUBJECT_DISTANCE: NUMBER(15) FOCAL_SPOT_SIZE: NUMBER(15) STORAGE_MEDIA_FILE_SET_UID: VARCHAR2(64) MIRC_DOC_URI: VARCHAR2(2000) DICOM_FILE_URI: VARCHAR2(2000) ACQUISITION_DATETIME: VARCHAR2(50) IMAGE_COMMENTS: VARCHAR2(4000) IMAGE_RECEIVING_TIMESTAMP: DATE CURATION_STATUS: VARCHAR2(20) CURATION_TIMESTAMP: DATE VISIBILITY: VARCHAR2(20) ANNOTATION: VARCHAR2(20) SUBMISSION_DATE: DATE DICOM_SIZE: NUMBER(15) IMAGE_LATERALITY: VARCHAR2(16) TRIAL_DP_PK_ID: NUMBER(15) PATIENT_ID: VARCHAR2(64) STUDY_INSTANCE_UID: VARCHAR2(500) SERIES_INSTANCE_UID: VARCHAR2(64) PATIENT_PK_ID: NUMBER(15) STUDY_PK_ID: NUMBER(15) PROJECT: VARCHAR2(200) VERSION: NUMBER(15) = 0 ACQUISITION_MATRIX: NUMBER(15) = 0 DX_DATA_COLLECTION_DIAMETER: NUMBER(15) = 0

«PK»+ PK_IMAGE_PK_ID(NUMBER)

«index»+ ACQUISITION_MATRIX_IDX(NUMBER)+ CONTRAST_BOLUS_ROUTE_IDX(VARCHAR2)+ CURATION_T_INDX(DATE)+ DX_DATA_COLLECTION_DIAMETER(NUMBER)+ GENERAL_IMAGE_SEARCH(NUMBER, NUMBER, NUMBER, VARCHAR2, VARCHAR2, DATE)+ GI_GS_DS_INDX(NUMBER, NUMBER, VARCHAR2)+ GI_PPKID_INDX(NUMBER)+ GI_SPKID_INDX(NUMBER)+ GI_TDPKID_INDX(NUMBER)+ IMAGE_FK_SERIES_PK_ID(NUMBER)+ IMAGE_SOP_INSTANCE_UID(VARCHAR2)+ IMAGE_VISIBILITY_IND(VARCHAR2)+ SLICE_THICKNESS_IDX(NUMBER)

«FK»+ FK_G_SERIES_PK_ID(NUMBER)

CT_IMAGE

«column» KVP: NUMBER(15) SCAN_OPTIONS: VARCHAR2(16) DATA_COLLECTION_DIAMETER: NUMBER(15) RECONSTRUCTION_DIAMETER: NUMBER(15) GANTRY_DETECTOR_TILT: NUMBER(15) EXPOSURE_TIME: NUMBER(15) X_RAY_TUBE_CURRENT: NUMBER(15) EXPOSURE: NUMBER(15) EXPOSURE_IN_MICROAS: NUMBER(15) CONVOLUTION_KERNEL: VARCHAR2(16) REVOLUTION_TIME: NUMBER(15) SINGLE_COLLIMATION_WIDTH: NUMBER(15) TOTAL_COLLIMATION_WIDTH: NUMBER(15) TABLE_SPEED: NUMBER(15) TABLE_FEED_PER_ROTATION: NUMBER(15) CT_PITCH_FACTOR: NUMBER(15) ANATOMIC_REGION_SEQ: VARCHAR2(500)*FK IMAGE_PK_ID: NUMBER(15)*PK CT_IMAGE_PK_ID: NUMBER(15) VISIBILITY: VARCHAR2(20)

«PK»+ PK_CT_IMAGE_PK_ID(NUMBER)

«index»+ CONVOLUTION_KERNEL_IDX(VARCHAR2)+ CT_IMAGE_IMAGE_PK_ID_INDX(NUMBER)+ KVP_IDX(NUMBER)+ VISIBILITY_CT_IMAGE(VARCHAR2)

«FK»+ FK_IMAGE_PK_ID(NUMBER)

CLINICAL_TRIAL

«column»*PK TRIAL_PK_ID: INTEGER TRIAL_SPONSOR_NAME: VARCHAR(64) TRIAL_PROTOCOL_ID: VARCHAR(64) TRIAL_PROTOCOL_NAME: VARCHAR(64) TRIAL_COORDINATING_CENTER: VARCHAR(64)

«PK»+ PK_TRIAL_PK_ID(INTEGER)

TRIAL_SITE

*PK TRIAL_SITE_PK_ID: INTEGER TRIAL_SITE_ID: VARCHAR(64) TRIAL_SITE_NAME: VARCHAR(64) FK TRIAL_PK_ID: INTEGER

+ PK_TRIAL_SITE_PK_ID(INTEGER)

+ FK_TRIAL_PK_ID(INTEGER)

Annotation

- aimVersion: String- cagridId: Integer- codeMeaning: String- codeValue: String- codingSchemeDesignator: String- codingSchemeVersion: String [0..1]- comment: String [0..1]- dateTime: Date- name: String- precedentReferencedAnnotationUID: String [0..1]- uniqueIdentifier: String

+ GetAnatomicEntityCollection() : AnatomicEntity[]+ GetCalculationCollection() : Calculation[]+ GetEquipment() : Equipment+ GetImagingObservationCollection() : ImagingObservation[]+ GetUser() : User+ SetAnatomicEnti tyCollection(AnatomicEntity[]) : void+ SetCalculationCollection(Calculation[]) : void+ SetEquipment(Equipment) : void+ SetImagingObservationCol lection(ImagingObservation[]) : void+ SetUser(User) : void

+ GetAimVersion() : String+ GetCodeMeaning() : String+ GetCodeValue() : String+ GetCodingSchemeDesignator() : String+ GetCodingSchemeVersion() : String+ GetComment() : String+ GetDateTime() : Date+ GetName() : String+ GetPrecedentReferencedAnnotationUID() : String+ GetUniqueIdentifier() : String+ IsAllowModification() : boolean

+ SetAimVersion(String) : void+ SetAllowModification(boolean) : void+ SetCodeMeaning(String) : void+ SetCodeValue(String) : void+ SetCodingSchemeDesignator(String) : void+ SetCodingSchemeVersion(String) : void+ SetComment(String) : void+ SetDateTime(Date) : void+ SetName(String) : void+ SetPrecedentReferencedAnnotationUID(String) : void+ SetUniqueIdentifier(String) : void

ImageAnnotation

+ GetGeometricShapeCollection() : GeometricShape[]+ GetImageReferenceCol lection() : ImageReference[]+ GetInferenceCollection() : Inference[]+ GetPatient() : Person+ GetSegmentationCollection() : Segmentation[]+ GetTextAnnotationCol lection() : TextAnnotation[]+ SetGeometricShapeCollection(GeometricShape[]) : void+ SetImageReferenceCollection(ImageReference[]) : void+ SetInferenceCollection(Inference[]) : void+ SetPatient(Person) : void+ SetSegmentationCollection(Segmentation[]) : void+ SetTextAnnotationCollection(TextAnnotation[]) : void

Calculation

- algorithmName: String [0..1]- algorithmVersion: String [0..1]- cagridId: Integer- codeMeaning: String- codeValue: String- codingSchemeDesignator: String- codingSchemeVersion: String [0..1]- description: String- mathML: String [0..1]- uid: String

+ GetCalculationResultCol lection() : CalculationResult[]+ GetReferencedCalculationCollection() : ReferencedCalculation[]+ GetReferencedGeometricShapeCol lection() : ReferencedGeometricShape[]+ SetCalculationResultCollection(CalculationResult[]) : void+ SetReferencedCalculationCol lection(ReferencedCalculation[]) : void+ SetReferencedGeometricShapeCollection(ReferencedGeometricShape[]) : void

+ GetAlgorithmName() : String+ GetCodeMeaning() : String+ GetCodeValue() : String+ GetCodingSchemeDesignator() : String+ GetCodingSchemeVersion() : String+ GetDescription() : String+ GetMathML() : String+ GetUID() : String

+ SetAlgorithmName(String) : void+ SetCodeMeaning(String) : void+ SetCodeValue(String) : void+ SetCodingSchemeDesignator(String) : void+ SetCodingSchemeVersion(String) : void+ SetDescription(String) : void+ SetMathML(String) : void+ SetUID(String) : void

Circle

+ GetCenter() : SpatialCoordinate+ GetRadiusPoint() : SpatialCoordinate+ SetCenter(SpatialCoordinate) : void+ SetRadiusPoint(SpatialCoordinate) : void

MultiPoint

Polyline

Ellipse

+ GetEllipseCol lection() : SpatialCoordinate[]+ SetEllipseCollection(SpatialCoordinate[]) : void

ImagingObservation

- annotatorConfidence: Double [0..1]- cagridId: Integer- codeMeaning: String- codeValue: String- codingSchemeDesignator: String- codingSchemeVersion: String [0..1]- comment: String [0..1]- isPresent: boolean [0..1]- label: String

+ GetImagingObservationCharacteristicCollection() : ImagingObservationCharacteristic[]+ SetImagingObservationCharacteristicCol lection(ImagingObservationCharacteristic[]) : void

+ GetAnnotatorConfidence() : Double+ GetCodeMeaning() : String+ GetCodeValue() : String+ GetCodingSchemeDesignator() : String+ GetCodingSchemeVersion() : String+ GetComment() : String+ GetConfidence() : Double+ GetLabel() : String+ IsIsPresent() : boolean

+ SetAnnotatorConfidence(Double) : void+ SetCodeMeaning(String) : void+ SetCodeValue(String) : void+ SetCodingSchemeDesignator(String) : void+ SetCodingSchemeVersion(String) : void+ SetComment(String) : void+ SetConfidence(Double) : void+ SetIsPresent(boolean) : void+ SetLabel(String) : void

CalculationResult

- cagridId: Integer- numberOfDimensions: Integer- type: CalculationResultIdentifier- uni tOfMeasure: String

+ GetDataCollection() : CalculationData[]+ GetDimensionCollection() : Dimension[]+ SetDataCollection(CalculationData) : void+ SetDimensionCollection(Dimension[]) : void

+ GetCodeValue() : String+ GetNumberOfDimensions() : Integer+ GetType() : String+ GetUnitOfMeasure() : String

+ SetCodeValue(String) : void+ SetNumberOfDimensions(Integer) : void+ SetType(String) : void+ SetUnitOfMeasure(String) : void

User

- cagridId: Integer- loginName: String- name: String- numberWithinRoleOfClinicalTrial: Integer [0..1]- roleInTrial: String [0..1]

+ GetLoginName() : String+ GetName() : String+ GetNumberWithinRoleOfClinicalTrial() : Integer+ GetRoleInTrial() : String

+ SetLoginName(String) : void+ SetName(String) : void+ SetNumberWithinRoleOfCl inicalTrial(Integer) : void+ SetRoleInTrial(String) : void

ImageReference

- cagridId: Integer

DICOMImageReference

+ GetPresentationStateCollection() : PresentationState []+ GetStudy() : ImageStudy+ SetPresentationStateCol lection(PresentationState []) : void+ SetStudy(ImageStudy)

WebImageReference

- uri: String

«property get»+ GetURI() : String

«property set»+ SetURI(String) : void

ReferencedCalculation

- cagridId: Integer- uniqueIdentifier: String

+ GetUniqueIdentifier() : String

+ SetUniqueIdentifier(String) : void

Point

+ GetCenter() : SpatialCoordinate+ SetCenter(SpatialCoordinate) : void

ImagingObservationCharacteristic

- annotatorConfidence: Double [0..1]- cagridId: Integer- codeMeaning: String- codeValue: String- codingSchemeDesignator: String- codingSchemeVersion: String [0..1]- comment: String [0..1]- label: String

+ GetCharacteristicQuantificationCollection() : CharacteristicQuanti fication []+ SetCharacteristicQuanti ficationCol lection(CharacteristicQuanti fication []) : void

+ GetAnnotatorConfidence() : Double+ GetCodeMeaning() : String+ GetCodeValue() : String+ GetCodingSchemeDesignator() : String+ GetCodingSchemeVersion() : String+ GetComment() : String+ GetLabel() : String

+ SetAnnotatorConfidence(Double) : void+ SetCodeMeaning(String) : void+ SetCodeValue(String) : void+ SetCodingSchemeDesignator(String) : void+ SetCodingSchemeVersion(String) : void+ SetComment(String) : void+ SetLabel(String) : void

AnatomicEntity

- annotatorConfidence: Double [0..1]- cagridId: Integer- codeMeaning: String- codeValue: String- codingSchemeDesignator: String- codingSchemeVersion: String [0..1]- isPresent: boolean [0..1]- label : String

+ GetAnatomicEntityCharacteristicCol lection() : AnatomicEnti tyCharacteristic[]+ SetAnatomicEnti tyCharacteristicCollection(AnatomicEnti tyCharacteristic[]) : void

+ GetAnnotatorConfidence() : Double+ GetCodeMeaning() : String+ GetCodeValue() : String+ GetCodingSchemeDesignator() : String+ GetCodingSchemeVersion() : String+ GetLabel() : String+ IsIsPresent() : boolean

+ SetAnnotatorConfidence(Double) : void+ SetCodeMeaning(String) : void+ SetCodeValue(String) : void+ SetCodingSchemeDesignator(String) : void+ SetCodingSchemeVersion(String) : void+ SetIsPresent(boolean) : void+ SetLabel(String) : void

Dimension

- cagridId: Integer- index: Integer- label : String- size: Integer

+ GetIndex() : Integer+ GetLabel() : String+ GetSize() : Integer

+ SetIndex(Integer) : void+ SetLabel(String) : void+ SetSize(Integer) : void

CalculationData

- cagridId: Integer- value: Double

+ GetCoordinateCol lection() : Coordinate[]+ SetCoordinateCollection(Coordinate[]) : void

+ GetValue() : Double

+ SetValue(Double) : void

Coordinate

- cagridId: Integer- dimensionIndex: Integer- posi tion: Integer

+ GetDimensionIndex() : Integer+ GetPosition() : Integer

+ SetDimensionIndex(Integer) : void+ SetPosition(Integer) : void

SpatialCoordinate

- cagridId: Integer- coordinateIndex: Integer

+ GetCoordinateIndex() : Integer

+ SetCoordinateIndex(Integer) : void

TwoDimensionSpatialCoordinate

- imageReferenceUID: String- referencedFrameNumber: Integer- x: Double- y: Double

+ GetImageReferenceUID() : String+ GetReferencedFrameNumber() : Integer+ GetX() : Double+ GetY() : Double

+ SetImageReferenceUID(String) : void+ SetReferencedFrameNumber(Integer) : void+ SetX(Double) : void+ SetY(Double) : void

Equipment

- cagridId: Integer- manufacturerModelName: String [0..1]- manufacturerName: String- softwareVersion: String [0..1]

+ GetManufacturerModelName() : String+ GetManufacturerName() : String+ GetSoftwareVersion() : String

+ SetManufacturerModelName(String) : void+ SetManufacturerName(String) : void+ SetSoftwareVersion(String) : void

GeometricShape

- cagridId: Integer- includeFlag: boolean- lineColor: String [0..1]- lineOpacity: String [0..1]- lineStyle: String [0..1]- lineThickness: String [0..1]- shapeIdenti fier: Integer

+ GetSpatialCoordinateCol lection() : SpatialCoordinate[]+ SetSpatialCoordinateCollection(SpatialCoordinate[]) : void

+ GetLineColor() : String+ GetLineOpacity() : String+ GetLineStyle() : String+ GetLineThickness() : String+ GetShapeIdenti fier() : Integer+ IsIncludeFlag() : boolean

+ SetIncludeFlag(boolean) : void+ SetLineColor(String) : void+ SetLineOpacity(String) : void+ SetLineStyle(String) : void+ SetLineThickness(String) : void+ SetShapeIdenti fier(Integer) : void

CharacteristicQuantification

- annotatorConfidence: Double [0..1]- cagridId: Integer- name: String

+ GetAnnotatorConfidence() : Double+ GetName() : String

+ SetAnnotatorConfidence(Double) : void+ SetName(String) : void

Inference

- annotatorConfidence: Double [0..1]- cagridId: Integer- codeMeaning: String- codeValue: String- codingSchemeDesignator: String- codingSchemeVersion: String [0..1]- imageEvidence: boolean

+ GetAnnotatorConfidence() : Double+ GetCodeMeaning() : String+ GetCodeValue() : String+ GetCodingSchemeDesignator() : String+ GetCodingSchemeVersion() : String+ IsImageEvidence() : boolean

+ SetAnnotatorConfidence(Double) : void+ SetCodeMeaning(String) : void+ SetCodeValue(String) : void+ SetCodingSchemeDesignator(String) : void+ SetCodingSchemeVersion(String) : void+ SetImageEvidence(boolean) : void

Segmentation

- cagridId: Integer- referencedSopInstanceUID: String- segmentNumber: Integer- sopClassUID: String- sopInstanceUID: String

+ GetImagingObservation() : ImagingObservation+ SetImagingObservation(ImagingObservation) : void

+ GetReferencedSopInstanceUID() : String+ GetSegmentNumber() : Integer+ GetSopClassUID() : String+ GetSopInstanceUID() : String

+ SetReferencedSopInstanceUID(String) : void+ SetSegmentNumber(Integer) : void+ SetSopClassUID(String) : void+ SetSopInstanceUID(String) : void

AnatomicEntityCharacteristic

- annotatorConfidence: Double [0..1]- cagridId: Integer- codeMeaning: String- codeValue: String- codingSchemeDesignator: String- codingSchemeVersion: String [0..1]- label : String

+ GetCharacteristicQuantificationCollection() : CharacteristicQuanti fication []+ SetCharacteristicQuanti ficationCol lection(CharacteristicQuanti fication []) : void

+ GetAnnotatorConfidence() : Double+ GetCodeMeaning() : String+ GetCodeValue() : String+ GetCodingSchemeDesignator() : String+ GetCodingSchemeVersion() : String+ GetLabel() : String

+ SetAnnotatorConfidence(Double) : void+ SetCodeMeaning(String) : void+ SetCodeValue(String) : void+ SetCodingSchemeDesignator(String) : void+ SetCodingSchemeVersion(String) : void+ SetLabel(String) : void

ReferencedGeometricShape

- cagridId: Integer- referencedShapeIdentifier: Integer

+ GetReferencedShapeIdenti fier() : Integer

+ SetReferencedShapeIdentifier(Integer) : void

ThreeDimensionSpatialCoordinate

- frameOfReferenceUID: String- x: Double- y: Double- z: Double

+ GetFrameOfReferenceUID() : String+ GetX() : Double+ GetY() : Double+ GetZ() : Double

+ SetFrameOfReferenceUID(String) : void+ SetX(Double) : void+ SetY(Double) : void+ SetZ(Double) : void

Scale

- comment: String [0..1]- description: String [0..1]- value: String

+ GetComment() : String+ GetDescription() : String+ GetValue() : String

+ SetComment(String) : void+ SetDescription(String) : void+ SetValue(String) : void

Quantile

- bin: Integer

+ GetBin() : Integer

+ SetBin(Integer) : void

Numerical

- operator: ComparisonOperators [0..1]- ucumString: String- value: Double

+ GetOperator() : ComparisonOperators+ GetucumString() : String+ GetValue() : Double

+ SetOperator(ComparisonOperators) : void+ SetucumString(String) : void+ SetValue(Double) : void

Interv al

- maxOperator: ComparisonOperators- maxValue: Double- minOperator: ComparisonOperators- minValue: Double- ucumString: String

+ GetMaxOperator() : ComparisonOperators+ GetMaxValue() : Double+ GetMinOperator() : ComparisonOperators+ GetMinValue() : Double+ GetUcumString() : String

+ SetMaxOperator(ComparisonOperators) : void+ SetMaxValue(Double) : void+ SetMinOperator(ComparisonOperators) : void+ SetMinValue(Double) : void+ SetUcumString(String) : void

NonQuantifiable

- codeMeaning: String- codeValue: String- codingSchemeDesignator: String- codingSchemeVersion: String [0..1]

+ GetCodeMeaning() : String+ GetCodeValue() : String+ GetCodingSchemeDesignator() : String+ GetCodingSchemeVersion() : String

+ SetCodeMeaning(String) : void+ SetCodeValue(String) : void+ SetCodingSchemeDesignator(String) : void+ SetCodingSchemeVersion(String) : void

AimStatus

- annotationVersion: double- authorizedBy: String [0..1]- cagridId: Integer- codeMeaning: String- codeValue: String- codingSchemeDesignator: String- codingSchemeVersion: String [0..1]

+ GetAnnotationVersion() : double+ GetCodeMeaning() : String+ GetCodeValue() : String+ GetCodingSchemeDesignator() : String+ GetCodingSchemeVersion() : String

+ SetAnnotationVersion(double) : void+ SetCodeMeaning(String) : void+ SetCodeValue(String) : void+ SetCodingSchemeDesignator(String) : void+ SetCodingSchemeVersion(String) : void

PresentationState

- cagridId: Integer- sopInstanceUID: String

+ GetSopInstanceUID() : String

+ SetSopInstanceUID(String) : void

OversightAuthority

RegulatoryAssessment

+ + + + +

RegulatoryApplication

+ identifier: II+ typeCode: CD

RegulatoryApplicationSponsor

+

RegulatoryAuthority

+

+

ReviewableUnit

+ typeCode: CD

Submission

+

+ + +

SubmissionUnit

+ effectiveDateRange: IVL<TS.DATETIME>

+ receiptDate: TS.DATETIME+ typeCode: CD

Organization

+ + + + +

+

OversightCommittee

+

+

StudyOv ersightAuthority

Document

+ typeCode: CD

DocumentIdentifier

+ + +

StudyRegistry

+ +

ContrastMaterialCDERRegulatedItem

+ classCode: DSET<CD>+ codeModifiedText: ST+ expirationDate: TS.DATE.FULL+ lotNumberText: ST.SIMPLE+ pre1938Indicator: BL+ typeCode: CD::ContrastMaterial+ actual Indicator: BL+ code: CD+ description: ST+ effectiveDateRange:

IVL<TS.DATETIME>+ formCode: CD

DocumentVersion

+ bibliographicDesignation: ST+ date: TS.DATETIME+ keywordCode: DSET<CD>+ keywordText: DSET<ST>+ numberText: ST .SIMPLE+ officialTitle: ST+ revisionReason: ST+ text: ED+ /uniformResourceLocator: URL

ProductizedBiomarkerTest

Quanti tative Technology Informatics Platform (new in Q-Tip)

Common Sub-Domain (mostly from BRIDG)

Image Storage (from NBIA)

Regulatory Sub-Domain (extends from BRIDG)

Study Conduct Sub-Domain (extends from BRIDG)

Image Annotation and Markup (from AIM)

Legend

Biomarker

ContextForUse

IndicatedBiology

Pathology

Physiology

Interv ention

AssayMethod

SubjectPrep

ImageFormation

ImageAnalysis

ImageInterpretation

CDRHRegulatedItem

PerformanceEvidenceOfTest

GroupStatisticOfTests

+ new operation() : void

QualificationFullDataPackage

510(k)OrPMA

ReferenceDataSet

TestableHypothesis

StatisticalAnalysisPlan

BatchAnalysisScript

POFDSElements

-annotation

1

-aimStatus

0..1

isa

isa

-anatomicEntityCharacteristic

0..1

isSubjectOf

-characteristicQuantificationCollection

0..*

-anatomicEnti ty

1

-anatomicEnti tyCharacteristicCol lection

0..*

-calculation

1

-referencedGeometricShapeCollection

0..*

-imagingObservation

0..1

-referencedGeometricShape0..1

-calculationResult1

-calculationDataCol lection0..*-calculationData1

-coordinateCollection

1..*

isa

0..*

is a functionperformed by

{functions as}

1

-annotation

1

-user 0..1

-dicomImageReference

1

-presentationStateCollection

0..*

-annotation

1

-inferenceCollection

0..*

-annotation

1

-calculationCollection

0..*

-annotation1

-imagingObservationCollection0..*

-annotation 1

-equipment 0..1

-annotation1

-anatomicEnti tyCollection

0..*

isa

-geometricShape

1

-spatialCoordinateCol lection

1..*isa

-calculation1 -referencedCalculationCollection 0..*

+FK_TRIAL_DP_PK_ID 0..*

(TRIAL_DP_PK_ID = TRIAL_DP_PK_ID)

+PK_TRIAL_DP_PK_ID 1

isa

isa

+FK_PATIENT_PK_ID

0..*(PATIENT_PK_ID = PATIENT_PK_ID)+PK_PATIENT_PK_ID

1

+FK_TRIAL_SITE_PK_ID 0..*

(TRIAL_SITE_PK_ID = TRIAL_SITE_PK_ID)

+PK_TRIAL_SITE_PK_ID

1

+FK_TRIAL_PK_ID

0..*(TRIAL_PK_ID = TRIAL_PK_ID)+PK_TRIAL_PK_ID

1

+FK_IMAGE_PK_ID

0..*(IMAGE_PK_ID = IMAGE_PK_ID)+PK_IMAGE_PK_ID

1

+FK_GS_STUDY_PK_ID

0..*(STUDY_PK_ID = STUDY_PK_ID)

+PK_STUDY_PK_ID

1

+FK_G_SERIES_PK_ID

0..*

(GENERAL_SERIES_PK_ID = GENERAL_SERIES_PK_ID)+PK_G_SERIES_PK_ID

1

-imageAnnotation

1

isAssociatedWith

-person1

-imagingObservation 1

-imagingObservationCharacteristicCol lection0..*

-calculation1-calculationResultCollection0..*

isa

-imageAnnotation1

-imageReferenceCol lection

1..*

isa

-dicomImageReference 1

-imageStudy1

-imageAnnotation1

mayInclude

-segmentationCollection0..*

-segmentation1

maySupport

-imagingObservation 0..1

isa

isa

isa

-imageAnnotation1

-geometricShapeCollection0..*

isa

isa

-calculationResult1-dimensionCollection

1..*

1..*

is submitted by

{submits}

1

-imagingObservationCharacteristic

0..1

-characteristicQuantificationCollection

0..*

0..*

areDrawnFrom

1..*

0..*

clears/approves

1

1

summarizesPerformanceOf

1..*

1..*

indicatesUseOf1..*

supportsClearanceOrApprovalOf

isSpecificBy

isOrganizedAccordingTo

1..*

supports

1..*

1..*

accumulates

1

speci fies

1..* defineTermsFor

1

isa

isa

isa

1informs

1

0..*

providesSourceDataFor

0..*

isposedBy

1..*

is a version of

{has as aversion}

1

1..*is grouped by

{groups} 1

0..*

is grouped into

{groups}

0..1

1..*

is grouped into

{groups}

0..1

0..*is grouped into

{groups}

1

1..*

is evaluated in{evaluates}

0..1

0..*

is performed by

{performs}

1

0..*

is a functionperformed by

{functions as}0..1

0..1

is a functionperformed by

{functions as}

1

0..*is assignedby

{assigns}

0..1

speci fies

speci fies

0..*

includes

{is includedin}

1..*

0..*

has as subject

{is the subject for}

1

0..*

is assignedby

{assigns}

0..10..*

is managed by

{manages}

1

is the subject for

1..*

isDescribedInTermsOf

1..*

1..*

isMeasuredBy

1..*

speci fies

1

establ ishProofFor

1..*

0..*

identifies

{is identifiedby}

1

15

Execute is where analyses of Reference Data Sets take place. It is based on MIDAS and the associated Batchmake” capability but extends it for QI-Bench. The storage model is optimized for metadata storage and grid computing.

16

Execute First Reference Data Sets• Pilot 3A

– 156 lesions for evaluation (1A read 15)• Pivotal 3A

– 408 lesions for evaluation (1A read 40)• Study 1C

– 2364 lesions for evaluation (1C is set to read 66)• Study 1187

– 7122 lesions for evaluation • Available: RIDER, IDRI, MSKCC “1B”, …

17

Execute roadmap• Script to write “Image Formation” content into Biomarker Database for provenance

of Reference Data Sets: Application for pulling in data from the “Image Formation” schema to populate the biomarker database. This data will originate from the DICOM imagery imported into QI-Bench.

• Laboratory Protocol for the NBIA Connector and Batch Analysis Service: Laboratory protocol to describe the use of the NBIA Connector and the Image Formation script to import data into QIBench and use of the Batch Analysis Service for server-side processing.

• Support change analysis biomarkers serial studies (up to two time points in the current period, extensible to additional in subsequent development iterations): Support experiments including at minimum two time points. An example of this is the change in volume or SUV, rather than (only) estimation of the value at one time point.

• Document and solidify the API harness for execution modules of the Batch Analysis Service: This task will include the documentation and complete specification of the Batchmake Application API.

• Support scripted reader studies: Support reader studies through worklist items specified via AIM templates as well as Query/Retrieve via DICOM standards for interaction with reader stations. ClearCanvas will serve as the target reader station for the first implementation.

• Generate output from the LSTK module via AIM template (as opposed to hard-coded): Generate annotation and image markup output from reference algorithms (i.e., LSTK for volumetric CT and Slicer3D for SUV) based on AIM templates instead of the current hard-coded implementation. An AIM Template is an .xsd file.

• Re-work the NBIA Connector to run in the context of Formulate: This task will include refactoring and stabilization of the NBIA Connector in order to incorporate its functionality into Formulate.

18

AnalyzeCurrent Prototype Capabilities

19

Analyze MVT provides good framework, but with gaps• Lesion tracking• Other modalities and measures, e.g., SUV via FDG-PET

• Properly functioning multiple regression and N-way ANOVA• Support Clinical Performance assessment (i.e., in addition to current

Technical Performance)– Outcome studies– Integrated genomic/proteomic correlation studies– Group studies for biomarker qualification

• Serial studies / change analysis• Persistent database• Scale-up to handle thousands of cases (10’s thousands of lesions)

• Deploy as Web app20

Analyze Figures of Merit and Descriptive Statistics• Collaborative activity underway to converge definitive

descriptive statistics for technical and clinical performance• Approaches to defining and administering compliance in

relationship with QIBA profiles.Done 1. Process the lesion reads on the same 40 lesions used in the 1A pivotal as a 7th reader

using 1A STATA method and compare results. Next-up 3. Process the 6 selected lesions from the MVT demonstrator using

the 1A STATA method and compare results. [MVT<->1A STATA]4. Process the lesion reads on the same 6 lesions used in the MVT

demo set as a 7th reader and compare results. [QI-Bench<->MVT]5. Convert 1A STATA analysis to R and compare the results on the

408. [STATA<->R]6. Extend MVT to use the created R scripts (and fill other gaps).7. Re-do analyses to verify that results come out the same.

Perform and analyze other studies (e.g., 1C, 3A, 1187, other modes, etc. using STATA analysis method.

Mid-term Convert to R-based, MVT-implemented analysis.

As Funding Allows

Completion of tool box to include all of the descriptive statistics determined in the discussions / workshops.

21

PackageStructure submissions according to eCTD, HL7 RCRIM, and SDTMSection 2 Summaries

2.1. Biomarker Qualification Overview2.1.1. Introduction2.1.2. Context of Use2.1.3. Summary of Methodology and Results2.1.4. Conclusion2.2. Nonclinical Technical Methods Data2.2.1. Summary of Technical Validation Studies and Associated Analytical Methods2.2.2. Synopses of individual studies2.3. Clinical Biomarker Data2.3.1. Summary of Biomarker Efficacy Studies and Associated Analytical Methods2.3.2. Summary of Clinical Efficacy [one for each clinical context]2.3.3. Synopses of individual studies

Section 3 Quality<used when individual sponsor qualifies marker in context of a specific NDA>

Section 4 Nonclinical Reports4.1. Study reports4.1.1. Technical Methods Development Reports4.1.2. Technical Methods Validation Reports4.1.3. Nonclinical Study Reports (in vivo)4.2. Literature references

Section 5 Clinical Reports5.1. Tabular listing of all clinical studies5.2. Clinical study reports and related information5.2.1. Technical Methods Development reports5.2.2. Technical Methods Validation reports5.2.3. Clinical Efficacy Study Reports [context for use]5.3. Literature references

22

PackageStandards Mapping

Domain SDTMVariable Name

Variable

Label

Definition

ACRIN DICOM NBIA Data

Element

BRIDG Datatype

Controlled

Terms, Codelist

or Format

Role Implementation Notes

Core

DM BRTHDTC

Patients Date of Birth

Date/time of birth of subject

(0010,0030)

Patient's Date of Birth

BiologicEntity. birthDate*

CHAR* NA Record Qualifier

Permissible

ISO8601 & 21090

Point to NCIt, RadLex,

etc. hereCDASH/SDTM

Variables

ACRINReference

DICOMTag SDTM

Role

NBIA DE

23

PackageNCI – CPATH – CDISC CRF WG

24

PackageWeb-enabled service for compiling results

25

Open source model• www.qi-bench.org domain• BSD license• Extending rather than forking assets

– Engaging with CBIIT OSDI program• QI-Bench specific assets in publicly accessible

repositories and full access to development tools through www.qi-bench.org

• Project wiki at www.qi-bench.org/wiki

2626

Development Lifecycle Process for Centrally Developed Portions

27

High-level relationship among development processes (modeled after corresponding process flow at caBIG to allow effective integration)

27

QI-Bench Developer’s Resources

282828

2929

Year OutlookTime frame ContentOctober 2011 • 3A Pilot data curation

• Specify feasibility established with proof of concept using BioPortal and QIBO• Data center upgrades (now including 24 Processor cores, 40 GB Main Memory, 12 TB Disk storage,

Linux, Windows Server, and Mac OS X)

November 2011 • QI-Bench developer environment (Git, Jira, dashboards)• Execute upgrade• 3A Challenge Launch

December 2011 • “Image Formation”: scripts and spreadsheet for provenance of Reference Data Sets • Laboratory Protocol (Formulate, Execute, Analyze)

January 2012 • Support change analysis / serial studies (up to two time points, extensible to additional in subsequent development iterations) (Formulate, Execute, Analyze)

February 2012 • Specify working model (including QIB Ontology and AIM template builder 18 or 23 as possible)• Document and solidify the API harness for execution modules of the Batch Analysis Service

(Execute)

March 2012 • Drive reading stations by making worklist items from within Batchmake scripts (Formulate, Execute)

April 2012 • Support DICOM Query/Retrieve from Reference Data Set Manager (to complete the Reader Studies support package) (Execute, Analyze)

May 2012 • Extend harness API for AIM templates (including generate output from the LSTK module via AIM template) (Execute)

Summer 2012 • Formulate working model (including re-worked NBIA Connector and other content as above)

3030

Summary:QI-Bench Contributions• We make it practical to increase the magnitude of data for increased

statistical significance. • We provide practical means to grapple with massive data sets.• We address the problem of efficient use of resources to assess limits of

generalizability. • We make formal specification accessible to diverse groups of experts that are

not skilled or interested in knowledge engineering. • We map both medical as well as technical domain expertise into

representations well suited to emerging capabilities of the semantic web. • We enable a mechanism to assess compliance with standards or

requirements within specific contexts for use.• We take a “toolbox” approach to statistical analysis. • We provide the capability in a manner which is accessible to varying levels of

collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access.

3131

3232