advancements and trends in medical image analysis … · university of british columbia, canada,...

ADVANCEMENTS AND TRENDS IN MEDICAL IMAGE ANALYSIS USING DEEP LEARNING

A presentation by Shekoofeh Azizi

University of British Columbia, Canada, Ph.D. 2014-2018 Electrical and Computer Engineering Isfahan University of Technology, Iran, M.Sc. 2011-2013 Computer Engineering / Hardware Design Isfahan University of Technology, Iran, B.Sc. 2007-2011 Computer Engineering / Hardware Engineering

2

Philips Research North America, 2015-now National Institutes of Health (NIH), 2015-now

MICCAI Student Board Officer, 2016-now Women in MICCAI, 2017-now

3

Size is related to the number of collaborators

Academic Collaborators

Industrial/Clinical Collaborators

UBC SFU VGH

Philips NIH

Queen’s University Univ. of Western Ontario Robarts Research

Technical Univ. of Munich ETH Zurich

NVidia IBM

Sejong Univ., Korea

Univ. of Colorado

OUTLINE

4

Deep Learning Medical Imaging Challenges and Opportunities

Vision

DEEP LEARNING

5

ARTIFICIAL INTELLIGENCE VS. DATA SCIENCE

6

AI is a technique which enables machines to mimic the “human behaviour”

Hey Google, Weather in Victoria

It’s 17°C and sunny in

Victoria!

User Google Home Google Home

Voice Services

Voice to Command Voice Output Brain/Model

Artificial Intelligence (AI)

ARTIFICIAL INTELLIGENCE VS. DATA SCIENCE

7

Machine Learning (ML)

Deep Learning (DL)

Data Science

Data Science is about processes and systems to extract knowledge or insights from data in various forms. Machine Learning is the connection between data science and artificial intelligence since machine learning is the process of learning from data over time.

MACHINE LEARNING (ML)

9

Reinforcement Learning

Andrew Ng, Machine Learning Course, Coursera.

Machine learning is the process of learning from data over time.

DEEP LEARNING (DL)

Inspired by the functionality of our brains

10

Square ?

Number of sides: 4?

Closed form shape?

Perpendicular sides?

Equal sides?

DEEP LEARNING (DL)

11

Feature Learning

Cat

Dog

DEEP LEARNING (DL)

12

WHY DEEP LEARNING?

13

Consistent improvement over the state-of-the-art across a large variety of domains.

Over 14 million images and 20 thousand categories.

WHY DEEP LEARNING?

14

Drive.ai’s self-driving car handle California city streets on a rainy night.

Tensorflow Object Detection API, 2015.

HEALTHCARE AND MEDICAL IMAGE ANALYSIS

15

THE ROLE OF IMAGING IN HEALTHCARE

16

Diagnosis Quantification Planning Monitoring Intervention

Slide Credit: NVidia

OPPORTUNITIES FOR AI IN RADIOLOGY

17

Reconstruction

AI for Image Reconstruction from Sensors

Analysis

ML/DL for Medical Image Analysis

Big Data

Pattern Recognition

Medical Report

Natural Language Processing

Machine learning software will serve as a very experienced clinical assistant, augmenting the doctor and making workflow more efficient and accurate.

APPLICATION OF AI IN MEDICAL IMAGE ANALYSIS

18

3D Ultrasound Volumetric Segmentation Project Clara, NVidia

Brain MRI Segmentation

• Assign each pixel of the image to a class

• In computer vision: need to generalize to different scenes, lightning, pose, corner-cases.

• In medical imaging: need to be precise at pixel level, account for variations in scan quality, artifacts, anatomy.

• Images vs. Volumes

Segmentation:

Razzak, Muhammad Imran, "Deep Learning for Medical Image Processing: Overview, Challenges and the Future." In Classification in BioApps, pp. 323-350. Springer, Cham, 2018.

Ker, Justin, et al. "Deep learning applications in medical image analysis." IEEE Access 6 (2018): 9375-9389.

APPLICATION OF AI IN MEDICAL IMAGE ANALYSIS

19

• Predicting Lung Cancer Using CT Scan

• Red is showing Cancer Region

• Accuracy 0.86

• Kaggle Competition (1 million)

© http://blog.kaggle.com/2017/06/29/2017-data-science-bowl-predicting-lung-cancer-2nd-place-solution-write-up-daniel-hammack-and-julian-de-wit/

Detection/Classification:

CHALLENGES

• Requires extensive inter-organization collaboration

• Data annotation:

– Noise and sparse labeling

– Tedious and expensive

– Rare disease

• Data variability

• Interpretability of the decision making model and acceptance by health profession

20

Medical Doctors

Medical Physicists

Computer Scientist

TEMPORAL ENHANCED ULTRASOUND

Prostate Cancer Diagnosis Using

21

Example of Medical Image Analysis Using Deep Learning:

TEMPORAL ENHANCED ULTRASOUN (TeUS)

22

[Moradi’07, Moradi’09, Imani’15, Khojaste’15, Ghavidel’16 ]

S. Azizi, et al., “Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks and tissue mimicking simulations,” Journal of Computer Assisted Radiology and Surgery (IJCARS): MICCAI’16 special issues, 2017.

TEMPORAL ENHANCED ULTRASOUN (TeUS)

23

Cancer

Benign

Feature Learning

Classification

[Moradi’07, Moradi’09, Imani’15, Khojaste’15, Ghavidel’16 ]

PROSTATE CANCER GRADING USING TeUS CHALLENGES

24

GS 3+3 GS 4+4 GS 4+3 Benign GS 3+4

...

Clinically significant

Clinically less significant

© Correas et al. 2013, Iczkowski et al. 2011.


PROSTATE CANCER GRADING USING TeUS CHALLENGES

25

ROIs with unknown pathology: other tissue types.

Exact location of the cancer: unknown.

Exact location of the Gleason patterns: unknown.

Benign or other tissue types?

?

?

GS 3+3 GS 4+4 GS 4+3 Benign GS 3+4

...

Clinically significant

Clinically less significant

© Correas et al. 2013, Iczkowski et al. 2011.


FEATURE LEARNING + DISTRIBUTION LEARNING

26

GS 4+4 Benign GS 3+3 GS 4+3 GS 3+4

Feature 1

Feat

ure

2

Feature 1

Feat

ure

2

Feature 1

Feat

ure

2

Feature 1

Feat

ure

2

Feature 1

Feat

ure

2

Feature 1

Feat

ure

2

Cluster of Gleason Pattern 3

Other Tissue Type


Benign Cluster

Feature Space


FEATURE LEARNING + DISTRIBUTION LEARNING

27

Training Dataset

20 ROIs

Target

Deep Belief Network (DBN)

Visible Layer Hidden Layers

Feature Space

GS3 GS4

Distribution Learning (F1,F2)

Clustering Model

Trained Deep Network

Clustering

Model

Test Data ?

?

?

?

?

? ?

?

?

?

?

?

?

?

??

?

?

?

?

Feature 1

Feat

ure

2



Benign Cluster


Beamformer

Back-end Signal Processing

Scan Conversion

B-m

od

e Im

age

Be

amfo

rme

d

RF D

ata

Radio Frequency (RF) : − Richer source of information than B-mode. − Not accessible on commercial scanners.

DATA VARIABILITY: RF VS. B-MODE

28 S. Azizi, et al., “Transfer learning from RF to B-mode temporal enhanced ultrasound features for prostate cancer detection,” Journal of Computer Assisted Radiology and Surgery: IPCAI’17 special issues, 2017.

S. Azizi, et al., “Investigating deep recurrent neural networks for prostate cancer detection: analysis of temporal enhanced ultrasound,” IEEE Transaction on Medical Imaging (TMI), 2018.

TRANSFER LEARNING: RF VS. B-MODE TeUS

29

Unlabeled B-mode TeUS Data RF TeUS Data

. . .

. . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

RF-mimicking TeUS or

𝐁𝐦𝐨𝐝𝐞 TeUS

B-mode TeUS

Preprocessing and Feature Extraction

RF TeUS

Objective function: Reconstruction error + KL divergence + Loss function

𝐃𝐊𝐋 𝐑𝐅, 𝐁𝐦𝐨𝐝𝐞 = − 𝐑𝐅 𝐢 𝐥𝐨𝐠𝐁𝐦𝐨𝐝𝐞(𝐢)𝐢

TRANSFER LEARNING: RF VS. B-MODE TeUS

30

Unlabeled B-mode TeUS Data RF TeUS Data

. . .

. . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

RF-mimicking TeUS

B-mode TeUS


RF TeUS

Labeled B-mode TeUS Data


Transfer learning network

Joint Classification Network

Benign vs. Cancer

RF-mimicking TeUS


TRANSFER LEARNING: RF VS. B-MODE TeUS DECISION MAKING MODEL

31

Performance: Area under ROC Curve = 0.96 Run-time = 1.66 ± 0.32 second for 100 frames


S. Azizi, et al., “Transfer learning from RF to B-mode temporal enhanced ultrasound features for prostate cancer detection,” Journal of Computer Assisted Radiology and Surgery: IPCAI’17 special issues, 2017.

MODEL INTERPRETATION

32

Benign Gleason Pattern 3 Gleason Pattern 4

Layer 1: 100 hidden neurons



Trai

ne

d D

BN

Bac

k P

rop

agat

ion

Absolute Difference

Low-frequency components

Visible Layer 50 spectral features


33

Medical Doctors Medical Physicists

Computer Scientist


34

Cancer Benign

Tissue response = f (Acoustic signal, Tissue microstructure,…)

Cell Nuclei (Scatterers)

Speckle Cell Nuclei

Speckle

Hunt, J. W., et. al. “The subtleties of ultrasound images of an ensemble of cells: simulation from regular and more random distributions of scatterers.” Ultrasound in medicine & biology, 21(3), 329-341, 1995.

Feature Extraction

Finite Element

Simulations

Nuclei Location Extraction

Digital Pathology

Temporal Ultrasound Generation

Ultrasound Simulations

(Field II)

Time


35

- K. Iczkowski, et al., "Digital quantification of five high-grade PCa patterns, including the cribri-form pattern, and their association with adverse outcome", American Journal of Clinical Pathology (2011). (University of Colorado)

- S. Bayat, et al., “Tissue mimicking simulations for temporal enhanced US-based tissue typing”, SPIE 2017.

Cancer Normal

HOW WILL AI IMPACT THE HEALTHCARE LANDSCAPE? 36

DEEP LEARNING MOMENTUM BUILDING

37

Medical Imaging Papers Using DL

Slide Credit: NVidia, DL for Health Informatics - Daniele Ravi, et. al., IEEE Journal of Biomedical and Health Informatics, Vol. 21, No. 1, January 2017

AI Across Healthcare Academic Pubs.

CAMBRIAN EXPLOSION

38 Slide Credit: NVidia

UNCERTAINTY VS. ACCURACY

• Uncertainty

• Is it hard to say I don’t know?

• Human level accuracy

• Noise

39 Who Said What: Modeling Individual Labelers Improves Classification, Guan et al., AAAI (Google Brain)

Gal, Yarin, and Zoubin Ghahramani. "Dropout as a Bayesian approximation: Representing model uncertainty in deep learning." international conference on machine learning. 2016.


40

• Uncertainty



• Noise

Who Said What: Modeling Individual Labelers Improves Classification, Guan et al., AAAI (Google Brain)

Gal, Yarin, and Zoubin Ghahramani. "Dropout as a Bayesian approximation: Representing model uncertainty in deep learning." international conference on machine learning. 2016.


41

Video of the first self-driving car crash that killed a pedestrian in the US shows how the autonomous Uber failed to slow down before it hit a 49-year-old woman walking her bike across the street. It has raised fresh questions about why the vehicle did not stop when a human entered its path.

• Uncertainty



• Noise

Greenspan, Hayit,, et al, "Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique." IEEE Transactions on Medical Imaging 35.5 (2016): 1153-1159.

Ker, Justin, et al. "Deep learning applications in medical image analysis." IEEE Access 6 (2018): 9375-9389.

INTERPRETABILITY

• How well can we get along with machines that are unpredictable?

• A patient who is being told that he/she must undergo chemotherapy is unlikely to accept the answer, “The machine learning algorithm said so, based on previous case data and your current condition.”

42

RESEARCH VISION

43

CIHR CHRP: Artificial Intelligence, Health and

Society

CIFAR for AI

Collaborative Research and Development (CRD)

Grants

• Effective measurement of uncertainty, discovering the source of it and integrating proper solutions in deep learning-based decision making models.

uncertainty Interpretability ?

CONCLUSION

44

DEEP LEARN NG Accuracy

Uncertainty

Interpretability

What I learned from AI in Medical Image Analysis:

Between Hopes and Fears

THANK YOU! QUESTIONS?

advancements and trends in medical image analysis … · university of british columbia, canada,...

Documents