AI in Digital Pathology

** Note to instructor: most of these slides are comprised of custom figures I created; the figures contain a minimal amount of labelling. I fully understand pechakuchashould have only one phrase per slide, but these labels are here to replace my “pointing at slide” and similar body language / gestures, as I cannot present this physically. My original plane was to omit them completely. I also put citations and credits at the bottom of the speaker notes to hopefully it easier to mark. Thanks for understanding! **

On with the presentation:

• Today we will be taking a look at a key piece that can bring us one step closer tosolving the cancer puzzle.

• Modern pathology is:• the gold standard for cancer diagnosis [1]• reaching new heights due to [2]:

• slide digitization• artificial intelligence

1

[1] Rodriguez-Canales, J., Eberle, F. C., Jaffe, E. S., & Emmert-Buck, M. R. (2011). Whyis it crucial to reintegrate pathology into cancer research?. BioEssays : news andreviews in molecular, cellular and developmental biology, 33(7), 490–498.https://doi.org/10.1002/bies.201100017

[2] Litjens, Geert et al. (2017). A survey on deep learning in medical image analysis.Medical Image Analysis, 2017(42), 60-88.https://doi.org/10.1016/j.media.2017.07.005

Image credits: John Hopkins Medicine

1

Gynecologicuterine, cervical, ovarian

Skin, melanoma

Genitourinarybladder, kidney, prostate, testicular

Bloodleukemia, lymphoma, myeloma

Breast

Endocrine

Head + neckmouth, throat, nasalBrain

Lung

Liver, pancreatic

Gastrointestinalcolorectal, stomach, anal

Boneosteosarcoma, soft tissue

Anatomy

• Cancer can occur in many locations in the human body• different organs• different tissues

• The most common cancers are [3]:1. Lung2. Breast3. Colorectal4. Prostate5. Skin (non-melanoma)6. Stomach

[3] World Health Organization. (2018, September 12). Cancer. https://www.who.int/news-room/fact-sheets/detail/cancer

Image credits: One Walk

2

PhysiologyMitosis

Growth Phase 1Growth

Phase 2

DNA Synthesis

Quiescent Stage

Healthy Cell Cycle

• At the bottom level, cancer is caused by malfunction in the cell cycle• Regular cells

• need growth factors to propagate through the cycle and multiply• stop multiplying when crowded• are programmed to die off – apoptosis

• These factors are part of the control system of the body

[4] Lodish H et al. (2000). Molecular Cell Biology 4th Edition. W. H. Freeman.

3

PathologyGenetically altered cell

Uncontrolled proliferation

Benign tumor

Malignant / invasive tumor

Metastasis

Blood flow

• Cancer is essentially uncontrolled cell proliferation [5]• Cancer can be benign or malignant based on whether it spreads and invades

surrounding tissues [5]• The metastasis mechanism allows malignant cells to spread to other areas of the

body

[5] Cooper GM. (2000). The Cell: A Molecular Approach 2nd Edition. Sunderland.

4

Procedure (diagnosis)

Biopsy Staining & slicing

Initial testing

Tissue sample Tissue slides

Pathology diagnosis

Treatment options

• Following physical, lab, and / or imaging tests, suspected cancer tissue is extracted in a biopsy [6]

• The stained tissue is examined under a microscope• a diagnosis and treatment recommendation is made based on qualitative

and quantitative visual criterion [6]

[6] Mayo Clinic. (n.d.) Cancer – diagnosis & treatment. https://www.mayoclinic.org/diseases-conditions/cancer/diagnosis-treatment/drc-20370594

5

Histology

Benign breast tumor Malignant breast tumor Malignant brain tumor

Normal colon

Benign colon

• This is what the standard (hematoxylin & eosin stain) stained tissue slide looks like under a microscope

• diagnosis is made on [7]:• cell appearance• tissue appearance• patterns such as gland formation

• these images are examples of histological scans

[7] The Royal College of Pathologists. (n.d.) Histopathology. https://www.rcpath.org/discover-pathology/news/fact-sheets/histopathology.html

Image credit: University of California Davis Medical Center, Wikipedia Commons

6

Whole Slide Imaging (WSI)

early-1990s

Cases

Clinical outcomesdiagnosis, prognosis, etc.

Feedback

WSI scanner

Training Data

Labeldiagnosis, prognosis, etc.

AIFeedback

• One of the main drivers behind digital pathology’s rise is whole slide imaging [1][8]• All of the benefits of WSI directly contribute to the feasibility of AI in digital

pathology• availibility of training data

• Observe that both pathologists and AI essentially learn with experience and feedback

[8] Pantanowitz, Liron et al. (2018). Twenty years of digital pathology: an overview of the road travelled, what is on the horizon, and the emergence of vendor-neutral archives. Journal of Pathology Informatics, 9(40), 60-88. https://doi.org/10.4103/jpi.jpi_69_18

Image credit: Leica Biosystems, IEEE Conference on Computer Vision and Pattern Recognition

7

Machine Learning

late-1990s

Training Data

Engineered Feature Vector

cell size

nucleus size

cellularity

cell deaths

# mitotic fig.

inflam. Index

…Image Processing Algorithms

0 = benign

1 = malignant

Neural Network

Output

�⃗� = 𝜎(𝑾 �⃗� + 𝑏)

• Classical machine learning takes a vector of feature values extracted from the data as input [1]

• which is propagated through a series of “layers”,• each of which comprise of weighted multiplication, addition of bias,

and then a non-linear activation function (e.g. arctan) [1]• the training process uses feedback to tune the weights and biases

Image credit: IEEE Conference on Computer Vision and Pattern Recognition

8

Deep Learning

2000s

Training Data

0 = benign

1 = malignant

Neural Network

Output

�⃗� = 𝜎(𝑾 �⃗� + 𝑏)

Learned Feature Extractor

Pixels Edges Shapes

• Instead of an engineered feature vector, deep learning uses more layers to perform feature extraction itself [1]

• initial layers pick up simple patterns, deeper layers learn more complex representations

• Harder to train than machine learning, but more accurate [1]

Image credit: IEEE Conference on Computer Vision and Pattern Recognition

9

Convolutional Neural Nets

𝑥 𝑥

𝑥 𝑘 𝑘

𝑘𝑦

𝑥 𝑥

𝑥 𝑥

𝑦

3 x 3 Convolutional Layer 2 x 2 Max-Pooling Layer

𝑦 , = 𝑘 , ∗ 𝑥 , = (𝑘 , )(𝑥 , )

,

𝑦 , = max,(𝑥 , )

• Notice in our neural network that just 7 pixels required a large number of weights• Convolutional neural networks use kernel convolution to share weights across the

entire image [1]• since patterns in images are space invariant

• Max pooling is used to further minimize the number of weights by reducing dimensions [1]

Image credit: IEEE Conference on Computer Vision and Pattern Recognition

10

Convolutional Neural Nets Cont’d

Output

0 = benign

1 = malignant

Slide-level classification

Sliding-window mask

Tiling

• Typical CNNs have a few convolution and max pooling layers that are followed by fully connected layers (similar to our earlier neural networks) [1]

• We can perform a variety of useful digital pathology tasks with this simple classifier:

• slide-level classification• create a mask of benign versus malignant areas

Image credit: IEEE Conference on Computer Vision and Pattern Recognition

11

Recurrent CNN

Output

Stream of input data

• One problem with CNNs are their limited “field of view”• Recurrent CNNs are basically CNN state machines, or “CNNs with memory”

• we can increase the contextual sensitivity of our network by showing a recurrent network a slideshow of tiles, so it can process more than one tile at a time

Image credit: IEEE Conference on Computer Vision and Pattern Recognition

12

U-NetOutputInput

• For segmentation, we can do better than just a sliding window classifier• The U-net converts the image into a segmentation map of the same resolution [1]

• has bridging connections so that general shapes are preserved for reconstruction

Image credit: Towards Data Science

13

Encoder-Decoder NetworkLatent SpaceInput Output

• An encoder-decoder network is like a mirrored pair of CNNs that compresses images into a lower-dimensioned latent space and then decompresses [1]

• This is a form of unsupervised learning, in which we tell the network to have the same output as input, but let it discover its own way of compressing [1]

Image credit: Towards Data Science

14

Encoder NetworkLatent SpaceInput

Neural Network

0 = benign

1 = malignant

Output

• One useful way to use an encoder-decoder network is to throw away the decoder portion and use the latent space as a feature vector for other networks [1]

• This is essentially a learned feature extractor• but now we don’t have to train the extractor and network at the same

time, so the network converges faster

Image credit: Towards Data Science

15

Encoder-Decoder NetworkLatent Space OutputNoise

• Another useful way of using an encoder-decoder network is to throw away the encoder, and use the decoder as a generative network [1]

• Since the latent space is made up of parameters describing an image, we can feed noise into it to generate random images from scratch

• this is very useful for creating “fake” data to augment a training set

Image credit: Towards Data Science

16

Generative Adversarial Network

Noise

Generator Discriminator

Fake Image

Real Image

0 = fake

1 = real

Output

• Generative adversarial networks are also like back-to-back CNNs, but flipped [1]• the Generator is like a counterfeiter trying to trick the discriminator into

thinking its generated “fake” images are real• the Discriminator is constantly training to get better at classifying real

versus fake• This model is very popular for training set augmentation

Image credit: Towards Data Science

17

In the Real World…

• AI in digital pathology is already making waves• Large scale challenges such as lymph node metastasis detection challenge

CAMELYON 17 have yielded algorithms that outperform a panel of pathologists [1]

• FDA approvals of AI applications in pathology and beyond are increasing

Image credit: CAMELYON 17, Business Wire

18

Future Uses

Rural Medicine

More economical

More versatile

More accessible

• Here is one of my outlooks on where AI in digital pathology could be headed:• we have the potential to replace a lot of special (and expensive) pathology

tests with a software program on a USB stick• economical• accessible• versatile• thus opens up a lot of possible applications in rural medicine where

supplies are scarce

19

Summary

• In summary, we discussed the emergence of AI in digital pathology• what is cancer and pathology / histology?• the history• some AI theory• some AI techniques• applications and real world impacts

• Thank you all for your attention

20