prognostic and diagnostic evaluation of histopathologic ...€¦ · current method for assessing...
TRANSCRIPT
![Page 1: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/1.jpg)
Prognostic and Diagnostic Evaluation of HistopathologicImages by Multispectral Imaging
Michael Feldman, MD, [email protected]
Assistant Professor PathologyUniversity Pennsylvania
“Cytomics on Slide”
Or
Slow Cytometry…
![Page 2: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/2.jpg)
Current method for assessing tumor cell signaling or other proteins by IHC- Immunostain for proteins- Pathologist-dependent, subjective (0-3+)
- “quantified” by stain area, intensity and estimated percent- Very subjective and not reproducible
- Proteins assessed one at a time
Assessment of inhibition of tumor cell signaling- Similar problems- Need objective quantification of changes in cell signaling
The Need
![Page 3: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/3.jpg)
Platform for assessing activity of TC signaling pathways
Immunohistologic staining of pathway componentsPathways analyzed on a cellular basis (“cytometry”)Results quantitated computationally (“automated”)Multiple pathways analyzed together (“multiparameter”)
Components1. Multiparameter immunostaining2. Multispectral image capture3. Image processing to resolve individual stains based on spectra4. Tumor segmentation5. Computational identification of nuclei and cells in image6. Computational assignment of immunostains to each nucleus or cell7. Data display and Analysis
![Page 4: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/4.jpg)
Stain A
Stain B
Stain C
1. Multiparameter immunostaining
A/C
A/B A/B
Paraffin section
Current State
3 Colors, no more than 2 colors in same spatial compartment
In Development
3 Colors in same spatial compartment
Future
High order multiplexing with nanotechnology
![Page 5: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/5.jpg)
2. Multispectral Hardware
Dispersive ElementsPrism – PARISS
http://www.lightforminc.com/
Dispersive
Tunable Liquid FiltersCRI -
www.cri-inc.com
![Page 6: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/6.jpg)
Light has no color. “Color” is an observer interpretation.These two yellows appear identical to the human eye, yetthey have very different spectral components.
Color and Spectra
![Page 7: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/7.jpg)
MSI vs. Traditional Color Imaging
Inte
nsity
Spectral
400 500 600 700
Spectroscopy captures the entire spectrum (light intensity as a function of wavelength).
Ired
= 253Igreen
= 203Iblue
= 112
RGB
But with RGB-based instrumentation, this complex spectrum will be described using only 3 values (bins) averaged over large spectral regions.
Inte
nsity
![Page 8: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/8.jpg)
Multispectral Imaging
450 nm
500 nm
550 nm
600 nm
650 nm
CCD
MelanomaTake images at different wavelengths using a CCD
Assemble the data into a “cube” in computer memory
This creates an optical spectrum at every pixel of the image
550 600 650
I
450 500
Spectra
Wavelength (nm)
![Page 9: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/9.jpg)
“extract” spectra of individual stains
teach computer thespectral profile ofchromogen/fluorophoreA, B, C
A B C
3. Image processing to resolve individual stains based on spectra
![Page 10: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/10.jpg)
4. Tumor Segmentation
Manual Segmentation Machine Segmentation
![Page 11: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/11.jpg)
CAD – Computer Assisted Dx
Can we develop software to identify Prostate Cancer in whole digitized slides?Utility
Disease level Segmentation for IHC/IF, CAD for Rescreening or Primary Screening
![Page 12: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/12.jpg)
Introduction
Most of the image is benignCan we reduce image size by excluding benign areasCombine this with scale information to greatly increase the efficiency of the detection procedure
![Page 13: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/13.jpg)
Introduction
Each “pass” of the image rejects pixels, and then only the positive pixels are analyzed at higher scalesThis allows us to efficiently analyze the image at higher scales by only looking at “interesting” pixelsIncreasing accuracy does NOT increase execution time
![Page 14: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/14.jpg)
Introduction
Viola and Jones [1] introduced the “attention cascade”By quickly rejecting image regions that are definitely negative using a minimum of features, we can concentrate computation on regions that might be positive
![Page 15: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/15.jpg)
Methods
![Page 16: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/16.jpg)
Methods
To describe each pixel, we extract ~ 600 image features from the imageThere are three categories of features:
Statistical FeaturesCo-occurrence (texture) FeaturesGabor Filter Features
Each of these is extracted from the three channels of the image, at three different window sizes
![Page 17: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/17.jpg)
Methods
Probability density functions are constructed using the ground truth pixels provided by a pathologistThese PDFs are used to assign a likelihood value to each pixel based on the feature value
Standard Deviation
![Page 18: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/18.jpg)
Methods
The individual feature likelihoods are combined by AdaBoost [3] to get a likelihood ensembleA small number of features are used at scale 0 to obtain the ensemble, which is then thresholdedThe process begins over again at the next scale; only pixels labeled as positive at the previous scale are analyzed
![Page 19: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/19.jpg)
Results
Graph of the computation time used to visualize efficiency in using the cascadeVisual comparison of the likelihood ensembles with the tumor maskReceiver operating characteristic (ROC) curves were used to measure accuracyScatter plots visualize the separation of classes
![Page 20: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/20.jpg)
Results
170180
190
200210
220
20
40
60
80
100
120
20
40
60
80
100
120
140
160
180
120140
160180
200220
240
0
20
40
60
80
100
120
0
50
100
150
200
250
Separation at Scale 0
0
50
100
150
200
20
40
60
80
100
120
140
60
80
100
120
140
160
180
200
220
Separation at Scale 1 Separation at Scale 2
![Page 21: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/21.jpg)
Higher scale analyses
![Page 22: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/22.jpg)
Higher scale
![Page 23: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/23.jpg)
CAD ConclusionsUsing the Hierarchical cascade allows for fast, accurate analysis of large biomedical imagesSuch a methodology could be employed in a number of scale-sensitive image analysis systemsBoth the cascade and the analysis itself can be applied to various pathologies and imaging modalitiesThe separation of classes as the scales increase indicates that more discriminatory information is available at higher scalesHigher scales studies demonstrate the capacity to distinguish stroma vs benign glands from Gleason grade 3 and grade 4 carcinoma
![Page 24: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/24.jpg)
5. Computational identification of nuclei in image using Farsight software
![Page 25: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/25.jpg)
![Page 26: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/26.jpg)
breast tumor stainedfor p-ERK (DAB) & hematoxylin
p-ERK- stromal cells (grey)
segmentation of nuclei(based on hematoxylin)
p-ERK+ tumor cells (yellow)
6. Computational assignment of immunostains to each nucleus
![Page 27: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/27.jpg)
7. Data display & Analysis:Frequency histogram of intensity of p-ERK staining of stromal and tumor cell nuclei in a breast tumor
tumor cell nuclei
stromal cell nuclei
![Page 28: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/28.jpg)
400X 400X
Pre-therapy On-therapy (1 week)
Pharmacodynamics:p-ERK immunostaining in thyroid carcinoma treated with sorafenib
![Page 29: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/29.jpg)
Frequency histogram of intensity of p-ERK staining of thyroid carcinoma nuclei
on-therapypre-therapy
![Page 30: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/30.jpg)
Hematoxylin Ki-67p-ERK
Two-parameter analysis: Germinal center stained p-ERK (SG blue), Ki-67 (VIP) and hematoxylin
A pathologist would find it very hard to classify nuclei as positive or negative from RGB image
![Page 31: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/31.jpg)
0
20
40
60
80
100
120
140
0 20 40 60 80 100 120 140 160 180
p-erkKi
67
2-antigen scatter plot (from 1 image)
Nuclear segmentationp-ERK+/Ki-67- (yellow)p-ERK-/Ki-67+ (blue)p-ERK+/Ki-67+ (magenta)
Analysis of antigen association (apparently present) in cells in tissues
![Page 32: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/32.jpg)
Triple stain for breast (ER/PR/Her2)
PR
ER
![Page 33: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/33.jpg)
Breast Carcinoma
Lapatinib – Dual inhibitor EGFR and Her2Who responds?How to select?
Co-expression of ErbB2 and IGFR Poor response to Trastuzumab (Herceptin)Favorable response to LapatinibLow PTEN and ErbB2 poor response to Trastuzumab(Herceptin)
Key is going to be use of multiple biomarkersExtend segmentation beyond nuclei to include membrane
![Page 34: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/34.jpg)
MSI Applications
ImmunohistochemistryRoutine testsNew tests
Immunoflourescence – Clinical TrialsBackground Fluorescence
![Page 35: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/35.jpg)
Lymphovascular Invasion Oral SCC
Cytokeratin – DAB (brown)
CD34 – Vector SG (blue)
Podoplanin – Vector VIP (purple)
![Page 36: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/36.jpg)
Prostate Cancer – Triple stain
![Page 37: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/37.jpg)
SLN Melanoma
![Page 38: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/38.jpg)
Autoflourescence
Original Spectrally separated
![Page 39: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/39.jpg)
Future directions
Develop automated analysis packagePathway analysis (5-6 colors) on FFPE tissue
Endothelial cellsTumor cells
More routine analysisER, PR, Her2, EGFR, cKIT…C4d in transplant rejection
Interact with Clinical Trial Workspace
![Page 40: Prognostic and Diagnostic Evaluation of Histopathologic ...€¦ · Current method for assessing tumor cell signaling or other proteins by IHC - Immunostain for proteins-Pathologist-dependent,](https://reader035.vdocuments.us/reader035/viewer/2022062922/5f0802ad7e708231d41fe14f/html5/thumbnails/40.jpg)
Collaborators
MSI workPenn - Bill Lee, Wiem LassoudRPI - Badri Roysam, Gang LinCRI - Richard Levenson, Cliff Hoyt
CADAnant Madhabushi
Scott Doyle