Towards Robust Medical Image Segmentation

Shaoting ZhangCBIM Center

Computer Science DepartmentRutgers University

IntroductionBackground

• Segmentation (finding 2D/3D region-of-interest) is a fundamental problem and bottleneck in many areas.

• We focus on learning-based deformable models with shape priors (2D contour or 3D mesh).

Chest X-ray

Lung CADLiver in whole-

body CT (PET-CT)

Rat brain structures in MR Microscopy

2

IntroductionBackground

• End-to-end, automatic, accurate, efficient.

• Robustness– Handle weak or misleading

appearance cues.– Handle diseased cases (e.g.,

with tumor/cancer).– Leverage shape priors to

improve the robustness (Active Shape Model, T. Cootes, CVIU’95; 3D ASM for cardiac segmentation, Y. Zheng, TMI’08)

Segmentation system

3

Liver shape variations[Springer images]

IntroductionResearch Void

• Limitations of existing shape prior methods:– Assume Gaussian errors → Sensitive to outliers– Assume unimodal distribution of shapes → Cannot

handle large shape variations, e.g., multimodal– Only keep major variation → Lose local shape detail

4

IntroductionResearch Void

• Handling gross errors or outliers. – RANSAC + ASM [M. Rogers, ECCV’02]– Robust Point Matching [J. Nahed, MICCAI’06]

• Handling multimodal distribution of shapes.– Mixture of Gaussians [T. F. Cootes, IVC’97]– Manifold learning for shape prior [Etyngier, ICCV’07]– Patient-specific shape [Y. Zhu, TMI’10]

• Preserving local shape details.– Sparse PCA [K. Sjostrand, TMI’07]– Hierarchical ASM [D. Shen, TMI’03]

5

Need to solve all three

challenges simultaneously

in practice

MethodsSegmentation framework

6

Landmark Detectors

Offline Learning

Boundary Detectors

Runtime Segmentation

New Image

Model Initialize

Iterative Deformation and Shape Refinement

Final Results

Shape Priors via Sparse Shape Representation

Image Data and Manually Labeled Ground Truths

Critical anatomical landmarks.

MethodsSegmentation framework

• Initialization: Landmark detectors + shape prior– Automatic, fast (AdaBoosting/PBT/random forest + 3D Haar).

• Deformation: Boundary detectors + shape prior– Extend the landmark detectors to locate the sub-surfaces.

Critical anatomical landmarks.

7

Lung Heart

Rib

Colon

Abdomen

7

MethodsShape prior using sparse shape representation

• Our shape prior is based on two observations:– An input shape can be approximately represented by a

sparse linear combination of training shapes.– The given shape information may contain gross errors, but

such errors are often sparse....

≈

8

MethodsShape prior using sparse shape representation

• Formulation:–

• Sparse linear combination:–

...≈

...≈

Dense xSparse xAligned shape data matrix DWeight x

Input y

Global transformationparameter

Number of nonzero elements

Global transformationoperator

9

MethodsShape prior using sparse shape representation

• Non-Gaussian errors:–

10

MethodsShape prior using sparse shape representation

• Why it works?– Robust: Explicitly modeling “e” with L0 norm constraint.

Thus it can detect gross (sparse) errors, i.e., non-Gaussian– General: No assumption of any parametric distribution

model (e.g., a unimodal distribution assumption in ASM). Thus it can model large shape variations.

– Lossless: It uses all training shapes. Thus it is able to recover detail information even if the detail is not statistically significant in training data.

11

MethodsOptimization

• Convex relaxation [Candes, Tao, T-IT’06]:

• Tuning parameters (λ1 and λ2) are not difficult to choose, compared to k1 and k2. – λ1 x║ ║1, λ1 controls the sparsity of x. – λ2 e║ ║1, λ2 controls the sparsity of e.

• One group of parameters work well for all images.• Efficient convex optimization: fast proximal gradient

methods (e.g., FISTA)12Shaoting Zhang

MethodsScalability

13Shaoting Zhang

... ≈... ...

Dictionary size = 64 #Coefficients = 1000 #Input samples = 1000

1. To efficiently handle many training samples - dictionary learning:

2. To efficiently handle thousands of vertices - mesh partitioning:

Applications – Part I 2D lung localization in X-ray

(Lung computer-aided diagnosis system, Siemens)• Handling gross errors

Detection PA ASM RASM NN TPS Sparse1 Sparse2

Procrustes analysisActive Shape ModelRobust ASMNearest NeighborsThin-plate-splineWithout modeling “e”

Proposed method

14

Applications – Part I2D lung localization in X-ray

• Multimodal shape distribution

Detection PA ASM/RASM NN TPS Sparse1 Sparse2

15

Applications – Part I2D lung localization in X-ray

• Recover local detail information

Detection PA ASM/RASM NN TPS Sparse1 Sparse2

16

Applications – Part I2D lung localization in X-ray

• Sparse shape components

• ASM modes:

17

≈

0.5760

+ +

0.2156 0.0982

Applications – Part I2D lung localization in X-ray

• Mean values and standard deviations. ~1,000 cases.

Left lung

Right lung

1)PA, 2)ASM, 3)RASM, 4)NN, 5)TPS, 6)Sparse1, 7)Sparse2

µσ

18

Applications – Part II3D liver segmentation in low-dose CT

19

• Use case: Improved interpretation of PET-CT, with Siemens– PET: Measure functional process (metabolic activity), oncology– CT: Provide anatomical information, co-registered with PET– Issues: High variations of activities across different organs– Solution: Segment organs in CT for organ-specific interpret.– Challenge: Low dose results in low contrast & fuzzy boundary

Applications – Part II3D liver segmentation in low-dose CT

Procrustesanalysis Sparse shape Ground truth

Initialization

Deformation

20

Same landmarks + different shape priors

Same deformation module

Applications – Part II3D liver segmentation in low-dose CT

Procrustesanalysis

Sparse shape

21

Applications – Part II3D liver segmentation in low-dose CT

Procrustesanalysis

Sparse shape

22

Applications – Part II3D liver segmentation in low-dose CT

Procrustesanalysis

Sparse shape

23

Applications – Part II3D liver segmentation in low-dose CT

Procrustesanalysis

Sparse shape

24

Applications – Part II3D liver segmentation in low-dose CT

ASM-type [Zhan’09] Sparse shape Ground truth

• Shape refinement during segmentation

25

Applications – Part II3D liver segmentation in low-dose CT

• Quantitative results: surface distances. ~80 cases

Mean value and standard deviation (voxel)

26

Applications – Part IIISegmentation of Multiple Brain Structures in MRI

Rat brain structures Drug addiction and alcoholism

(with Brookhaven National Lab)

Human brain, 34 structures Alzheimer's disease

(with GE Global Research)

27

Summary of Robust Segmentation• Robustly handle abnormal cases, such as diseased cases

(liver tumor). Critical to healthcare applications such as computational diagnosis systems.

• Patent with Siemens. Used in several clinical applications. Key contribution for our awarded NSF-MRI grant (’12-’16).

• Relevant publications:– First author papers

• MICCAI 2012, 2011 (MICCAI Young Scientist Award Finalist)• CVPR 2011• Two journal papers in Medical Image Analysis

– Second author paper• Medical Physics 2013 (with my co-mentored student, G. Wang)• ISBI 2013, oral (with my co-mentored student, Z. Yan)28