enhanced image modeling for em/mpmenhanced image...

Enhanced Image Modeling for EM/MPMEnhanced Image Modeling for EM/MPM Segmentation of 3D Materials Image Data

Dae Woo Kim, Mary L. ComerSchool of Electrical and Computer EngineeringSchool of Electrical and Computer Engineering

Purdue University

Presentation Outline

- Motivation

- SEM Imaging Modes

2D d 3D Bl i M d l- 2D and 3D Blurring Models

- Expectation-Maximization/Maximization of the Posterior Marginals

(EM/MPM) Segmentation

- 2D and 3D Joint Deconvolution/Segmentation (JDS)

- New prior model: Minimum Area Increment (MAI)

3D EM/MPM- 3D EM/MPM

- Results & Conclusions

Motivation

• Scanning electron microscope (SEM) images have blurring due in part to complexScanning electron microscope (SEM) images have blurring due in part to complexelectron interactions during acquisition

• One particular problem that arises during segmentation is necking: the merging ofti l th t d t t t h i th i i l i d tparticles that do not appear to touch in the original image data

• We incorporate model for blurring degradation into the original EM/MPM method inorder to reduce neckingg

• We also introduce a new prior model called minimum area increment to reduce necking

Current model in EM/MPM has smoothing parameter βCurrent model in EM/MPM has smoothing parameter β

(a) Original image (b) Ground truth (c) Original (d) Original

3

(a) Original image (b) Ground truth (c) Original EM/MPM (β=3.0)

(d) Original EM/MPM (β=1.2)

SEM Imaging Modes

• Secondary Electrons (SE):

XSEBSE BSE

PE

AE

Secondary Electrons (SE):Due to SE’s low energy, they can escape only from a thin surface layer of a few nanometers. In this mode, blurring degradation can be modeled with a

R

SEBSE AE, g g2D blurring filter.

• Backscattered Electrons (BSE) :

ctro

n ra

nge ( )

Information depth in BSE mode is deeper than in SE mode. If we capture electrons with small energies below 1keV, we can make the exit depth of

Ele

g pBSE have the same order as of SE. Therefore, we can conclude that the interactions for low-energy electrons can be modeled with a 2D filter while the interactions for high-energy electrons can be modeled with 3D blurring filter.

Diffusion cloud of electron range R for normal incidence of the primary electron (PE)1.

41L. Reimer. Scanning Electron Microscopy: Physics of Image Formation and Microanalysis , 2nd Edition. Springer-Verlag, Berlin, 1998

2D and 3D Blurring Filter Coefficients

• 2D filter coefficients :2D filter coefficients : The lateral number of generated SE can be modeled as an exponential.

ated

SE 10000

8000

r of g

ener 6000

4000

• 3D filter coefficients :We propose 3D filter which has coefficients as below: N

umbe

r

Lateral distance from impact

2000

0-400 -300 -200 -100 0 100 200 300 400

The lateral distribution of generated

Lateral distance from impact point [nm]

gSE (Monte Carlo simulation, silicon, 5kV)2

52 Günter Wilkening, Ludger Koenders. Nanoscale Calibration Standards And Methods: dimensional and related measurements in the micro and nanometer range, 1st Edition, WILEY-VCH, Weinheim, 2005

Original Image Models for EM/MPM

• Use the Markov Random Field as the prior model

• Use the Gaussian distribution

• Use Bayes’ rule to combine the these two models into the posterior distribution function

data term regularization term

6

EM/MPM Segmentation

• Use the Maximizer of the Posterior Marginal (MPM) criterion as the optimization objective. – Minimizes the expected number of misclassified pixels.

• Use the Expectation/Maximization (EM) algorithm to estimate model parameters.- The unknown parameter vector contains means and variances for the Gaussian image model.

7

2D JDS(Joint Deconvolution/Segmentation) Method : Definition

• We define label field x, observed image y and blurring vector hWe define label field x, observed image y and blurring vector h.

- Let the set of all lattice point S be [1, · · · ,M]2 and the order of the pixel of the label field x and the observed image y be raster scan order as below:label field x and the observed image y be raster scan order as below:

- We can make the blurring matrix H having window size (2W + 1) × (2W + 1) be avector h using raster scan order, so that

8

2D JDS Method : Image Model & Posterior Model 3

• Image Model :g

• Posterior Model:

93 D.W. Kim and M.L. Comer, “Joint deconvolution/segmentation of microscope image of materials,” in IEEE Statistical Signal Processing Workshop, Ann Arbor, MI, USA, August 2012.

2D JDS Method : EM algorithm

• In EM iteration we can get closed form solution of variance But for theIn EM iteration we can get closed form solution of variance. But for the mean, we get L linear equations from which we can obtain estimates of the means

10

3D JDS Method : Definition

• We define label field xn and observed image yn of the n-th slice in a stack of image. n g yn gAnd we define 3D blurring vector h3D.

xn : label field of the n-th imageyn : observed n-th image.h3D : blurring vector having coefficient h3D(s1,s2,m)

yT

yn

y1

11

3D JDS Method : Image Model & Posterior Model

• New Image Model :g

• New Posterior Model:• New Posterior Model:

12

Results : Test Sequence

Slice166(Bottom) Slice167 Slice168Slice166(Bottom) Slice167 Slice168

Slice169 Slice170(Top)

• Series of five René 88 DT images The light colored phase is γ' the gray13

• Series of five René 88 DT images. The light-colored phase is γ the graymatrix is γ. We applied our method from the bottom image to the top.

Results : 3D Blurring Image Model

(c) Original EM/MPM (β=3.0)PMP 5 35%

(b) Ground Truth(a) Original imageSli 170 PMP = 5.35%Slice 170

(d) blurring image model (β=3.0, ω=0.10, δ =0.5)

(e) Preprocessed ImageMATLAB ‘deconvlucy’

(ω=2 0)

(f) Preprocessed EM/MPM (β=3.0)

PMP = 4 76%




















(d) blurring image model (β=3.0, ω=1.10, δ =0.5)(d) 3D JDS EM/MPM(β=3.0, ω=0.52, δ =0.5)

PMP = 4 08%

14PMP (percentage of misclassified pixels)

(ω 2.0) PMP = 4.76%PMP = 4.08%

MAI(Minimum Area Increment)

• To further reduce object necking, we propose a minimum area incrementconstraint This assigns a penalty for the merging of two or more large objectsconstraint. This assigns a penalty for the merging of two or more large objects

• Connecting point: A point where two or more disconnected areas of thesame class exist in a pre-defined neighborhood around the pointsame class exist in a pre defined neighborhood around the point

4-neighbor 12-neighbor

• C id 4 i hb fi ti Th t i l i th l ft fi i t

neighborhood neighborhood

• Consider a 4-neighbor configuration: The center pixel in the left figure is nota connecting point; the center pixel in the right figure is a connecting point

1 1 001 1 1 0012 class example 1 1 0

1 0

0 0

0

0 0 11

0

1

0

00

0

00xr

0 0 1

0

1 1 0

1 0

0 0

0

0 0 11

1

1

1

00

0

00xr

0 1 1

0

p0 : class 01 : class 1

Minimum area increment

15

Minimum area incrementwindow size ws = 5 Not connecting point

when either xr = 0 or 1Connecting pointwhen either xr = 0 or 1

Area Increment Measuring Function

• Area increment measuring function gws,r(xr) : The increase in area of theg f gws,r( r)largest-area region in a window of size ws×ws around pixel location r if theclass label assigned to pixel r is xr ,

- If one class is a background class (assume this is class 0), then we letIf one class is a background class (assume this is class 0), then we letgws,r(0)=0 for all r- If pixel r is not a connecting point then gws,r(xr)=0

• Consider the following 3-class example, with class 0 a background class

3 l l 1 1

1

11

1

xr

1

1 1 2

1

2

11

1

1

1

2

2

2xr

1 1

1 1 2

1

1

2

1 1 11

1

1

1

21

2

2xr

1 1

1

1 1

1

1 1

1

1 11

1

1

1

11 11xr

1 1

1

3 class exampleblank : class 0

1 : class 12 : class 2

11

g2,r(1) = 0g2,r (2) = 0

11

g2,r(1) = 4g2,r(2) = 1

1 1 111

g2,r(1) = 0g2,r(2) = 1

1 111

g2,r(1) = 9g2,r(2) = 0

Minimum area incrementwindow size ws = 5

16

New prior model: MRF and MAI

• We propose new prior model by incorporating MAI constraint into existingMRF prior model as below:

• To make proposed prior model more effective, we applied SA(simulated annealing) scheme. We gradually increase the β value of the classes which have no necking problemhave no necking problem.

17

Results : MAI

(c) Original EM/MPM (β 3 0) PMP 6 37%

(b) Ground Truth(a) Original imageSli 017

no MAI MAI

(β=3.0) PMP = 6.37%Slice 017

(d) 2D JDS

τ = 1.5ws = 712-neighbor

method with…

g

23th EM iteration 23th EM iteration24th EM iteration 24th EM iteration25th EM iteration 25th EM iteration26th EM iteration 26th EM iteration27th EM iteration 27th EM iteration28th EM iteration 28th EM iteration29th EM iteration 29th EM iteration30th EM iteration 30th EM iteration

18

23th EM iterationβ(0) = 2.3, β(1) = 3.0















PMP =2.97%


PMP=2.75%

Results : MAI

(b) Ground Truth(a) Original imageSli 170

(c) Original EM/MPM (β 3 0) PMP 5 35%Slice 170 (β=3.0) PMP = 5.35%

no MAI MAI(d) 2D JDS

τ = 1.5ws = 712-neighbor

method with…

g

23th EM iteration 23th EM iteration24th EM iteration 24th EM iteration25th EM iteration 25th EM iteration26th EM iteration 26th EM iteration27th EM iteration 27th EM iteration28th EM iteration 28th EM iteration29th EM iteration 29th EM iteration30th EM iteration 30th EM iteration

19
















PMP=4.35%


PMP=3.70%

3D EM/MPM with JDS

• Image Model :

• Posterior Model:Posterior Model:

20

2D EM/MPM VS 3D EM/MPM

2D EM/MPM 3D EM/MPM

yT

2D EM/MPM 3D EM/MPM

IyT

yn

ImageModel

y1

Prior

pixel ofnext frame

PriorModel pixel of

previous frame

• 3D EM/MPM needs a large amount of memory. So we apply 3D EM/MPMmethod for 3 frames and save the result of the middle frame and then move tothe next 3 frames which are one frame shifted from the previous 3 frames

21

the next 3 frames which are one frame shifted from the previous 3 frames.

Results : 3D JDS with 3D EM/MPM

3D JDS with 3D EM/MPM( = 1 5 = 0 75 ω = 0 3

3D JDS with 2D EM/MPM ( = 1 5 ω = 0 3 δ = 0 1

NiAlCr ground truth( = 1.5, = 0.75, ω = 0.3, δ = 0.1, PMP = 6.03%)

( = 1.5, ω = 0.3, δ = 0.1, PMP = 6.31%)

slice 027

3D JDS with 3D EM/MPM ( = 1.5, = 0.75, ω = 0.5, δ = 0.5, PMP = 4.24%)

3D JDS with 2D EM/MPM ( = 1.5, ω = 0.5, δ = 0.5,

PMP = 4.38%)

Rene88 slice170

ground truth

22

δ 0.5, PMP 4.24%))

Results : 3D EM/MPM with JDS and MAI

NiAlCr slice number 1 to 59 (59slices, Image Size 194 X 149 pixels)

3D JDS & MAI i h 3D EM/MPMEM/MPM 3D JDS & MAI 2D EM/MPM

23

3D JDS & MAI with 3D EM/MPM(β= 1.5, ω = 0.3, δ = 0.1, τ = 1.5)

Running time : 6638sec (82x)

EM/MPM (β= 1.5)

Running time4 : 81sec 4 Intel i7 CPU 2.4GHz, Memory 8GB

3D JDS & MAI 2D EM/MPM(β= 1.5, ω = 0.3, δ = 0.1, τ = 1.5)

Running time : 2260sec (28x)

Results : 3D EM/MPM with JDS and MAI

Rene88 slice number 143 to 188 (46slices, Image Size 194 X 149 pixels)

3D JDS & MAI i h 3D EM/MPMEM/MPM 3D JDS & MAI 2D EM/MPM

24

3D JDS & MAI with 3D EM/MPM(β= 1.5, ω = 0.5, δ = 0.5, τ = 1.0)

Running time : 5175sec

EM/MPM (β= 1.5)


3D JDS & MAI 2D EM/MPM(β= 1.5, ω = 0.5, δ = 0.5, τ = 1.0)


Conclusions

• In this research, we propose a blurring model to improve pixel mis-In this research, we propose a blurring model to improve pixel misclassification originating from blurring in SEM images. The proposed methodincorporates physical modeling of electron interactions into a blurring imagemodelmodel

• We also propose a new prior model including minimum area incrementt i t d l it ith SA hconstraint and apply it with SA scheme.

• In addition, we apply JDS and MAI in the 3D EM/MPM.

• Experimental results demonstrate that the proposed methods can be used toreduce necking in the segmentation of microscope images of materialsg g p g

25

Th kThank you

Appendix B: EM estimation for the new image model (1/4)model (1/4)

• The EM algorithm is an iterative procedure. At each iteration expectation step and maximization step are performed. In the expectation step the following function is computed.

• In the maximization step, we can estimate θ(p) which maximize Q(θ(p), θ( 1))θ(p-1))


• Similarly, by differentiating with parameter σkwe can get


Therefore,


• Let

then we can getg

enhanced image modeling for em/mpmenhanced image...

Documents