Image Segmentation and Image Segmentation and RegistrationRegistration

Rachel Jiang

Department of Computer Science

Ryerson University

2006

Rachel Jiang, 2006

2

Rigid registration Rigid registration

• Intensity-based methods – Have been very successful

• Monomodal• Multimodal

– Assuming a global relationship between the… images to register

– Deriving a suitable similarity measure• Correlation coefficient• Correlation ratio• Mutual information• Block matching• …

Rachel Jiang, 2006

3

Methods for fusing Methods for fusing imagesimages

• There are three general methods for fusing images from different (or the same) image modalities: – landmark matching

• include external fiducial landmarks or anatomic landmarks.

– surface matching • uses an algorithm that matches different images of the

same patient surface.

– intensity matching.

Rachel Jiang, 2006

4

intensity matchingintensity matching

• uses mutual intensity information to co-register different images.

• The matched intensities may come from the same scanner – two different MRI scans acquired on different

days– from different modalities such as MRI and PET.

Rachel Jiang, 2006

5

Rigid Body ModelRigid Body Model

• Most constraint model for medical imaging

• asserts that the distance and internal angles within images can not be changed during the registration

Rachel Jiang, 2006

6

Non rigid modelNon rigid model

• can detect and correct discrepancies of small spatial extent, by deforming one of the images (source) to match the other (reference).

• Spatial deformation model can be based on different physical properties like elasticity or viscosity, or their generalizations and simplifications.

• Deformation is driven by external forces, which tend to minimize image differences, measured by image similarity measures.

Rachel Jiang, 2006

7

Mutual informationMutual information

• The mutual information of two images is the amount of information that one image contains about the other or vice versa.

• Transforming one image with respect to the other such that their mutual information is maximized. The images are assumed to be registered.

Rachel Jiang, 2006

8

Mutual informationMutual information

• A function of transformation between the images– an algorithm that searches maximum value for a

function that gives the alignment information between images

– different transformation estimates are evaluated• (those transformation estimates will result in varying

degrees of overlap between the two images)– (MI as a registration criteria is not invariant to the overlap between

images)

Rachel Jiang, 2006

9

MIMI

Rachel Jiang, 2006

10

MI in 2D/3D space MI in 2D/3D space

– In 3D space• We attempt to find the registration by maximizing the

information that one volumetric image provides about the other.

– In 2D space• Two curve that are to be matched as the reference

curve and the current active evolving curve.• We seek an estimate of the transformation that

registers the reference curve and the active curve by maximizing their mutual information

Rachel Jiang, 2006

11

Degree of freedomDegree of freedom

• Six degree of freedom charactering the rigid movements– 3 describe the Rotation– 3 describe Translation

Rachel Jiang, 2006

12

Similarity measuresSimilarity measures

• Common registration methods can be grouped as– Feature based techniques

• Rely on the presence and identification of natural landmarks or fiducial marks in the input dataset to determine the best alignment

– Intensity-based measures• Operate on the pixel/voxel intensities directly

– Varies statistics are calculated by using the raw intensity values

Rachel Jiang, 2006

13

Taking geometric Taking geometric constraintsconstraints

• Special land marks on human body

• Manually embed land marks

• Spacial correlations

• …

Rachel Jiang, 2006

14

Correlation ratio Correlation ratio

• Given two images I and J, the basic principle of the CR method is to search spatial transformation T and an intensity mapping f such that, by displacing J and remapping its intensities, the resulting image f(JoT) be as similar as possible to I.

Rachel Jiang, 2006

15

Example of Image Example of Image SegmentationSegmentation

• Bone fracture detection

Rachel Jiang, 2006

16

Image Segmentation Image Segmentation TechniquesTechniques

• threshold techniques– make decisions based on local pixel information

• edge-based methods– Weakness: broken contour lines causes failure

• region-based techniques– partitioning the image into connected regions by grouping neighbouring pixels of

similar intensity levels. – Adjacent regions are then merged under certain criterion. Criteria create

fragmentation or overlook blurred boundaries and overmerge.

• Active contour models

Rachel Jiang, 2006

17

Deformable modelsDeformable models

– Snakes/Balloons/Deformable Templates• provide a curve as a compromise between regularity of the curve

and high gradient values among the curve points. – (Kass et al., 1988; Cohen, 1991; Terzopoulos, 1992)

– Level set methods• Level Set Methods are numerical techniques which can follow the

evolution of interfaces. These interfaces can develop sharp corners, break apart, and merge together.

– (Osher and Sethian, 1988; Sethian, 2001)

– Geodesic Active Contour • take the advantages of both Snake and Level set

methods – (Caselles et al., 1995; Malladi et al, 1995; Sapiro, 2001)

Rachel Jiang, 2006

18

The Snake formulaThe Snake formula

Rachel Jiang, 2006

19

Snake: image forceSnake: image force

Rachel Jiang, 2006

20

Snake: exampleSnake: example

Rachel Jiang, 2006

21

Level Set MethodLevel Set Method

• Level Set Methods – provide formulation of propagating interfaces, a

mathematical formulation and numerical algorithm for tracking the motion of curve and surfaces

• (Osher and Sethian, 1988; Sethian, 2001)

– For segmenting several objects simultaneously or an objects with holes, it is possible to model the contour as a level set of a surface, allow it to change its topology in a nature way

• (Cohen, 1997).

Rachel Jiang, 2006

22

Level SetLevel Set

Rachel Jiang, 2006

23

Formula of Level Set Formula of Level Set methodmethod

• Curve evolving formula:

Rachel Jiang, 2006

24

LSM: exampleLSM: example

Rachel Jiang, 2006

25

LSM: More ExampleLSM: More Example

Rachel Jiang, 2006

26

Geodesic Active Contour Geodesic Active Contour (GAC)(GAC)

– presents some nice properties• the initialization step does not impose any significant constraint• can deal successfully with topological changes, • finding the global minimum of energy minimizing curve can be solved by

mapping the boundary detection problem into a single minimum problem. • The new model mathematically inherit

– the way handling the topological changes from the Level Set– the minimizing deformation energy function with ‘internal’ and ‘external’ energies

along its boundary from the traditional Snake. – This model inherit the advantages of LS and ‘Snakes’ by transform mathematical

formulation of Snake – Lagrenge formula with PDEs

– The theory behind the GAC is the use of partial differential equations and curvature-driven flows.

• (Caselles et al., 1995; Malladi et al, 1995; Sapiro, 2001)

Rachel Jiang, 2006

27

Segmentation result using Segmentation result using ACMACM

Rachel Jiang, 2006

28

Surface reconstructionSurface reconstruction

Rachel Jiang, 2006

29

Mathematical Morphology Mathematical Morphology

– provides the foundation for measuring topological shape, size, location.

• The theory behind mathematical morphology is defining computing operations by primitive shapes

– Georges Matheron, Jean Serra and their colleagues of Centre de Morphologie Mathematique

– G.Matheron 1975, Serra, 1982, Vicent, 1990…

– offer several robust theories and algorithms• to implement on digital images to extract complex features

– uses ‘Set Theory’ as the foundation for its functions. The simplest functions to implement are ‘Dilation’ and ‘Erosion’.

Rachel Jiang, 2006

30

Erosion and Dilation (1D)Erosion and Dilation (1D)

Rachel Jiang, 2006

31

Erosion and Dilation (2D)Erosion and Dilation (2D)

Rachel Jiang, 2006

32

Erosion & Dilation Erosion & Dilation exampleexample

Rachel Jiang, 2006

33

Opening & ClosingOpening & Closing

Rachel Jiang, 2006

34

Shape OperatorsShape Operators

Shapes are usually combined by means of :

X X X Xc2 1 1 2\ X2X1

• Set Intersection (occluded objects):

X X1 2X1 X2

• Set Union (overlapping objects):

Rachel Jiang, 2006

35

DilationDilation

•

••

•

•

••

•••

••

••

••B

A BA

Rachel Jiang, 2006

36

DilationDilation

xB)(

BA

A

Aa

aBb

b BABA

)()(

Rachel Jiang, 2006

37

ExtensitivityExtensitivity

•

•

•

•

••

••

••

••

B

ABA

••

Rachel Jiang, 2006

38

ErosionErosion

•

•

•

•

••••

••

B

A BA

• •••

Rachel Jiang, 2006

39

ErosionErosion

•

•

•

•

••••

••

BA

• ••••

•

•

•

••••)1,0(1A )0,0(A

Rachel Jiang, 2006

40

ErosionErosion

xB)(

A

BA

))({ 2 ABZxBA x

Rachel Jiang, 2006

41

Opening and ClosingOpening and Closing

Opening and closing are iteratively applied dilation and erosion

Opening

Closing

BBABA )(

BBABA )(

Rachel Jiang, 2006

42

Opening and ClosingOpening and Closing

xB

BA

ABBABA )(

}{ ABx

xx

BBA

Rachel Jiang, 2006

43

Opening and ClosingOpening and Closing

They are idempotent. Their reapplication has not further effects to the previously transformed result

BBABA )(

BBABA )(

Rachel Jiang, 2006

44

Rachel Jiang, 2006

45

WatershedWatershed

Rachel Jiang, 2006

46

Capturing the shape priorCapturing the shape prior

• the curve C and the transformation S, R, T is calculated such that the curve Cnew = SRC + T and C* are perfectly aligned.

• The minimization problem now can be solved by finding steady state solutions to the following system

Rachel Jiang, 2006

47

Minimization processing Minimization processing systemsystem

Rachel Jiang, 2006

48

Distance measureDistance measure

• d(x,y) = d(C* ,(x,y)) is the distance of the point (x,y) from C*

• The function d is evaluated at

• SRC(p) + T

Rachel Jiang, 2006

49

Minimizing Energy functionMinimizing Energy function