Industrial Problem‐Solving Workshop on Medical Imaging 

Problem # 2Rapid Modeling of Internal Structures of p g

Deformable Organs (i.e. Liver)

Edward Xishi Huang and James DrakeThe Hospital for Sick Children

Problem definition 

Accurate estimation of deformation of the soft organ’s internal structures between two images acquired at differentinternal structures between two images acquired at different conditions

What information are used?! Point landmarks  [1] Vessel segments and curves [1]

Surface information

Problem definition 

Application: Tumor treatment using high‐intensity focused ultrasound (HIFU) thermal proceduresultrasound (HIFU) thermal procedures. 

• Real‐time image guidance is required for targeting. • Intraoperative imaging systems:

– Low resolution and quality images– Long acquisition time– Compatibility with other equipments in the operation room.

Solution: Updating preoperative treatment plans and physical deformation models based on intraoperatively acquired images.gThe idea: Employing boundary conditions (i.e. curves of vessels and bifurcations of vessels, landmarks, and surface information ) to constrain the solution of the deformableinformation ) to constrain the solution of the deformable model of organ.

Problem definition 

Registering internal structures and surface information of the two set of images acquired from the soft organtwo set of images acquired from the soft organ. 

Image Registration

Given a reference image R and a template image T, find aGiven a reference image R and a template image T, find a reasonable transformation y, so that the transformed image T[y]is similar to R [2].

Problem 1

Modeling internal structures in 2D images using landmarks and vessel segments with known end pointsand vessel segments with known end points 

9

10

C 9

10

6

7

8

6

7

8

C1

3

4

5

D

B

4

5

B1

A

( )S x( )S x′ ′

1

2

3

A

1

2

3

D1

A1

0 2 4 6 8 100

0 2 4 6 8 100

Problem 1

Solution

L d k ( ) ( )d f l′ ′Landmarks:  

Vessel Curves:

( ) ( ), and , for 1,2,...,l l l lx x x y l n′ ′ =

( ) and ( )y S x y S x′ ′ ′= =Vessel Curves:

Displacement Function:

( ) and ( )y S x y S x

( )( , ) , ( , )u x y v x yp

Landmarks: 

( )( ) ( )y y

( ) ( ), ( , ), ( , )l l l l l l l lx y x u x y y v x y′ ′ ⇐ + +

Any point on  the curve:  ( ) ( ), ( , ), ( , )x y x u x y y v x y′ ′ ⇐ + +

Problem 1

Solution

Th f i b i i i d f l d k hiThe cost function to be minimized for landmarks matching:

( ) ( ){ }2 21( , ) ( , ) ( , )

n

l l l l l l l lJ u v x x u x y y y v x y′ ′= − − + − −∑

The cost function to be minimized for curve matching:

{ }1l=

( ) ( ) ( )( )( ) ( ) ( ) ( )x S x x u x S x S x v x S x⇒ + +This point corresponds to

( ) ( ) ( )( ), ( ) , ( ) , ( ) , ( )x S x x u x S x S x v x S x⇒ + +

( ) ( ) ( )( )( ), ( ) , ( ) , , ( )x S x x u x S x S x u x S x′⇒ + +( ) ( ) ( )( )( )( )( ) ( )

2

1

2

2 ( , ) , ( ) ( ) , ( )x

x

J u v S x u x S x S x v x S x dx′= + − −∫

Problem 1

Solution

Th l f i b i i i d f l d k dThe total cost function to be minimized for landmarks and curve matching: 

1 1 2 2( , ) ( , ) ( , )J u v w J u v w J u v= +

Having the variational model in optimization model form:

0 ( , )N

i iu Ax By x a x yϕ= + + +∑01

01

( )

( , )

i iiN

i ii

y y

v Cx Dy y b x y

ϕ

ϕ

=

=

= + + +

∑

∑are radial basis functions. 

The cost function in terms of unknown parameters 

( 1,2,..., )i i Nϕ =

( )0 0 1 1min , , , , , , ,..., , ,...,N NJ A B C D x y a a b b

Problem 2

Modeling internal structures in 2D images using landmarks and vessel segments with one end pointand vessel segments with one end point. 

9

B8

9

( )S x

6

7

8

6

7

8

( , )e ex y( )′ ′

( )S x′ ′

4

5

3

4

5

B1

1 1( , )x y′ ′

( , )e ex y′ ′

1 2 3 4 5 6 7 8 92

3A

1A A→1 2 3 4 5 6 7 8 9

2

3A11 1( , )x y

1 1( , )y

1

?!A AB→→

Problem 2

Solution

i k i b f d d d b dd d( ) is unknown point to be found and needs to be added to the optimization function as follows:

( , )e ex y

( )( ) ( )( )

( )( ) ( )

2 22

2

( , , ) , ( ) , ( )

( ) ( ) ( )

e e e e e e e e e

xe

J x u v x x u x S x y y v x S x

S x u x S x S x v x S x dx

′ ′= − − + − −

′+ + − −∫ ( )( ) ( )1

, ( ) ( ) , ( )x

S x u x S x S x v x S x dx+ +∫

Problem 3

Modeling internal structures in 3D image

L d k Th 2D iLandmarks: The same as 2D images

( ) ( ) ( ){ }1

2 2 2

( , , )n

J u v w =

∑

Curves: 

( ) ( ) ( ){ }2 2 2

1( , , ) ( , , ( , ,l l l l l l l l l l l l l l l

lx x u x y z y y v x y z z z v x y z

=

′ ′ ′− − + − − + − −∑

Parametric cubic representation: piecewise spline3 2( ) x x x xx t a t b t c t d= + + +3 2

3 2

( )

( )

( )

x x x x

y y y y

z z z z

y t a t b t c t d

z t a t b t c t d

= + + +

= + + +

Problem 4

Registration based on surface matching of the object in two set of imagesset of images

( ), , ( , )x y S x y ( ), , ( , )x y S x y′ ′ ′ ′ ′

( )( , , ), ( , , ), ( , , )u x y z v x y z w x y z( )( , , ), ( , , ), ( , , )u x y z v x y z w x y z

Problem 4

The cost function to be minimized:

( ), , ( , )x y S x y ( ), , ( , )x y S x y′ ′ ′ ′ ′

( ) ( )( ) ( )2

( , , )

( ) ( ) ( ) ( )

J u v w

S S S S S d d

=

′∫Another solution: deformable surface modeling using active 

conto rs [3]

( ) ( )( ) ( ), , ( , ) , , , ( , ) ( , ) , , ( , )S x u x y S x y y v x y S x y S x y w x y S x y dx dyΩ

′ + + − −∫

contours [3]

Impact

FEM‐based image registration.

I i b d [4]Intensity based [4]

[ ]( ) 1min ( ) , ;2

TJ u D T u R u Kuα α += + ∈ℜ[ ]( )2

[ ]( ),( ) 0D T u RJ u Ku

u uα

∂∂= + =

∂ ∂u u∂ ∂

[ ]( ),1( )D T u R

Ku f u∂

= = −∂

‐ Multimodality

uα ∂

‐ Local minima 

Impact

FEM‐based image registration.

F b dFeature based

Internal structures and surface information of the deformable organ provides necessary boundary condition to solve theorgan provides necessary boundary condition to solve the FEM‐based deformation equation.

Example: Linear finite element model of the brain shift and deformation [3].

Impact

FEM‐based image registration.

References

[1] Dietlind Zühlke, Sven Arnold, Gernoth Grunst, Peter Wißkirchen, “Intra‐interventional registration of 3D ultrasound to models of the vascularinterventional registration of 3D ultrasound to models of the vascular system of the liver” GMS CURAC 2007, Vol. 2(1)

[2] J. Modersitzki, Numerical Methods for Image Registration. New York:Oxford, 2004.o O o d, 00

[3] M. Ferrant, A. Nabavi, B. Macq, F. Jolesz, R. Kikinis, and S. Warfield, “Registration of 3D Intraoperative MR Images of the Brain Using a Finite‐Element Biomechanical Model,” IEEE Trans. on Medical Imaging, vol. 20, , g g, ,no. 12, pp. 1384–1397, 2001.

[4] B. Marami, S. Sirouspour, and D. Capson, “Model‐Based Deformable Registration of Preoperative 3D to Intraoperative Low‐Resolution 3D and g p p2D Sequences of MR Images,” in MICCAI 2011, pp. 460–467, 2011.

Group Members

Edward Xishi Huang

C Sean BohunC. Sean Bohun

Tian Chen

Jamil Jabbour

Ken Jackson

Iain Moyles

Cartihk Sharma

Yongji Tan

Bahram Marami

Blah

Motivation

Modelling a three-dimensional image of the liver iscomputationally expensive (and has challenging physics)

Image data arrives in several two-dimensional cross sections(slices) as the body is scanned

Each slice is identified by unique markers (landmarks andcurves)

If all of the markers from one slice map to all be on atransformed slice then we can consider a series oftwo-dimensional transformations

Blah

Setup

LandmarksThe N distinguishing points on image

CurvesThe κ distinguishing curves (vessels) on image

Assume a transformation from“pre-image” (X ,Y ) to“post-image” (X , Y ) via a transformation (X , Y )

X = X +∑j

ajφj(X ,Y ),

Y = Y +∑j

bjφj(X ,Y ),

φj are radial basis functions (RBF)

φj(X ,Y ) = exp

(−

(X − Xj)2 + (Y − Yj)

2

σ

)(Xj ,Yj) are the L allocation points to form a basis set (L ≤ N)

Blah

Setup

For curves we assume there exists a mapping for curve pointson the pre-image (xx , yy) given by yy = S(xx)

Similarly there exists a map yy = S(xx) for the curve pointson the post-image (xx , yy)

Often the form of S and S will be through an interpolation(cubic splines)

The curves have the requirement that the endpoints arelandmarks

Blah

Pre Image

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Student Version of MATLAB

Blah

Post Image

187 187.5 188 188.5 189 189.5 190 190.5256

256.5

257

257.5

258

258.5

259

259.5

260

Student Version of MATLAB

Blah

Optimization

We wish to determine the coefficients for the RBF thattransforms the pre-image into the post-image

We determine these by minimizing an error function composedof matching landmarks and curves

Therefore we consider1 Landmark Error

We wish all terms of the form X − X to be small so that thelandmarks are close

2 Curve Error

We wish the curve points to be close as well i.e. xx − xx andS(xx) − S(xx) are smallThe predicted curve S(xx) = S(xx) +

∑j bjφj(xx , S(xx))

Blah

Solution

187 187.5 188 188.5 189 189.5 190 190.5256

256.5

257

257.5

258

258.5

259

259.5

260

189.2 189.4 189.6 189.8 190 190.2

258.6

258.8

259

259.2

259.4

259.6

259.8

Blah

Oops...

186 186.5 187 187.5 188 188.5 189 189.5 190 190.5254

255

256

257

258

259

260

261

188.8 189 189.2 189.4 189.6 189.8 190 190.2 190.4

258

258.5

259

259.5

260

260.5

Blah

Constrained Optimization

There are many local minima to this problem with largeenergies

In fact, there are many local minima with small energies thatdon’t represent the “true” solution (lack of global minimizer?)

Idea: Build in a constraint for each curve that forces the areaunder the post-image curve to match that of the predictedcurve ∫ xx2

xx1

(S(xx) − S(xx) −∑j

bjφj) dxx = 0

Blah

Solution

187 187.5 188 188.5 189 189.5 190 190.5256

256.5

257

257.5

258

258.5

259

259.5

260

189.4 189.5 189.6 189.7 189.8 189.9 190 190.1 190.2

258.6

258.8

259

259.2

259.4

259.6

259.8

Blah

Conclusions

Created image data

Performed an optimization to recover RBF coefficients thattransform pre-image to post-image

Included integral constraints to reduce set of minimizers

Future Work

Apply to real landmark data

Consider weightings carefully

Center of mass type penalty systemVoronoi diagrams

Allow for curve endpoints to move