S. Bertoluzza, G. Maggi

Dr. Tomatis, Istituto Tumori in Milan

Radiation therapy programmed by a machine which is given Targets to irradiate regions to avoid

Identify and isolate anatomical structures to input such machine

Segment an image = tag each pixel/voxel with a semantic label

Quo vadis, Atlas based segmentation di T. Rohlfing, R. Brand, R.

Menzel, D.B. Russakoff, C.R. Maurer jr

Intensity based classification Group the gray level space in clusters Each cluster is identified with a label Each pixel is tagged with the label of the

corresponding cluster

Ok for classifying tissue types Not Ok when classifying anatomical

structures

Make the geometry of the image evolve using Image properties (gradient) Constraints (smoothness of the segmented curves)

Level set method Active contours Snakes ...

Template image T, already segmented

Find a map transforming the template image in the image R to be segmented

Tranfer on R the segmentation of T

= image domain L = label set = {e1,e2,e3,...,eN}

A: >L map tagging each position with a

label T: >R+: template image

T segmented exactely by superposing A

Atlas: (T,A)

R: -> R+ - image to be segmented

Find mapping : -> such that T0 are as close as possible

Choose A0 as segmentation of R

Segmentation is reduced to registration

Choice of atlas has substantial impact on the result of the segmentation algorithm

Four strategies Single atlas for all images to be segmented Choice of best atlas for each given image into an

atlas set Average atlas for all the images to be segmented Simultaneous use of multiple atlases

Strategia #Atlas x image Type of atlas Assignment of atlas to F

IND Single Individual Fixed

SIM Single Individual Variable

AVG Single Average Fixed

MUL Multiple Individual Fixed

Selection of an individual atlas Registration of all the raw images with the

selected atlas (individual atlas = obtained by a single raw

image)

Set of atlases Segment R using all atlases Select the atlas that gives the best risult

Maximum similarity Minimum deformation

Construct an artificial tipical image T from a set of images

Construct corresponding atlas Register all images with T

Set of atlases Register R using each atlas Tag pixels with labels usin all information

Ex: voting strategy tag with label chosen by highest number of atlases

Grey scale images real valued functions R: -> R+, T: -> R+

Find mapping : R -> T such that T0 and R

are as close as possible

Huge number of approaches Huge number of algorithms See [Brown 92; Zitova,Flusser 03]

Huge number of algorithms obtained by combining Different distance functionals Different transformation classes Different image models Different optimization algorithms

Focus on the distance functional

Rigid transformations (translations, rotations) Affine transformation

Non rigid transformations

Splines, B-splines [Thevenaz, Unser 98] Thin plate splines [Bookstein, 89] Interpolating wavelets [S.B., G. Maggi 13] ... Parametric transformations

N-dimensional parameter space Parameter -> mapping

(Ex: = i ei )

= (x;)

x : spatial coordinate (in R2 )

: parameter (in RN ) : R2 x RN function (represents the selected class

of transformations)

Im: space of images

d: Im x Im -> R functional measuring the discrepancy between the two images

: Im -> R defined as (X) = d(X,R)

c : RN -> R cost functional c() = d(T0 ,R) = (T0 )

Many different possibilities Least square error Human Visual System model distance [Mannos, 74] Structural similarity index SSI [Wang et al, 04, Brunet, Vrscay, Wang 11] Besov functional norm Besov norm + divisive renormalization (generalised

SSI) [S.B., G. Maggi, 14] Mutual information [Viola, Wells 97; Thevenaz, Unser 98] ...

Find in RN = arg min c() Unconstrained optimization problem

Several possible optimization algorithms Choice depends on the characteristics of

Transformation Distance functional

Need rules to Evaluate images out of their domain of definition

(extrapolation) Compute values of (deformed) images at pixels

(interpolation) Compute the values of derivative of images at

pixels (numerical differentiation)

Many different possibilities

[Citare: ....]

X,Y quantized random variables

measure of the mutual dependence of X and Y

====

===ij ji

jiji yYpxXp

yYxXpyYxXpYXMI

][][],[

log],[),(

Experiment: choose a pixel randomly

T -> XT: gray level of T R -> XR: gray level of R

Compute MI(XR,XT)

Need to estimate associated probabilities

Quantized RV: histograms

Probabilities

pix

TiT N

ihxXp )(][ ==pix

RiR N

ihxXp )(][ ==

pix

TjRiT N

jihxXxXp ),(],[ ===

})(:{#)( iT xpixTpixih == })(:{#)( iR xpixRpixih ==

})(,)(:{#),(, jiTR xpixTxpixRpixjih ===

Domain of trivial histogram is

Trivial histogram gives rise to a discontinuos cost function

Image is sampling of an continuous image Domain of continuous histogram is [0,255]

Estimate (a discretized version of) the

continuous histogram and relative probability

{ }255,,1,0

[Parzen, 1962] Gaussian window Continuous joint histogram

Joint probability

dxk

xTkxR

kh TR

=

)()(1),( 2,

2

)( xex =

=

',' ,

,, )','(

),(),(

TR

TRTR h

hp

Mutual information of the two images

Cost function = -MI(R,T)

Extremely robust

Computing the histogram is overly expensive

=

)()(

),(log),(),( ,,

TR

TRTR pp

ppTRMI

Straightforward implementation for cycle over gray level bins look for pixels contributing to the given bin

Take advantage of matlab vectorial operations

for cycle over pixels compute contribution of given pixels to different

bins with vectorial operations

is numerically compactly supported

Computational cost of mutual information

(the computation also takes into account computing the Frchet derivative)

64x64 128x128 256x256 512x512

LS 0.0468 0.1248 0.3588 1.56

MI 1.1544 4.4304 17.7061 72.0101

Histogram 1.1856 4.1808 16.6765 65.9104

[Bovik, Wang 02]

Consider three discrepancies in images Luminance (mean) Contrast (variance) Correlation

Combine the three components in one index measuring the similarity between R and T

T,R images =

iiTN

T 1 =i

iRNR 1

22 )(1

1 = i iTTT

Ns 22 )(

11 = i iR

RRN

s

= i iiRTRRTT

Ns ))((

11

,

Luminance

Contrast

Correlation

122

12),(cTR

cTRTRl++

+=

222

22),(csscssTRc

TR

TR

+++

=

3

3,2),(csscs

TRsTR

TR

++

=

Structural Similarity Index

Simplification ( )

),(),(),(),( TRsTRcTRlTRSSIM =

2/32 cc =

),(),(

22),(

21

222

2,

122

1

TRSTRScss

cscTR

cTRTRSSIMTR

TR

=

=+++

++

+=

S1(T,R) and S2(T,R) are not metrics, but

We set

d1(T,R) and d2(T,R) are metrics [Brunet, Vrscay, Wang 12]

),(1),( 11 RTSRTd =),(1),( 22 RTSRTd =

Straightforward calculation yields

( , )

122

12

21 ),(

cRT

cRTRTd

++

+=

2

2020

2

2002

2 ),(cRT

cRTRTd

++

+=

TTT =0 RRR =0

Combine d1 and d2 to obtain a class of metrics

Parameters p1 and 1>0 and 2>0

If d1 and d2 are metrics then d is a metric

ppp TRdTRdRTd /12211 )),(),((),( +=

Functional analysis norm, generalizing Sobolev norms

Wavelet decomposition of T

(Tj: component at the frequency band [2j,2j+1])

Besov norm

==j

jj

j TtTj

,

=j

pp

ljjspj

BTT s

p

/1)2/1/1( )22( 2,2

Combine Besov norm with divisive renormalization

Wavelet decomposition of T and R

Generalized SSI distance

p

j

p

jj

jjj

RT

RTRTd

/12/

22

2

),(

+

=

SSI metric

Generalized SSI metric

+

=

j

p

jj

jjj

p

RT

RTRTd

2/

22

2

),(

2/

1

2020

1

200

1

2/

122

12

1),(

pp

p

cRT

cRT

cRT

cRTRTd

++

++

++

+=

Code to test new registration algorithms

Flexible algorithm design allowing to switch one of the ingredient without modifying the others

New distance functionals / image models / transformation classes can be incorporated easily

dxxJxTxTDc );())(();()( =

Most optimization algorithms need to compute

Chain-rule

I II III

c

I. Frchet derivative of the functional

II. Gradient of the image

III. Jacobian (w.r. to the parameter) of the transformation

);( xXD

)(yT

);( xJ

Image module

Transformation module

Distance module

Optimization is dealt with by the Matlab Optimization Toolbox

Input: The parameter

Output:

The grid G obtained by deforming the reference grid via the mapping

The values assumed by the jacobian (w.r. to the parameter ) of the function at the points of the reference grid

Input: A sampled image X A grid G of arbitrary points (representing the

center of the deformed pixels)

Output: An interpolated (or extrapolated) value of X at the

points of G A value of X at the points of G

Input An arbitrary image X sampled at the points of the

reference grid

Output: The value of the cost function c(X) The value at the points of the reference grid of

(the Frchet derivative of the cost functional at X)

To evaluate c()

Call the mapping module to obtain the deformed grid G

Call the image model module to sample T at the grid G -> T = deformed image sampled at the reference grid

Call the distance module to evaluate c() = d(T;R).

To evaluate c Call the mapping module and evaluate J (x; )

at the points of the reference grid Call the image module and evaluate T at the

points of the deformed grid Call the distance module and evaluate D (T,x) at

the points of the reference grid Evaluate the integral by a chosen quadrature rule

Flexible algorithm allowing to switch one of the ingredient without modifying the others

New distance functionals / image models / transformation classes can be incorporated easily

Idea can be extended to the computation of the Hessian

REFERENCE IMAGE TEMPLATE IMAGE

R obtained from T by applying deformation with Photoshop

No noise Distance: LS

No noise Distance: SSIM p=2

No noise Distance: GSSIM p=2

Gaussian noise Distance: LS

Gaussian noise Distance: SSIM p=2

Gaussian noise Distance: GSSIM p=2

Computational cost: evaluation of the cost functional

(*) computed on a machine twice as fast

64 x 64 128x128 256x256 512x512

LS 0.2815 0.35938 0,73438 3.0156

SSIM 0.23438 0.67188 1.8906 6.5625

G-SSIM 1.125 1.5313 2.9531 12.875

MI(*) 2.106 8.0965 31.902 134.41

REFERENCE IMAGE TEMPLATE IMAGE

CT studies, of the same patient by different CT machines

Quality of the results

Unreg. LS SSIM G-SSIM

Pearson 0.245 0.858 0.872 0.926

Overlap 0.492 0.992 0.993 0.992

M1 0.47 0.833 0.846 0.85

M2 0.253 0.868 0.861 0.933

Image registration based on a generalized SSI metricProblemImage segmentationSeveral approachesSeveral approachesAtlas based segmentationAtlasAtlas based segmentationAtlas selectionAtlas selection strategiesSingle individual atlasVariable individual atlasSingle average atlasMultiple individual atlasImage registrationIngredientsThe transformationsParametric transformationsThe distanceThe distanceThe optimization problemImage modelMutual InformationImage as random variableEstimate probabilityContinuous vs quantized imageParzen probability estimationMI of R and TComputing histogram in MatlabComputational costThe distance: SSIMSSIDiscrepancyStructural Similarty IndexThe SSI metricsThe SSI metricsThe SSI metricsThe Besov normThe generalized SSI metricCompare the two metricsCode: Modular frameworkGradient of the cost functionalThree componentsThree modulesTransformation moduleImage moduleThe distance moduleCombining the modulesCombining the modulesModular registration frameworkTestsResultsResultsResultsResultResultResultTestsTestsTestsTests