revisiting horn and schunck: interpretation as gauß-newton...

MICCAI 2010 Tutorial

Intensity-based Deformable Registration

Similarity Measures

Christian Wachinger

Computer Aided Medical Procedures (CAMP), TU München, Germany

Quantify the similarity between images

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 2

• Do images contain the same object?

Image retrieval

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 3

• How are images correctly aligned?

Image registration

Quantify the similarity between images

Requirements on similarity measure

• Extremum for correctly aligned images

• Smooth, best convex

• Fast computation

• Differentiable

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 4

Optimization

Energy ModelTransformation

Model

Transfo.

Source Image

Target Image

Difference Measure

Regularization Term

Overview

1. Standard similarity measures

2. Probabilistic framework

3. Pre-processing steps

4. Recent approaches

5. Linear vs. non-linear registration

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 5

Standard Similarity Measures

PART I

9/24/2010 6MICCAI 2010 Tutorial: Intensity-based Deformable Registration

• SSD-SAD

• Cross Correlation

• Mutual information

• Derivatives

Difference Measures

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 7

Volume X Volume T(Y)

i

ii yxN

SSD 2)(1

Sum of Squared Differences

i

ii yxN

SAD1

Sum of Absolute Differences:

Less sensitive on large intensity

differences than SSD

Volume X Volume T(Y)

Limitations of SSD

• Illumination change affects similarity function

• Idea: normalization of images

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 8

SSD

rotation

SSD

rotation

SSD on Normalized Images

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 9

Normalized Cross Correlaiton (NCC)

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 10

Volume X Volume T(Y)

i yx

ii yyxx

NNCC

))((1

Normalized Cross Correlation:

Expresses the linear relationship between voxel

intensities in the two volumes

pixels ofNumber :

deviation Standard:

Mean:

N

x

x

NCC - example

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 11

Source: wikipedia

Sets of (x,y) points, with the NCC of x and y for each set.

Multi-Modal Registration

• More complex intensity relationship

• Approaches:

– Simulate one modality from the other one

– Apply sophisticated similarity measures

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 12

?

CT MR

Information Theoretic Approach

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 13

Histogram calculation

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 14

Image Histogram

Bins

4 3 3 6

1/4 3/16 3/16 3/8

counts

probs

Joint histogram calculation

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration15

Image X Image Y

Y

X

1

1

1

2 1 3

Overlap

Joint Histogram

Information Theoretic Approach

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 16

Source: W. Wells, MICCAI 2009

Registered Not Registered

Joint Histogram

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 17

px(i)

py(i)

pxy(i,j)

px(i)

py(i)

pxy(i,j)

X and Y identical X and Y misaligned

Joint Histogram

• Histogram for images from different Modalities

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 18

Joint HistogramTarget Image

Source ImageNot Aligned

Aligned

Source: PhD Thesis, L. Zöllei

Joint Histogram

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 19

Intensities of Source X

Intensities of Target Y

SSD OptimumY = X

NCC OptimumY = a*X + b

Information Theoretic Approach

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 20

• How to quantify the quality of alignment between MR and CT?

Measure the structure of the joint distribution

• How to measure the structure?

Shannon Entropy

Source: W. Wells, MICCAI 2009

Entropy

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 21

Shannon Entropy, developed in the 1940s(communication theory)

i

ii ppH log

i

pi

i

pi

uniform distribution→ maximum entropy

any other distribution→ less entropy

Maximize or minimize entropy?

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 22

Registered Not Registered

Mutual Information (MI)

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 23

i j yx

xy

xyjpip

jipjip

YXHYHXHYXMI

)()(

),(log),(

),()()(),(

• Maximized if X and Y are perfectly aligned

• H(X) and H(Y) help to make the measure more robust

• Maximization of mutual information leads to minimization of joint entropy

Historical Note

• Minimum Entropy Registration– Collignon A., Vandermeulen, D., Suetens, P., and Marchal, G. 3D

multi-modality medical image registration using feature space clustering. CVRMED April 1995.

• Maximum Mutual Information Registration– Viola, P. and Wells, W.. Alignment by maximization of mutual

information. In Proceedings of the 5th International Conference of Computer Vision, June 20 – 23, 1995.

– Collignon A, Maes F, Delaere D, Vandermeulen D, Suetens P, Marchal G, Automated multi-modality image registration based on information theory. IPMI June 26, 1995.

– Viola, P. Alignment by maximization of Mutual Information. MIT PhD Thesis, June 1995.

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 24

Normalization of MI

• Problem: changing overlap affects MI

• Entropy Correlation Coefficient (ECC)

• Normalized MI (NMI)

• Revisiting overlap invariance

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 25

C. Studholme, D.L.G.Hill, D.J. Hawkes, An Overlap Invariant Entropy Measure of 3D Medical Image Alignment, Pattern Recognition, Vol. 32(1), Jan 1999, pp 71-86.

Nathan D. Cahill, Julia A. Schnabel, J. Alison Noble, David J. Hawkes, "Revisiting overlap invariance in medical image alignment," MMBIA, 2008

Maes F, Collignon A,Vandermeulen D, Marchal G, Suetens P. Multimodality image registration by maximization of mutual information, TMI, 1997

NMI – change of overlap

• Two images

• T2 has higher information content

• Overlap changes for deformable registration?

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 26

m n

C. Studholme, D.L.G.Hill, D.J. Hawkes, An Overlap Invariant Entropy Measure of 3D Medical Image Alignment, Pattern Recognition, Vol. 32(1), Jan 1999, pp 71-86.

Improvements to MI

• Density estimation– Parzen window– Partial volume distribution– Uniform volume histogram– NP windows

• Spatial information• Alternative entropy measures

• Tutorial at MICCAI 2009: Information theoretic similarity measures for image registration and segmentation: Maes, Wells, Pluimhttp://ubimon.doc.ic.ac.uk/MICCAI09/a1882.html

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 27

Derivatives of Similarity Measures

• General form of derivative of similarity metrics

– SSD:

– MI

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 28

Hermosillo, G., Chefd'Hotel, C., Faugeras, O.: Variational Methods for Multimodal Image Matching, International Journal of Computer Vision 50(3) (2002)

Derivatives of Similarity Measures

• General form of derivative of similarity metrics

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 29

dxI3 dyI3

dxI2 dyI2

dxI1 dyI1

Derivatives of Similarity Measures

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 30

Images

X

Y

•

Update

=

=

dxI3 dyI3

dxI2 dyI2

dxI1 dyI1

Derivatives of Similarity Measures

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 31

Images

X

Y

•

Update

=

=

dxI3 dyI3

dxI2 dyI2

dxI1 dyI1

Derivatives of Similarity Measures

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 32

Images

X

Y

•

Update

=

=

dxI3 dyI3

dxI2 dyI2

dxI1 dyI1

Overview

1. Standard similarity measures

2. Probabilistic framework

3. Pre-processing steps

4. Recent approaches

5. Linear vs. non-linear registration

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 33

Probabilistic Framework for Image Registration

PART II

9/24/2010 34MICCAI 2010 Tutorial: Intensity-based Deformable Registration

• Viola, PhD thesis, 1995

• Roche et al., 2000

• Zöllei, PhD thesis, 2006

Probabilistic Framework for registration

• Maximum Likelihood Estimation (MLE)– Probability for the model m having the observations a

– Log-likelihood function

• Formulate registration as likelihood maximization

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 35

MLE framework for registration

• Model

• Probability function

• Log-likelihood function

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 36

• Stationary white Gaussian noise

• Log-likelihood with Gaussian noise

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 37

Assuming f to be the identity

• Stationary white Gaussian noise

• Log-likelihood with Gaussian noise

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 38

SSD

Intensity relationships

• Identity : SSD

• Affine : NCC

• Functional : Correlation Ratio

• Statistical : Mutual Information

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 39

Viola, P. Alignment by maximization of Mutual Information. MIT PhD Thesis, June 1995.Roche et al., Unifying maximum likelihood approaches in medical image registration, 2000

Pre-Processing Steps

PART III

9/24/2010 41MICCAI 2010 Tutorial: Intensity-based Deformable Registration

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 42

Similarity Measure

Optimization

Registration FrameworkImages

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 43

Similarity Measure

Optimization

Registration FrameworkImages

?

Pre-processing

1. Image gradients

2. Entropy images

3. Phase

4. Multi-resolution

5. Attribute vectors

Image Gradients

• Normalized gradient fields

• Maximize square of cosine

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 44

Haber, E. and Modersitzki, J., Intensity gradient based registration and fusion of multi-modal images, MICCAI 2006

“two images are considered similar, if intensity changes occur at the same locations“

Image Gradients

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 45

Synthetic images Gradient images

Image Gradients

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 46

Gradient of smoothed images Gradient images

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 47

Similarity Measure

Optimization

Registration FrameworkImages

?

Pre-processing

1. Image gradients

2. Entropy images

3. Phase

4. Multi-resolution

5. Attribute vectors

Entropy Images

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 48

0 2 4 6 8 10 12 14 16 180

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0 2 4 6 8 10 12 14 16 180

0.02

0.04

0.06

0.08

0.1

0.12

0.14

H

H

Input Image Patches PDF Entropy Image

Wachinger, Navab, Structural Images for Image Registration, MMBIA, 2010.

Entropy Images

• Examples from the RIRE dataset http://www.insight-journal.org/rire/

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 49

CT T1 T2 PD

Original

Entropy

Wachinger, Navab, Structural Images for Image Registration, MMBIA, 2010.

Entropy Images

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 50

Synthetic images Entropy images

Wachinger, Navab, Structural Images for Image Registration, MMBIA, 2010.

Entropy Images

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 51

Entropy on smoothed images Entropy images

Wachinger, Navab, Structural Images for Image Registration, MMBIA, 2010.

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 52

Similarity Measure

Optimization

Registration FrameworkImages

?

Pre-processing

1. Image gradients

2. Entropy images

3. Phase

4. Multi-resolution

5. Attribute vectors

Phase-base registration

• Fourier representation

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 53

M. Mellor and M. Brady. Phase mutual information as a similarity measure for registration. Medical Image Analysis, 2005.

Dashed: MISolid: Phase MI

Optical Flow with Phase

• Idea: replace the assumption of brightness constancy with phase constancy

• Error for chaning noise and illumination

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 54

L. Wietzke, G. Sommer, O. Fleischmann, The Geometry of 2D Image Signals, CVPR, 2009

blue: gradientred: phase

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 55

Similarity Measure

Optimization

Registration FrameworkImages

?

Pre-processing

1. Image gradients

2. Entropy images

3. Phase

4. Multi-resolution

5. Attribute vectors

Multi-Resolution Registration

• Perform registration on multiple resolutions

1. Smooth

2. Downsample

• Advantages:

– Speed: down-sampled images

– Convergence: smoother cost func

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 56

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 57

Similarity Measure

Optimization

Registration FrameworkImages

?

Pre-processing

1. Image gradients

2. Entropy images

3. Phase

4. Multi-resolution

5. Attribute vectors

Attribute Vectors

• Registration with attribute vectorsa = [ gradient, intensity, GMI ]

• GMI = geometric moment invariants

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 58

Shen, D. and Davatzikos, C., HAMMER: hierarchical attribute matching mechanism for elastic registration, IEEE Transactions on Medical Imaging, 2002M. Wacker and F. Deinzer. Automatic robust medical image registration using a new democratic vector optimization approach with multiple Measures, MICCAI, 2009

Brain Image GMI self-similarity

Recent Approaches

PART IV

9/24/2010 59MICCAI 2010 Tutorial: Intensity-based Deformable Registration

Recent examples...

• mainly compare to MI as state of the art

• address the following problems of MI

– intensity does not represent tissue (e.g. US-CT)

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 60

CT US US-CT

Recent examples...

• mainly compare to MI as state of the art

• address the following problems of MI

– intensity does not represent tissue (e.g. US-CT)

– intensity non-uniformity (bias)

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 61

Minimizing Residual Complexity

• Motivation: MI fails for these images

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 62

Myronenko A., Song X.: "Intensity-based Image Registration by Minimizing Residual Complexity", IEEE Trans. on Medical Imaging, 2010

Model

Minimizing Residual Complexity

• Compress the difference image

• Similarity measure

• Related to entropy of difference image

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 63

Myronenko A., Song X.: "Intensity-based Image Registration by Minimizing Residual Complexity", IEEE Trans. on Medical Imaging, 2010

Buzug et al.: Image registration: convex weighting functions, CVRMed, 1997

Local and Global Statistics

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 64

Z. Yi and S. Soatto. Nonrigid Registration Combining Global and Local Statistics, CVPR, 2009.

global local

Local and Global Statistics

• Combining local and global densities

• : manually, emphasize on local or global influence

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 65

Zhuang, X. and Hawkes, D. and Ourselin, Unifying Encoding of Spatial Information in Mutual Information for Nonrigid Registration, IPMI, 2009Loeckx, Slagmolen, Maes, Vandermeulen, Suetens, Nonrigid image registration using conditional mutual information, IPMI, 2007

Linear vs. Non-Linear Registration

PART V

9/24/2010 66MICCAI 2010 Tutorial: Intensity-based Deformable Registration

Linear vs. Non-Linear Registration

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 67

dxI3 dyI3

dxI2 dyI2

dxI1 dyI1

Variational

Affine

• Deformable registration is more sensitive to errors from poor modeling

• Averaged out in linear registration

x1

xn

y1

yn

x1

xn

y1

yn

1

1

1

1

...

...

......

...

...

11

11

11

2 * 2562

2 *

25

62

6

2 *

25

62

•

2 * 2562

Linear vs. Non-Linear Registration

• Choice of similarity measure has influence on the optimization strategy

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 68

fy

fz

fx

hy

hz

hx

=

EXAMPLE: Gauß-Newton Optimization with SSD

=

Linear vs. Non-Linear Registration

• Overlap invariance only for linear ?

– MI vs. NMI

– Joint Entropy vs. MI

• Transition between linear and non-linear

– Parameterization

– Regularization

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 69

Summary

• A large variety of concepts for measuring the similarity is available in the literature

• Choose similarity measure that best fits the application

• Appropriate choice is more important for deformable registration

9/24/2010 MICCAI 2010 Tutorial: Intensity-based Deformable Registration 70

9/24/2010 71MICCAI 2010 Tutorial: Intensity-based Deformable Registration