Purely evidence-based multi-scale cardiac tracking using optic flow

Hans van Assen1, Luc Florack1,

Avan Suinesiaputra2, Jos Westenberg2,

Bart ter Haar Romeny1

1 Biomedical Image Analysis, Biomedical Engineering, Technical University Eindhoven, Eindhoven, Netherlands

2 Div. Image Processing, Dept. Radiology, Leiden University Medical Center, Leiden, Netherlands

Contents Introduction Tissue Function Tagged MRI Proposed Method

Classical Optic Flow Constraint Equation Multi-scale Optic Flow Constraint Equation Application to Image Tuples

Results Conclusion

Introduction

Cardiac pathologies can alter LV contraction patterns, e.g.: valvular aorta stenosis myocardial infarction hyperobstructive cardiomyopathy

Aim: extract and analyse local cardiac tissue function from MR image sequences

Global tissue function Global tissue function often measured

based on contours using CINE data

Wall thickness Wall thickening Stroke volume Ejection fraction Wall dynamics

Local tissue function

Local tissue function can be derived from CINE data using

Nonrigid registration1

Deformable models2

Drawback: local function is derived from global observations + interpolation

1 Chandrashekara et al, LNCS 3504, 20052 Bistoquet et al, IEEE TMI 26(9), 2007

G. Hautvast, TU/e-BME - Philips MS

Local function from tagged MRI Zerhouni3 introduced tagging

in 1988

Axel4 introduced SPAtial Modulation of Magnetisation (SPAMM) in 19893 Zerhouni et al, Radiology 169, 19884 Axel et al, Radiology 172, 1989

Tagged MRI Tagging pattern inherent in the tissue

moves along with tissue Enables local motion analysis

Motion Analysis from Tagging Sparse analysis

followed by interpolation and regularisation:

Finite Element Models5,6

“Virtual tags”7

Constraints: Motion field smoothness Tissue incompressibility

5 Young, Med Image Anal 3(4), 19996 Haber et al, LNCS 2208, 20027 Axel et al, LNCS 3504, 2005

Motion Analysis from Tagging

Dense analysis by

Optic Flow8,9

HARmonic Phase (HARP)10

Multiscale Optic Flow11,12

Nonrigid registration13,14

8 Prince & McVeigh, IEEE TMI 11(2), 19929 Gupta & Prince, 14th Int. Conf. IPMI, 199510 Osman et al, MRM 42, 199911 ter Haar Romeny, Front-End Vision & Multi-Scale Analysis, Springer 200412 Suinesiaputra et al, LNCS 2878, 200313 Sanchez-Ortiz et al, LNCS 3749, 2005 14 Chandrashekara et al, LNCS 3504, 2005

Proposed method

Novelties Multi-scale OFCE14,* on two time-synchronous

sequences with perpendicular tags No aperture problem Multi-scale paradigm Automatic scale selection, per pixel Sine HARP angle images

14 Florack et al, IJCV 27(3), 1998* OFCE = optic flow constraint equation

Classical OFCE (Horn & Schunck15)

Assumption (L is intensity, t is time)

And when

Using 0-order Taylor expansion (subtract L)

(notice: 1 equation and 2 unknowns, u and v)15 Horn and Schunck, Artif. Intell. 17, 1981

The isophote landscape of an image changes drastically when we change our aperture size. This happens when we move away or towards the scene with the same camera. Left: observation of an image with = 1 pix, isophotes L=50 are indicated. Right: same observation at a distance twice as far away. The isophotes L=50 have now changed.

scalarflow densityflow

Scalar images: intensity is kept constant with the divergenceDensity images: intensity ‘dilutes’ with the divergence

Two types of images need to be considered:

The Lie derivative (denoted with the symbol v) of a

function Fg with respect to a vectorfield v is defined as

vFg. The optic flow constraint equation (OFCE) states

that the luminance does not change when we take the

derivative along the vectorfield of the motion:

vFg 0

vFg F.v

v Div v v. 0

Multi-scale optic flow constraint equation:

For scalar images:

For density images:

The velocity field is unknown, and this is what we want to recover from the data. We like to retrieve the velocity and its derivatives with respect to x, y, z and t. We insert this unknown velocity field as a truncated Taylor series, truncated at first order.

‘Spurious resolution’: artefact of the wrong aperture

What is the best aperture?

Aliasing, partial volume effect

Regularization is the technique to make data behave well when an operator is applied to them. A small variation of the input data should lead to small change in the output data.

Differentiation is a notorious function with 'bad behaviour'.

2 4 6 8 10 12

4

3

2

1

1

2

3

10 20 30

1

0.5

0.5

2 1 1 2

0.2

0.4

0.6

0.8

2 1 1 2 3 4 5

0.2

0.4

0.6

0.8

1

Some functions that can not be differentiated.

The formal mathematical method to solve the problems of ill-posed differentiation was given by Laurent Schwartz:

A regular tempered distribution associated with an image is defined by the action of a smooth test function on the image.TL

Lxx x

i1...inTL 1n

Lxi1...inx xThe derivative is:

Fields Medal 1950 for his work on the theory of distributions. Schwartz has received a long list of prizes, medals and honours in addition to the Fields Medal. He received prizes from the Paris Academy of Sciences in 1955, 1964 and 1972. In 1972 he was elected a member of the Academy. He has been awarded honorary doctorates from many universities including Humboldt (1960), Brussels (1962), Lund (1981), Tel-Aviv (1981), Montreal (1985) and Athens (1993).

Laurent Schwartz (1915 - 2002)

Mathematics Smooth test function

Computer vision Kernel, filter

Biological vision Receptive field

xL0x, y G x, y; L0x, y

xG x, y;

Multi-scaledifferentialoperators

Multi-scale OFCE Florack, Niessen and Nielsen came up with two new

notions: the optic flow constraint equation is of an observed physical

system: Gaussian differential operators the velocity can be different in every pixel, so: the derivative with

respect to the (unknown) velocity field must be zero. The derivative with respect to a velocity field is called a

Lie-derivative. The Lie derivative of a function F with respect to a

vectorfield must be zero: vF 0

For scalar images:

For density images:

vF F.v 0

v

.v v v 0

The multi-scale OFCE

For scalar images:

For scalar images the observed (convolved with the aperture function) the optic flow constraint equation (OFCE) is written as:

F.vg x 0

from which we get by partial integration:F

.g v x 0

Lx x ', y y ', t t 'gx, y, t. vx, y, t x ' y ' t ' 0

, or

.vg x 0

The multi-scale OFCE

For density images:

For density images the observed (convolved with the aperture function) the optic flow constraint equation (OFCE) is written as:

from which we get by partial integration: , or

g . v x 0

Lx x ', y y ', t t 'gx, y, tvx, y, t x ' y ' t ' 0

Approximation of the velocity field

The velocity field {u,v} is unknown. We will approximate it, put it in the equation and solve for it.

We can approximate it to zero'th order:

and to first order:u t ut x ux y uy, v t vt x vx y vy, w t wt x wx y wyu, v, w

u, ux, uy, ut, v, vx, vy and vt

We have 8 unknowns, and need equations to solve them (per pixel):

vF 0 Lxg. v 0

vFx 0 Lxx g. v 0

vFy 0 Lxy g. v 0

vF 0 Lxt g. v 0

Four equations are given by:

The remaining 4 have to come from external information.

The normal constraint in 2D is expressed as

n.v0 11 0L.v 0 or v Lx u Ly 0

where Lx and Ly are constant. So we get four more equations:

v Lx u Ly 0

x v Lx u Ly 0

y v Lx u Ly 0

t v Lx u Ly 0

This gives 8 equations with 8 unknowns to be solved in every pixel:

But:

Case: density images. Note the third order derivatives and scales σ.

Suinesiaputra et al. 2005

Limitation 1

Assumption does not hold, due to

spin-lattice relaxation (T1)

Harmonic Phase techniqueSpatial domainSPAMM images

Fourier domainfilter first harmonic peak

Spatial domain Sine of HARP angle images

Limitation 2

Twice as many unknowns as equations(aperture problem)

Limitation 2

Proposed “normal flow constraint” 11

is erroneous

11 Suinesiaputra et al, LNCS 2878, 2003

so we may not add the second second set of 4 equations ...

Normal flow

Flow direction is colour-coded.Should be one color during horizontal movement and one color during vertical movement.

Ball moving first horizontally and then vertically. Gradient on the ball in radial directions.

1st order multiscale OFCE

Application to image tuples

Image 1

Image 2

Images 1 & 2

with

Application to image tuples

Application to image tuples

Solve for every pixel in every frame in multiple scale space

automatically select proper scale:how far is the coefficient matrix off from being singular?

Automatic scale selection

Stability of matrix C can be calculated with (Squared) Frobenius norm:

Condition number:

is an eigen value of C

C is a m x n matrix

Scale selection:

The condition number of the coefficient matrix exhibits an optimum over scale in many pixels, given the local density of texture.

0 5 10 15 20scaleindex

0.1

1

10

100

norm

0.8

3.0

Artificially created test image sequencefor validation purposes

Scale selection map(Frobenius norm)

Automatic scale selection Using condition number

Spatial scale in

vertical tagging

image

Spatial scale in

horizontal tagging

image

Temporal scale

Results

SPAMMSine of HARPMasked sine of HARP

Tuple Optic Flow vs Normal FlowTuple Optic Flow

Tuple Optic Flow vs PC-MRI

Conclusion We developed a tracking method that

Works at pixel resolution Yields displacements and their differential

structure (important for strain, strain rates, tissue acceleration)

Is straightforwardly extensible to 3D Uses multi-scale paradigm with automatic

scale-selection Does not need constraints regularization

Future work

Evaluate quantitatively on large data set Compute strain, strain rate, tissue

acceleration Extend to 3D using true 3D data

(currently implemented) Classify strain patterns and tissue function Accelerate computation

Acknowledgements The Netherlands Organisation for Scientific Research

(NWO) is greatfully acknowledged for financial support of Luc Florack, PhD (VICI award)

BSIK is greatfully acknowledged for financial support of Hans van Assen, PhD

BioMedical ImageAnalysis Group,TU Eindhoven