A. Criminisi, T. Sharp and K. Siddiqui

Properties of our algorithmProperties of our algorithm

• efficient on high-res./nD images (~milliseconds)• easy to edit and fix• accurate (e.g. handling of thin structures)• robust to noise• edge sensitive• captures uncertainty (probabilistic output)

State of the art segmentation State of the art segmentation algorithmsalgorithms

• QPBO• Tree-reweighted mess. passing• Random Walker• Min-cut/max-flow• Belief propagation• Level Sets• Geodesic active contours• Region growing• K-means

Most based on complex energy minimization -> slow.

Input image User-entered brush strokes Likelihood of Fg v Bg

This is noisy!In order to obtain segmentation we need to encourage spatial smoothness

Notation

Point position

Image intensities

Output segmentation

Appearance likelihood

Appearance likelihood is computed from histograms of intensities accumulated under the two brush strokes

Legend• Green indicates high probability of Fg• Red indicates high probability of Bg• Grey for uncertain

A very efficient way of propagating image information around

Image Binary mask

unit tangent vector

Geodesic distance (Euclidean for =0)

Coronal CT view of r. kidney User-drawn brush stroke Binary mask of brush stroke Gradient and tangent vectors

Output geodesic distance D(p)

Output geodesic distanceOutput geodesic distance

Forward pass: (top-left to bottom-right)

Backward pass: (bottom-right to top-left)

Properties of algorithm • Contiguous memory access• Parallelizable• GPU-friendly

• no region growing• no FMM• no level set

InputInput

Image (8bpp) after W/L mapping!

GDT raster scan algorithmGDT raster scan algorithm

with

GPU algorithm implemented on NVidia processors using the CUDA language.GPU algorithm implemented on NVidia processors using the CUDA language.

Typical CT image resolution = 512 X 512

Timings

Downward pass:

with

…similarly for the other three passes.

The downward pass. The red column shows pixels processed by the current thread. The distance values in the top (green) row Have already been computed. Distances along the arrow directions Are computed from texture reads as in the equation on the left panel.

In 2D four passes are necessary: top-bottom, bottom-top, left-right, right-left

Downward pass:

1 2 3y

x

1 2 3

4 5 6

7 8 9

dire

ction

of ra

ster

sca

n

GPU algorithm implemented on NVidia processors using the CUDA language.GPU algorithm implemented on NVidia processors using the CUDA language.

Pass 1 of 6:In 3D six passes are necessary.

with

Pass 1 of 6:

y

xz…similarly for the other five passes.

How do we impose smoothness ?

Input binary mask M

withSigned distance from boundary

Signed distance Ds from boundary (in green) Signed distance Ds (zoomed)

For ease of explanation we focus on a toy 2D example here.

Geodesic morphology

Signed distance Ds from boundary (green)Input binary mask M

Geodesi

c d

ilati

on

Geodesi

c ero

sion

Eroded mask

Dilated mask

In each of the eroded and dilated masks part of the noisehas been removed. Now we need to combine the two so asto remove all of the noise.

Geodesic morphology, real example

Th

is is all e

xtre

mely

fast to

com

pu

te.

Distance Ds

Geodesic erosion

Geo. dilation , small d Geo. dilation , large d

Eroded maskDilated mask

Final, symmetric signed distance

Symmetric signed distance

The new distance is much smoother than the original one because the effect of noise has been reduced.

Original signed distance

The GSF operator produces the final, noise-free mask Ms as:

Input binary mask M Symmetric signed distance Thresholding at 0 Final noise-free mask

More examples with different noise patterns

Input Distance Filtered image Input Distance Filtered image

Binary case was

Real-valued caseinput GSF

GGDT:GGDT:GeneralizedGeneralizedGeodesic DistanceGeodesic Distance

Ms: output segmentation

GSF( )

Ms: Output of GSF for small

Ms: Output of GSF for large

smoothsmooth

lockedlocked

Larger values of produce smoother segmentations

The actual algorithm

Output of GSF for fixed

Appearance lik.

Soft input mask

Distance term with

Data likelihood

I. Compute pixel-wise likelihoods from user hints

combine

User brushes Appearance likelihood L a Distance term L d

II. Segmentation from pixel-wise likelihoods

Output of GSF for fixed

Geodesic symmetric distance

Segmentation via GSF

GSF( )

Appearance likelihood La

Output segmentation Ms

Distance signal Ld

Aortic aneurism

The aorta, bottom of heart and pelvis have been segmented in 3D by our technique.The whole process (including user interaction) takes only a few seconds.

Three views of the segmented aorta. Bony structures (Bg) are shown faded to provide spatial context. Input is CT.

Carotid arteries

The carotids have been segmented interactively by our technique. Segmenting such long and thin structures is usually a problem for other existing algorithms.

Three views of segmented carotids. Bony structures (Bg) are shown faded to provide spatial context. Input is CT.

Studying the interaction between aorta and spine

The aorta and its thin bifurcations have been segmented by our technique in only a few seconds.

Three views of segmented aorta. The spine and heart (Bg) are shown faded to provide spatial context. Input is CT.

A liver tumor

The tumor has been segmented with a single user click. The whole operationhas taken only a few milliseconds. Now that the tumor has been isolated statistics about itsdensity, shape and texture are easily computed (right panel).

Axial CT slice. The user segments a tumor in only a few milliseconds Automatic measurements of tumor

Segmenting different structures in MR images

Segmentation of noisy images

Very noisy input image. Input is CT. Our segmentation

Encouraging spatial smoothness is especially useful when dealing with very noisy images.

Very noisy input image. Input is MR Our segmentation

Segmentation of noisy images

Encouraging spatial smoothness is especially useful when dealing with very noisy images.

Testing accuracy in low-contrast images

Very low-contrast input image Our segmentation

The geodesic term in the definition of the geodesic distance enables segmentation in low contrast images.

Capturing uncertainty in low-contrast regions

In low gradient regions the algorithm correctly returns low confidence segmentation

In low gradient regions the algorithm correctly returns low confidence segmentation

Sharp segmentation (high confidence) becauseof strong gradients

Sharp segmentation (high confidence) becauseof strong gradients

Images and ground-truth from the standard GrabCut test dataset Not much of a difference in terms of accuracy.

[ Rother, C. Kolmogorov, V and Blake, A. GrabCut: Interactive foreground extraction using iterated graph cut. SIGGRAPH 2004. ][Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., Rother, C.: A comparative study of energy minimization methods for markov random fields. In: ECCV. (2006)]

• up to 60X speed up compared to min-cut.• with similar accuracy

[ Rother, C. Kolmogorov, V and Blake, A. GrabCut: Interactive foreground extraction using iterated graph cut. SIGGRAPH 2004. ]

[ Bai, X. and Sapiro, G. A geodesic framework for fast interactive image and video segmentation and matting. ICCV 2007, Rio, Brasil. ]

• up to 30X speed up compared to Bai et al., while avoiding topology issues.• also, Bai et al. do not impose spatial smoothness