visualization and haptics for interactive medical image analysis171287/fulltext01.pdf ·...
TRANSCRIPT
ACTAUNIVERSITATISUPSALIENSISUPPSALA2008
Digital Comprehensive Summaries of Uppsala Dissertationsfrom the Faculty of Science and Technology 386
Visualization and haptics forinteractive medical image analysis
ERIK VIDHOLM
ISSN 1651-6214ISBN 978-91-554-7067-8urn:nbn:se:uu:diva-8409
Dissertation presented at Uppsala University to be publicly examined in Häggsalen,Ångströmlaboratoriet, Polacksbacken, Uppsala, Friday, February 8, 2008 at 10:15 for thedegree of Doctor of Philosophy. The examination will be conducted in English.
Abstract
Vidholm, E. 2008. Visualization and Haptics for Interactive Medical Image Analysis.(Visualisering och Haptik för Interaktiv Medicinsk Bildanalys). Acta UniversitatisUpsaliensis. Digital Comprehensive Summaries of Uppsala Dissertations from the Faculty ofScience and Technology 386. 83 pp. Uppsala. ISBN 978-91-554-7067-8.
Modern medical imaging techniques provide an increasing amount of high-dimensional andhigh-resolution image data that need to be visualized, analyzed, and interpreted for diagnosticand treatment planning purposes. As a consequence, efficient ways of exploring these imagesare needed. In order to work with specific patient cases, it is necessary to be able to workdirectly with the medical image volumes and to generate the relevant 3D structures directly asthey are needed for visualization and analysis. This requires efficient tools for segmentation,i.e., separation of objects from each other and from the background. Segmentation is hard toautomate due to, e.g., high shape variability of organs and limited contrast between tissues.Manual segmentation, on the other hand, is tedious and error-prone. An approach combiningthe merits from automatic and manual methods is semi-automatic segmentation, where theuser interactively provides input to the methods. For complex medical image volumes, theinteractive part can be highly 3D oriented and is therefore dependent on the user interface.This thesis presents methods for interactive segmentation and visualization where true 3D
interaction with haptic feedback and stereo graphics is used. Well-known segmentationmethods such as fast marching, fuzzy connectedness, live-wire, and deformable models, havebeen tailored and extended for implementation in a 3D environment where volumevisualization and haptics are used to guide the user. The visualization is accelerated withgraphics hardware and therefore allows for volume rendering in stereo at interactive rates. Thehaptic feedback is rendered with constraint-based direct volume haptics in order to conveyinformation about the data that is hard to visualize and thereby facilitate the interaction. Themethods have been applied to real medical images, e.g., 3D liver CT data and 4D breast MRdata with good results. To provide a tool for future work in this area, a software toolkitcontaining the implementations of the developed methods has been made publicly available.
Keywords: medical image analysis, haptic rendering, volume visualization, interactivesegmentation
Erik Vidholm, Centre for Image Analysis, Box 337, Uppsala University, SE-75105Uppsala, Sweden
© Erik Vidholm 2008
ISSN 1651-6214ISBN 978-91-554-7067-8urn:nbn:se:uu:diva-8409 (http://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-8409)
Distributor: Uppsala University Library, Box 510, SE-751 20 Uppsalawww.uu.se, [email protected]
Till minne av min pappa
List of enclosed papers
This thesis is based on the following papers, which are referred to in the text
by their Roman numerals.
I E. Vidholm, X. Tizon, I. Nyström, and E. Bengtsson. Hapticguided seeding of MRA images for semi-automatic segmentation.In Proceedings of IEEE International Symposium on Biomedical
Imaging (ISBI), pp. 288-291, IEEE, 2004.
II E. Vidholm and I. Nyström. A haptic interaction technique forvolume images based on gradient diffusion. In Proceedings of
WorldHaptics, pp. 336-341, IEEE, 2005.
III F. Malmberg, E. Vidholm, and I. Nyström. A 3D live-wiresegmentation method for volume images using haptic interaction.In Proceedings of Discrete Geometry for Computer Imagery
(DGCI), pp. 663-673, LNCS 4245, Springer Verlag, 2006.
IV E. Vidholm, S. Nilsson, and I. Nyström. Fast and robust semi-automatic liver segmentation with haptic interaction. In Proceed-
ings of Medical Image Computing and Computer-Assisted Inter-
vention (MICCAI), pp. 774-781, LNCS 4191, Springer Verlag,
2006.
V E. Vidholm and I. Nyström. Haptic interaction with deformablemodels for 3D liver segmentation. In Proceedings of MICCAI
Workshop: Interaction in Medical Image Analysis and Visualiza-
tion, pp. 41–48, 2007.
VI E. Vidholm, M. Golubovic, S. Nilsson, and I. Nyström. Accurateand reproducible semi-automatic liver segmentation using hapticinteraction. In Proceedings of SPIE Medical Imaging: Visualiza-
tion, Image-guided Procedures, and Modeling, 2008. In press.
VII E. Vidholm, A. Mehnert, E. Bengtsson, M. Wildermoth, K.
McMahon, S. Wilson, and S. Crozier. Hardware acceleratedvisualization of parametrically mapped dynamic breast MRIdata. In Proceedings of MICCAI Workshop: Interaction in
Medical Image Analysis and Visualization, pp. 33–40, 2007.
VIII E. Vidholm, F. Malmberg, I. Nyström, and E. Bengtsson. A toolkitfor interactive medical image visualization and segmentation withhaptics. Submitted for journal publication.
Reprints of Paper I and Paper II was made with kind permissions from the IEEE.
Reprints of Paper III and Paper IV was made with kind permission of Springer Science
and Business Media.
The author has significantly contributed to the work performed in all the papers.
In Paper III, the contribution was mainly on the implementation and in Paper VII
the author designed and implemented the visualization part. For the other papers, the
author has had the major responsibility for method development, implementation, and
writing.
Faculty opponent is Professor Gábor Székely, ETH Zürich.
Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.1 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2 Medical image analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2.1 Computed tomography . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.2 Magnetic resonance imaging . . . . . . . . . . . . . . . . . . . . . . 11
1.3 Interactive segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4 Volume visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.5 Haptic interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.6 Goals of this thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2 Image analysis tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1 Digital images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Spatial filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.1 General smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.2 Bilateral filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.3 Gradient extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Gradient vector flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4 Digital distance transforms . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3 Interactive segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2 Segmentation by region growing . . . . . . . . . . . . . . . . . . . . . . . 21
3.3 Live-wire segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.4 Fast marching segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.4.1 Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.4.2 Fast marching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.5 Fuzzy connectedness segmentation . . . . . . . . . . . . . . . . . . . . . 26
3.5.1 Fuzzy set theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.5.2 Fuzzy connectedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.6 Segmentation by deformable models . . . . . . . . . . . . . . . . . . . . 29
3.6.1 Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.6.2 Simplex meshes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.6.3 Deformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.7 Evaluation of segmentation methods . . . . . . . . . . . . . . . . . . . . 33
4 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.1 Multi-planar reformatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 Volume rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.3 Hardware accelerated direct volume rendering . . . . . . . . . . . . . 38
4.4 Surface rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.5 Stereo graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5 Computer haptics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.1 Haptic devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.2 Haptic displays and software . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.3 Haptic rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.3.1 Haptic surface rendering . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.3.2 Haptic volume rendering . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.3.3 Proxy-based volume haptics . . . . . . . . . . . . . . . . . . . . . . 47
5.4 Volume haptics for medical image segmentation . . . . . . . . . . . 50
6 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.1 Haptics for seeding of region growing algorithms . . . . . . . . . . 51
6.1.1 Seeding of vessels in MRA . . . . . . . . . . . . . . . . . . . . . . . 51
6.1.2 Volume haptics based on GVF . . . . . . . . . . . . . . . . . . . . . 52
6.2 An extension of live-wire to 3D . . . . . . . . . . . . . . . . . . . . . . . . 53
6.3 Interactive deformable models . . . . . . . . . . . . . . . . . . . . . . . . . 56
6.4 Accurate and reproducible liver segmentation . . . . . . . . . . . . . 58
6.5 Visualization of dynamic breast MRI . . . . . . . . . . . . . . . . . . . . 62
6.6 The WISH toolkit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
7.1 Summary of contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
7.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
7.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8 Summary in Swedish . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
1. Introduction
The increasing capabilities of medical imaging devices has strongly facilitated
diagnosis and surgery planning. During the last decades, the technology has
evolved enormously, resulting in a never-ending flow of high-dimensional and
high-resolution data that need to be visualized, analyzed, and interpreted. The
development of computer hardware and software has given invaluable tools
for performing these tasks, but it is still very hard to exclude the human oper-
ator from the decision making. The process of stating a medical diagnosis or
to conduct a surgical planning is simply too complex to fully automate. There-
fore, interactive or semi-automatic methods for image analysis and visualiza-
tion are needed where the user can explore the data efficiently and provide
his or her expert knowledge as input to the methods. The work in this thesis
is about development and application of such methods with an emphasis on
visualization and haptics for improved interaction.
1.1 Thesis outline
The outline of the thesis is as follows. First, the key areas and the specific
goals are summarized in the following introductory sections. In Chapter 2,
the basic image analysis methods used in this work are described. Chapter 3
presents interactive segmentation. Volume visualization and volume haptics
are described in Chapter 4 and Chapter 5, respectively. The scientific contri-
butions of this work are summarized in Chapter 6, and Chapter 7 concludes
the thesis summary with conclusions and suggestions for future work. A sum-
mary in Swedish is given in Section 8.
1.2 Medical image analysis
Digital image analysis is the field of extracting relevant information from dig-
ital images [44, 15]. Besides medicine, which is the application area consid-
ered in this thesis, it is also used in many other different areas in science and
industry, e.g., astronomy, remote sensing, defense, and security. The image
material considered in this work consist of three-dimensional (3D) and four-
dimensional (4D) medical images obtained with either computed tomography
(CT) or magnetic resonance imaging (MRI).
9
1.2.1 Computed tomography
CT is probably the most well-known 3D medical imaging technique
and builds on transmission measurements of X-rays for several different
angles around the object under examination. The X-ray source is located
on a ring that rotates around the object. On the other side of the ring,
opposite the source, there is a detector array. From the transmission
measurements, mathematical image reconstruction is performed to produce
a 2D cross-section image of the object. By repeating this procedure, a stack
of 2D slices that can be regarded as a 3D image is obtained. A common
slice-size is 512×512 pixels. Modern multi-slice CT-scanners are capable of
acquiring multiple slices at a time with the use of up to 64 detector rings.
CT is used for diagnosing a variety of diseases in the brain, chest, heart, and
abdomen. It is also commonly used for imaging of complex fractures. With the
use of contrast agents, the images can be optimized for different organs. CT is
unique in the way that the image values are defined in the absolute Hounsfield
scale. Here, the radiodensity of distilled water at standard pressure and tem-
perature (STP) is defined as zero Hounsfield units (HU) and the radiodensity
of air at STP is defined as -1000 HU. Figure 1.1 shows an example of a CT
image.
Figure 1.1: An axial slice (left) and a coronal slice (right) of an abdominal CT volume.
Dark pixels represent low density and bright pixels represent high density.
10
1.2.2 Magnetic resonance imaging
MRI is a standard medical imaging technique for both anatomical and func-
tional studies. Medical MRI relies on the relaxation properties of excited hy-
drogen nuclei in water and lipids. The object that is about to be imaged is
placed in a static magnetic field (typical strength 1.5 Tesla), resulting in that
the magnetic moment of the atom nuclei aligns with the field direction. By
applying a perturbation with another magnetic field in the form of a radio
pulse, the spin of the nuclei is excited and causes the magnetic moments to
point in the direction of the perturbing field. When the radio pulse is switched
off, the spins will return to their equilibrium state. During this relaxation, ra-
dio frequency signals are emitted. From these signals, that are detected by the
MRI equipment, the time it takes for the magnetic moments to realign with
the static magnetic field can be calculated. Since the relaxation times vary
between tissues, contrast is obtained in the reconstructed images. Figure 1.2
shows examples of MR images1.
Figure 1.2: Left: An axial slice of an MR image of the abdomen. Right: A sagittal
slice of an MR image of the brain.
An important difference between MRI and CT is that MR images reflect
the chemical structure of tissue whilst CT images reflect the radiodensity of
tissue. Therefore, MRI can provide good contrast between soft tissues and is
suitable for brain imaging in order to, e.g., find tumors and detect the small
changes in the brain caused by a stroke. MRI is also commonly used for imag-
ing of the heart, abdomen, and joints. Contrast agents for enhancement of
specific organs are common, e.g., in MR angiography where blood vessels are
imaged. A modern technique for diagnosing breast cancer uses dynamic con-
trast enhanced MRI (DCE-MRI), where the enhancement pattern over time is
analyzed to differentiate between benign and malign tissue.
1The dataset in Figure 1.2 (right) is courtesy of Dirk Bartz, VCM, University of Tübingen,
Germany.
11
1.3 Interactive segmentation
One of the most important steps in medical image analysis is segmentation,
i.e., the process of partitioning the contents of an image into objects of interest
and background in order to facilitate further analysis and information extrac-
tion. Segmentation is needed in diagnostics, therapy monitoring, surgery plan-
ning, and several other medical applications. To manually segment the struc-
tures of interest in medical datasets is a very tedious and error-prone proce-
dure, while fully automatic segmentation is, despite decades of research, still
seen as an unsolved problem. Therefore, many methods are semi-automatic,
i.e., the segmentation algorithms are provided with high-level knowledge from
the user by some means of interaction. A successful semi-automatic method
combines the outstanding capabilities of the human to recognize and locate
objects in images with the computer’s ability to quickly perform tedious and
time-consuming tasks such as counting and exact object delineation. The in-
teractive part is highly dependent on the user interface. Interfaces that rely
on 2D interaction have many drawbacks when the data is 3D, since it is not
straight-forward how to map 2D interaction into 3D space.
1.4 Volume visualization
In order to visualize volume images on a computer screen, some sort of pro-
jection has to be performed. A common and simple way is to show 2D cross-
sections (slices) of the dataset. The slices can be orthogonal to the main axis
of the digitization grid, or arbitrarily positioned and oriented. If several slices
are viewed at once, this is called multi-planar reformatting. Another approach
is to extract and display surfaces representing certain intensity values in the
volumetric dataset. A more general, but also more complex approach, is to
use direct volume rendering (DVR). In this case, different colors and opaci-
ties are assigned to each voxel in the volume in order to visualize overlapping
objects. With modern graphics hardware, DVR can be performed in real-time
on standard workstations.
In order to provide a more realistic 3D visualization, stereo graphics can be
used where two renderings of the volume image are performed, one for the
left eye and one for the right eye. These images are then displayed to the user
by using, e.g., time-multiplexing or color-multiplexing.
12
Figure 1.3: SenseGraphics 3D-IW haptic display with a PHANToM Omni haptic de-
vice. The haptic device is positioned beneath a semi-transparent mirror and the med-
ical image is visualized with time-multiplexed stereo graphics projected through the
mirror in order to obtain co-localization of haptics and graphics.
1.5 Haptic interaction
The word haptic stems from the greek word haptesthai which means “to
touch”. In psychology, haptics is the study of touching behavior and haptic
perception. Haptic perception can be divided into tactile perception and kines-thetic perception. Tactile perception involves the receptors in the skin that al-
low us to feel temperature and pressure in order to determine, e.g., the texture
and roughness of surfaces. The kinesthetic component involves the receptors
in muscles and joints which are used for body control and for determining
weights and friction.
In computer science, haptics refers to the use of tactile and kinesthetic feed-
back as a human-computer interface component, i.e., rendering of virtual ob-
jects with force feedback. The task here is to generate intuitive force feedback
when the user moves the haptic device so that it comes in contact with an
object. An advantage with haptic interaction is that it provides the unique pos-
sibility of simultaneous exploration and manipulation of data. In this work,
the focus is on haptic rendering of medical volume images in order to, e.g.,
feel boundaries between organs and provide input to segmentation algorithms.
The aim with the haptic feedback is to convey more information about the data
than can be obtained with only visual rendering.
13
1.6 Goals of this thesis
The main goal of this thesis is to provide new intuitive ways of interaction for
efficient, accurate and high-precision medical image segmentation by using
a 3D user interface capable of haptic feedback and stereo visualization, see
Figure 1.3. The cover image illustrates two examples of interactive medical
image segmentation with haptic feedback performed in this work.
The use of haptic interaction for segmentation is a young research field with
few contributions. This means that there are no standard methods or widely
spread software packages available to start from. Therefore, to reach the main
goal stated above, the following tasks need to be accomplished:
• Development of new methods and tailoring of existing methods,
where visualization, haptic rendering, and interactive segmentation
are integrated.
• Application and evaluation of the methods on real medical image
data.
• Efficient implementations of the methods in a framework for research
and development. This framework should be possible to extend and
easy to integrate into other software packages.
The emphasis in this work lies on the development, implementation, appli-
cation, and evaluation of the methods. The author has not been involved in
the image acquisition and is not an expert in the medical field. However, the
applied parts of the work involve strong collaborations with research groups
at the Department of Oncology, Radiology, and Clinical Immunology at Upp-
sala University Hospital and the ITEE Biomedical Engineering group at The
University of Queensland, Brisbane, Australia.
14
2. Image analysis tools
In this chapter, a presentation of digital images and the basic image analysis
tools used in this work is given.
2.1 Digital images
A general digital image may be regarded as a discrete integer-valued function
f (x, t,b), where x = (x,y,z) are the spatial coordinates, t is time, and b is the
spectral (color) component. In the work described here, mainly 3D gray-level
images are considered, which will be reflected in the notation and terminol-
ogy. A digital image consists of a finite number of elements, each with a
x
y
z
Δx
Δ
Δ
y
z
Figure 2.1: Left: A 3D image and the commonly used coordinate system. Right: A
voxel with size (Δx,Δy,Δz).
particular location and value. In a digitization grid of size M×N×P, thereare MNP elements and the spatial coordinates x belong to the range
[0,M− 1]× [0,N − 1]× [0,P− 1]. These elements are called pixels (picture
elements) in 2D and voxels (volume picture elements) in 3D, see Figure 2.1.
The gray-level value at each pixel or voxel is referred to as the intensity or
brightness. In computer memory, the intensity is most commonly represented
with signed or unsigned integer data types having a certain bit-depth N, mean-
ing that f ∈ [0,2N−1] (unsigned) or f ∈ [−2N−1,2N−1−1] (signed). Common
bit-depths are 8 and 16. A binary image has only two intensities representing
background and object, respectively.
15
If a voxel has the same size in all spatial directions, i.e., Δx = Δy = Δz, thegrid is uniform (isotropic). In medical images it is common that the voxels are
elongated in the z-direction, i.e., the grid is non-uniform (anisotropic). This
has to be considered when implementing both image analysis and visualiza-
tion algorithms.
The connectivity of voxels is of importance for many image analysis algo-
rithms. In the standard digitization grid, a voxel has 26 neighbors consisting
of six face neighbors, twelve edge neighbors, and eight vertex neighbors, see
Figure 2.2.
Figure 2.2: In a 3×3×3 neighborhood in the standard digitization grid, each voxel has
six face neighbors (left), twelve edge neighbors (middle), and eight vertex neighbors
(right).
2.2 Spatial filtering
Filtering of images may be performed for purposes such as noise reduction
and edge enhancement. A spatial filtering method involves the use of a filter
mask or kernel that covers a certain neighborhood N of the center voxel that
is the origin of the mask, see Figure 2.3.
w(1,−1)w(0,−1)
w(0,0)w(−1,0) w(1,0)
w(−1,−1)
w(0,1)w(−1,1) w(1,1)
f(x,y−1)f(x−1,y−1)
f(x−1,y+1) f(x,y+1)
f(x+1,y)f(x,y)f(x−1,y)
f(x+1,y−1)
f(x+1,y+1)
Figure 2.3: 2D example of a 3×3 filter mask (left) and the corresponding image pixels
under the filter mask (right).
16
The size of the filter mask is application dependent. For each voxel, a value
based on the neighbors are calculated and used as output. A linear filter con-
tains coefficients that are multiplied with the corresponding values in the input
image and the result is a weighted sum of the voxels covered by the mask, i.e.,
a convolution:
g(x) = ∑s∈N
w(s) f (x+ s),
where w is the filter mask, f the input image, and g is the output image. Non-
linear filters output values based on other criteria, e.g., the median value or the
maximum value.
2.2.1 General smoothing
Smoothing filters are used for blurring and noise reduction. There exist both
linear and non-linear versions. The output of a smoothing, linear spatial filter
is simply a weighted average of the voxels covered by the filter mask, i.e.,
g(x) = ∑s∈N w(s) f (x+ s)∑s∈N w(s)
.
The most basic linear smoothing operation is the averaging filter where all
filter coefficients w(s) = 1. This means that all voxels contribute equally much
to the output. A better approach is Gaussian smoothing, where the filter mask
is a discrete approximation of a Gaussian function with a certain standard
deviation σ :
w(s) = e−12 ||s||2/σ2
.
Here, the central voxel contributes most and the voxels distant from the center
contribute less. To obtain a good approximation, the size of the filter mask
should increase with the value of σ . A Gaussian smoothing filter is symmetric,
which means that it can be efficiently implemented as three consecutive one-
dimensional convolutions.
2.2.2 Bilateral filtering
A problem with smoothing filters is that they do not take image content into
account. In many applications, edges are important and hence should be pre-
served. By using a linear smoothing filter, an edge is blurred equally much
as a homogeneous region. In order to mitigate this problem, methods for
edge-preserving smoothing have been developed. One common method is
anisotropic diffusion [36] which is based on solving a partial differential equa-
tion (PDE).
Another approach is bilateral filtering [47]. The main idea here is to not only
use the spatial distance between voxels for calculating the filter coefficients,
but also the range distance, i.e., the intensity difference for gray-level images.
As an example, a filter kernel composed of two Gaussian functions can be
17
used, one for the domain (σd) and one for the range (σr). The filter produces
the output
g(x) = ∑s∈N e−12 ||s||2/σ2
d · e− 12 ( f (x+s)− f (x))2/σ2
r · f (x+ s)
∑s∈N e−12 ||s||2/σ2
d · e− 12 ( f (x+s)− f (x))2/σ2
r.
When implementing this filter, a new filter mask is computed at each voxel po-
sition. In Figure 2.4, a 2D illustration of bilateral filtering compared to Gaus-
sian smoothing is shown. The Gaussian smoothing removes a lot of detail
while the bilateral filter smoothes only in more homogeneous regions.
Figure 2.4: Bilateral filtering compared to Gaussian smoothing. Left: Maximum in-
tensity projection (MIP) of an MR angiography image. Middle: Result of Gaussian
smoothing with σ = 3 voxels. Right: Result of bilateral filtering with σd = 3 voxels
and σr = 20.
We used bilateral filtering for pre-processing in Paper IV, Paper V, and
Paper VI. The implementation is also included in our toolkit described in Pa-
per VIII.
2.2.3 Gradient extraction
Many image analysis algorithms, particularly segmentation algorithms, are
dependent on high-quality edge information. The gradient magnitude ||∇ f ||is a good indicator of edges. A standard method for estimating the gradient of
an image is to use centered differences, i.e.,
∇ f ≈
⎛⎜⎝
f (x+1,y,z)− f (x−1,y,z)Δx
f (x,y+1,z)− f (x,y−1,z)Δy
f (x,y,z+1)− f (x,y,z−1)Δz
⎞⎟⎠ .
This approximation can easily be implemented with three spatial filters.
If the image data is noisy, the gradient extraction becomes more robust if
some smoothing is performed before computing the gradient, e.g., Gaussian
smoothing or bilateral filtering. Another option is to involve smoothing
directly in the design of the gradient filter by using, e.g., derivatives of
Gaussian functions.
18
2.3 Gradient vector flow
As discussed in Section 2.2.3, the gradient and the gradient magnitude are
important for many algorithms. However, the gradient is a local property and
may not provide sufficient global information. An example is segmentation
with deformable models [21, 32]. Here, the gradient magnitude can serve as
an external image force that drives deformable contours or surfaces in order
to find a certain object in the image. If the initial model is initialized far away
from the object of interest, the model will probably fail or have very slow con-
vergence because the object edges are not found. This is because the capture
range of the gradient magnitude is very limited. This problem was addressed
by Xu and Prince [53] when they introduced gradient vector flow (GVF). The
basic idea behind GVF is to propagate edge information from strong bound-
aries into the inner part of homogeneous regions by gradient diffusion.
The 3D GVF vector field V(x) = (u(x),v(x),w(x)) minimizes the energy
functional
E =∫
R3μ||∇V||2 + ||∇g||2||V−∇g||2dx,
where g is an edge map computed from the original image by using, e.g., a
gradient magnitude filter. Note that the ∇ operator is applied to each com-
ponent of V separately. The first term, ||∇V||2, is called the smoothing term
since it alone will produce a smoothly varying vector field, while the second
term, ||∇g||2||V−∇g||2, forces the vector field to be close to ∇g in regions
where ||∇g|| is large. The regularization parameter μ controls the effect of the
smoothing term. Figure 2.5 shows an example of GVF.
Figure 2.5: Left: A slice of an edge map of an abdominal MR image. Right: Magnified
view of the GVF field computed from the edge map.
19
By using calculus of variations, it can be shown that V must satisfy the
Euler-Lagrange equation
μ∇2V−||∇g||2(V−∇g) = 0, (2.1)
where ∇2 is also applied to each component of V separately. The solution
to Equation (2.1) can be obtained by treating V as a function of time, t, andfinding the steady-state solution of
Vt = μ∇2V−||∇g||2(V−∇g). (2.2)
Equation (2.2) comprises three decoupled scalar PDEs that can be discretized
with finite differences and solved using standard numerical PDE solvers. In
higher dimensions, more sophisticated numerical solvers must be considered
in order to reduce computation time.
In this thesis, GVF is used both for haptic rendering and for driving de-
formable models, see Papers II, V, VI, and VIII. In a related publication [51],
we implemented and evaluated a number of numerical techniques for GVF
computation. Our implementations allow for computation times 1–2 orders of
magnitude faster than straight-forward implementations.
2.4 Digital distance transforms
Methods for measuring distances are useful image analysis tools, e.g., for
shape description. A distance transform (DT) is applied to a binary image and
computes the distance from each object element to the closest background ele-
ment. By regarding the object as background and vice versa, the distance from
each background element to the closest object element is computed. The re-
sult of a DT is a distance map, i.e., a gray-level image containing the distance
values. A signed DT assigns to each element the minimum distance from that
element to the nearest border element of the object. In this case, the distance
values have different signs depending on whether the element belong to the
object or to the background.
The DTs can be computed in different ways. In this work, we have used
weighted (chamfer) DTs [4] and DTs computed with the fast marching
method, see Section 3.4. The DTs have been used for deformable model
segmentation and for haptic rendering, see Papers IV, V, VI, and VIII.
20
3. Interactive segmentation
This chapter gives a brief introduction to interactive segmentation and descrip-
tions of the segmentation methods that have been used in this work.
3.1 Introduction
Image segmentation is the process of separating objects of interest from each
other and from the background and is one of the most important topics in
image analysis. Since segmentation often is an initial step, all further analysis
and information extraction will depend on the result of the segmentation.
In medical applications, segmentation is needed for basic tasks such as vol-
ume and area measurements and more complex tasks such as 3Dmodel extrac-
tion for surgical planning and image-guided therapy. There are several things
that make medical image segmentation hard, e.g., similar intensity patterns
for different tissues (low contrast) and bad image quality due to noise from
the imaging system. Another problem is the high shape variability of organs
making it hard to incorporate a priori information.
Over the recent decades, huge research efforts have been put on developing
automatic algorithms for medical image segmentation. Despite these efforts,
a general automatic algorithm is still not developed. Manual segmentation, on
the other hand, is very tedious and error-prone due to inter- and intra-observer
variability. Therefore, many methods are semi-automatic, i.e., the segmenta-
tion algorithms make use of input from the user [34]. The interaction can
involve manual delineation of an organ in selected 2D slices, positioning of
differently labeled seed-regions in different organs, or simply post-processing
of the result of an automatic method [20].
3.2 Segmentation by region growing
The basic idea in region growing is to group voxels into larger regions based
on predefined criteria. Starting from a set of seed-voxels [1], regions are grown
by appending neighboring voxels that have similar properties as the seeds.
The selection of seeds can be performed manually or automatically. The def-
inition of similarity is the key issue and is dependent on the application. In
the simplest case, the similarity measure is based on the intensity difference
between the seed-voxels and the voxels that are candidates for being included
21
in the region. An important issue in region growing is when to terminate the
growing. One possibility is to stop when no more voxels fulfill the similarity
criteria, but this often results in over-segmentation (leaking) due to low con-
trast. Therefore, it is common to use global measures based on the size and
shape of the region.
In Paper I, we use a simple region growing algorithm, where the similarity
criteria is based on a non-linear mapping of intensity values. To avoid leak-
ing, the maximum size of the grown region is controlled. More sophisticated
region-growing algorithms based on fast marching and fuzzy connectedness
were used in Paper IV and Paper V. These algorithms are described in Sec-
tion 3.4 and Section 3.5.
3.3 Live-wire segmentation
Live-wire [13, 12] is a semi-automatic segmentation method for 2D images
and slice-wise segmentation of 3D images. It is based on shortest path calcu-
lation in a graph representation of the image. For every edge in the graph, a
cost is assigned to represent the likelihood that the edge belongs to a desired
boundary in the image. To segment an object, the user places a seed-point on
the object boundary. All possible minimum-cost paths from the seed-point to
all other points in the image are computed via Dijkstra’s algorithm [8]. As
the user moves the cursor in the image, the minimum-cost path, referred to as
the live-wire, from the current position of the cursor back to the seed-point
is displayed in real-time, see Figure 3.1. The idea is to have low cost at the
desired boundary in order to make the live-wire snap onto it. When the user is
satisfied with a live-wire segment, he continues by placing a new seed-point.
In this way, the entire object boundary can be traced with a rather small num-
ber of live-wire segments. The design of the cost function is the crucial part
of the algorithm and how it should be chosen is discussed in, e.g., [13]. Com-
monly, a combination of intensity measures and gradient magnitude measures
are used.
Live-wire is a method for 2D segmentation, but it can be also be used for
3D segmentation. Most 3D extensions are based on using the standard live-
wire method on a subset of 2D slices in the 3D volume, and then reconstruct-
ing the entire object based on this information [40, 11, 16]. Even though the
reconstruction algorithms might take 3D information into account, all user in-
teraction is performed in 2D. This sometimes cause problems, e.g., when the
object topology changes between slices.
In Paper III, we suggest a more direct 3D approach where live-wire curves
are connected to form surfaces. For this, we use the Image Foresting Trans-
form (IFT) [10] which is an extension of Dijkstra’s algorithm that allows an
arbitrary number of seed-points and finding of the shortest path from all image
points to any of the seed-points.
22
Figure 3.1: Liver segmentation in a CT image with the live-wire method. The user
interactively positions seed-points (blue) on the liver boundary. As the user moves the
cursor, the shortest path (yellow) from the last seed-point to the current cursor position
is displayed in real-time. When a new seed-point is set, the current live-wire segment
is fixed (red).
3.4 Fast marching segmentation
In the live-wire method, a discrete shortest-path problem is solved by using
Dijkstra’s algorithm. Similar algorithms have also been used successfully for
computing discrete approximate solutions to continuous problems. Examples
are fast marching methods and level set methods [43].
3.4.1 Formulation
Consider a closed surface Γ ∈ Rn propagating in its normal direction n with
speed F(x). The front only propagates outwards, i.e., F > 0. To character-
ize the position of the expanding front, the time of arrival u(x) of the front
can be computed as it crosses each point x. Basic relations between distance,
speed, and time, gives that the speed F(x) must be inversely proportional to
the gradient, i.e.,
∇u(x) = C(x)n,
where C(x) = 1/F(x) can be regarded as a slowness or cost function. This
leads to the boundary value problem
||∇u(x)||= C(x), u(x) = 0 on Γ, (3.1)
which is known as the Eikonal equation and is the underlying equation in fast
marching methods.
23
The relation between fast marching methods and level set methods can be
seen through an alternative approach of derivation. If the initial position of
the front is embedded as the zero level set of a function φ ∈ Rn+1, the time
dependent initial-value problem
φt +F ||∇φ ||= 0, (3.2)
is obtained. This is the level set equation and it describes the time evolution
of the level set function φ in such a way that the zero level set φ = 0 is always
identified with the propagating surface. This formulation is more general than
fast marching since it allows the speed function F to be both positive and
negative. Hence, it is also more computationally demanding.
3.4.2 Fast marching
In the work described here, we have focused on fast marching methods which
essentially are numerical schemes for solving Equation (3.1).
The central idea behind the fast marching method is to systematically con-
struct the solution u in a downwind fashion, i.e., to propagate information out-
wards from the boundary condition, from smaller values of u to larger values.
This requires upwind difference schemes in place of classical approximations
such as centered differences. A commonly used upwind gradient approxima-
tion is the Godunov scheme
||∇u|| ≈
⎡⎢⎣
max(D−xi jku,−D+x
i jku,0)2+
max(D−yi jku,−D+y
i jku,0)2+
max(D−zi jku,−D+z
i jku,0)2
⎤⎥⎦
1/2
,
where i, j, k denotes the indices in the discretization and D+ and D− are stan-
dard forward and backward difference operators, e.g.,
D+xi jku =
ui+1, j,k−ui, j,k
Δx, and
D−xi jku =
ui, j,k−ui−1, j,k
Δx.
Using this scheme, the discrete approximation of Equation (3.1) becomes
C2i jk =max
(ui, j,k−ui−1, j,k
Δx,ui, j,k−ui+1, j,k
Δx,0
)2
+
max
(ui, j,k−ui, j−1,k
Δy,ui, j,k−ui, j+1,k
Δy,0
)2
+
max
(ui, j,k−ui, j,k−1
Δz,ui, j,k−ui, j,k+1
Δz,0
)2
.
(3.3)
This equation can be solved for u by using the procedure in [22].
24
The fast marching algorithm is accelerated by limiting the computational
domain to an area in the proximity of the front, the so called narrow band.While this narrow band is marching forward, the values of existing points are
accepted and new points are brought into the narrow band structure. Because
of upwinding, no point can be affected by grid points with larger values of u.Therefore, the point where u is minimal will be accepted and its downwind
neighbors are updated. In the Dijkstra-like algorithm, the narrow band points
are labeled as Trial points and the accepted points as Accepted. Points that
have not been reached by the front yet are labeled Faraway, see Figure 3.2.
The Trial points in the narrow band are stored in a priority queue based on
Accepted
Trial (narrow band)
Faraway
Figure 3.2: Fast marching front propagation. The computational domain is limited
to a narrow band in the proximity of the front. The narrow band contains the Trialpoints and is represented by a minimum heap data structure. In each iteration, the Trialpoint with the smallest time of arrival value is labeled as Accepted and its Farawayneighbors are brought into the narrow band.
a minimum heap data structure, a binary tree where the node with minimum
value is always located at the root. Accessing the root has complexity O(1)and point insertion/update has complexity O(logn) if there are n points in the
heap. The overall complexity is O(N logN) for grids with N elements, see
Algorithm 1.
The common use of fast marching methods in image segmentation is to de-
sign a proper cost functionC, provide a set of seed-points representing the ini-
tial front, and then propagate the front until a certain arrival time is reached.
The cost image C should be designed to achieve low costs in homogeneous
parts and high costs at edges, see Figure 3.3. In order to find a proper arrival
time threshold to obtain the final segmentation, visual inspection can be used.
More automatic methods are based on, e.g., global measures of the voxels be-
longing to the front [54]. Note also that by using a constant cost function, e.g.,
C = 1, the fast marching algorithm can be used to compute an approximation
of the Euclidean distance map from the set of seed-points.
25
Algorithm 1: Fast marching
begininitialization;
forall p in the domain doset u(p) = ∞;
tag p as Faraway;forall p in the initial condition do
set u(p) = 0;
tag p as Accepted;forall p that are Accepted do
forall p′ that is a neighbor of p doif p′ is Faraway then
tag p′ as Trial;compute u(p′);insert p′ in the heap;
main loop;
while heap is not empty dopop the root of the heap and store it in p;tag p as Accepted;forall p′ that is a neighbor of p do
if p′ is Faraway thentag p′ as Trial;compute u(p′);insert p′ in the heap;
else if p′ is Trial thenrecompute u(p′);update the heap;
end
Our fast marching implementation follows Algorithm 1 and utilizes the Go-
dunov scheme for upwind gradient approximation. To efficiently solve the dis-
cretized Eikonal equation (3.3) for u, we use the procedure in [22]. We used
our implementation for liver segmentation in Paper IV and Paper VI. It is also
part of the toolkit in Paper VIII.
3.5 Fuzzy connectedness segmentation
Most segmentation algorithms give a crisp output, i.e., a set of points that
either belong to the object or to its complement. In cases when the objects
of interest show intensity patterns distinctly different from those of other
26
Figure 3.3: Fast marching segmentation of the liver. Left: Seed-regions placed inside
the liver in an CT image. Middle: The cost image. Right: The resulting time-of-arrival
map with overlaid contour obtained by thresholding.
objects, crisp segmentation is often a good option. This is seldom the case
though, especially not for medical images where artifacts such as blur, noise
and background variation are introduced by the imaging device. The image
points rather belong to an object to a certain degree, i.e., they are fuzzy. Fun-damental work on the fuzzy topology and geometry of image subsets was
made by Rosenfeld. In [38], he reviews work in this field and its application
in image processing. For the work described here, a couple of basic notions
are required to present fuzzy connectedness and the segmentation evaluation
in Section 3.7.
3.5.1 Fuzzy set theory
For any reference set X , a fuzzy subset A of X is a set of ordered pairs
A= {(x,μA(x))|x ∈ X},
where μA : X �→ [0,1] is the membership function of A in X . The value μArepresents the degree of belongingness for x inA. If μA(x) �= 0 for any x ∈ X ,
A is said to be non-empty. In the definition list below, A and B are fuzzy
subsets of the reference set X and x is any point in X .
Fuzzy set union A∪B:
μA∪B(x) = max(μA(x),μB(x)).
Fuzzy set intersection A∩B:
μA∩B(x) = min(μA(x),μB(x)).
Fuzzy set difference A−B:
μA−B(x) = max(μA(x)−μB(x),0).
27
Fuzzy set complement A:
μA(x) = 1−μA(x).
Fuzzy set cardinality |A|:|A|= ∑
x∈XμA(x).
Finally, a fuzzy relation ρ in X is a fuzzy subset of X×X :
ρ = {((x,y),μρ(x,y))|(x,y) ∈ X×X}, where μρ : X×X �→ [0,1].
3.5.2 Fuzzy connectedness
Fuzzy connectedness [50] defines how image elements hang together in a
fuzzy setting. The aim of FC is to capture the global hanging togetherness
between grid points through local hanging togetherness defined by the fuzzy
relations adjacency μad j(x,y) and affinity μa f f (x,y). The adjacency depends
on the spatial distance between grid points. Hard adjacency relations are com-
monly used, e.g., μad j(x,y) = 1 if x and y are face neighbors and zero other-
wise. The affinity takes into account the adjacency of the grid points as well
as their similarity in intensity. The closer they are and the more similar in in-
tensity they are, the higher the affinity between them. The selection of fuzzy
adjacencies is essential and can be compared to the cost function selection in
fast marching.
In order to assign a strength of connectedness between any pair of grid
points (x,y), all possible paths of points between x and y are considered.
These paths are made up of links between successive neighboring grid
points (xi,xi+1) along the path where the strength of each link is the affinity
μa f f (xi,xi+1). The strength of a path is the strength of its weakest link. The
strength of connectivity between x and y is associated with the maximum
strength of all possible paths.
For image segmentation, FC can be applied and implemented in a way sim-
ilar to fast marching. The user specifies a proper adjacency, select a set of
seed-points inside the object of interest, and then compute an FC map by us-
ing a modification of Algorithm 1. The propagation of time in fast marching
is simply replaced by propagation of affinity in FC. Here, a maximum heap
is needed since the propagation will be from higher connectedness values to
smaller. The resulting connectedness map is a fuzzy segmentation where the
values between zero and one indicate the degree of belongingness to the object
of interest.
An extension to FC is relative FC [49], where extraction of objects is per-
formed by competing multiple initial regions against each other. In the work
by Nyúl et al. [33], various implementations of FC for interactive 3D segmen-
tation are reviewed.
28
Our implementation of FC is based on small modifications of the fast
marching implementation and it is also included in the toolkit described in
Paper VIII. In Paper V, we used the FC algorithm to create external forces
for driving a deformable model.
3.6 Segmentation by deformable models
Deformable models for image segmentation were introduced in 1988 by Kass,
Witkin, and Terzopoulos [21]. They dealt with 2D active contours (snakes).
The concept has been extended to 3D deformable surfaces in various ways, an
overview can be found in [32].
3.6.1 Formulation
Common for most deformable model algorithms is that the models are driven
by minimization of an energy functional E . The energy functional usually
consists of several terms including at least one internal shape regularizing
term Eint and one external term Eext based on image data, i.e.,
E(S) = Eint(S)+Eext(S),
where S is the deformable surface. The key is to build the energy functional
in such a way that the desired solution S∗ coincides with the global minimum
of E , i.e., S∗ =minS E(S). For given internal and external terms, the minimiza-
tion equation does not in general have an analytical solution. Therefore, it is
necessary to discretize and solve it numerically. The most common method
is to rewrite the problem by using the Euler-Lagrange equation ∇E(S) = 0
in order to obtain a stationary equation that describes a force equilibrium be-
tween internal regularizing forces and external forces. The resulting problem
is then discretized by using finite differences, initialized with an approximate
solution, and solved iteratively. With a proper initialization, the model will
converge towards a minimum.
The two main characteristics of a deformable surface model are its geomet-
rical representation and its law of deformation. The geometrical representation
sets the degrees of freedom and the topological flexibility of the model while
the law of deformation tells how the model should be deformed in order to fit
the underlying image data. Among the continuous representations, there are
parametric models and implicit models. Common discrete representations are
unstructured meshes and polygonal meshes. In this work, Delingette’s discrete
simplex mesh representation [7] has been used.
29
3.6.2 Simplex meshes
A k-simplex mesh is a discrete surface mesh where each vertex is linked to
k +1 neighboring vertices by an edge. In 3D, a surface model is realized as a
2-simplex mesh where each vertex has three neighbors. An important property
of the simplex mesh representation is the duality with triangulations. There
exists a dual triangle for each mesh vertex and a dual triangulation vertex for
each mesh face, see Figure 3.4. Each simplex vertex pi has three neighbor-
Figure 3.4: Left: Close-up of a 2-simplex mesh also showing the dual triangulation.
Right: A sphere represented by a 2-simplex mesh consisting of 320 vertices and 162
faces.
ing vertices (pN1(i),pN2(i),pN3(i)). These points define a triangle in the tangent
plane with normal vector ni:
ni =(pN2(i)−pN1(i))× (pN3(i)−pN1(i))||(pN2(i)−pN1(i))× (pN3(i)−pN1(i))||
.
The simplex mesh geometry involves the circle (Ci,ri) circumscribed to the
triangle (pN1(i),pN2(i),pN3(i)) and the sphere (Oi,Ri) circumscribed to the four
vertices (pi,pN1(i),pN2(i),pN3(i)), see Figure 3.5. The orthogonal projection Fi
of pi onto the tangent plane triangle can be represented with the barycentric
coordinates called the metric parameters, i.e.,
Fi = ε1ipN1(i) + ε2ipN2(i) + ε3ipN3(i), ε1i + ε2i + ε3i = 1.
The simplex angle φi ∈ [−π,π] is defined by
sin(φi) =ri
Ri· sign((pN1(i)−pi) ·ni),
cos(φi) =||Ci−Oi||
Ri· sign((Ci−Oi) ·ni).
By drawing the projection of the sphere in the plane (Fi,Ci,pi), the simplex
angle appears as the angle between the segments that join pi to the projection
30
N1(i)p
pN2(i)
pN3(i)
ip
F
O
C
Ri
i i
i
i
d
φi
n i
L( i di, ), iφr
ir
Figure 3.5: Simplex mesh geometry. The circle that circumscribes the three neighbor-
ing vertices (pN1(i),pN2(i),pN3(i)) has center Ci and radius ri. The sphere that circum-
scribes the vertex pi and its neighbors has center Oi and radius Ri. Fi is the orthog-
onal projection of pi onto the tangent plane and di = ||Fi−Ci||. The elevation of piwith respect to the tangent plane is L(ri,di,φi). The red triangle represents the plane
(Fi,Ci,pi). In the projection onto this plane, the simplex angle appears as the angle
between the segments that join pi to the projection of the circle.
of the circle. The height of pi with respect to the tangent plane is defined
through the elevation function
L(ri,di,φi) =(r2i −d2
i ) tan(φi)
±√
r2i +(r2i −d2i ) tan2(φi)+ ri
,
where di = ||Fi−Ci||. This means that pi can be represented in terms of its
three neighbors:
pi = Fi +L(ri,di,φi)ni = ε1ipN1(i) + ε2ipN2(i) + ε3ipN3(i) +L(ri,di,φi)ni.
While the metric parameters control the projection of pi onto the tangent
plane, the simplex angle controls the local mean curvature Hi = 1/Ri through
the simple relationship
Hi =sinφi
ri.
31
3.6.3 Deformation
Mesh deformation is dependent on which law of motion that is selected. For
discrete models, Newtonian evolution is commonly used, i.e., each vertex is
regarded as a point-mass influenced by internal forces, external forces, and
damping. The differential equation for a unit mass vertex is
d2pi
dt2=−γ
dpi
dt+(Fint)i +(Fext)i,
i.e., a force equilibrium equation where t is the time, γ is the damping factor,
Fint are the internal forces, and Fext are the external forces. By discretization
with central finite differences, the following iterative scheme is obtained:
pt+1i = pt
i +(1− γ)(pti−pt−1
i )+αi(Fint)ti +βi(Fext)t
i.
The weights αi and βi control the level of the internal and external force com-
ponents, respectively.
For the simplex mesh, the internal forces are decomposed into a tangential
force and a normal force:
Fint = Ftangent +Fnormal.
The tangential force is based on a set of reference metric parameters ε·i. Thesedefine a reference projection Fi in the tangent plane and the resulting force is
simply a spring force between Fi and Fi, i.e., Ftangent = Fi−Fi. If uniformly
spread vertices are desired, the reference projection should be the centroid of
the neighbor triangle: ε1i = ε2i = ε3i = 1/3. A curvature based tangential force
can be used in order to concentrate vertices in areas where a denser mesh is
needed. The reference metric parameters are then updated based on the mean
curvature deviation vector
δ |H|i =
⎛⎜⎜⎜⎝
|HN1(i)|−|Hi||Hi|
|HN2(i)|−|Hi||Hi|
|HN3(i)|−|Hi||Hi|
⎞⎟⎟⎟⎠ ,
where |Hi|= (|HN1(i)|+ |HN2(i)|+ |HN3(i)|)/3 is the absolute mean curvature.
The normal force component is based on a reference simplex angle φi that
gives the resulting force Fnormal = (L(ri,di, φi)− L(ri,di,φi))ni. Different
choices of φi give rise to different shape constraints. To obtain smooth
surfaces, a C2 continuity constraint can be used where φi is defined as the
average of the simplex angles at neighboring vertices. Another option is to
use a global shape constraint by setting φi = φ0, where φ0 is a constant.
In Paper V and Paper VIII we present our implementation of deformable
simplex meshes. It follows the theory and equations above with Newtonian
32
evolution as the law of deformation. For the internal forces, we use a C2 con-
straint in the normal direction and a curvature-based force in the tangential
direction. Which external forces to use is dependent on the application. In our
implementation, the external forces are applied at vertex pi as
(Fext)i = ∑k
wki(fki ·ni)ni,
where fki are the forces computed from image data and wki are weights. To
avoid instabilities, the external forces are applied only in the normal direc-
tion [7]. The forces we use are:
Gradient magnitude force: Attracts the vertex to the
voxel gi with maximum gradient magnitude in a specified
neighborhood:
fi = gi−pi.
Inflation force: Makes the vertex propagate in its positive
or negative normal direction depending on if the intensity
value I(pi) is higher or lower than the constant I0:
fi = sign(I(pi)− I0).
Potential field force: A force proportional to the gradient
of a potential field P, e.g., a distance map:
fi = ∇P(pi).
Vector field force: A force based on the vector field V,
e.g., GVF:
fi = V(pi).
Our implementation also involves external forces based on haptic interaction,
see Section 6.3.
3.7 Evaluation of segmentation methods
An important step in the development of segmentation methods is the evalua-
tion. There are mainly two questions that need to be answered:
1. How accurate is the segmentation result?
2. With which precision can the segmentation result be repeated?
Udupa et al. [48] recently presented a framework for comparing and evalu-
ating segmentation methods with focus on medical applications. This frame-
work provides tools for evaluation of segmentation efficacy, i.e., precision, ac-curacy, and efficiency. It is also suggested how to perform statistical analysis
of the data. The framework is developed to be able to handle fuzzy segmenta-
tions where crisp segmentations become a particular case.
33
A fuzzy segmentation of object O obtained with method M is denoted by
CMO = (C, fO), where C is a 3D grid and for any x ∈ C, fO(x) ∈ [0,1] is the
degree of belongingness to the object. In order to evaluate the accuracy of a
segmentation method, the results need to be compared against a ground truth.
Since it is uncommon to have completely true segmentations available, surro-
gates of truth are used instead, e.g., manual delineations. If multiple manual
delineations CmanOk
= (C, fOk) exist, a fuzzy surrogate of truth Ctrue = (C, ftrue)can be generated by averaging, i.e.,
ftrue(x) =1
N
N
∑k=1
fOk(x),
where N is the number of delineations. The segmentation accuracy is then
obtained by computing the sensitivity as the true positive volume fraction
T PV FMO (O) =
|CT P||Ctrue| =
|CMO ∩Ctrue||Ctrue| ,
and computing the specificity as one minus the false positive volume fraction:
1−FPV FMO (O) = 1− |CFP|
|Ctrue|= 1− |C
MO −Ctrue||Ctrue|
.
A completely accurate segmentation method will have both specificity and
sensitivity equal to one, see Figure 3.6 for an illustration of the accuracy com-
ponents in the crisp case. For the definitions of union, intersection, comple-
ment, and cardinality for fuzzy sets, see Section 3.5.
C
C
C
C
C
FN
FP
TP
true
OM
Figure 3.6: 2D illustration of the accuracy components in the crisp case. The sensitiv-
ity is computed as |CT P|/|Ctrue| and the specificity is computed as 1−CFP/|Ctrue|.
34
Segmentation precision is a measure of repeatability, i.e., how sensitive the
result is to the operator input, e.g., seeding. Both inter- and intra-operator pre-
cision must be considered. A full evaluation of precision also considers how
the patient positioning in the scanner affects the result (inter-scan precision).
The suggested precision metric for two fuzzy segmentations CMO1
and CMO2
ob-
tained with method M at different occasions is
PRM(O) =|CM
O1∩CM
O2|
|CMO1∪CM
O2| .
PRM(O) represents the amount of tissue common to both CMO1
and CMO2, the
intersection of the sets, as a fraction to the total amount of tissue found in the
union of CMO1
and CMO2.
We used this framework for evaluation of our fast marching segmentation
and deformable model segmentation in Paper IV, V, and VI. It is also imple-
mented in the WISH toolkit described in Paper VIII.
35
4. Visualization
Visualization is the process of creating images to convey information. In com-
puter science, this involves rendering images onto a computer screen by us-
ing computer graphics techniques and graphical user interfaces. The aim of
a visualization is to reduce the huge amount of information contained in a
multi-dimensional dataset in order to make it perceptible for the user. Inter-
active visualizations that can be manipulated and examined in real-time are
preferable.
In medical image visualization, 2D, 3D, and 4D images, and sometimes
images of even higher dimensionality need to be visualized. 2D images are
straight-forward to visualize by displaying the images on the screen together
with a user interface that allows the user to select a proper contrast/brightness
window and to magnify or minify the image. For 3D and 4D images, more
sophisticated visualization techniques are required in order to not overwhelm
the user with information. In this chapter, we describe the methods for medical
image visualization that have been used in this work.
4.1 Multi-planar reformatting
Multi-planar reformatting (MPR) is a classic visualization technique for view-
ing 3Dmedical images. Arbitrarily positioned and oriented 2D planes are used
to visualize multiple cross-sections of the 3D dataset. The common applica-
tion of MPR is to display the three principal planes (axial, coronal, and sagit-
tal) next to each other along with a user interface that allows for translation
of the planes. It is also possible to display the planes in their correct 3D posi-
tion and orientation, see Figure 4.1. The simplicity of this approach, and the
fact that it is so widely used make it a standard tool in most medical image
visualization applications. MPR is a standard tool and was used in all papers
included in this work. In the WISH toolkit described in Paper VIII, the MPR
viewer has support for rendering of overlays to, e.g., display segmentation
results.
4.2 Volume rendering
Volume rendering techniques [25, 9] is a family of rendering techniques where
the full 3D dataset is considered. The parts of the volume important to the
37
Figure 4.1: A CT image of the abdomen visualized with multi-planar reformatting.
user are displayed while uninteresting parts are made transparent or semi-
transparent by tuning of application specific transfer functions. The most com-
mon approach to perform volume rendering is by using ray-casting. Through
each pixel in the image plane, a ray is cast from the view position into the
volume, see Figure 4.2. For all voxels along the ray, the scalar value is sam-
pled and used for lookup in a transfer function. The resulting color of the
image plane pixel is determined by the selected compositing technique. Com-
mon compositing techniques are alpha blending, maximum intensity projec-
tion (MIP), and iso-surface extraction. Because no geometric representation of
the data is needed, the term direct volume rendering (DVR) is commonly used.
To perform DVR in software is computationally demanding and not suitable
for interactive renderings. By using hardware accelerated DVR, described in
the next section, it is possible to speed up the rendering to interactive rates on
ordinary workstations.
4.3 Hardware accelerated direct volume rendering
In recent years there has been a great development of consumer graphics hard-
ware where the fixed-function pipeline in the graphics processing unit (GPU)
has been replaced by programmable vertex processors and fragment proces-
38
(0,0,1)
(0,1,0)
(0,1,0)
(1,1,0)
(1,0,0)
(1,1,1)
(1,0,1)(τ )
c
(τ)
Image plane
dr0
(0,0,0)
rr e
Figure 4.2: Volumetric ray-casting. Through each pixel in the image plane, a ray is cast
from the camera position c into the volume. In the GPU implementation, the volume
is represented by a 3D texture. For each fragment generated in the rasterization of the
bounding box, the entry point r0 and direction d = r0− c of the ray are interpolated.
sors. These can be customized for the application by using shader programs.The vertex shader is responsible for vertex transformation, projection, and for
specifying which varying vertex attributes (e.g., colors, normals, and texture
coordinates) should be interpolated across the primitive we are drawing. The
fragment shader is active in the primitive rasterization stage and is responsible
for writing a color to the frame buffer based on interpolated attributes from the
vertex shader and additional uniform data, e.g., textures.
In order to generate the fragments (pixels) needed for the ray-casting, a
proxy geometry is needed, e.g., the bounding box of the volume. Each vertex
of the box is assigned a texture coordinate in [0,1]× [0,1]× [0,1]. These val-
ues are interpolated across the box faces during rasterization in order to obtain
the entry position r0 of the ray. The ray direction is computed as d = r0− c,where c is the camera position in texture coordinates. The ray-casting is then
performed in the fragment shader on the GPU by sampling the 3D texture
representation of the volume along the parametric ray r(τ) = r0 + τ d||d|| by in-
crementing τ with a suitable step length Δτ until r(τ) is outside the volume,
see Figure 4.2. This technique is the most common today [24, 46]. In ear-
lier approaches, when the programmability of the GPUs were limited, it was
common to slice the dataset with back-to-front ordered, viewplane-aligned
polygons [52].
With this general ray-casting engine it is possible to use different sample
compositing techniques along the ray. In our implementation described in Pa-
per VII we use a specialized maximum intensity projection (MIP) for color-
correct visualization of 4D MR breast images. In our toolkit (Paper VIII) we
have implemented three additional compositing modes: standard MIP, front-
39
Figure 4.3: Left: Volume rendering with alpha compositing of a CT scan of the head
and upper thorax. Right: Ray-casting in iso-surface mode of the fast marching seg-
mentation result of a liver.
to-back alpha blending, and iso-surface extraction with shading. Figure 4.3
shows some rendering examples1.
4.4 Surface rendering
In surface rendering, polygonal surfaces are extracted from the dataset and
displayed by using standard computer graphics techniques. Because of sim-
plicity and hardware issues, triangles are commonly used. The most well-
known algorithm for surface extraction from volumetric data is Marching
Cubes (MC) [27]. MC is designed for iso-surface extraction, i.e., surfaces with
the same scalar value, see Figure 4.4. Eight voxels at a time are examined. De-
pending on whether the voxels are inside or outside the selected iso-surface,
a configuration number is computed and the corresponding triangulation is
obtained through table lookup. Because eight voxels are examined for being
inside or outside, there are 28 = 256 possible configurations.
Surface normals are obtained by computing the gradient of the data. By
performing tri-linear interpolation of positions and gradients, smooth shaded
surfaces can be obtained. The MC algorithm is also useful for rendering of
binary images obtained through, e.g., segmentation.
1Anders Persson, CMIV, Linköping, is acknowledged for providing the dataset in Figure 4.3
(left). The same dataset is also used in Figure 4.4 and Figure 4.5.
40
Figure 4.4: Left: Iso-surface extraction of the skin in a CT image performed with
marching cubes. Right: A close-up of the ear where the large amount of generated
triangles are visible.
4.5 Stereo graphics
The standard way of rendering images onto a computer screen only provides
212D information. Perspective projection, occlusion, and shading effects pro-
vide quite strong depth cues, but the image is still 2D. In order to provide a
more realistic 3D perspective, stereo graphics can be used where two images
are presented to the user, one for the left eye and one for the right eye. These
images show the same scene from a slightly different perspective based on the
horizontal distance between the eyes, the disparity. To present a correct stereo
effect to the user, the left image must be visible to the left eye only, and vice
versa. To obtain this, there are a number of techniques available [41].
In color-multiplexed (anaglyph) displays, the left and right images are fil-
tered with near complementary colors and the user wears corresponding filter
glasses. A drawback with this technique used to be the limited color render-
ing, but with modern notch filters this problem is mitigated. Another technique
is polarization-multiplexing where polarizing filters are used for rendering of
the image pairs and the user wears polarizing glasses. This technique requires
that the screen preserves the polarization.
With time-multiplexing, the image pairs are rendered rapidly in alternate
refresh cycles and the user wears active shutter glasses that are synchronized
to the rendering. In this stereo rendering technique, the frame rate must be
twice as high compared to mono rendering. It is also common with cross-talk
(ghosting) effects due to non-perfect shuttering and slow phosphor decay of
the cathode ray tube (CRT) monitor. Time-multiplexing has been used in this
work. It is part of the haptic displays described in Section 5.2.
Location-multiplexing means that the images are rendered at different loca-
tions and sent to the eyes in separate channels. An example is head-mounted
displays (HMDs). A more simple example of location-multiplexing is illus-
trated in Figure 4.5, where it is possible to obtain a stereoscopic view by
41
crossing the eyes. In [41], a good overview of stereoscopic rendering and 3D
displays is given, including modern auto-stereoscopic display technology.
Figure 4.5: Stereo pair of an iso-surface rendering of a CT image. The stereo effect is
obtained by crossing the eyes.
42
5. Computer haptics
In this chapter an introduction to computer haptics is given. First, some haptic
devices, display solutions, and softwares for haptics are reviewed. The follow-
ing sections is about haptic rendering techniques with focus on volume haptics
and its use in medical segmentation.
5.1 Haptic devices
A haptic device, according to its name, should be capable of both tactile and
kinesthetic feedback, but most commercially available devices today provide
only kinesthetic feedback. Well-known devices are the PHANToM series from
Sensable Technologies1, the Omega.X family from Force Dimension2, and the
recently released low-priced Falcon from Novint3, see Figures 5.1 and 5.2.
These devices are impedance driven, i.e., they follow the rule position in –
force out. An admittance driven device follow the rule force in – position out.
The number of dimensions the haptic device can monitor and control is re-
ferred to as the degrees of freedom (DoF). The DoF need not to be the same for
input and output. The PHANToM desktop device takes a 3D position and three
rotations for input (6DoF) and for output it provides a force vector (3DoF).
Devices that have six DoF in and six DoF out provide more complexity and
are hence more expensive. Examples of six DoF in and six DoF out devices
are the PHANToM Premium 1.5/6DoF and PHANToM Premium 3.0/6DoF.
The design of a haptic device varies, but in most cases they are constructed
with a stylus or a ball that the user holds in her hands. A single point, the
haptic probe, is located at the tip of the stylus or at the center of the ball and
serves as the interface to the haptic device.
The haptic devices that have been used in this work are the PHANToM
Desktop and the PHANToM Omni, see Figure 5.1. Some features of these
devices are summarized in Table 5.1.
1URL: http://www.sensable.com2URL: http://www.forcedimension.com3URL: http://home.novint.com
43
Figure 5.1: The PHANToM Desktop device (left) and the PHANToM Omni device
(right).
Figure 5.2: The Omega.X device from Force Dimension (left) and the Falcon device
from Novint (right).
5.2 Haptic displays and software
A desktop haptic devices can simply be positioned next to the keyboard and
mouse and used as an additional advanced interaction device. This way of
usage is common in the test phase of application development when the pro-
grammer need to test, restart, and recompile the application frequently.
In our work we have used special haptic displays from the Swedish com-
panies Reachin4 and SenseGraphics5 where the 3D capabilities of the device
are used to a higher extent. Both companies provide display solutions where
the haptic device is positioned beneath a semi-transparent mirror. The graph-
ics is rendered with time-multiplexed stereo and projected through the mirror
4URL: http://www.reachin.se5URL: http://www.sensegraphics.com
44
Table 5.1: Features of the PHANToM Desktop and the PHANToM Omni devices.
PHANToM Desktop PHANToM OmniWorkspace 160 × 120 × 120 mm 160 × 120 × 70 mm
Weight 2.86 kg 1.79 kg
Position resolution 0.023 mm 0.055 mm
Maximum force 7.9 N 3.3 N
Continuous force 1.75 N 0.88 N
Stiffness 1.86 N/mm (x-axis) 1.26 N/mm (x-axis)
2.35 N/mm (y-axis) 2.31 N/mm (y-axis)
1.48 N/mm (z-axis) 1.02 N/mm (z-axis)
Interface Parallel port IEEE-1394 Firewire� port
Platform Intel PC Intel PC
in order to obtain co-localization of haptics and graphics, see Figure 5.3 and
Figure 1.3.
Both display vendors also provide programming API:s for 3D visualization
applications with haptics. Reachin provide their Reachin API and SenseG-
raphics has their dual commercial and GPL (open source) licensed software
H3D API. Both API:s have been used in this work, but the WISH toolkit de-
scribed in Paper VIII has been implemented in the H3D API. For research
purposes, an open source solution is more convenient.
Figure 5.3: The Reachin desktop display with a PHANToM Desktop haptic device.
45
A low-level API for programming haptic applications is OpenHaptics from
Sensable. OpenHaptics has similar syntax as OpenGL, making it familiar to
graphics programmers. The H3D API (up to version 1.5) uses OpenHaptics
for haptic rendering.
5.3 Haptic rendering
Haptic interaction with objects in a 3D computer graphics environment in-
volves the task of generating intuitive force feedback when the user moves
the haptic probe so that it comes in contact with an object. Depending on
the application, there are different object representations to interact with. We
can have explicit (polygonal) surfaces, implicit surfaces, and, as in this work,
volumetric data. In order to obtain perceptually convincing haptic feedback,
the haptic update rate should be approximately 1 kHz. The haptic rendering
techniques described in this chapter is point-based haptic feedback with three
DoF.
5.3.1 Haptic surface rendering
An algorithm for surface haptics should generate force feedback when the
haptic probe comes in contact with, or penetrates, a surface. The first algo-
rithm for haptic interaction with polygonal surfaces was the penalty method
described in [31]. Since an impedance controlled haptic device cannot explic-
itly control the position of the haptic probe, the rendering algorithms must
allow for penetration of surfaces. In the penalty method, the force feedback
generated is proportional to the distance from the haptic probe to the closest
point on the polygon that has been penetrated. This method is unstable and is
not commonly used today. By instead introducing a virtual probe that can be
completely controlled by the algorithm, the force feedback can be based on the
distance between the true probe and a virtual probe. This was first suggested
in [56], where a so-called god-object was constrained to stay on the rendered
surface. This algorithm was further refined in [39], where the god-object was
replaced by a finite-sized spherical proxy point. These techniques are referred
to as constraint-based since the surfaces constrain the movements of the god-
object and the proxy. In the proxy-method, the connection between the haptic
probe and the proxy consists of a spring-damper, i.e., a virtual coupling device
consisting of a spring and a damper, see Figure 5.4. The rendering equation
for proxy-based surface haptics is
f =−k(x−p)− γ(x− p), (5.1)
where x is the probe position, p the proxy position, k the stiffness of the spring-
coupler, γ the damping coefficient, and x and p the velocities of the probe and
46
Figure 5.4: Proxy-based surface haptics. The haptic probe (white) is connected to a
virtual proxy (black) through a spring-damper. Note that the distance between the
probe and the proxy is exaggerated.
the proxy, respectively. In many applications, the damping term is set to zero
since it is hard to determine an optimal value [35].
The distance between the probe and the proxy is in general small. For in-
stance, if the user applies a force of 1 N and the spring-damper stiffness is
500 N/m, the probe–proxy distance is 1/500 m=2 mm. To reduce the feeling
of offset, the graphical representation of the haptic probe can be rendered at
the proxy position.
5.3.2 Haptic volume rendering
One option to perform haptic rendering of volumetric data is to locally es-
timate surfaces and use them as an intermediate representation for a surface
haptics technique [23]. For direct haptic rendering of volumetric data, one of
the first algorithms was suggested by Avila and Sobierajski [2]. They proposed
a direct force mapping based on the current position and velocity of the haptic
probe along with local data measures, e.g., density value and gradient:
f = �F (x, x,V (x),∇V (x)) ,
where V is the volumetric data. A similar six DoF haptic rendering scheme
for direct display of vector data was suggested by Iwata and Noma [19]. A
drawback with these techniques is that they add energy to the system [35], and
therefore are noise sensitive and tend to result in vibrations. However, they
constitute an important step in the development of haptic volume rendering
techniques.
5.3.3 Proxy-based volume haptics
State-of-the art in direct volume haptics is derived from the proxy-based meth-
ods for surface haptics [56, 39] and was introduced by Lundin [30]. Here, the
interpolated gradient direction at the proxy position is used as a surface normal
to define a surface to which the proxy is constrained. By using haptic transfer
47
functions, the stiffness and friction of the surface can be modulated. Ikits [18]
describes a general framework for three DoF haptic rendering of volumetric
data based on a constrained proxy-point. The basic idea here is to choose an
appropriate local reference frame (LRF) and generate constraints for proxy-
movements in the LRF. The proposed method works for arbitrary LRFs, and
as an example it is shown how to haptically render diffusion tensor MRI data.
Common for these techniques is that they require the axes in the LRF to be or-
thogonal. In 2005, Lundin [29] presented a more general framework based on
four basic haptic primitives: point, line, plane, and directed force. This frame-
work allows for direct volume haptics in non-orthogonal LRFs and provides
an intuitive abstraction layer for proxy-based volume haptics. The primitive-
haptics framework was further developed in [28] and summarized in Lundin’s
PhD thesis [35].
The implementation of proxy-based direct volume haptics used in this work
has mainly been based on the framework in [18]. The fundamentals of this
framework is described below.
Let {e0,e1,e2} denote the LRF, pt the proxy position at time step t, xt the
probe position, and d = (xt − pt−1) the displacement of the probe relative to
the previous proxy position. In each iteration of the haptic loop, the proxy is
moved in small steps according to local data measures, haptic transfer func-
tions, and rendering parameters. Allowed proxy movements are defined by
motion rules for each axis in the LRF. The proxy position at time step t is
computed as
pt = pt−1 +2
∑i=0
Δpiei,
where Δpi is a motion rule function of the displacement di = d · ei. The re-
sulting force is computed as f t =−k(xt −pt), where k is the stiffness of the
spring-coupler. Figure 5.5 gives an illustration of the algorithm. The motion
rule function for a unilateral constraint along axis i is defined by
Δpi =
{di if di > 0
max(|di|− si/k,0) if di ≤ 0,
where si is the strength of the constraint, in this case the force threshold that
the user must apply to move in the direction −ei. Along +ei, there is free
motion. This is the motion rule commonly used for surface simulation with
axis i being the normal direction, see Figure 5.6. For a bilateral constraint,we have
Δpi = sign(di)max(|di|− si/k,0),
which is used, e.g., to simulate friction of a surface in the directions orthogonal
to the surface normal.
In addition to the constraints above, we also define the motion rule function
for a directed force:Δpi = si/k +di,
48
e
p
e0
t−1
2
p t
f txt
e1d
Figure 5.5: Proxy-based haptic volume rendering. The proxy is moved from pt−1 to
pt according to motion rules specified for each axis in the LRF. The resulting force is
proportional to the displacement (xt −pt).
d
0
0e
ks /
0ds0/
0e
k0
dd
Figure 5.6: Proxy-based volume haptics with a unilateral constraint for surface simula-
tion. Here, the gradient is used to compute the normal direction, i.e., e0 =−∇ f /||∇ f ||.In order to move in the direction −e0, the user has to apply a force such that
|d0| > s0/k. To the left, |d0| < s0/k which gives Δp0 = 0, i.e., the proxy will not
move. To the right, |d0| > s0/k which gives Δp0 = |d0| − s0/k, i.e., the proxy will
move Δp0e0.
which can be used, e.g., to follow a vector field. In our implementation of this
framework, the strengths si of the involved constraints are controlled through
haptic transfer functions, e.g., sigmoid functions based on the image intensity
I(p) at the proxy position:
si = si(I(p)) =A
1+ e−(I(p)−β )/α ,
where A is the maximum strength, β controls the center of the function, and αcontrols the width of the function. In Paper V and Paper VIII, we combine mo-
tion rules into haptic modes for haptic volume rendering of surfaces, viscosity,
potential fields, and vector fields. See also Section 6.3 and Section 6.6.
49
5.4 Volume haptics for medical image segmentation
Volume haptics for image segmentation is not yet widely used, but there has
been some work conducted in this field. Harders and Székely [17] showed
by conducting user studies how haptic feedback significantly improved man-
ual center-line extraction of tubular structures in MR images. The aim was
to initialize a deformable surface for segmenting the small intestines. They
performed an approximate segmentation through thresholding and computed
a distance map of the thresholded dataset. The haptic feedback was then com-
puted from the gradient of the distance map by using a direct force mapping
technique. Spuhler [45] presents methods for improved interactive centerline
extraction by using haptic guidance and snakelets for application in virtual en-
doscopy. Giess et al. [14] employed haptics for positioning of landmarks in a
3D segmentation application for liver surgery planning. Senger [42] describes
an immersive segmentation environment where proxy-based haptic feedback
is used to provide a sense of touch for segmented structures and to control a
seeded region-growing algorithm.
In this work, proxy-based volume haptics has been used in several different
ways for interactive segmentation. In Paper I, the method in [30] was used for
tracing of vessels in MR angiography images. The aim here was to facilitate
seeding of vessels for separation of arteries and veins. The same volume hap-
tics implementation was also used in Paper III for guiding the user to draw
live-wire curves in 3D. In Paper II, we made an implementation of the frame-
work in [18] in order to perform haptic rendering based on GVF fields. This
implementation was extended in Paper V where it was used for exploration of
external force image data for deformable model segmentation.
50
6. Contributions
In this chapter, the methods and results described in detail in the appended pa-
pers are presented briefly. The theory behind the methods has been described
in the previous chapters, and therefore the focus in this chapter is on the ap-
plication of the methods and the results.
6.1 Haptics for seeding of region growing algorithms
As mentioned in the introduction, semi-automatic segmentation methods
should combine the outstanding capabilities of the human to recognize
objects with the capabilities of the computer to perform the tedious task of
exact delineation. An example of recognition is seeding, i.e., positioning of
seed-points and seed-regions inside the object of interest.
6.1.1 Seeding of vessels in MRA
In Paper I, we present a method for seeding of vessels in whole-body CE-
MRA images directly in 3D guided by a hardware accelerated MIP and vol-
ume haptics. We use an implementation of Lundin’s proxy-based direct vol-
ume haptics method [30] where frictional surfaces are simulated by using
the gradient as surface normal. Thereby we are able to render stable haptic
feedback based directly on the image data, which differs from previous at-
tempts to trace tubular structures where direct force mapping from segmented
data [3, 17] has been used. The resulting haptic feedback allows the user to
trace the vessels, place seed-points with different labels (colors), and expand
them to larger seed-regions with a region growing method. The goal is to use
the seeds as input to a gray-scale connectedness algorithm (similar to FC)
in order to separate the main arteries from each other and from other bright
objects.
Previously, the seeding was performed with mouse interaction in an MPR
view. An advantage of the haptic interface and interaction compared to the
previous method, is a large increase of the number of placed seed-points and
thereby faster convergence of the gray-scale connectedness segmentation. An-
other advantage is the possibility to trace vessels directly in 3D which facili-
tates seeding of vessels with complex orientation. Figure 6.1 shows a screen-
shot from the application and a vessel separation result.
51
Figure 6.1: Left: A screenshot from the application. The user is seeding the vascular
tree in a head and upper thorax data set guided by volume haptics and a hardware
accelerated MIP. Right: Final gray-scale connectedness segmentation result.
During the work with this application we made the first implementations of
proxy-based volume haptics and hardware accelerated volume rendering that
became fundamental tools in coming projects.
6.1.2 Volume haptics based on GVF
In Paper II, we introduce proxy-based volume haptics rendering based on GVF
fields. The application here is again seeding, but now for interactive liver seg-
mentation from MR images. The aim is to initialize a seed-region inside the
liver and then perform segmentation with the fast marching method. A draw-
back with the fast marching method is that it is prone to leaking if the ini-
tialization is asymmetric and/or positioned too close to the object boundary.
Therefore we provide haptic feedback to guide the user to stay centered inside
the liver in order to facilitate a symmetric initialization. Existing methods,
including our gradient-based feedback used in Paper I, provides the desired
feedback only for narrow and elongated objects as vessels, but not for wider
objects as the liver since feedback is provided only immediately at the bound-
ary where the gradient is large enough.
To overcome this limitation, we suggest a rendering method based on GVF.
The initial idea with GVF was to increase the capture range of the gradient for
segmentation with deformable models by propagating the gradient informa-
tion of strong boundaries into the inner part of homogeneous regions. Here,
we use this property to provide haptic feedback that allows the user to stay in-
52
side the object while still feeling its boundaries. In this work we use the proxy-
based volume haptics framework by Ikits [18] described in Section 5.3.3. The
first component in our LRF is determined by the direction of the GVF field
V at the proxy position, i.e., e0 = V(p)/|V(p)|. The orthogonal directions are
constructed so that e1 ·e0 = 0, |e1|= 1, and e2 = e0×e1. We apply a unilateral
constraint with strength s0 that obstructs proxy movements along the GVF
field. In the orthogonal directions we use low-strength bilateral constraints.
The idea is to provide feedback that allows the user to move freely inside
the object without getting out. Therefore, the strength s0 is modulated with a
sigmoid shaped transfer function based on the magnitude of the GVF field:
s0 = s0(|V(p)|) = Tmin +Tmax−Tmin
1+ e−(|V(p)|−β )/α ,
i.e., s0 ∈ [Tmin,Tmax]. The parameters α and β are interactively controlled by
the user. This way, we obtain a high strength close to object boundaries where
the GVFmagnitude is high and a lower strength far away from the boundaries.
The suggested haptic rendering was tested on both synthetic and real data,
see Figure 6.2. From the force magnitude graph we see that the feedback
behaves as desired. The magnitude of the rendered force increases when we
move the haptic probe from the middle of the object towards the border. This
is because the constraint strength s0 increases with GVF magnitude. When we
move along the border there are only small changes in magnitude and while
we move along the GVF field into the center the force magnitude is very small.
The peak to the right occurs when we move from the center and finally leave
the object.
A drawback with this method at the time of submission of Paper II was
the time-consuming computation of the GVF field. In our work [51], we use
more advanced numerical schemes in order to speed up the computations.
With these schemes, the GVF equation was solved 1–2 orders of magnitude
faster than with the commonly used straight-forward implementation.
6.2 An extension of live-wire to 3D
In Paper III, we present a new 3D extension of the live-wire method. In the
2D live-wire method, the user places seed-points that are connected by optimal
minimum-cost paths. Our idea for a 3D extension is to let the user draw live-
wire curves which are connected by discrete minimum-cost surfaces, a process
we call bridging. The aim is to segment entire objects by drawing a relatively
small number of live-wire curves on the boundary of the object.
The live-wire method involves Dijkstra’s algorithm to compute shortest
paths in a graph representation of the image. To convert a 3D image into a
graph, we have chosen to place a node at the center of each voxel and cre-
ate edges to all its 26 neighbors. This definition does not use oriented edges
53
0 5 10 150
0.5
1
1.5
Time (s)
Fo
rce
mag
nit
ud
e (N
)
Figure 6.2: Top left: A slice of the test volume and the path traveled with the hap-
tic device. Top right: The magnitude of the rendered force along the path. Bottom:
Screenshot from the application where the user has seeded the liver in an MR image
guided by the GVF-based haptic feedback.
as in [13], but it has the advantage that the resulting graph has relatively few
nodes. This is important since the bottleneck of Dijkstra’s algorithm is main-
taining the heap containing the nodes.
For actually drawing the curves interactively in 3D, we have implemented
a user interface where the user has two options: (1) place seed-points freely
in the volume guided by volume haptics and volume rendering, and (2) draw
the curve onto an arbitrarily oriented slice of the volume, see Figure 6.3. The
haptic feedback in the first case is proxy-based volume haptics tuned to feel
the surface of the object, and in the second case the slice plane is a haptic
surface that the user can feel while drawing.
The bridging algorithm for connecting two curves is based on the image
foresting transform (IFT) [10]. The IFT is essentially Dijkstra’s algorithm for
54
Figure 6.3: Illustration of the 3D live-wire method. Left: Placing seed-points freely in
the volume using volume rendering and volume haptics to locate the boundary of the
object. Right: Drawing a live-wire curve onto an arbitrarily oriented slice.
(a) (b) (c) (d)
Figure 6.4: Illustration of the bridging algorithm. (a) A synthetic spherical object. (b)
Two closed live-wire curves on the boundary of the object. (c) Result of connecting
the two curves with the IFT. (d) Result of our algorithm including rasterization.
shortest path calculation, modified to allow an arbitrary number of seed-points
in order to find the shortest path from all points in the image to any of the seed-
points. With the IFT, we compute the optimal curves from each voxel in one
of the curves, using all voxels of the other curve as seed-points. Since the
curves we compute are independent of each other, the result is generally not a
tunnel-free surface. To fill the gaps between the curves we define a polygonal
surface between adjacent curves. These polygons are then rasterized to obtain
a discrete surface, see Figure 6.4. Note that all points on the second curve are
not necessarily the closest point to a point on the first curve. Therefore we run
the bridging algorithm from both directions and use the union of the results.
In this way, we are guaranteed that the result always includes the two original
curves.
55
6.3 Interactive deformable models
In Paper V we present our framework for interactive segmentation with de-
formable models. The framework involves a standard 2-simplex mesh data
structure and deformation engine, see Section 3.6.2. The main contribution
here is the new interaction possibilities. By using our 3D interface we can
perform initialization of the model close to the structure of interest. This is
an important property since the sensitivity to initialization is commonly re-
garded as the main drawback with deformable model segmentation. In addi-
tion to the enhanced initialization we have the possibility to interact with the
model by providing interaction forces during deformation in order to speed
up convergence and to obtain a more accurate segmentation. As described in
Section 3.6.3, a deformable model is typically driven by a number of image
based external forces. In general, it is not possible to visualize all of the un-
derlying data that drives the model. Therefore, we also use volume haptics as
an additional channel to convey information about the data. This facilitates
the parameter tuning, which is regarded as another drawback with deformable
model segmentation.
Deformation engine
In our implementation, the deformation engine is based on Newtonian evolu-
tion and the internal regularizing forces are based on curvature adaption in the
tangential direction and a C2 constraint in the normal direction. We use the
four external forces that are described in Section 3.6.3: gradient magnitude
force, inflation force, potential field force, and vector field force.
Mesh interaction
The most basic interaction with the mesh is for initialization where the haptic
device is used to grab and translate the initial mesh. In order to provide inter-
action forces during deformation, we allow the user to select parts of the mesh
with the haptic probe, see Figure 6.5. The selected vertices are then affected
by interaction forces computed in a similar way as the image based external
forces, i.e.,
(Finteraction)i = wi(fprobe ·ni)ni,
with
fprobe = (r−pi)/||r−pi||.
Here, wi is a weight, r is the position of the probe, pi is the position of the
current vertex, and ni is the normal direction at pi. The sign of wi controls the
direction of the force, i.e., if the user will pull or push the selected sub-mesh.
This interaction is performed with haptic feedback where the force rendered
56
Figure 6.5: Left: Initialization of the model with a sphere consisting of 42 vertices and
80 faces. The fuzzy connectedness map is overlaid on the original CT image. Right:
Interaction forces are applied to selected vertices (yellow).
to the user is the reaction force averaged over all selected vertices, i.e.,
Freaction =− 1
Nselected∑i∈S
(Finteraction)i,
where the set S contains the indices for the Nselected selected vertices.
Volume haptics
For the purpose of rendering the different types of external force data, the
volume haptics framework described in Section 5.3.3 was used to implement
four haptic modes. Note that in Paper V, only three modes were implemented,
but here we also include the additional viscosity mode that was added in
Paper VIII. The modes are:
Surface mode: In surface mode, a unilateral constraint is used in the normal
direction e0 = ∇I(p)/||∇I(p)|| and bilateral frictional constraints are used in
the orthogonal directions. The user sets the strength of the normal constraint
s0 and the friction coefficient μ through haptic transfer functions, i.e.,
s0 = s0(I(p)) and μ = μ(I(p)). The strengths of the frictional constraints are
computed as s1 = s2 = |f · e0|μ , where f is the current force generated by the
spring-coupler.
Viscosity mode: The aim with this mode is to provide an intensity-modulated
resistance force. Three bilateral constraints with equal strength are used. The
strength is controlled with a transfer function based on the intensity at the
proxy position. This mode is orientation independent which means that we
57
can choose an arbitrary LRF.
Potential mode: In the potential mode, the main direction is computed
as e0 = ∇P(p), where P is a potential field, e.g., a signed distance map.
The constraint is either unilateral or a directed force. The strength s0 is
controlled with a transfer function based on the gradient magnitude, i.e.,
s0 = s0(||∇P(p)||). For the orthogonal directions e1 and e2, bilateral frictionalconstraints are used.
Vector mode: The vector mode is identical to the potential mode, except
that the direction of the vector field V is used as the main direction, i.e.,
e0 = V(p)/||V(p)||.
Results
To demonstrate the system, we made a small-scale experiment where we per-
formed interactive segmentations of the liver in 10 abdominal contrast en-
hanced venous phase CT images. For each image, we have a manual de-
lineation of the liver made by a radiologist using the routine software of a
Siemens Leonardo workstation. The time required for the manual delineation
was between 5 and 18 minutes per dataset
To drive the model, we chose an inflation force based on a fuzzy connect-
edness (FC) map that we compute prior to initialization of the model. Seeding
of the FC map is performed interactively by using manual seeding of interior
liver voxels using the haptic probe. The deformable model is interactively ini-
tialized by a sphere, see Figure 6.5. When the deformation engine is activated,
the initial model propagates in its normal direction based on the FC map and
the internal curvature based forces adapt the mesh resolution locally to the
object. Typically, the initial mesh is coarse, and therefore global mesh refine-
ments through subdivision are also needed. By applying interaction forces
during deformation and performing global mesh refinements, the segmenta-
tion result is obtained in 2–3 minutes. The resulting mesh is voxelized in or-
der to measure the accuracy against the manual segmentations. This is done
according to the framework described in Section 3.7. The results show that we
obtain an average sensitivity of 90% and an average specificity of 99% for the
10 datasets.
6.4 Accurate and reproducible liver segmentation
In Paper IV, we describe an interactive fast marching method for liver seg-
mentation. Four users independently performed segmentation of the liver in 52
abdominal contrast enhanced venous phase CT images from 26 patients with
58
either carcinoid or endocrine pancreas tumor that underwent two CT exami-
nations with time interval 3–17 months. The aim was to measure the volume
of the liver because disease progress is correlated to liver enlargement.
The fast marching method performs well, but in some cases there are prob-
lems with leakage due to low contrast between the liver and surrounding tis-
sue, especially between the liver and the heart. In Paper VI, we combine the
fast marching segmentation with a subsequent deformable simplex mesh seg-
mentation in order to mitigate the leakage problem.
Fast marching segmentation
The fast marching method in Paper IV involves interactive initialization of
the method (seeding) with the haptic interface, computation of a cost image,
and fast marching propagation of seed-regions with automatic time-of-arrival
thresholding. In the initialization step, the user performs seeding of the liver in
an MPR view where a haptic spring-force guides the user to draw seeds in the
desired slice. The reason why we did not use the GVF-based volume haptics
here was because our efficient GVF solver [51] was not developed yet. With
the standard solver, the computation time required would simply be too long
for the large number of datasets.
The average seeding time for the four users was 40 seconds and the average
number of initialized seed-voxels per dataset was just below 3000. Based on
gray-level statistics from the seed-regions, the cost image was computed in
four steps: (1) edge-preserving smoothing using bilateral filtering (see Sec-
tion 2.2.2), (2) voxel-wise gray-level transformation based on statistics from
the seed-regions, (3) gradient magnitude extraction, and (4) weighted sum of
the result from (2) and (3). See Figure 6.6. The result from the fast marching
method, as described in Section 3.4, is a time-of-arrival map that need to be
thresholded to obtain a crisp segmentation. To automatically find a threshold
value, we examine the average cost of the front voxels during propagation
and choose the time value with maximum average cost as threshold. This is
illustrated in Figure 6.7 together with a segmentation result.
Deformable model segmentation
In Paper VI we use our deformable model framework to fit simplex meshes to
the fast marching results in order to mitigate the leakage problem and obtain
a more accurate segmentation. From the fast marching result, we compute
a signed distance map with positive values outside the contour and negative
values inside the contour. This distance map is regarded as a potential field
and is used as an external force to drive the deformable simplex mesh. The
aim is to fit the mesh to the fast marching segmentation and make use of
the built-in shape regularization of the mesh to avoid capturing the irregular
erroneous regions caused by leaking. In addition to the potential force, the
59
Figure 6.6: Cost image generation. (a) Original CT image. (b) Result of bilateral fil-
tering. (c) Result of voxel-wise gray-level transformation. (d) Gradient magnitude of
(c). (e) The resulting cost image, a weighted sum of (c) and (d). (f) Gray-level profile
for the edge in (e).
Figure 6.7: Left: The average cost of the front voxels at each time instance. The time
of arrival at the maximum peak is used as a threshold to obtain the final segmentation.
Right: A slice of one of the 52 datasets with overlaid segmentation result (blue) and
the seed regions (red).
60
Figure 6.8: Left: Surface rendering of the fast marching segmentation result for one of
the livers. Note the leakage problem between the liver and the heart. Right: Segmen-
tation result obtained with the deformable mesh. Here, the irregular region caused by
leaking is removed because of the built-in shape regularization of the mesh.
user also provides interaction forces during deformation. In this experiment,
one user performed liver segmentation with the deformable simplex mesh for
two of the four sets of fast marching segmentations.
Results
We quantitatively evaluate both segmentation precision and accuracy using the
framework described in Section 3.7. For 23 of the datasets, we have manual
delineations performed by two radiologists using the routine software of a
Siemens Leonardo workstation. From these delineations, we construct a fuzzy
ground truth per dataset by averaging the manual delineations as described in
Section 3.7. The simplex mesh segmentations were voxelized in order to be
compared with the true segmentations.
The mean interaction time for seeding of the fast marching method was 40
seconds per dataset with standard deviation (SD) 23 seconds. For the subse-
quent deformable mesh segmentation, we have a mean interaction time of 93
seconds (SD 16 seconds). The interaction time required for the manual delin-
eation was between 5 and 18 minutes. For the two sets of 23 manual segmenta-
tions performed by the radiologists, we obtain a mean precision of 88.9% with
coefficient of variation (CV) 1.9%. The mean precision of the fast marching
method is 96.9% (CV 3.8%), which is considerably higher. For the two sets
of simplex mesh segmentations, we obtain a mean precision of 97.8% (CV
0.5%) which indicates a high reproducibility. For the fast marching method,
the average sensitivity is 93% and the specificity is close to 100%, i.e., we
have very few false positive voxels. When we apply the deformable mesh seg-
mentation, we obtain an average sensitivity of 96%, i.e., an increase of three
percentage points while the high specificity is maintained. See Figure 6.8 for
illustrations of the results.
61
6.5 Visualization of dynamic breast MRI
In Paper VII, we present a new approach for visualization of dynamic contrast-
enhanced MRI images of the breast. DCE-MRI is emerging as a useful clini-
cal tool in the diagnosis and staging of breast cancer. It involves acquisition of
T1-weighted images of the breast before and after the injection of a contrast
agent. The signal intensity over time turns out to be an important criterion for
differentiating benign lesions from malign lesions. The enhancement curves
for most cancers exhibit rapid initial uptake and then washout whilst benign
tissue have either no enhancement, or slowed continued enhancement with de-
layed washout. Methods for quantitative analysis of the enhancement curves
exist, but the most commonly adopted approach in routine clinical practice
is the qualitative approach. This involves the tedious and error-prone manual
review of the raw 4D data and derived subtraction images (post-contrast mi-
nus pre-contrast) to identify suspicious areas, and a qualitative evaluation of
the shape of the associated enhancement curves. The high dimensionality and
high resolution of the data overwhelms the user with information.
In our work, we reduce the dimensionality of the data by parameteriza-
tion of the voxel-wise enhancement curves. Intensity observations over sev-
eral time points are reduced to a smaller number of parameters that can be
color-coded in hue-saturation-value (HSV) color space [15] to permit a 3D vi-
sualization of the 4D data. The parameterization and subsequent HSV color-
coding, described in Paper VII, results in a color volume where voxels with
rapid initial enhancement appear brighter and more saturated whilst the na-
ture and degree of intermediate to late enhancement is reflected in the hue
channel. To visualize this color volume, we perform a maximum intensity
projection (MIP) in HSV space. Since we want the rendering to be fast, we
implement a ray-casting engine on the GPU, see Section 4.3. We store the
color-coded volumes using an RGBα representation where the RGB part con-
tains the converted HSV triple, and the α-component contains the intensity
V. This representation allows for quick access to the intensity and no HSV
to RGB conversions are needed during rendering. Since the color volume has
high intensity in regions with high contrast agent uptake and almost zero in-
tensity elsewhere, direct visualization of this image can be hard to interpret
due to the lack of geometric context. We mitigate this problem by blending
the color volume with the pre-contrast volume. Figure 6.9 shows a screen-
shot from our software which also includes a slice-based interface for contour
delineation of regions of interest (ROIs).
Results
We made an experiment based on DCE-MR image data from 14 subjects. The
data were chosen to include 9 subjects with enhancing lesions subsequently
confirmed by histopathology to be cancer, and 5 subjects with enhancing le-
62
Figure 6.9: Screenshot from the DCE-MRI Viewer showing one of the color-coded
datasets. The uptake of contrast medium over time in each voxel of the volume is con-
verted to a HSV color encoded volume that is visualized with hardware accelerated,
color-correct maximum intensity projection (MIP). The MIP can be rotated and ex-
plored dynamically giving a direct overview the volume. It is also possible to browse
the slices of the volume and to draw contours defining regions of interest.
sions subsequently confirmed by histopathology to be benign. The data in-
cludes screensaves of the suspicious regions of interest originally traced by
a radiologist and the corresponding findings. One of the malignant data sets
was used to familiarize the operator with our software and the nature of the
visualization for benign and malignant tissue. The operator was asked to de-
lineate suspicious lesions for the remaining 13 subjects and to report an over-
all finding of either malignant or benign. Our software permitted the operator
to examine the color MIP interactively and to define ROIs within individual
2D slices. The results show that the operator correctly identified eight of the
ten malignant cases identified by the original interpreting radiologist, i.e. 80%
sensitivity for malignancy, and two of the three benign cases; i.e. 67% speci-
ficity. The results also show that with respect to the histopathology findings,
the operator had a sensitivity of 100% and a specificity of 80%, whilst the
original interpreting radiologist had a sensitivity of 100% and specificity of
only 60%.
63
6.6 The WISH toolkit
During our work with research and development of methods for segmentation,
visualization, and haptic rendering we have assembled a fairly large code li-
brary. With the aim to make this code library more homogeneous, usable, and
maintainable we have created a toolkit called WISH—Interactive Segmen-
tation with Haptics. The design and contents of this toolkit is described in
Paper VIII.
The core of the WISH software consists of a stand-alone C++ class library
for image analysis, visualization, and volume haptics. This means that the al-
gorithms can be integrated into applications written in C++ or even used as
command-line tools. However, in order to realize the haptic-enabled interac-
tive tool we aim for, we have chosen to create an interface between the core
functionality of WISH and the open source H3D API from SenseGraphics,
see Section 5.2. H3D API is implemented in C++ and is based on the X3D1
scene-graph standard. It also supports Python2 scripting.
The interface between the WISH core and the H3D API is called
WISHH3D, and consists of H3D API scene graph nodes for the image
processing filters, segmentation algorithms, visualization algorithms,
and volume haptics algorithms in WISH. The nodes are compiled into a
separate dynamically linked library that is imported into the main H3D API
application at startup. In this way, the WISHH3D nodes can be used and
controlled through X3D or Python. For our graphical user interface (GUI),
we use the TkInter library in Python. See Figure 6.10 for an illustration of the
toolkit components and Figure 6.11 for the GUI.
The WISH toolkit is a research tool that is intended for quick development
and prototyping of new interactive segmentation methods with multi-sensory
feedback. WISH integrates volume haptics and volume visualization with in-
teractive segmentation methods such as fast marching, deformable simplex
meshes, and live-wire. It also includes efficient implementations of several
other image analysis tools. The WISH toolkit is publicly available and can be
downloaded from http://www.cb.uu.se/research/haptics.
1URL: http://www.web3d.org2URL: http://www.python.org
64
wishH3DVolume
wishH3DVolumeSlicer
wishH3DVolumeRenderer
wishH3DVolumeHaptics
wishH3DFastMarchingFilter
C++
Python
X3D
TkInter
H3D API
WISHH3D
WISH
OpenHaptics
OpenGL
Hapticdevice
GUI
Hapticdevice
Graphicswindow
...
Figure 6.10: The relations between WISH and the H3D API. WISHH3D is compiled
into a dynamically linked library with nodes that is imported into the H3D API. The
main scene is set up with X3D. Python is used to set up and manage the connections
between the TkInter GUI and the nodes and also the connections between nodes. On
the user input side we have the GUI and the haptic device. On the output side we have
OpenGL graphics and haptic feedback.
65
Figure 6.11: Part of the WISH TkInter GUI.
66
7. Conclusions
In this chapter, the work in this thesis is concluded with a summary of the
contributions and suggestions for future work.
7.1 Summary of contributions
A number of segmentation methods have been developed and tailored for im-
plementation in an environment with support for haptic interaction. These are
fast marching, fuzzy connectedness, 3D live-wire, and deformable simplex
meshes. Some of them (fast marching, fuzzy connectedness) require basic in-
teraction for facilitating navigation and seeding, while others (live-wire, de-
formable meshes) require more complex interaction. The performance of the
deformable meshes and the fast marching method was evaluated on a fairly
large set of real image data with convincing results.
For visualization, a fast ray-casting engine was implemented on the GPU. It
was used to render parametrically mapped time sequences of breast MRI data
with a color-correct maximum intensity projection (MIP). The potential of
this novel visualization technique was verified with experiments on real data.
The ray-casting implementation is general and allows for complex volume
visualization at interactive rates.
Stable proxy-based haptic rendering for volumetric data was implemented
and specially designed for use in interactive segmentation. In a first appli-
cation, this type of volume haptics was used for tracing of vessels in MRA
images. The encouraging results from this work led to the development of
a new rendering mode based on gradient vector flow (GVF) for facilitating
segmentation initialization in generally shaped structures. For the deformable
model method, the volume haptics engine was extended with a number of
haptic modes designed for exploration of external force data.
Most of the methods developed in this work have been assembled in a soft-
ware toolkit for interactive segmentation with haptic interaction. The aim with
this software is to provide a development framework that in the future can be
used for implementation of new image analysis and visualization methods that
can benefit from multi-sensory interaction.
67
7.2 Future work
An important topic for future research is evaluation. The evaluation and val-
idation we have made of the segmentation results indicate that haptics has a
clear potential for interactive segmentation. However, this potential must also
be quantitatively evaluated. An issue for such an evaluation is to find ways
to perform fair comparisons between methods. Methods designed for user in-
teraction with haptic feedback are often very different from methods that use
standard interaction techniques.
Another concern for future work is the importance of collaboration. The
multi-disciplinary nature of this field means that there are many methods and
techniques to consider how to combine. In each discipline, we need to find the
methods that are most suitable for interactive medical image analysis and if
they do not exist they should be developed. Therefore, one main suggestion to
move this research forward is strong collaboration between groups specialized
in haptics, visualization, and image analysis. It should be emphasized that this
collaboration to a high degree also should involve the physicians who are the
potential demanding end-users.
Some specific ideas for future work concern improvement and further de-
velopment of the presented methods. It would be interesting to combine the
3D live-wire method with segmentation based on graph cuts [5]. Live-wire
could be used for marking the boundary between object and background and
use this as input to the graph cuts algorithm.
A desirable extension to the deformable mesh framework is to be able
to handle other topologies than spherical. It would also be interesting to
work with multiple coupled objects and to incorporate statistical shape
constraints [6]. Another idea is to include more haptic tools for local
refinement of the model. In the LiverPlanner [37], such tools are described,
but not with haptic interaction.
The GPU-accelerated ray-casting engine should be extended to handle large
datasets. Due to the high resolution of medical images acquired with the latest
imaging equipment it is not possible to store the full volumes in GPUmemory.
There are techniques that combines efficient data reduction and memory man-
agement in order to render very large datasets [26]. An improvement concern-
ing rendering quality is to implement pre-integrated volume rendering [55].
The volume haptics framework should be generalized to work with non-
orthogonal reference frames. Here, the plan is to use the Volume Haptics
Toolkit (VHTK) [28, 35] for primitives-based haptics available for the H3D
API.
68
7.3 Conclusion
This work has contributed to research in three different fields: image analysis,
visualization, and haptics. In the application to interactive medical image seg-
mentation, methods in these fields have been closely integrated. To conclude,
we go back to the list of tasks in Section 1.6:
• New methods and improvements of existing methods for visualiza-
tion, haptic rendering, and interactive segmentation have been devel-
oped.
• The methods have been applied and evaluated on real medical image
data.
• A software framework with efficient and general implementations of
the methods has been made available for future research and develop-
ment. This framework is based on open source code and is therefore
easy to extend and integrate into other packages.
69
8. Summary in Swedish
Dagens utrustning för att skapa tredimensionella (3D) bilder av
människokroppens inre är oumbärlig för att diagnostisera och planera
behandling av t. ex. cancer. Vanliga avbildningstekniker är datortomografi
(eng. computed tomography, CT) där kroppen avbildas med röntgenstrålar
från många olika vinklar runt kroppen och magnetresonanstomografi (MRT)
där väteatomers egenskaper i olika vävnader i kroppen avbildas med en
magnetkamera. Dessa tekniker har utvecklats i snabb takt de senaste åren och
varje dag genereras mycket stora mängder bilddata som ska analyseras och
tolkas. Figur 1.1 och Figur 1.2 visar exempel på CT- och MR-bilder.
Att utvinna information ur digitala bilder kallas för digital bildanalys. I
medicinska tillämpningar kan bildanalys hjälpa till med uppgifter som t. ex.
volymmätning av organ baserat på bildinformation eller att konstruera 3D-
modeller av organ för användning inom kirurgiplanering och behandling. Ett
centralt problem som ofta måste lösas är segmentering, dvs. att separera or-
gan från varandra och från bildbakgrunden. Segmentering är ofta ett tidigt steg
i bildanalysprocessen och ligger till grund för all fortsatt informationsutvin-
ning och beräkning. Att segmentera flera organ manuellt i högupplösta bilder
är mycket tidskrävande och tröttsamt vilket innebär att det är lätt att göra fel.
Tyvärr är det inte heller möjligt att låta datorer utföra segmenteringen helt au-
tomatiskt eftersom kontrasten mellan olika organ i bilderna kan vara väldigt
låg och formen på organ kan variera mycket mellan patienter.
För att underlätta arbetet kan man istället använda interaktiva eller semi-
automatiska metoder där datorn gör en stor del av jobbet, men där användaren
kan styra och övervaka datorns arbete. Denna doktorsavhandling handlar om
utvecklandet av sådana metoder där den klassiska typen av interaktion med
mus och tangentbord ersätts med direkt 3D-interaktion som låter användaren
känna på bilderna med hjälp av haptisk återkoppling (kraftåterkoppling). För
att visualisera bilderna på skärmen används stereografik som ger en förstärkt
djupupplevelse jämfört med vanlig visualisering. Figur 1.3 och Figur 5.3 visar
den typ av arbetsstation som har använts.
Forskningsarbetet bakom avhandlingen innefattar tre områden: bildanalys,
visualisering och haptik. Inom bildanalysen har ett antal välkända
segmenteringsmetoder anpassats och byggts ut för att kunna användas
tillsammans med den här typen av interaktion. I de enklaste fallen så
handlar det om att användaren snabbt och effektivt kan placera ett stort antal
markörer i bilden och utifrån dessa segmenterar datorn det organ man är
71
intresserad av. Två mer avancerade metoder illustreras på omslagsbilden.
I den vänstra bilden jobbar användaren med en deformerbar ytmodell.
Modellen består av ett antal små ytelement som sitter ihop med varandra
enligt fördefinierade villkor. Genom att koppla bildinformation till varje
punkt på modellen kan man få den att anpassa sig till bilden. I detta exempel
försöker användaren få den att anpassa sig till levern i en CT-bild genom
att påverka den med det haptiska instrumentet. Bilden till höger illustrerar
en metod som heter live-wire. Här placerar användaren en punkt på konturen
av levern och när han sedan flyttar markören så försöker algoritmen hitta
den bästa vägen mellan den placerade punkten och markören. Med rätt
parameterinställningar så ligger den bästa vägen längs leverkonturen och
därmed kan hela konturen segmenteras med bara några få punktplaceringar.
Här känner också användaren organet med det haptiska instrumentet och kan
därför glida längs konturen.
Flera metoder för visualisering av volymdata har också utvecklats. Förutom
den klassiska tekniken att visa genomskärningsbilder så har också hårdvaru-
accelererad direkt volymrendrering (DVR) använts. Med direkt menas att in-
gen ytrepresentation av bilden behövs utan man jobbar direkt med bilddatat.
DVR är ett sätt att simulera genomlysning av den tredimensionella bilden för
att visa inre strukturer. Genom att ställa in s. k. överföringsfunktioner mel-
lan det verkliga bilddatat och den bilden som ska visas kan olika organ syn-
liggöras med olika genomskinlighet och färg. Denna metod är beräkningsin-
tensiv och lämpar sig i vanliga fall inte för interaktiv visualisering med hög
kvalitet. Men genom att utnyttja den höga beräkningskapacitet som finns i da-
gens grafikhårdvara (GPU) kan man utföra DVR i tillräckligt hög hastighet,
även med sterografik. Figur 4.3 och Figur 4.5 visar några exempel. Bland an-
nat så har DVR anpassats för att visualisera tidssekvenser av MR-bilder på
bröst som ett led i utvecklingen av en metod för att lättare upptäcka bröst-
cancer.
Den tredje delen är haptisk rendrering, dvs. metoder för att generera
kraftåterkoppling från de medicinska bilderna. Här har direkt volymhaptik
använts som i likhet med direkt volymrendrering arbetar direkt på
bilddatat. Det haptiska instrumentet har en interaktionspunkt, en pekare,
där kraftåterkopplingen ges. När användaren för pekaren genom datasetet
så kan man med en analys av pekarens omgivning avgöra om det finns en
yta eller en övergång mellan olika organ. Baserat på denna information
kombinerat med användarens rörelser genereras en kraft som användaren
känner. Haptiken har använts bland annat för att kunna följa blodkärl i bilder
och markera olika delar av kärlträdet för att senare kunna dela upp det i olika
delar för fortsatt analys. Ett annat exempel är utforskning av datat som driver
de deformerbara modellerna. Här har man ofta flera olika bilder som driver
modellen och det är svårt att visualisera dem samtidigt. Genom att istället
kunna känna på datat kan man underlätta parameterinställningarna och uppnå
ett bättre segmenteringsresultat.
72
Alla dessa metoder har applicerats och testats på riktiga medicinska bilder
och de resultat som har uppnåtts är positiva. Till exempel så har snabb och
effektiv segmentering av levern i ett stort antal CT-bilder genomförts med hög
noggrannhet och precision. Metoderna har även implementerats och samlats i
ett mjukvarupaket som har gjorts tillgängligt för fri nedladdning. Metoderna
kan på detta sätt komma till användning för framtida forskning och utveckling
av förbättrade bildanalysmetoder med 3D-visualisering och haptik.
73
Acknowledgments
This work was performed between February 2003 and January 2008. Five
years that have been very enriching. I have learned a lot about many different
things, I have seen interesting places, and most important, I have met so many
interesting persons. I have really enjoyed this time and this is because of you.
Special thanks to:
• My supervisor Ingela Nyström for friendship, support and good collabora-
tion. You have taught me so much and always been very encouraging and
helpful.
• My supervisor Ewert Bengtsson for giving me the opportunity to do this
work. You have given me a lot of invaluable advice and support.
• The reference group: Stefan Seipel, Gunnar Jansson, Lennart Thurfjell, and
Hans Frimmel for good advice and interesting discussions.
• Filip Malmberg for good collaboration and proofreading. Good luck with
your PhD! I will be happy to give help and advice if you need it.
• Stina Svensson and Bosse Nordin for helping me with the language and
formulations in the thesis.
• Sven Nilsson for good collaboration and for teaching me about the contents
of the images that I have been looking at every day.
• Andrew Mehnert and family in Brisbane for friendship, infinite hospitality,
and good collaboration.
• All other co-authors and collaborators: Xavier Tizon, Per Sundqvist, Milan
Golubovic, Michael Wildermoth, Kerry McMahon, Steven Wilson, Stuart
Crozier, Jonas Agmund, and Tomas Bjerner.
• Daniel Evestedt and the rest of the SenseGraphics team.
• Lena Wadelius for making CBA feel like home.
• The “CBA permanents”: Ewert Bengtsson, Gunilla Borgefors, Ingela Nys-
tröm, Lena Wadelius, Olle Eriksson, Tommy Lindell, and Bosse Nordin for
making CBA a wonderful working place.
• My friends and colleagues at (or related to) CBA that I have been traveling
with, playing golf with, having friday beer with, visiting in Australia,
watching football with, having lunch with, sharing office with, working
with, going to the movies with, skiing with, creating “Blaskan” with,
...: Ida-Maria Sintorn, Ola Weistrand, Kristin Norell, Magnus Gedda,
Robin Strand, Stina Svensson, Patrick Karlsson Edlund, Hamid Sarvé,
Filip Malmberg, Milan Gavrilovic, Khalid Niazi, Bettina Selig, Maria
75
Axelsson, Amin Allalou, Amalka Pinidiyaarachchi, Joakim Lindblad,
Nataša Sladoje, Carolina Wählby, Petra Philipsson, Mats Erikson, Roger
Lundqvist, Felix Wehrmann, Xavier Tizon, Mattias Aronsson, Hamed
Hamid Muhammed, Pasha Razifar, Mathias Raspe, Joel Kullberg,
Suthakar Somaskandan, Anders Hast, Lars Pettersson, Catherine Östlund,
Gerhard Bax, Celine Fouard, Jocelyn Chanussot, Susana Mata, Julia
Åhlen, Roger Hult, Stefan Seipel, Lucia Ballerini, Ewert Bengtsson,
Gunilla Borgefors, Ingela Nyström, Lena Wadelius, Olle Eriksson,
Tommy Lindell, Bosse Nordin, and many others.
♥ Min familj och mina vänner. Speciellt tack till Mamma och Sven-Erik för
allt ert stöd och till Terese för att du är underbar och har fått mig att koppla
av och tänka på annat under dessa intensiva månader.
Uppsala, January 2008
Erik
76
Bibliography
[1] R. Adams and L. Bischof, Seeded region growing, IEEE Transactions on Pat-
tern Analysis and Machine Intelligence 16 (1994), no. 6, 641–647.
[2] R. S. Avila and L. M. Sobierajski, A haptic interaction method for volumevisualization, Proceedings of IEEE Visualization, 1996, pp. 197–204.
[3] D. Bartz and Ö. Gürvit, Haptic navigation in volumetric datasets, Proceed-ings of the PHANToM User Research Symposium, 2000.
[4] G. Borgefors, On digital distance transforms in three dimensions, Computer
Vision and Image Understanding 64 (1996), no. 3, 368–376.
[5] Y. Boykov and M. P. Jolly, Interactive graph cuts for optimal boundary andregion segmentation of objects in N-D images, Proceedings of InternationalConference on Computer Vision, 2001, pp. 105–112.
[6] M. J. Costa, H. Delingette, S. Novellas, and N. Ayache, Automatic segmenta-tion of bladder and prostate using coupled 3D deformable models, Pro-ceedings of Medical Image Computing and Computer-Assisted Intervention,
LNCS, no. 4791, 2007, pp. 252–260.
[7] H. Delingette, General object reconstruction based on simplex meshes, In-ternational Journal of Computer Vision 32 (1999), no. 2, 111–146.
[8] E. W. Dijkstra, A note on two problems in connexion with graphs, Nu-
merische Mathematik 1 (1959), no. 1, 269––271.
[9] R. A. Drebin, L. Carpenter, and P. Hanrahan, Volume rendering, Computer
Graphics 22 (1988), no. 4, 65–74.
[10] A. X. Falcão and F. P. G. Bergo, Interactive volume segmentation with dif-ferential image foresting transforms, IEEE Transactions on Medical Imaging
23 (2004), no. 9, 1100–1108.
[11] A. X. Falcão and J. K. Udupa, A 3D generalization of user-steered live-wiresegmentation, Medical Image Analysis 4 (2000), no. 4, 389–402.
[12] A. X. Falcão, J. K. Udupa, and F. K. Miyazawa, An ultra-fast user-steeredimage segmentation paradigm: Live-wire on the fly, IEEE Transactions on
Medical Imaging 19 (2000), no. 1, 55–62.
77
[13] A. X. Falcão, J. K. Udupa, S. Samarasekara, S. Sharma, B. E. Hirsch, and R. A.
Lotufo, User-steered image segmentation paradigms: Live wire and Livelane, Graphical Models and Image Processing 60 (1998), no. 4, 233–260.
[14] C. Giess, H. Evers, S. Hassfeld, G. Glombitza, R. De Simone, and H-P. Meinzer,
Haptic volume rendering for 3D segmentation as a prerequisite for surgi-cal planning, Proceedings of the 1st International Workshop on Haptic Devices
in Medical Applications, 1999, pp. 102–108.
[15] R. C. Gonzalez and R. E. Woods, Digital image processing, third ed., Prentice
Hall, Inc., 2008.
[16] G. Hamarneh, J. Yang, C. McIntosh, and M. Langille, 3D live-wire-basedsemi-automatic segmentation of medical images, Proceedings of SPIEMed-
ical Imaging: Image Processing, 2005, pp. 1597–1603.
[17] M. Harders and G. Székely, Improving medical segmentation with hapticinteraction, Proceedings of IEEE Virtual Reality, 2002, pp. 243–250.
[18] M. Ikits, J. D. Brederson, C. D. Hansen, and C. R. Johnson, A constraint-basedtechnique for haptic volume exploration, Proceedings of IEEE Visualization,
2003, pp. 263–269.
[19] H. Iwata and H. Noma, Volume haptization, Proceedings of IEEE Symposium
on Research Frontiers in Virtual Reality, 1993, pp. 16–23.
[20] Y. Kang, K. Engelke, and W. A. Kalender, Interactive 3D editing tools forimage segmentation, Medical Image Analysis 8 (2004), no. 1, 35–46.
[21] M. Kass, A. Witkin, and D. Terzopoulos, Snakes: Active contour models,International Journal of Computer Vision 1 (1988), no. 4, 321–331.
[22] R. Kimmel and J. A. Sethian, Optimal algorithm for shape from shadingand path planning, Journal of Mathematical Imaging and Vision 14 (2001),
no. 3, 237–244.
[23] O. Körner, M. Schill, C. Wagner, H. J. Bender, and R. Männer, Haptic volumerendering with an intermediate local representation, Proceedings of Inter-
nationalWorkshop on Haptic Devices inMedical Applications, 1999, pp. 79–84.
[24] J. Krüger and R. Westermann, Acceleration techniques for GPU-based vol-ume rendering, Proceedings of IEEE Visualization, 2003, pp. 287–292.
[25] M. Levoy, Display of surfaces from volume data, IEEE Computer Graphics
& Applications 8 (1988), no. 3, 29–37.
[26] P. Ljung, Efficient methods for direct volume rendering of large data sets,Ph.D. thesis, Linköping University, 2006.
78
[27] W. E. Lorensen and H. E. Cline, Marching cubes: A high resolution 3D sur-face reconstruction algorithm, Computer Graphics 21 (1987), no. 4, 163–
169.
[28] K. Lundin, M. Cooper, A. Persson, D. Evestedt, and A. Ynnerman, Enablingdesign and interactive selection of haptic modes, Virtual Reality (2006), In
press, DOI: 10.1007/s10055-006-0033-7.
[29] K. Lundin, B. Gudmundsson, and A. Ynnerman, General proxy-based hapticsfor volume visualization, Proceedings of World Haptics, 2005, pp. 557–560.
[30] K. Lundin, A. Ynnerman, and B. Gudmundsson, Proxy-based haptic feedbackfrom volumetric density data, Proceeding of Eurohaptics, 2002, pp. 104–109.
[31] T. H. Massie and J. K. Salisbury, The PHANToM haptic interface: A devicefor probing virtual objects, Proceedings of ASME Symposium on Haptic In-
terfaces for Virtual Environment and Teleoperator Systems, 1994.
[32] J. Montagnat, H. Delingette, and N. Ayache, A review of deformable sur-faces: topology, geometry and deformation, Image and Vision Computing
19 (2001), no. 14, 1023–1040.
[33] L. G. Nyúl, A. X. Falcão, and J. K. Udupa, Fuzzy-connected 3D image seg-mentation at interactive speeds, Graphical Models 64 (2002), no. 5, 259–281.
[34] S. D. Olabarriaga and A.W. M. Smeulders, Interaction in the segmentation ofmedical images: A survey, Medical Image Analysis 5 (2001), no. 2, 127–142.
[35] K. Lundin Palmerius, Direct volume haptics for visualization, Ph.D. thesis,
Linköping University, 2007.
[36] P. Perona and J. Malik, Scale-space and edge detection using anisotropicdiffusion, IEEE Transactions on Pattern Analysis and Machine Intelligence 12(1990), no. 7, 629–639.
[37] B. Reitinger, A. Bornik, R. Beichel, and D. Schmalstieg, Liver surgery plan-ning using virtual reality, IEEE Computer Graphics and Applications 26(2006), no. 6, 36–47.
[38] A. Rosenfeld, Fuzzy geometry: An updated overview, Information Sciences
110 (1998), no. 3–4, 127–133.
[39] D. C. Ruspini, K. Kolarov, and O. Khatib, The haptic display of complexgraphical environments, Proceedings of ACM SIGGRAPH, 1997, pp. 345–
352.
[40] A. Schenk, G. Prause, and H-O. Peitgen, Efficient semi-automatic segmenta-tion of 3D objects in medical images, Proceedings of Medical Image Com-
puting and Computer-Assisted Intervention, LNCS, no. 1935, 2000, pp. 186–
195.
79
[41] O. Schreer, P. Kauff, and T. Sikora (eds.), 3D Videocommunication: Algo-rithms, concepts and real-time systems in human centred communication,Wiley, 2005.
[42] S. Senger, Integrating haptics into an immersive environment for the seg-mentation and visualization of volumetric data, Proceedings of WorldHap-
tics, 2005, pp. 487–490.
[43] J. A. Sethian, Level set methods and fast marching methods, Cambridge
University Press, 1999.
[44] M. Sonka, V. Havlac, and R. Boyle, Image processing, analysis, and machinevision, third ed., Brooks/Cole Publishing, 2007.
[45] Christoph Spuhler, Interactive centerline finding in complex tubular struc-tures, Ph.D. thesis, ETH Zürich, 2006.
[46] S. Stegmaier, M. Strengert, T. Klein, and T. Ertl, A simple and flexible volumerendering framework for graphics-hardware-based raycasting, Proceed-ings of International Workshop on Volume Graphics, 2005.
[47] C. Tomasi and R. Manduchi, Bilateral filtering for gray and color images,Proceedings of International Conference on Computer Vision, 1998, pp. 839–
846.
[48] J. K. Udupa, V. R. Leblanc, Y. Zhuge, C. Imielinska, H. Schmidt, L. Currie,
Y. Jin, B. E. Hirsch, and J. Woodburn, A framework for evaluating imagesegmentation algorithms, Computerized Medical Imaging and Graphics 30(2006), no. 2, 75–87.
[49] J. K. Udupa, P. K. Saha, and R. A. Lotufo, Relative fuzzy connectedness andobject definition: Theory, algorithms, and applications in image segmen-tation, IEEE Transactions on Pattern Analysis and Machine Intelligence 24(2002), no. 11, 1485–1500.
[50] J. K. Udupa and S. Samarasekera, Fuzzy connectedness and object defini-tion: Theory, algorithms, and applications in image segmentation, Graph-
ical Models and Image Processing 58 (1996), no. 3, 246–261.
[51] E. Vidholm, P. Sundqvist, and I. Nyström, Accelerating the computation of3D gradient vector flow fields, Proceedings of the 18th International Confer-
ence on Pattern Recognition, 2006, pp. 677–680.
[52] R. Westermann and T. Ertl, Efficiently using graphics hardware in volumerendering applications, Proceedings of ACM SIGGRAPH, 1998.
[53] C. Xu and J. L. Prince, Snakes, shapes, and gradient vector flow, IEEE
Transactions on Image Processing 7 (1998), no. 3, 359–369.
80
[54] J. Yan and T. Zhuang, Applying improved fast marching method to echocar-dial boundary detection in echocardiographic images, Pattern Recognition
Letters 24 (2003), no. 15, 2777–2784.
[55] Q. Zhang, R. Eagleson, and T. M. Peters, Rapid voxel classification method-ology for interactive 3D medical image visualization, Proceedings of Med-
ical Image Computing and Computer-Assisted Intervention, LNCS, no. 4792,
2007, pp. 86–93.
[56] C. Zilles and J. Salisbury, A constraint-based god-object method for hap-tic display, Proceedings of IEEE/RSJ International Conference on Intelligent
Robots and Systems, Human Robot Interaction, and Cooperative Robots, 1995,
pp. 146–151.
81
Related work
In addition to the papers included in this thesis, the author has also written or
contributed to the following publications:
Reviewed publications
• E. Vidholm, P. Sundqvist, and I. Nyström. Accelerating the computation of3D gradient vector flow fields. In Proceedings of International Conference
on Pattern Recognition (ICPR), pp. 667-680, IEEE, 2006.
• E. Vidholm and J. Agmund. Fast surface rendering for interactive medicalimage segmentation with haptic feedback. In Proceedings of SIGRAD, pp.
33-39, 2004.
Non-reviewed publications
• S. P. Nilsson, A. Bergman, B. Eriksson, I. Nyström, E. Vidholm. Semi-automatic liver volumetry in CT images from patients with neuroendocrinetumours. In Proceedings of European Congress Of Radiology (ECR), 2007.
• E. Vidholm. Haptic interaction with deformable simplex meshes for 3Dsegmentation. In Proceedings of Swedish symposium on image analysis
(SSBA), pp. 1-4, 2007.
• F. Malmberg, E. Vidholm, and I. Nyström. Live-wire based interactive seg-mentation of volume images using haptics. In Proceedings of Swedish sym-
posium on image analysis (SSBA), pp. 57-60, 2006.
• E. Vidholm and I. Nyström. Haptic volume rendering based on gradientvector flow. In Proceedings of Swedish symposium on image analysis
(SSBA), pp. 97-100, 2005.
• E. Vidholm and X. Tizon. Facilitating semi-automatic segmentation ofMRA volumes using haptics. In Proceedings of Swedish symposium on
image analysis (SSBA), pp. 162-165, 2004.
83
Acta Universitatis UpsaliensisDigital Comprehensive Summaries of Uppsala Dissertationsfrom the Faculty of Science and Technology 386
Editor: The Dean of the Faculty of Science and Technology
A doctoral dissertation from the Faculty of Science andTechnology, Uppsala University, is usually a summary of anumber of papers. A few copies of the complete dissertationare kept at major Swedish research libraries, while thesummary alone is distributed internationally through theseries Digital Comprehensive Summaries of UppsalaDissertations from the Faculty of Science and Technology.(Prior to January, 2005, the series was published under thetitle “Comprehensive Summaries of Uppsala Dissertationsfrom the Faculty of Science and Technology”.)
Distribution: publications.uu.seurn:nbn:se:uu:diva-8409
ACTAUNIVERSITATISUPSALIENSISUPPSALA2008