principles and advanced_methods_in_medical_imaging_and_image_analysis_(wsp,_2008)

PRINCIPLES AND ADVANCED METHODS IN

MEDICAL IMAGING AND IMAGE ANALYSIS

This page intentionally left blankThis page intentionally left blank

NEW JERSEY • LONDON • SINGAPORE • BEIJING • SHANGHAI • HONG KONG • TAIPEI • CHENNAI

World Scientific

PRINCIPLES AND ADVANCED METHODS IN

MEDICAL IMAGING AND IMAGE ANALYSIS

ATAM P DHAWANNew Jersey Institute of Technology, USA

H K HUANGUniversity of Southern California, USA

DAE-SHIK KIMBoston University, USA

British Library Cataloguing-in-Publication DataA catalogue record for this book is available from the British Library.

For photocopying of material in this volume, please pay a copying fee through the CopyrightClearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission tophotocopy is not required from the publisher.

ISBN-13 978-981-270-534-1ISBN-10 981-270-534-1ISBN-13 978-981-270-535-8 (pbk)ISBN-10 981-270-535-X (pbk)

Typeset by Stallion PressEmail: [email protected]

All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means,electronic or mechanical, including photocopying, recording or any information storage and retrievalsystem now known or to be invented, without written permission from the Publisher.

Copyright © 2008 by World Scientific Publishing Co. Pte. Ltd.

Published by

World Scientific Publishing Co. Pte. Ltd.

5 Toh Tuck Link, Singapore 596224

USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601

UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

Printed in Singapore.

PRINCIPLES AND ADVANCED METHODS IN MEDICAL IMAGING ANDIMAGE ANALYSIS

ChingTing - Principles and Adv Methods.pmd 6/18/2008, 9:12 AM1

January 23, 2008 11:18 WSPC/SPI-B540:Principles and Recent Advances fm

ToMy wife, Nilam,

for her support and patience;and my sons, Anirudh and Akshay,

for their quest for learning.(Atam P Dhawan)

ToMy wife, Fong,for her support;

and my daughter, Cammy; and my son, Tilden,for their young wisdom.

(HK Huang)

ToMy daughter, Zeno,

for her curiosity(Dae-Shik Kim)

v


Preface and Acknowledgments

We are pleased to bring “Principles and Advanced Methods inMedical Imaging and Image Analysis”, a volume of contributorychapters, to the scientific community. The book is a compilationof carefully crafted chapters written by leading researchers in thefield of medical imaging they have put in a great deal of effortin contributing the various chapters. This book can be used asa research reference or a text book for graduate level courses inbiomedical engineering and medical sciences.

The book is a unique combination of chapters describing theprinciples as well as state-of-the-art advanced methods in medicalimaging and image analysis for selected applications. Though com-puterized medical imaging has a very wide spectrum of applicationsin diagnostic radiology and medical research, we have selected asubset of important imaging modalities with specific applicationsthat are significant in medical sciences and clinical practice. Thetopics covered in the chapters have been developed with a naturalprogression of understanding, keeping in mind future technologicaladvances that are expected to have a major impact in clinical prac-tice and the understanding of complex pathologies. We hope thatthis book will provide a unique learning experience from theoreti-cal concepts to advanced methods and applications to researchers,clinicians and students.

vii


viii Preface and Acknowledgments

We are very grateful to our contributors who are internationallyrenowned experts and experienced researchers in their respectivefields within the wide spectrum of medical imaging and comput-erized medical image analysis. We also gracefully acknowledge thesupport provided by the editorial board and staff members of WorldScientific Publishing. Special thanks to Ms CT Ang for her guidanceand patience in preparing this book.

We hope that readers will find this book useful in providing aconcise version of important principles, advances, and applicationsin medical imaging and image analysis.

Atam P DhawanHK Huang

Dae-Shik Kim


Contributors

Walter J Akers, PhDStaff Scientist, Optical Radiology LaboratoryDepartment of RadiologyWashington University School of MedicineUniversity of Washington at St Louis

Elsa Angelini, PhDEcole Nationale Supérieure desTélécommunicationsParis, France

Leonard Berliner, MDDepartment of RadiologyNew York Methodist Hospital, NY

Sharon Bloch, PhDOptical Radiology Laboratory, Department of RadiologyWashington University School of MedicineUniversity of Washington at St Louis

Christos Davatzikos, PhDDirector, Section of Biomedical Image AnalysisAssociate Professor, Department of RadiologyUniversity of Pennsylvania

ix


x Contributors

Mathieu De Craene, PhDComputational Imaging LabDepartment of Informationand Communication TechnologiesUniversitat Pompeu Fabra, Barcelona

Atam P Dhawan, PhDProfessor, Department of Electricaland Computer EngineeringProfessor, Department of Biomedical EngineeringNew Jersey Institute of Technology

Qi Duan, PhDDepartment of Biomedical EngineeringColumbia University

Alejandro F Frangi, PhDComputational Imaging LabDepartment of Informationand Communication TechnologiesUniversitat Pompeu Fabra, Barcelona

Shunichi Homma, MDMargaret Millikin Hatch ProfessorDepartment of MedicineColumbia University

HK Huang, DScProfessor and Director, Imaging Informatics DivisionDepartment of Radiology, Keck School of MedicineDepartment of Biomedical EngineeringViterbi School of EngineeringUniversity of Southern California

Dae-Shik Kim, PhDDirector, Center for Biomedical ImagingAssociate Professor, Anatomy and NeurobiologyBoston University School of Medicine


Contributors xi

Elisa E Konofagou, PhDAssistant ProfessorDepartment of Biomedical EngineeringColumbia University

Andrew Laine, PhDProfessorDepartment of Biomedical EngineeringColumbia University

Angela R Laird, PhDAssistant Professor, Department of RadiologyUniversity of Texas Health Sciences CenterSan Antonio

Maria YY Law, PhDAssociate ProfessorDepartment of Health Technology and InformaticsThe Hong Kong Polytechnic University

Heinz U Lemke, PhDResearch Professor, Department of RadiologyUniversity of Southern CaliforniaLos Angeles, CA

Guang Li, PhDMedical Physicist, Radiation Oncology BranchNational Cancer Institute,NIH, Bethesda, Maryland

Brent J Liu, PhDAssistant Professor and Deputy Director of InformaticsDepartment of Radiology, Keck School of MedicineDepartment of Biomedical EngineeringViterbi School of EngineeringUniversity of Southern California


xii Contributors

Tianming Liu, PhDCenter for BioinformaticsHarvard Medical SchoolDepartment of RadiologyBrigham and Women’s Hospital, MA

Sachin Patwardhan, PhDResearch Scientist, Department of RadiologyMallinckrodt Institute of RadiologyUniversity of Washington, St Louis

Xiaochuan Pan, PhDProfessorDepartment of RadiologyCancer Research CenterThe University of Chicago

Itamar Ronen, PhDAssistant ProfessorCenter for Biomedical ImagingDepartment of Anatomy and NeurobiologyBoston University School of Medicine

Yulin Song, PhDAssociate ProfessorDepartment of RadiologyMemorial Sloan-Kettering Cancer CenterNew Jersey

Song Wang, PhDDepartment of Electricaland Computer EngineeringNew Jersey Institute of Technology

Pat Zanzonico, PhDMolecular Pharmacology and ChemistryMemorial Sloan-Kettering Cancer CenterNew York


Contributors xiii

Zheng Zhou, PhDManagerImaging Processing and Informatics LabDepartment of RadiologyUniversity of Southern California

Xiang Sean Zhou, PhDSenior Staff Scientist, Program ManagerComputer Aided Diagnosis and Therapy SolutionsSiemens Medical Solutions, Inc., Malvern PA

Lionel Zuckier, MDHead, Nuclear MedicineDepartment of RadiologyNew Jersey University for Medicine and Dentistry


Contents

Preface and Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiContributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

1. Introduction to Medical Imaging and ImageAnalysis: A Multidisciplinary Paradigm . . . . . . . . . 1Atam P Dhawan, HK Huang and Dae-Shik Kim

Part I. Principles of Medical Imaging and ImageAnalysis

2. Medical Imaging and Image Formation . . . . . . . . . . 9Atam P Dhawan

3. Principles of X-ray Anatomical ImagingModalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29Brent J Liu and HK Huang

4. Principles of Nuclear Medicine ImagingModalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63Lionel S Zuckier

5. Principles of Magnetic Resonance Imaging . . . . . . 99Itamar Ronen and Dae-Shik Kim

6. Principles of Ultrasound Imaging Modalities . . . . 129Elisa Konofagou

7. Principles of Image Reconstruction Methods. . . . . 151Atam P Dhawan

8. Principles of Image Processing Methods . . . . . . . . . 173Atam P Dhawan

xv


xvi Contents

9. Image Segmentation and Feature Extraction . . . . . 197Atam P Dhawan

10. Clustering and Pattern Classification . . . . . . . . . . . . 229Atam P Dhawan and Shuangshuang Dai

Part II. Recent Advances in Medical Imaging andImage Analysis

11. Recent Advances in Functional MagneticResonance Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267Dae-Shik Kim

12. Recent Advances in Diffusion MagneticResonance Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289Dae-Shik Kim and Itamar Ronen

13. Fluorescence Molecular Imaging: Microscopic toMacroscopic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311Sachin V Patwardhan, Walter J Akers andSharon Bloch

14. Tracking Endocardium Using Optical FlowAlong Iso-Value Curve . . . . . . . . . . . . . . . . . . . . . . . . . . 337Qi Duan, Elsa Angelini, Shunichi Homma andAndrew Laine

15. Some Recent Developments in ReconstructionAlgorithms for Tomographic Imaging . . . . . . . . . . . 361Chien-Min Kao, Emil Y Sidky, Patrick La Rivièreand Xiaochuan Pan

16. Shape-Based Reconstruction from NevoscopeOptical Images of Skin Lesions . . . . . . . . . . . . . . . . . . 393Song Wang and Atam P Dhawan

17. Multimodality Image Registration and Fusion . . . 413Pat Zanzonico

18. Wavelet Transform and Its Applications inMedical Image Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 437Atam P Dhawan


Contents xvii

19. Multiclass Classification for TissueCharacterization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455Atam P Dhawan

20. From Pairwise Medical Image Registration toPopulational Computational Atlases . . . . . . . . . . . . . 481M De Craene and AF Frangi

21. Grid Methods for Large Scale Medical ImageArchiving and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 517HK Huang, Zheng Zhou and Brent Liu

22. Image-Assisted Knowledge Discoveryand Decision Support in RadiationTherapy Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545Brent J Liu

23. Lossless Digital Signature EmbeddingMethods for Assuring 2D and 3D MedicalImage Integrity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573Zheng Zhou, HK Huang and Brent J Liu

Part III. Medical Imaging Applications, Case Studiesand Future Trends

24. The Treatment of Superficial Tumors UsingIntensity Modulated Radiation Therapy andModulated Electron Radiation Therapy . . . . . . . . . 599Yulin Song and Maria Chan

25. Image Guidance in Radiation Therapy. . . . . . . . . . . 635Maria YY Law

26. Functional Brain Mapping and ActivationLikelihood Estimation Meta-Analysis . . . . . . . . . . . . 663Angela R Laird, Jack L Lancaster and Peter T Fox

27. Dynamic Human Brain Mapping and Analysis:From Statistical Atlases to Patient-SpecificDiagnosis and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 677Christos Davatzikos


xviii Contents

28. Diffusion Tensor Imaging Based Analysisof Neurological Disorders . . . . . . . . . . . . . . . . . . . . . . . 703Tianming Liu and Stephen TC Wong

29. Intelligent Computer Aided Interpretationin Echocardiography: Clinical Needsand Recent Advances . . . . . . . . . . . . . . . . . . . . . . . . . . . 725Xiang Sean Zhou and Bogdan Georgescu

30. Current and Future Trends in RadiationTherapy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 745Yulin Song and Guang Li

31. IT Architecture and Standards for aTherapy Imaging and Model ManagementSystem (TIMMS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 783Heinz U Lemke and Leonard Berliner

32. Future Trends in Medical and MolecularImaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 829Atam P Dhawan, HK Huang and Dae-Shik Kim

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 845

January 22, 2008 12:2 WSPC/SPI-B540:Principles and Recent Advances ch01 FA

CHAPTER 1

Introduction to Medical Imagingand Image Analysis:

A Multidisciplinary Paradigm

Atam P Dhawan, HK Huang and Dae-Shik Kim

Recent advances in medical imaging with significant contributions fromelectrical and computer engineering, medical physics, chemistry, andcomputer science have witnessed a revolutionary growth in diagnosticradiology. Fast improvements in engineering and computing technolo-gies have made it possible to acquire high-resolution multidimensionalimages of complex organs to analyze structural and functional informationof human physiology for computer-assisted diagnosis, treatment evalua-tion, and intervention. Through large databases of vast amount of infor-mation such as standardized atlases of images, demographics, genomics,etc. new knowledge about physiological processes and associated patholo-gies is continuously being derived to improve our understanding of criti-cal diseases for better diagnosis and management. This chapter providesan introduction to this ongoing knowledge quest and the contents of thebook.

1.1 INTRODUCTION

In a general sense, medical imaging refers to the process involvingspecialized instrumentation and techniques to create images or rel-evant information about the internal biological structures and func-tions of the body. Medical imaging is sometimes categorized, in awider sense, as a part of radiological sciences. This is particularly

1


2 Atam P Dhawan, HK Huang and Dae-Shik Kim

relevant because of its most common applications in diagnostic radi-ology. In clinical environment, medical images of a specific organ orpart of the body are obtained for clinical examination for the diag-nosis of a disease or pathology. However, medical imaging tests arealso performed to obtain images and information to study anatom-ical and functional structures for research purposes with normal aswell as pathological subjects. Such studies are very important tounderstand the characteristic behavior of physiological processes inhuman body to understand and detect the onset of a pathology. Suchan understanding is extremely important for early diagnosis as wellas developing a knowledge base to study the progression of a diseaseassociated with the physiological processes that deviate from theirnormal counterparts. The significance of medical imaging paradigmis its direct impact on the healthcare through diagnosis, treatmentevaluation, intervention and prognosis of a specific disease.

From a scientific point of view, medical imaging is highly multi-disciplinary and interdisciplinary with a wide coverage of physical,biological, engineering and medical sciences. The overall technologyrequires direct involvement of expertise in physics, chemistry, biol-ogy, mathematics, engineering, computer science and medicine sothat useful procedures and protocols for medical imaging tests withappropriate instrumentation can be developed. The developmentof a specific imaging modality system starts with the physiologicalunderstanding of the biological medium and its relationship tothe targeted information to be obtained through imaging. Oncesuch a relationship is determined, a method for obtaining the tar-geted information using a specific energy transformation process,often known as physics of imaging, is investigated. Once a methodfor imaging is established, proper instrumentation with energysource(s), detectors, and data acquisition systems are designedand integrated to physically build an imaging system for imagingpatients to obtain target information in the context of a patholog-ical investigation. For example, to obtain anatomical informationabout internal organs of the body, X-ray energy may be used. TheX-ray energy, while transmitted through the body, goes throughattenuation based on the density of the internal structures. Thus,


Introduction to Medical Imaging and Image Analysis 3

the attenuation of the X-ray energy carries the target informationabout the density of internal structures which is then displayed as atwo-dimensional (in case of radiography or mammography) or mul-tidimensional (3D in case computed tomography (CT); 4D in case ofcine-CT) image. This information (image) can be directly interpretedby a radiologist or further processed by a computer for image pro-cessing and analysis for better interpretation.

With the evolutionary progress in engineering and computingtechnologies in the last century, medical imaging technologies havewitnessed a tremendous growth that has made a major impact indiagnostic radiology. These advances have revolutionarized health-care through fast imaging techniques; data acquisition, storage andanalysis systems; high resolution picture archiving and communi-cation systems; information mining with modeling and simulationcapabilities to enhance our knowledge base about the diagnosis,treatment and management of critical diseases such as cancer, car-diac failure, brain tumors and cognitive disorders.

Figure 1 provides a conceptual notion of the medical imagingprocess from determination of principle of imaging based on thetarget pathological investigation to acquiring data for image recon-struction, processing and analysis for diagnostic, treatment evalua-tion, and/or research applications.

There are many medical imaging modalities and techniquesthat have been developed in the past years. Anatomical structurescan be effectively imaged today with X-ray computed tomogra-phy (CT), magnetic resonance imaging (MRI), ultrasound, and opti-cal imaging methods. Furthermore, information about physiologi-cal structures with respect to metabolism and/or functions, can beobtained through nuclear medicine [single photon emission com-puted tomography (SPECT) and positron emission tomography(PET)], ultrasound, optical fluorescence, and several derivative pro-tocols of MRI such as fMRI, diffusion-tensor MRI, etc.

The selection of an appropriate medical imaging modality isimportant for obtaining the target information for a successfulpathological investigation. For example, if information has to beobtained about the cardiac volumes and functions associated with



Physiology and Understanding

of Imaging Medium

Principle of Imaging

Target Investigation

or Pathology

Physics of Imaging

Detector Physics

Imaging Instrumentation

Energy-SourcePhysics

Data Acquisition

Image Reconstruction

Image Processing

Interpretation: Diagnosis Evaluation

Intervention

Database and Computerized

Analysis

New Knowledge

Fig. 1. A conceptual block diagram of medical imaging process for diagnostic,treatment evaluation and intervention applications.

a beating heart, one has to determine the requirements and limita-tions about the spatial and temporal resolution for the target set ofimages. It is also important to keep in mind the type of pathologybeing investigated for the imaging test. Depending on the investi-gation, such as metabolism of cardiac walls, or opening and closingmeasurements of mitral valve, a specific medical imaging modality(e.g. PET) or a combination of different modalities (e.g. stress-PETand ultrasound) can be selected.

1.1.1 Book Chapters

In this book, we present a collection of carefully written chapters todescribe principles and recent advances of major medical imaging



modalities and techniques. Case studies and data analysis protocolsare also described for investigating selected critical pathologies. Wehope that this book will be useful for engineering as well as clinicalstudents and researchers. The book presents a natural progressionof technology development and applications through the chaptersthat are written by leading and renowned researchers and educa-tors. The book is organized in three parts: Principles of Imagingand Image Analysis (Chapters 2–10); Recent Advances in MedicalImaging and ImageAnalysis (Chapters 11–23); and Medical ImagingApplications, Case Studies and Future Trends (Chapters 24–32).

Chapter 2 describes some basic principles of medical imagingand image formation. In this chapter, Atam Dhawan has focused ona basic mathematical model of image formation for a linear spatiallyinvariant imaging system.

In Chapter 3, Brent Liu and HK Huang present basic principles ofX-ray imaging modalities. X-ray radiography, mammography, com-puted tomography (CT) and more recent PET-XCT fusion imagingsystems are described.

Principles of nuclear medicine imaging are described by LionelZuckier in Chapter 4 where he provides foundation and clinicalapplications of single photon emission tomography (SPECT) andpositron emission tomography (PET).

In Chapter 5, Itamar Ronen and Dae-Shik Kim describes sophis-ticated principles and imaging techniques of Magnetic ResonanceImaging (MRI). Imaging parameters and pulse techniques for use-ful MR imaging are presented.

Elisa Konofagou presents the principles of ultrasound imagingin Chapter 6. Instrumentation and various imaging methods withexamples are described.

In Chapter 7, Atam Dhawan describes the foundation of multi-dimensional image reconstruction methods. A brief introduction ofdifferent types of transform and estimation methods is presented.

Atam Dhawan presents a spectrum of image enhancement,restoration and filtering operations in Chapter 8. Image processingmethods in spatial (image) domain as well as frequency (Fourier)



domain are described. In Chapter 9, Atam Dhawan describes basicimage segmentation and feature extraction methods for representa-tion of regions of interest for classification.

In Chapter 10, Atam Dhawan and Shuangshuang Dai presentprinciples of pattern recognition and classification. Genetic algo-rithm based feature selection and nonparametric classification meth-ods are also described for image/tissue classification for diagnosticapplications.

Advances in MR imaging with respect to new methods and pulsesequences associated with functional imaging of brain are describedby Dae-Shik Kim in Chapter 11. Diffusion and diffusion-tensor basedmagnetic resonance imaging methods are described by Dae-ShikKim and Itamar Ronen in Chapter 12. These two chapters bring themost recent developments in functional brain imaging to investigateneuronal information including homodynamic response and axonalpathways.

Chapter 13 provides a spectrum of optical and fluorescenceimaging for 3-D tomographic applications. Through specific contrastimaging methods, Sachin Patwardhan, Walter Akers and SharonBloch explore molecular imaging applications.

In Chapter 14, Qi Duan, Elsa Angelini, Shunichi Homma andAndrew Laine presents recent investigations in dynamic ultrasoundimage analysis for tracking endocardium in 4D cardiac imaging.

Chien-Min Kao, Emil Y. Sidky, Patrick LaRiviere, and XiaochuanPan describe recent advances in model based multidimensionalimage reconstruction methods for medical imaging applications inChapter 15. These methods use multivariate statistical estimationmethods in image reconstruction.

Shape-based optical image reconstruction of specific entitiesfrom multispectral images of skin lesions is presented by Song Wangand Atam Dhawan in Chapter 16.

Clinical multimodality image registration and fusion methodswith nuclear medicine and optical imaging are described by PatZanzonico in Chapter 17. Pat emphasizes on clinical needs of local-ization of metabolic information with real time processing andefficiency requirements.



Recently wavelet transform has been extensively investigatedfor obtaining localized spatio-frequency information. The use ofwavelet transform in medical image processing and analysis isdescribed by Atam Dhawan in Chapter 18.

Medical image processing and analysis often require a multi-class characterization for image contents. Atam Dhawan presents aprobabilistic multiclass tissue characterization method for MR brainimages in Chapter 19.

In Chapter 20, Mathieu De Craene and Alejandro F Frangipresent a review of advances in image registration methods for con-structing standardized computational atlases.

In Chapter 21, HK Huang, Zheng Zhou and Brent Liu describeinformation processing and computational methods to deal withlarge image archiving and communication corresponding to largemedical image databases.

Brent Lu, in Chapter 22, describes knowledge mining and deci-sion making strategies for medical imaging applications in radiationtherapy planning and treatment.

With large image archiving and communication systems linkedwith large image databases, information integrity becomes a criticalissue. In Chapter 23, Zheng Zhou, HK Huang and Brent J Liu presentlossless digital signature embedding methods in multidimensionalmedical images for authentification and integrity.

Medical imaging applications in intensity modulated radiationtherapy (IMRT), a radiation treatment protocol, are discussed byYulin Song in Chapter 24.

In Chapter 25, Maria Law presents the detailed role of medicalimaging based computer assisted protocols for radiation treatmentplanning and delivery.

Recently developed fMR and diffusion-MR imaging meth-ods provide overwhelming volumes of image data. A produc-tive and useful analysis of targeted information extracted fromsuch MR images of brain is a challenging problem. In Chapter 26,Angela Laird, Jack Lancaster and Peter Fox describe recently devel-oped maximum likelihood estimation based “meta” analysis algo-rithms for the investigation of a specific pathology. In Chapter 27,



Christos Davatzikos presents dynamic brain mapping methods foranalysis of patient specific information for better pathological char-acterization and diagnosis. Tianming Liu and Stephen Wong, inChapter 28, explore a recently developed model-based image anal-ysis algorithms for analyzing diffusion-tensor MR brain images forthe characterization of neurological disorders.

Model-based intelligent analysis and decision-support tools areimportant in medical imaging for computer-assisted diagnosis andevaluation. Xiang Sean Zhou, in Chapter 29, presents specific chal-lenges of intelligent medical image analysis, specifically for the inter-pretation of cardiac ultrasound images. However, the issues raisedin this chapter could be extended to other modalities and applica-tions. In Chapter 30, Yulin Song and Guang Li present an overviewof future trends and challenges in radiation therapy methodsthat closely linked with high resolution multidimensional medicalimaging.

Heinz U Lemke and Leonard Berliner, in Chapter 31, describesspecific methods and information technology (IT) issues in dealingwith image management systems involving very large databasesand widely networked image communication systems.

To conclude, Chapter 32 presents a glimpse of future trends andchallenges in high-resolution medical imaging, intelligent imageanalysis, and smart data management systems.


CHAPTER 2

Medical Imaging and Image Formation

Atam P Dhawan

Medical imaging involves a good understanding of imaging mediumand object, physics of imaging, instrumentation, and often computerizedreconstruction and visual display methods. Though there are a number ofmedical imaging modalities available today involving ionized radiation,nuclear medicine, magnetic resonance, ultrasound, and optical methods,each modality offers a characteristic response to structural or metabolicparameters of tissues and organs of human body. This chapter providesan overview of the principles of medical imaging modalities and a basiclinear spatially invariant image formation model used for most commonimage processing tasks.

2.1 INTRODUCTION

Medical imaging is a process of collecting information about aspecific physiological structure (an organ or tissue) using a pre-defined characteristic property that is displayed in the form ofan image. For example, in X-ray radiography, mammography andcomputed tomography (CT), tissue density is the characteristicproperty that is displayed in images to show anatomical struc-tures. The information about tissue density of anatomical struc-tures is obtained by measuring attenuation to X-ray energy whenit is transmitted through the body. On the other hand, a nuclear

9


10 Atam P Dhawan

medicine positron emission tomography (PET) image may showglucose metabolism information in the tissue or organ. A PETimage is obtained by measuring gamma-ray emission from the bodywhen a radioactive pharmaceutical material, such as flurodeoxyglu-cose (FDG) is injected in the body. FDG metabolizes with the tis-sue through blood circulation eventually making it a source ofemission of gamma-ray photons. Thus, medical images may pro-vide anatomical, metabolic or functional information related toan organ or tissue. These images through proper instrumentationand data collection methods can be primarily reconstructed intwo- or three-dimensions and then displayed as multidimensionaldata sets.

The basic process of image formation requires an energy sourceto obtain information about the object that is displayed in the formof an image. Some form of radiation such as optical light, X-ray,gamma-ray, RF or acoustic waves, interacts with the object tissueor organ to provide information about its characteristic property.The energy source can be external (X-ray radiography, mammog-raphy, CT, ultrasound), internal [nuclear medicine: single photonemission computed tomography (SPECT); positron emission tomog-raphy (PET)], or a combination of both internal and external such asin magnetic resonance imaging where proton nuclei that are avail-able in the tissue in the body provides electromagnetic RF energybased signals in the presence of an external magnetic field and aresonating RF energy source.

As described above, image formation requires an energy source,a mechanism of interaction of energy with the object, an instru-mentation to collect the data with the measurement of energy afterthe interaction, and a method of reconstructing images of infor-mation about the characteristic property of the object from thecollected data.

The following imaging modalities are commonly used formedical applications today. The medical imaging modalitiesare briefly described below with their respective principles ofimaging.


Medical Imaging and Image Formation 11

2.2 X-RAY IMAGING

X-rays were invented by in Conrad Rontgen in 1895 who describedit as new kind of rays which can penetrate almost anything. Hedescribed the diagnostic capabilities of X-rays for imaging humanbody and received the Nobel prize in 1901. X-ray radiography is thesimplest form of medical imaging with the transmission of X-raysthrough the body which is then collected on a film or an array ofdetectors. The attenuation or absorption of X-rays is described bythe photoelectric and Compton effects providing more attenuationthrough bones than soft tissues or air.1−5

The diagnostic range of X-rays is used between 0.5A and 0.01A.A wavelength which corresponds to the photon energy of approx-imately 20 kev to 1.0 Mev. In this range, the attenuation is quitereasonable to discriminate bones, soft tissue and air. In addition,the wavelength is short enough for providing excellent resolutionof images even with sub mm accuracy. Shorter wavelengths thandiagnostic range of X-rays provides much higher photon energyand therefore less attenuation. Increasing photon energy makes thehuman body transparent for the loss of any contrast in the image.The diagnostic X-rays wavelength range provides higher energy perphotons and provides a refractive index of unity for almost all mate-rials in the body. This guarantees that the diffraction will not distortthe image and rays will travel in straight lines.1−8

X-ray medical imaging uses an external ionized radiation source,an X-ray tube to generate X-ray radiation beam that is transmit-ted through human body. Attenuation to X-ray radiation beam ismeasured to provide information about variations in the tissue den-sity that is displayed in X-ray 2D radiographs or 3D computedtomography (CT) images. The output intensity of a radiation beamparallel to x-direction for a specific y-coordinate location in theselected z-axial planar cross-section, Iout(y; x,z) would be given by:

Iout(y; x, z) = Iin(y; x, z)e− ∫µ(x,y;z)dx,

where µ(x, y, z) represents attenuation coefficient to the transmittedX-ray energy.


12 Atam P Dhawan

Fig. 1. An X-ray mammography image with microcalcification areas.

X-ray conventional radiography creates a 2D image of a 3D objectprojected on the detector plane. Figure 1 shows a 2D mammographyimage of a female breast. Several microcalcification areas can be seenin this image.

While 2D projection radiography may be adequate for manydiagnostic applications, it does not provide 3D qualitative and quan-titative information about the anatomical structures and associatedpathology that is necessary for diagnostics and treating a numberof diseases or abnormalities. Combining Radon transform acquir-ing ray-integral measurements with 3D scanning geometry, X-raycomputed tomography (CT) provides a three-dimensional recon-struction of internal organs and structures.9−11 X-ray CT has provento be very useful and sophisticated imaging tool in diagnostic radi-ology and therapeutic intervention protocols. The basic principle ofX-ray CT is the same as that of X-ray digital radiography: X-rays aretransmitted through the body and collected by an array of detectorsto measure the total attenuation along the X-ray path.8−11 Figure 2,shows a pathological axial image of the cardiovascular cavity of acadaver. The corresponding image obtained from X-ray CT is shownat the bottom of Fig. 2.



Fig. 2. Top: a pathological axial image of the cardiovascular cavity of a cadaver,bottom: the corresponding image obtained from X-ray CT.

2.3 MAGNETIC RESONANCE IMAGING

The principle of nuclear magnetic resonance for medical imag-ing was first demonstrated by Raymond Damadian in 1971 andPaul Lauterbur in 1973. Nuclear magnetic resonance (NMR) is aphenomenon of magnetic systems that possesses both a magneticmoment and an angular moment. In magnetic resonance imaging(MRI), the electromagnetic induction based signals at magnetic res-onance frequency in the radio frequency (RF) range are collectedthrough nuclear magnetic resonance from the excited nuclei withmagnetic moment and angular momentum present in the body.4−7

All materials consist of nuclei which are protons, neutrons ora combination of both. Nuclei that contain an odd number of pro-tons, neurons or both in combination possess a nuclear spin and amagnetic moment. Most materials are composed of several nucleiwhich have the magnetic moments such as 1H, 2H, 13C, 31Na, etc.When such material is placed number a magnetic field, randomlyoriented nuclei experience an external magnetic torque which aligns


14 Atam P Dhawan

the nuclei either in a parallel or an antiparallel direction in referenceto the external magnetic field. The number of nuclei aligned in par-allel is greater by a fraction than the number of nuclei aligned inan antiparallel direction and is dependent on the strength of theapplied magnetic field. Thus, a net vector results in the paralleldirection. The nuclei under the magnetic field rotate or precess likespinning tops precessing around the direction of the gravitationalfield. The rotating or precessional frequency of the spins is calledthe Larmor precession frequency and is proportional to the mag-netic field strength. The energy state of the nuclei in the antiparalleldirection is higher than the energy state of the nuclei in the paralleldirection. When an external electromagnetic radiation at the Larmorfrequency is applied through the RF coils (because the natural mag-netic frequency of these nuclei fall within the radiofrequency range),some of the nuclei directed in the parallel direction get excited andgo to the higher energy state, becoming in the direction antipar-allel to the external magnetic field to the antiparallel direction. Thelower energy state has the larger population of spins than the higherenergy states. Thus, through the application of the RF signal, the spinpopulation is also affected.

When the RF excitation signal is removed, the excited portionstend to return to their lower energy states through relaxation result-ing in the recovery of the net vector and the spin population. Therelaxation process causes the emission of a RF frequency signal at thesame Larmor frequency which is received by the RF coils to gener-ate an electric potential signal called the free-induction decay (FID).This signal becomes the basis of MR imaging.

Given an external magnetic field H0, the angular (Larmor) fre-quency, ω0 of nuclear precession can be expressed:

ω0 = γH0. (1)

Thus, the precession frequency depends on the type of nuclei with aspecific gyromagnetic ratio γ and the intensity of the external mag-netic field. This is the frequency on which the nuclei can receivethe radio frequency (RF) energy to change their states for exhibitingnuclear magnetic resonance. The excited nuclei return to the thermal



equilibrium through a process of relaxation emitting energy at thesame precession frequency, ω0.

It can be shown that during the RF pulse (nuclear excitationphase), the rate of change in the net stationary magnetization vector�M can be expressed as (the Bloch equation):

d �Mdt

= γ �M × �H, (2)

where �H is the net effective magnetic field.Considering the total response of the spin system in the presence

of an external magnetic field along with the RF pulse for nuclearexcitation followed by the nuclear relaxation phase, the change ofthe net magnetization vector can be expressed as Ref. [5]:

d �Mdt

= γ �M × �H − Mx�i + My�jT2

− (Mz − M0z )�k

T1, (3)

where �M0z is the net magnetization vector in thermal equilibrium

in the presence of an external magnetic field H0 only, and T1 and T2

are, respectively, the longitudinal (spin-lattice) and transverse (spin-spin) relaxation times in the nuclear relaxation phase when excitednuclei return to their thermal equilibrium state.

In other words, the longitudinal relaxation time, T1 representsthe return of net magnetization vector in z direction to its thermalequilibrium state while the transverse relaxation time, T2, representsthe loss of coherence or dephasing of spin leading to the net zero vec-tor in the x-y plane. The longitudinal and transverse magnetizationvectors with respect to the relaxation times in the actual stationarycoordinate system, can be given by:

�Mx,y(t) = �Mx,y(0)e−t/T2e−iω0t

�Mz(t) = �M0z (1 − e−t/T1 ) + �Mz(0)e−t/T1 (4)

where �Mx,y(0) = �Mx′,y′(0)e−iω0τp .

�Mx,y(0) represents the initial transverse magnetization vector withthe time set to zero at the end of the RF pulse of duration τp.

During imaging, the RF pulse, transmitted through an RF coilcauses nuclear excitation changing the longitudinal and transverse


16 Atam P Dhawan

magnetization vectors. After the RF pulse is turned off, the excitednuclei go through the relaxation phase emitting the absorbed energyat the same Larmor frequency that can be detected as an electricalsignal, called the free induction decay (FID). The FID is the raw NMRsignal that can be acquired through the same RF coil tuned at theLarmor frequency.

Let us represent a spatial location vector r in the spinning nucleisystem with a net magnetic field vector �Hr(r) and the correspondingnet magnetization vector �M(r, t), the magnetic flux φ(t) through theRF coil can be given as Ref. [5]:

φ(t) =∫

object

�Hr(r) · �M(r, t)dr, (5)

where the voltage induced in the RF coil, V(t) is the raw NMR signaland can be expressed (using Faraday’s Law) as:

V(t) = −∂φ(t)∂t

= − ∂

∂t

∫object

�Hr(r) · �M(r, t)dr. (6)

Figure 3 provides axial, coronal and sagittal cross-section MR imagesof a brain. Details of the gray and white matter structures are evidentin these images.

2.4 SINGLE PHOTON EMISSION COMPUTEDTOMOGRAPHY

In 1934, Jean Frederic Curie and Irene Curie discovered radiophos-phorous 32P, a radioisotope to demonstrate radioactivity decay.

Fig. 3. (from left to right): axial, coronal and sagittal cross-section MR images ofa human brain.



In 1951, radionucleotide imaging of thyroid was demonstrated byCassen through administration of iodine radioisotope 131I. Angerin 1952, developed a scintillation camera, also known as Angercamera with sodium iodide crystals coupled with photomultipliertubes. Kuhl and Edwards developed a transverse section tomog-raphy gamma ray scanner for radionuclide imaging in 1960s.12−15

Their imaging system included an array of multiple collimateddetectors surrounding a patient with rotate-translation motion toacquire projection for emission tomography. With the advances ofcomputer reconstruction algorithms and detector instrumentation,the gamma ray imaging is now known as single photon emissioncomputed tomography (SPECT) for 3D imaging of human organsthat is extended to even full body imaging. The radioisotopes areinjected in the body through the administration of radiopharmaceu-tical drugs that metabolize with the tissue making tissue a source ofgamma ray emissions. The gamma rays from the tissue pass throughthe body and are captured by the detectors surrounding the body toacquire raw data for defining projections. The projection data is thenused in reconstruction algorithms to display images with the helpof a computer and high-resolution displays. In SPECT imaging, thecommonly used radionuclides are Thallium 201Tl, Technetium 99mTc,Iodine 123I and Gallium 68Ga. These radionuclides decay by emittinggamma rays with photon energies ranging from 135 keV to 511 keV.The attenuation to gamma rays is similar in nature as of X-rays andcan be expressed as:

Id = I0e−τx,

where I0 is the intensity of gamma rays at the source, Id is the intensityat the detector after the gamma rays have passed the distance x inthe body with a linear attenuation coefficient ν that depends on thedensity of the medium and the energy of gamma ray photons.

Figure 4 shows 99mTl SPECT images of a human brain. It can benoticed that SPECT images are poor in resolution and anatomicalstructure as compared to CT or MR images. However, the SPECTimages show radioactivity distribution in the tissue representing aspecific metabolism or blood flow.


18 Atam P Dhawan

Fig. 4. SPECT image of a human brain.

2.5 POSITRON EMISSION TOMOGRAPHY

Positron emission tomography (PET) imaging methods were devel-oped in 1970s by a number of researchers including Phelp,Robertson, Ter-Pogossian and Brownell and several others.14,16 Theconcept of PET imaging is based on the simultaneous detection oftwo 511 keV energy photons traveling in the opposite direction. Thedistinct feature of PET imaging is its ability to trace radioactive mate-rial metabolized in the tissue to provide specific information aboutits biochemical and physiological behavior.

Some radioisotopes decay by emitting positive charged parti-cles called positrons. The emission of positron is accompanied bya significant amount of kinetic energy. After emission, a positrontravels typically for 1 mm – 3 mm losing some of its kinetic energy.The loss of energy makes the positron suitable for interaction with aloosely bound electron within a material for annihilation. The anni-hilation of the positron with the electron causes the formation oftwo gamma photons with 511 keV traveling in opposite directions(close to 180◦ apart). The two photons can be detected by two sur-rounding scintillation detectors simultaneously within a small time



window. This simultaneous detection within a small time window(typically in the order of nanoseconds) is called a coincidence detec-tion indicating the origin of annihilation along the line joining thetwo detectors involved in coincidence detection. Thus, by detectinga large number of coincidences, the source location and distributioncan be reconstructed through image reconstruction algorithms. Itshould be noted that the point of emission of a positron is differentfrom the point of annihilation with an electron. Though the imag-ing process is aimed at the reconstruction of source representing thelocations of emission of positrons, it is the locations of annihilationevents that are reconstructed as an image in the positron emissiontomography (PET). However, the distribution of emission eventsof positrons is considered to be close enough to the distribution ofannihilation events within a resolution limit.

The main advantage of PET imaging is its ability of extractingmetabolic and functional information of the tissue because of theunique interaction of positron within the matter of the tissue. Themost common positron emitter radionuclide used in PET imagingis fluorine 18F that is administered as fluorine labeled radiopharma-ceutical called fluorodeoxyglucose (FDG). The FDG images obtainedthrough PET imaging show very significant information about theglucose metabolism and blood-flow of the tissue. Such metabolisminformation has been proven to be of critical in determining the het-erogeneity and invasiveness of tumors.

Figure 5 shows a set of axial cross-section of brain PET imagesshowing glucose metabolism. The streaking artifacts and low resolu-tion details can be noticed in these images. The artifacts seen in PETimages are primarily because of low volume of data caused by thenature of radionuclide-tissue interaction and electronic collimationnecessary to reject the scattered events.

2.6 ULTRASOUND IMAGING

Sound or acoustic waves were successfully used in sonar technologyin military applications in World War II. The potential of ultrasound


20 Atam P Dhawan

Fig. 5. Serial axial images of a human brain acquired using FDG PET.

waves in medical imaging was explored and demonstrated by sev-eral researchers in the 1970s and 1980s including Wild, Reid, Frey,Greenleaf and Goldberg.17−20 Today, ultrasound imaging is success-fully used in diagnostic imaging of anatomical structures, blood flowmeasurements and tissue characterization. Safety, portability andlow-cost aspects of ultrasound imaging have made it a significantlysuccessful diagnostic imaging modality.

Sound waves are characterized by wavelength and frequency.Sound waves audible to the human ear are comprised of frequen-cies ranging from 15 Hz to 20 kHz. Sound waves with frequenciesabove 20 kHz are called ultrasound waves. The velocity of propaga-tion of sound in water and in most body tissues is about 1500 m/sec.Thus, the wavelength based resolution criterion is not satisfiedfrom electromagnetic radiation concept. The resolution capability ofacoustic energy is therefore dependent on the frequency spectrum.The attenuation coefficient in body tissues varies approximatelyproportional to the acoustic frequency at about 1.5 dB/cm/MHz.Thus, at much higher frequencies, the imaging is not meaningfulbecause of excessive attenuation. In diagnostic ultrasound, imag-ing resolution is limited by the wavelength. Shorter wavelengthsprovide better imaging resolution. Shorter waves can also penetrate



deeper into tissue. Since the velocity of sound waves in a specificmedium is fixed, the wavelength is inversely proportional to thefrequency. In medical ultrasound imaging, sound waves of 2 MHzto 10 MHz can be used but 2 MHz to 5 MHz frequencies are morecommon.

Let us assume that a transducer provides an accoustic signal ofs(x, y) intensity with a pulse ω(t) that is transmitted in a mediumwith an attenuation coefficient, µ and reflected by a biological tissueof reflectivity R(x, y, z) with a distance z from the transducer. Therecorded reflected intensity of a time varying accoustic signal, Jr(t)over the region R can then be expressed as:

Jr(t) = K∣∣∣∣∫ ∫ ∫

�

(e−2µz

z

)R(x, y, z)s(x, y)ω

(t − 2z

c

)dxdydz

∣∣∣∣ ,

(7)where K, ω(t) and c, respectively, represent a normalizing constant,received pulse and the velocity of the acoustic signal in the medium.

Using an adaptive time varying gain to compensate for the atten-uation of the signal, Eq. 7 for the compensated recorded reflectedsignal from the tissue, Jcr(t) can be simplified to:

Jcr(t) = K∣∣∣∣∫ ∫ ∫

�R(x, y, z)s(x, y)ω

(t − 2z

c

)dxdydz

∣∣∣∣or, in terms of a convolution as:

Jcr(t) = K∣∣∣∣R

(x, y,

ct2

)⊗ s(− x, −y)ω(t)

∣∣∣∣ , (8)

where ⊗ represents a 3D convolution. This is a convolution of areflectivity term characterizing the tissue and an impulse responsecharacterizing the source parameters.

Backscattered echo and Doppler shift principles are more com-monly used with the interaction of sound waves with human tissue.Sometimes, the scattering information is complemented with trans-mission or attenuation related information such as velocity in thetissue. Figure 6 shows a diastolic color Doppler flow convergence inthe apical four-chamber view of mitral stenosis.


22 Atam P Dhawan

Fig. 6. Adiastolic color Doppler flow image showing an apical four-chamber viewof mitral stenosis.

2.7 PRINCIPLES OF IMAGE FORMATION

It is usually desirable for an imaging system to behave like a lin-ear spatially invariant system. In other words, the response of theimaging system should be consistent, scalable and independent ofthe spatial position of the object being imaged. A system is said tobe linear if it follows two properties: scaling and superposition.1−3

In mathematical representation, it can be expressed as:

h{aI1(x, y, z) + bI2(x, y, z)} = ah{I1(x, y, z)} + bh{I2(x, y, z)}, (9)

where a and b area scalar multiplication factors, and I1(x, y, z) andI2(x, y, z) are two inputs to the system represented by the responsefunction h.

It should be noted that in real-world situations, it is difficult tofind a perfectly linear image formation system. For example, theresponse of photographic film or X-ray detectors cannot be linearover the entire operating range. Nevertheless, under constrained



conditions and limited exposures, the response can be practicallylinear.Also, a nonlinear system can be modeled with piecewise linearproperties under specific operating considerations.

In general, image formation is a neighborhood process. One canassume a radiant energy such as light source to illuminate an objectrepresented by the function f (α, β, γ). Using the additive propertyof radiating energy distribution to form an image, g(x, y, z) can bewritten as:

g(x, y, z) =∫ +∞

−∞

∫ ∞

−∞

∫ +∞

−∞h(x, y, z, α, β, γ , f (α, β, γ))dαdβdγ , (10)

where h(x, y, z, α, β, γ , f (α, β, γ)) is called the response function of theimage formation system. If the image formation system is assumedto be linear, the image expression becomes:

g(x, y, z) =∫ +∞

−∞

∫ ∞

−∞

∫ +∞

−∞h(x, y, z, α, β, γ)f (α, β, γ)dαdβdγ . (11)

The response function h(x, y, z, α, β, γ) is called the Point SpreadFunction (PSF) of the image formation system. The PSF dependson the spatial extent of the object and image coordinates systems.The expression h(x, y, z, α, β, γ) is the generalized version of the PSFdescribed for the linear image formation system that can be furthercharacterized as a spatially invariant (SI) or spatially variant (SV)system. If a linear image formation system is such that the PSF isuniform across the entire spatial extent of the object and image coor-dinates, the system is called a linear spatially invariant (LSI) system.In such a case, the image formation can be expressed as:

g(x, y, z) =∫ +∞

−∞

∫ ∞

−∞

∫ +∞

−∞h(x−α, y−β, z−γ)f (α, β, γ)dαdβdγ . (12)

In other words, for an LSI image formation system, the image is rep-resented as the convolution of the object radiant energy distributionand the PSF of the image formation system. It should be noted thatthe PSF is basically a degrading function that causes a blur in theimage and can be compared to the unit impulse response, a com-mon term used in signal processing. In other words, the acquired


24 Atam P Dhawan

image g(x, y, z) for a LSI imaging system, can be expressed as theconvolution of object distribution with the PSF as:

g(x, y, z) = h(x, y, z) ⊗ f (x, y, z) + n(x, y, z), (13)

where n(x, y, z) represents an additive noise term.Considering Fourier Transform, the above equation can be rep-

resented in frequency domain:

G(u, v, w) = H(u, v, w)F(u, v, w) + N(u, v, w), (14)

where G(u, v, w), H(u, v, w) and N(u, v, w) are, respectively, FourierTransform of g(x, y, z), f (x, y, z) and n(x, y, z) as:

G(u, v, w) =∫ ∞

−∞

∫ ∞

−∞g(x, y, z)e−j2π(ux+vy+wz)dxdydx,

H(u, v, w) =∫ ∞

−∞

∫ ∞

−∞h(x, y, z)e−j2π(ux+vy+wz)dxdydx,

N(u, v, w) =∫ ∞

−∞

∫ ∞

−∞n(x, y, z)e−j2π(ux+vy+wz)dxdydx. (15)

Image processing and enhancement operations can be easily andmore effectively performed on the above described representationof image formation through a LSI imaging system. However, thevalidity of such an assumption for imaging systems in the real worldmay be limited.

2.8 RECEIVER OPERATING CHARACTERISTICS (ROC)ANALYSIS AS A PERFORMANCE MEASURE

Receiver operating characteristic (ROC) analysis is considered a sta-tistical measure for studying the performance of an imaging or diag-nostic system with respect to its ability to detect a system’s ability todetect abnormalities accurately and reliably (true positive) withoutproviding false detections. In other words, ROC analysis providesa systematic analysis of sensitivity and specificity of a diagnostictest.1,8,21

Let us assume the total number of examination cases to be Ntot,out of which Ntp cases have positive true-condition with the actualpresence of the object and the remaining cases, Ntn, have negative



true-condition with no object present. Let us suppose these cases areexamined though the test for which we need to evaluate accuracy,sensitivity and specificity factors. Considering the observer does notcause any loss of information or misinterpretation, let Notp (truepositive) be the number of positive observations from Ntp positivetrue-condition cases and Nofn (false negative) be the number of neg-ative observation from Ntp positive true-condition cases. Also, letNotn (true negative) be the number of negative observations fromNtn negative true-condition cases and Nofp (false positive) be num-ber of positive observation from Ntn negative true-condition cases.Thus,

Ntp = Notp + Nofn

and

Ntn = Nofp + Notn. (16)

The following relationships can be easily derived from abovedefinitions.

(1) True positive fraction (TPF) is the ratio of the number of positiveobservations to the number of positive true-condition cases.

TPF = Notp/Ntp (17)

(2) False negative fraction (FNF) is the ratio of the number of nega-tive observations to the number of positive true-condition cases.

FNF = Nofn/Ntp (18)

(3) False positive fraction (FPF) is the ratio of the number of positiveobservations to the number of negative true-condition cases.

FPF = Nofp/Ntn (19)

(4) True negative fraction (TNF) is the ratio of the number of nega-tive observations to the number of negative true-condition cases.

TNF = Notn/Ntn (20)

This should be noted that:

TPF + FNF = 1(21)

TNF + FPF = 1.


26 Atam P Dhawan

FPF

TPF

b

a

c

Fig. 7. ROC curves with curve “a” indicating better overall classification abilitythan the curve “b” while the curve “c” shows the random probability.

Agraph between TPF and FPF is called a receiver operating char-acteristic (ROC) curve for a specific medical imaging or diagnostictest for detection of an object. It should also be noted that statisticalrandom trials with equal probability of positive and negative obser-vations would lead to the diagonally placed straight-line as the ROCcurve. Different tests and different observers may lead to differentROC curves for the same object detection. Figure 7 shows differentthree different ROC curves for a hypothetical detection/diagnosis.It can be noted that observer corresponding to curve “a” is far betterthan the observer “b.”

True positive fraction, TPF, is also called the sensitivity whilethe true negative fraction (TNF) is known as specificity of the testfor detection of an object. Accuracy of the test is given by a ratio ofcorrect observation to the total number of examination cases. Thus,

Accuracy = (TPF + TNF)/Ntot. (22)

In other words, different imaging modalities and observers may leadto different accuracy, sensitivity and specificity levels.

2.9 CONCLUDING REMARKS

This chapter presented basic principles of major medical imagingmodalities and a linear spatially invariant model of image formationthat is practically easier to deal with post-processing operations forimage enhancement and analysis. Though these assumptions maynot be strictly followed by the real world imaging scanners, medi-cal imaging systems often perform close to them. Medical imaging



modalities and image analysis systems are evaluated on the basis oftheir capabilities to detect true detections of pathologies while mini-mizing the false detections. Such performance evaluations are oftenconducted through receiver operating characteristic (ROC) curvesthat provides a very useful way of understanding detection capa-bility in terms of the sensitivity and specificity and the relation-ship of potential tradeoffs between true positive and false positivedetections. Quantitative data analysis with appropriate models canimprove image presentation (through better image reconstructionmethods), and image analysis with feature detection, analysis andclassification to improve the true positive rate while minimizingfalse positive rate of detection of a specific pathology for whichimaging tests are performed.

References

1. DhawanAP, Medical Image Analysis, John Wiley & Sons, Hoboken, 2003.2. Barrett H, Swindell W, Radiological Imaging: The Theory of Image Forma-

tion, Detection and Processing Volumes 1–2, Academic Press, New York,1981.

3. Bushberg JT, Seibert JA, Leidholdt EM, Boone JM, The Essentials of Med-ical Imaging, Williams & Wilkins, 1994.

4. Cho ZH, Jones JP, Singh M, Fundamentals of Medical Imaging, JohnWiley & Sons, New York, 1993.

5. Liang Z, Lauterbur PC, Principles of Magnetic Resonance Imaging, IEEEPress, 2000.

6. Lev MH, Hochberg F, Perfusion magnetic resonance imaging to assessbrain tumor responses to new therapies, J Moffit Cancer Center 5:446–450, 1998.

7. Stark DD, Bradley WG, Magnetic Resonance Imaging, 3rd edn., Mosby,1999.

8. Shung KK, Smith MB, Tsui B, Principles of Medical Imaging, AcademicPress, 1992.

9. Hounsfield GN, A method and apparatus for examination of a bodyby radiation such as X or gamma radiation, Patent 1283915, The PatentOffice, London, England, 1972.

10. Hounsfield GN, Computerized transverse axial scanning tomography:Part 1, description of the system, Br J Radiol 46: 1016–1022, 1973.

11. Cormack AM, Representation of a function by its line integrals withsome radiological applications, J Appl Phys 34: 2722–2727, 1963.


28 Atam P Dhawan

12. Cassen B, Curtis L, Reed C, Libby R, Instrumentation for 131I used inmedical studies, Nucleonics 9: 46–48, 1951.

13. Anger H, Use of gamma-ray pinhole camera for in vivo studies, Nature170: 200–204, 1952.

14. Brownell G, Sweet HW, Localization of brain tumors, Nucleonics 11:40–45, 1953.

15. Casey ME, Eriksson L, Schmand M, Andreaco M, Paulus M, DahlbornM, Nutt R, Investigation of LSO crystals for high spatial resolutionpositron emission tomography, IEEE Trans Nucl Sci 44: 1109–1113, 1997.

16. Kuhl E, Edwards RQ, Reorganizing data from transverse sections scansusing digital processing, Radiology 91: 975–983, 1968.

17. Fish P, Physics and Instrumentation of Diagnostic Medical Ultrasound, JohnWiley & Sons, Chichester, 1990.

18. Kremkau FW, Diagnostic Ultrasound Principles and Instrumentation,Saunders, Philadelphia, 1995.

19. Kremkau FW, Doppler Ultrasound: Principles and Instruments, Saunders,Philadelphia, 1991.

20. Hykes D, Ultrasound Physics and Instrumentation, Mosby, New York,1994.

21. Swets JA, Pickett RM, Evaluation of Diagnostic Systems, Academic Press,Harcourt Brace Jovanovich Publishers, New York, 1982.


CHAPTER 3

Principles of X-ray Anatomical ImagingModalities

Brent J Liu and HK Huang

This chapter provides basic concepts of various X-ray imaging modal-ities. The first part of the chapter addresses digital X-ray projectionradiography which includes digital fluorography, computed radiogra-phy, X-ray mammography, and digital radiography. The key compo-nents belonging to each of these imaging modalities will be discussedalong with basic principles to reconstruct the 2D image. The second partof the chapter focuses on 3D volume X-ray acquisition which includesX-ray CT, multislice, cine, and 4D CT. The image reconstruction methodswill be discussed along with key components which have advanced theCT technology to the present day.

3.1 INTRODUCTION

This chapter will present X-ray anatomical imaging modalitieswhich cover a large amount of the total number of diagnostic imag-ing procedures. X-ray projection radiography alone accounts for 70%of the total number of diagnostic imaging procedures. In this chapter,we will only focus on digital X-ray anatomical imaging modalities,which include digital fluorography, computed radiography, X-raymammography, digital radiography, X-ray CT, and multislice, cine,and 4D X-ray CT.

29


30 Brent J Liu and HK Huang

There are two approaches to convert a film-based image to digitalform. The first is to utilize existing equipment in the radiographicprocedure room and only change the image receptor component.Two technologies, computed radiography (CR) using the photostim-ulable phosphor imaging plate technology, and digital fluorography,are in this category. This approach does not require any modificationin the procedure room and is therefore more easily adopted for dailyclinical practice. The second approach is to redesign the conventionalradiographic procedure equipment, including the geometry of theX-ray beams and the image receptor. This method is therefore moreexpensive to adopt, but the advantage is that it offers special featureslike low X-ray scatter which would not otherwise be achievable inthe conventional procedure.

3.2 DIGITAL FLUOROGRAPHY

Since 70% of the radiographic procedures still use film as an outputmedium, it is necessary to develop methods to convert images onfilms to digital format. This section discusses digital fluorographywhich converts images to digital format utilizing a video cameraand A/D converter.

The video scanning system is a low cost X-ray digitizer whichproduces either a 512 K or 1 K digitized image with 8 bits/pixel. Thesystem consists of three major components: a scanning device witha video or a charge-coupled device (CCD) camera that scans theX-ray film, an analog/digital converter that converts the video sig-nals from the camera to gray level values, and an image memory tostore the digital signals from the A/D converter. The image stored inthe image memory is the digital representation of the X-ray film orimage in the image intensifier tube obtained by using the video scan-ning system. If the image memory is connected to a digital-to-analog(D/A) conversion circuitry and to a TV monitor, this image can bedisplayed back on the monitor (which is a video image). The mem-ory can be connected to a peripheral storage device for long-termimage archive. Figure 1 shows a block diagram of a video scanning


Principles of X-ray Anatomical Imaging Modalities 31

A/DImage

MemoryD/A

VideoMonitor

ImageProcessor/Computer

DigitalStorageDevice

VideoScanner

Digital Chain

Fig. 1. Block diagram of a video scanning system, the digital chain is a standardcomponent in all types of scanner.

system. The digital chain shown is a standard component in all typesof scanner.

Video scanner system can be connected to an image intensi-fier tube to form a digital fluoroscopic system. Digital fluorogra-phy is a method that can produce dynamic digital X-ray imageswithout changing the radiographic procedure room drasticallyfrom conventional fluorography. This technique requires an add-on unit in the conventional fluorographic system. Figure 2 showsa schematic of the digital fluorographic system with the followingmajor components:

(1) X-ray source : The X-ray tube and a grid to minimize X-raysscatter.

(2) Image receptor : The image receptor is an image intensifier tube.(3) Video camera plus optical system : The output light from the

image intensifier goes through an optical system, which allowsthe video camera to be adjusted for focusing. The amount oflight going into the camera is controlled by means of a lightdiaphragm. The camera used is usually a plumbicon or a CCD(charge couple device) with 512 or 1024 scan lines.



(1)

X-rayTube

Collimator Table Patient GridImageIntensitesTube

LightDiaphragm

TVCamara

Optics

DigitalChain

TVMonitor

(2) (3) (4)

Fig. 2. Schematic of a digital fluorographic system coupling the image intensifierand the digital chain. See text for key to numbers.

(4) Digital chain: The digital chain consists of an A/D converter,image memories, image processor, digital storage, and videodisplay. The A/D converter, the image memory, and the digi-tal storage can handle 512 × 512 × 8 bit image at 30 frames persecond, or 1024 × 1024 × 8 bit image at 7.5 frames per second.Sometime the RAID (redundant array of inexpensive disks) isused to handle the high speed data transfer.

Fluorography is used to visualize the motion of body compart-ments (e.g. blood flow, heart beat), the movement of a catheter, aswell as to pinpoint an organ in a body region for subsequent detaileddiagnosis. Each exposure required in a fluorography procedure isvery minimal compared with a conventional X-ray procedure.

Digital fluorography is considered to be an add-on systembecause a digital chain is added to an existing fluorographicunit. This method utilizes the established X-ray tube assembly,image intensifier, video scanning, and digital technologies. The out-put from a digital fluorographic system is a sequence of digitalimages displayed on a video monitor. Digital fluorography has anadvantage over conventional fluorography in that it gives a largerdynamic range image and can remove uninteresting structures inthe images by performing digital subtraction.



When image processing is introduced to the digital fluoro-graphic system, dependent on the application, other names areused, for example, digital subtraction angiography (DSA), digitalsubtraction arteriography (DSA), digital video angiography (DVA),intravenous video arteriography (IVA), computerized fluoroscopy(CF), and digital video subtraction angiography (DVSA).

3.3 IMAGING PLATE TECHNOLOGY

Imaging plate system, commonly called computed radiography(CR), consists of two components: the imaging plate and the scan-ning mechanism. The imaging plate (laser-stimulated luminescencephosphor plate) used for X-rays detection, is similar in principle tothe phosphor intensifier screen used in the standard screen/filmreceptor. The scanning of a laser-stimulated luminescence phosphorimaging plate also uses a scanning mechanism (Reader) similar tothat of a laser film scanner. The only difference is that instead ofscanning an X-ray film, the laser scans the imaging plate. This sec-tion describes the principle of the imaging plate, specifications ofthe system, and system operation.

3.3.1 Principle of the Laser-Stimulated LuminescencePhosphor Plate

The physical size of the imaging plate is similar to that of a con-ventional radiographic screen; it consists of a support coated witha photo-stimulable phosphorous layer made of BaFX : Eu2+(X= Cl,Br,I), Europium-activated barium-fluorohalide compounds.After the X-ray exposure, the photo-stimulable phosphor crystal isable to store a part of the absorbed X-ray energy in a quasistable state.Stimulation of the plate by a 633 nanometer wavelength helium-neon (red) laser beam leads to emission of luminescence radiationof a different wavelength (400 nanometer), the amount of which isa function of the absorbed X-ray energy [Fig. 3(B)].

The luminescence radiation stimulated by the laser scanning iscollected through a focusing lens and a light guide into a photomul-tiplier tube, which converts it into electronic signals. Figure 3(A)



X-ray Photons

Unused imaging plate

Recording the X-ray image

The laser beams extractthe X-ray image from theplate by converting it tolight photons which form alight image.

The small amount of re-sidual image on the plateis thoroughly erased byflooding the plate with light.

The erased plate can beused again.

Laser-Beam Scanning

Light

Reading

X-rayexposure

BaFXcrystalssupport

(A)

Erasing

Fig. 3. Physical principle of laser-stimulated luminescence phosphor imagingplate. Above: (A) From the X-ray photons exposing the imaging plate to the forma-tion of the light image. Below: (B) The wavelength of the scanning laser beam (b) isdifferent from that of the emitted light (a) from the imaging plate after stimulation(courtesy of J Miyahara, Fuji Photo Film Co Ltd).



Fig. 3. (Continued)

shows the physical principle of the laser-stimulated luminescencephosphor imaging plate. The size of the imaging plate can be 8×10,10 × 12, 14 × 14, or 14 × 17 square inches. The image produced is2 000 × 2 500 × 10 bits.

3.3.2 Computed Radiography System Block Diagramand its Principle of Operation

The imaging plate is housed inside a cassette just like a screen/filmreceptor. Exposure of the imaging plate (IP) to X-ray radiation resultsin the formation of a latent image on the plate (similar to the latentimage formed in a screen/film receptor). The exposed plate is pro-cessed through a CR Reader to extract the latent image — analogousto the exposed film developed by a film developer. The processedimaging plate can be erased by bright light and be used again. Theimaging plate can either be removable or nonremovable. An imageprocessor is used to optimize the display (e.g. lookup tables) basedon types of exam and body regions.

The output of this system can be one of two forms — a printedfilm or a digital image — the latter can be stored in a digital storagedevice and be displayed on a video monitor. Figure 4 illustrates the



1 2

3

4

CRT

To host computer

Controller

A/Dconverter

Semiconductor laser

Stimulable phospor detector

Fig. 4. Dataflow of an upright CR system with nonremovable imaging plates (IP).(1) Formation of the latent image on the IP. (2) The IP is scanned by the laser beam.(3) Light photons are converted to electronic signals. (4) Electronic signals are con-verted to digital signals which form a CR image (courtesy of Konica Corporation,Japan).

dataflow of an upright CR system with three unremovable imagingplates. Figure 5 shows the latest XG-5000 multiplate reader systemwith removable imaging plate and its components.

3.3.3 Operating Characteristics of the CR System

A major advantage of the CR system compared to the conven-tional screen/film system is that the imaging plate is linear andhas a large dynamic range between the X-ray exposure and therelative intensity of the stimulated phosphors. Hence, under asimilar X-ray exposure condition, the image reader is capable ofproducing images with density resolution comparable or superiorto those from the conventional screen/film system. Since the imagereader automatically adjusts the amount of exposure received bythe plate, over- or underexposure within a certain limit would notaffect the appearance of the image. This useful feature can best beexplained by the two examples given in Fig. 6.

In quadrant A of Fig. 6, example I represents the plate exposedto a higher relative exposure level but with a narrower exposurerange (103–104). The linear response of the plate after laser scan-ning yields a high level but narrow light intensity (photostimula-ble luminescence, PSL) range from 103–104. These light photons areconverted into electronic output signals representing the latent



Fig. 5. A Fuji XG-5000 CR System with the multiimaging plate reader and twoQA/Image Processing workstations (IIP and IIP Lite). Note that the second work-station shares the same database as the first workstation so that an X-ray techniciancan perform QA and image processing while another is operating the plate readerand processing the imaging plates.

image stored on the image plate. The image processor senses a nar-row range of electronic signals and selects a special look-up table[the linear line in Fig. 6(B)], which converts the narrow dynamicrange 103–104 to a large light relative exposure of 1 to 50 [Fig. 6(B)].If hardcopy is needed, a large latitude film can be used that cov-ers the dynamic range of the light exposure from 1 to 50, as shownin quadrant C, these output signals will register the entire opticaldensity (OD) range from OD 0.2 to OD 2.8 on the film. The total sys-tem response including the imaging plate, the look-up table, and the



Fig. 6. Two examples, I and II, illustrate the operating characteristics of the CRsystem and explain how it compensates for over and under exposures.

film subject to this exposure range is depicted as curve I in quad-rant D. The system-response curve, relating the relative exposureon the plate and the OD of the output film, shows a high gammavalue and is quite linear. This example demonstrates how the systemaccommodates a high exposure level with a narrow exposure range.

Consider example II, in which the plate receives a lower ex-posure level but with wider exposure range. The CR system auto-matically selects a different look-up table in the image processorto accommodate this range of exposure so that the output sig-nals again span the entire light exposure range form 1 to 50.



The system-response curve is shown as curve II in quadrant D. Thekey in selecting the correct look-up table is that the range of the expo-sure has to span the total light exposure of the film, namely from 1 to50. It is noted that in both examples, the entire useful optical densityrange for diagnostic radiology is utilized.

If a conventional screen/film combination system was used,exposure on example I in Fig. 6 would only utilize the higher opticaldensity region of the film, whereas in example II it would utilizethe lower region. Neither case would utilize the full dynamic rangeof the optical density in the film. From these two examples, it isseen that the CR system allows the utilization of the full optical den-sity dynamic range, regardless whether the plate is overexposed orunderexposed. Figure 7 shows an example comparing the resultsof using screen/film versus CR under identical X-ray exposures.

Fig. 7. Comparison of quality of images obtained by using (A) the conven-tional screen/film method and (B) CR techniques. Exposures were 70 kVp; 10 mAs,40 mAs, 160 mAs, 320 mAs on a skull phantom. It is seen that in this example thatthe CR technique is almost dose independent (courtesy of Dr S Balter).



The same effect is achieved if the image signals are for digital out-put, and not for hard copy film. That is, the digital image producedfrom the image reader and the image processor will also utilizethe full dynamic range from quadrant D to produce 10-bit digitalnumbers.

3.4 FULL-FIELD DIRECT DIGITAL MAMMOGRAPHY

3.4.1 Screen/Film and Digital Mammography

Conventional screen/film mammography produces a very highquality mammogram on an 8 sq in × 10 sq in film. Some ab-normalities in the mammogram require 50 µm spatial resolutionto be recognized. For this reason, it is difficult to use CR or a laserfilm scanner to convert a mammogram to a digital image, hinder-ing the integration of the modality images to PACS. Yet, mammo-graphy examinations account for about 8% of all diagnosticprocedures in a typical radiology department. During the past sev-eral years, due to much support from the National Cancer Instituteand the United States Army Medical Research and DevelopmentCommand, some direct digital mammography systems have beendeveloped by joint efforts between academic institutions and pri-vate industry. Some of these systems are in clinical use. In the nextsection, we describe the principle of digital mammography, a verycritical component in a totally digital imaging system in a hospital.

3.4.2 Full Field Direct Digital Mammography

There are two methods of obtaining a full field direct digital mam-mogram, one is the imaging plate technology described in Sec. 3.3but with higher resolution imaging plate of different materialsand higher quantum efficient detector systems. The other is theslot-scanning method. This section summarizes the slot scanningmethod.

The slot-scanning technology modifies the image receptor ofa conventional mammography system by using a slot-scanning



mechanism and detector system. The slot-scanning mechanismscans a breast by an X-ray fan beam and the image is recorded bya charged-couple device (CCD) camera encompassed in the Buckyantiscatter grid of the mammography unit. Figure 8 shows a pic-ture of a FFDDM system. The X-ray photons emitted from the X-raytube are shaped by a collimator to become a fan beam. The widthof the fan beam covers one dimension of the image area (e.g. x-axis)and the fan beam sweeps in the other direction (y-axis). The move-ment of the detector system is synchronous with the scan of thefan beam. The detector system of the FFDDM shown is composedof a thin phosphor screen coupled with four CCD detector arraysvia a tapered fiber optic bundle. Each CCD array is composed of1 100×300 CCD cells. The gap between any two adjacent CCD arrays

Fig. 8. A slot-scanning digital mammography system. The slot with 300 pixelwidth covering the x-axis (4 400 pixels). The X-ray beam sweeps (arrow) in the y-direction producing over 5 500 pixels. X: X-ray and collimator housing, C: breastcompressor.



requires a procedure called “butting” to minimize the loss of pixels.The phosphor screen converts the penetrated X-ray photons (i.e. thelatent image) to light photons. The light photons pass through thefiber optic bundle, reach the CCD cells, and then are transformedto electronic signals. The more light photons received by each CCDcell, the larger the signal is transformed. The electronic signals arequantized by an analog to digital converter to create a digital image.Finally, the image pixels travel through a data channel to the sys-tem memory of the FFDDM acquisition computer. Figure 9 showsa 4 K × 5 K × 12 bit digital mammogram obtained with the systemshown in Fig. 8. A screening mammography examination requiresfour images, two for each breast, producing a total of 160 Mbytes ofimage data.

Fig. 9. A 4 K × 5 K × 12 bit digital mammogram obtained with the slot-scanningFFDDM shown on a 2 K × 2.5 K monitor. The window at the upper part of the imageis the magnified glass showing a true 4 K × 5 K region (courtesy of Drs E Sicklesand SL Lou).



3.5 DIGITAL RADIOGRAPHY

During the past five years, research laboratories and manufacturershave devoted tremendous energy and resources investigating newdigital radiography systems other than CR. The main emphases areto improve the image quality and operation efficiency, and to reducethe cost of projection radiography examination. Digital radiography(DR) is an ideal candidate. In order to compete with conventionalscreen/film and CR, a good DR system should:

• Have a high detector quantum efficiency (DQE) detector with 2–3or higher line pair/mm spatial resolution, and a higher signal tonoise ratio.

• Produce digital images of high quality.• Deliver low dosage to patients.• Produce the digital image within seconds after X-ray exposure.• Comply with industrial standards.• Have an open architecture for connectivity.• Be easy to operate.• Be compact in size.• Offer competitive cost savings.

Depending on the method used for the X-ray photon conversion,DR can be categorized into direct and indirect image capture meth-ods. In indirect image capture, attenuated X-ray photons are firstconverted to light photons by the phosphor or the scintillator, fromwhich the light photons are converted to electronic signals to formthe DR image. The direct image capture method generates a digitalimage without going through the light photon conversion process.Figure 10 shows the difference between the direct and the indirectdigital capture method. The advantage of the direct image capturemethod is that it eliminates the intermediate step of light photonconversion. The disadvantages are that the engineering involved indirect digital capture is more elaborate, and that it is inherently dif-ficult to use the detector for dynamic image acquisition due to thenecessity of recharging the detector after each read out. The indirect



Selenium + Semiconductor Converts X-rays to Electrical Signals

Direct Digital Radiograph

Scintillator or Phosphor Converts X-rays to Light Photons

Light Photons to Electrical Signals

Indirect digital Radiograph

e

e

Light photons

(B) Indirect Image Capture

X-Rays

X-Rays

Scintillator or Phosphor ConvertsX-rays to Light Photons

Light Photons to Electrical Signals

Scintillator or Phosphor Converts X-rays to Light Photons

(A) Direct Image Capture

Fig. 10. Direct and indirect image capture methods in digital radiography.

capture method uses either the amorphous silicon phosphor or scin-tillator panels. The direct capture method uses the amorphous sele-nium panel. It appears that the direct capture method has the advan-tage over the indirect capture method since it eliminates the inter-mediate step of light photon conversion.

Two prevailing scanning modes in digital radiography are slotand areal scanning. The digital mammography system discussed inthe last section uses the slot-scanning method. Current technologyfor areal detection mode uses the flat-panel sensors. The flat-panelcan be one large or several smaller panels put together. The arealscan method has the advantage of being fast in image capture, butit also has two disadvantages, one being the high X-ray scattering.The second is the manufacturing of the large flat panels is technicallydifficult.

Digital radiography (DR) design is flexible which can be used asan add-on unit in a typical radiography room or a dedicated system.In the dedicated system, some design can be used both as a table topunit attached to a C-arm radiographic device or as an upright unitshown in Fig. 11. Figure 12 illustrates the formation of a DR image,comparing it with Fig. 4 on that of a CR image. A typical DR unitproduces a 2 000 × 2 500 × 12 bit image instantaneously after theexposure.



(A) Dedicated C-Arm System

(B) Dedicated Chest

(C) Add-On

Fig. 11. Three configurations of digital radiography design.

Unused IP IP with latent image IP with residue image

High Intensity LightX-ray

Digital Image

4 6 8 (100nm)

EmissionLight

Stimulatedlight

DR LaserReader

Fig. 12. Steps in the formation of a DR image, comparing it with that of a CR imageshown in Fig. 4.



3.6 X-RAY CT AND MULTISLICE CT

3.6.1 Image Reconstruction from Projections

Since most sectional images, like CT, are generated based on imagereconstruction from projections, we first summarize the Fourier pro-jection theorem, the algebraic reconstruction, and the filtered back-projection method before the discussion of imaging modalities.

3.6.1.1 The Fourier Projection Theorem

Let f (x, y) be a 2D cross-sectional image of a three-dimensionalobject. The image reconstruction theorem states that f (x, y) can bereconstructed from the cross-sectional one-dimensional projections.In general, 180 different projections in one degree increments arenecessary to produce a satisfactory image, and using more projec-tions always result in a better reconstructed image.

Mathematically, the image reconstruction theorem can bedescribed with the help of the Fourier transform (FT). Let f (x, y) rep-resent the two-dimensional image to be reconstructed and let p(x)be the one-dimensional projection of f (x, y) onto the horizontal axis,which can be measured experimentally (see Fig. 13, the zero degreeprojection). In the case of X-ray CT, we can consider p(x) as the totallinear attenuation of tissues transverses by a collimated X-ray beamat location x.

Then:

p(x, 0) =∫ +∞

−∞f (x, y)dy. (1)

The 1D Fourier transform of p(x) has the form:

P(u) =∫ +∞

−∞

(∫ +∞

−∞f (x, y)dy

)exp ( − i2πux) dx. (2)

Equations (1) and (2) imply that the 1D Fourier transform of a one-dimensional projection of a two-dimensional image is identical tothe corresponding central section of the two-dimensional Fouriertransform of the object. For example, the two-dimensional imagecan be a transverse (cross) sectional X-ray image of the body, and



2-D IFT

F(u,0)= (P(x,0))

1-D FT

1

1

2

3

4

X-rays

F(u, )= (P(x, ))

P(x, )

P(x,0)

Spatial Domain Frequency Domain

f(x, y)

2-D FT

2

2

= 0'...180'

= 0θ

θ

θ

θ θℑ

ℑ

'...180'

Fig. 13. Principle of the Fourier projection theorem for image reconstruction fromprojections. F(0,0) is at the center of the 2D FT, low frequency components arerepresented at the center region. The numerals represent the steps described in thetext.P(x, θ): X-rays projection at angle θ

F(u, θ): 1D Fourier transform of p(x, θ)IFT: Inverse Fourier transform

the one-dimensional projections can be the X-ray attenuation pro-files (projection) of the same section obtained from a linear X-rayscan at certain angles. If 180 projections at one degree incrementsare accumulated and their 1D FTs performed, each of these 180 1DFourier transform represents a corresponding central line of the two-dimensional Fourier transform of the X-ray cross-sectional image.The collection of all these 180 1D Fourier transform is the 2D Fouriertransform of f (x, y).

The steps of a 2D image reconstruction from its 1D projectionsshown in Fig. 13 are as follows:

(1) Obtain 180 1D projections of f (x, y), p(x, θ) where θ = 1, . . . , 180.(2) Perform the FT on each 1D projection.(3) Arrange all these 1D FTs according to their corresponding angles

in the frequency domain. The result is the 2D FT of f (x, y).(4) Perform the inverse 2D FT of (3), which gives f (x, y).



The Fourier projection theorem forms the basis of tomographicimage reconstruction. Other methods that can also be used to recon-struct a 2D image from its projections are discussed later in thischapter. We emphasize that the reconstructed image from projec-tions is not always exact; it is only an approximation of the originalimage. A different reconstruction method will give a slightly differ-ent version of the original image. Since all these methods requireextensive computation, specially designed image reconstructionhardware is normally used to implement the algorithm. The term“computerized (computed) tomography” (CT) is often used to rep-resent that the image is obtained from its projections using a recon-struction method. If the 1D projections are obtained from X-raytransmission (attenuation) profiles, the procedure is called XCT orX-ray CT. In the following sections, we summarize the algebraic andfiltered back-projection methods with simple numerical examples.

3.6.1.2 The Algebraic Reconstruction Method

The algebraic reconstruction method is often used for the recon-struction of images from an incomplete number of projections(i.e. <180◦). The result is an exact reconstruction (a pure chance) ofthe original image f (x, y). For a 512 × 512 image, it will require over180 projections, each with sufficient data points in the projection, torender a good quality image.

3.6.1.3 The Filtered (Convolution) Back-Projection Method

The filtered back-projection method requires two components, theback-projection algorithm, and the selection of a filter to modify theprojection data. The selection of a proper filter for a given anatomicalregion is the key in obtaining a good reconstruction from filtered(convolution) back-projection method. This is the method of choicefor almost all XCT scanners. The result of this method, is an exactreconstruction (again, by pure chance) of the original f (x, y). Themathematical formulation of the filtered back-projection method is



given in Eq. (3):

f (x, y) =∫ π

0h(t)∗m(t, θ)dθ, (3)

where m(t, θ) is the “t” sampling point at “θ” angle projection, h(t)is the filtered function, and “∗” is the convolution operator.

3.6.2 Transmission X-ray Computed Tomography (XCT)

3.6.2.1 Conventional XCT

A CT scanner consists of a scanning gantry housing an X-ray tubeand a detector unit, and a movable bed which can align a specificcross section of the patient with the gantry. The gantry provides afixed relative position between the X-ray tube and the detector unit.A scanning mode is the procedure of collecting X-ray attenuationprofiles (projections) from a transverse (cross) section of the body.From these projections, the CT scanner’s computer program or back-projector hardware reconstructs the corresponding cross-sectionalimage of the body. Figures 14 and 15 show the schematic of two mostpopular XCT scanners (third and fourth generations), both using anX-ray fan beam. These types of XCT take about 5 seconds for onesectional scan, and more time for image reconstruction.

3.6.2.2 Spiral (Helical) XCT

Three other configurations can improve the scanning speed: thehelical (spiral) CT, the cine CT (Sec. 3.6.2.3), and the multislice CT(Sec. 3.6.2.4). The helical CT is based on the design of the third-, orthe fourth-generation scanner, the cine CT uses a scanning electronbeam X-ray tube, and multi-slice CT uses a cone beam instead of afan beam.

The CT configurations shown in Figs. 14 and 15 have one com-mon characteristic: the patient’s bed remains stationary duringthe scanning; after a complete scan, the patient’s bed advances acertain distance and the second scan resumes. The start-and-stopmotions of the bed slow down the scanning operation. If the patient’s



Fig. 14. Schematic of the rotation scanning mode using a fan-beam X-ray. Thedetector array rotates with the X-ray tube as a unit.

bed can assume a forward motion at a constant speed while thescanning gantry rotated continuously, the total scanning time of amultiple section examination could be reduced. Such a configura-tion is not possible, however, because the scanning gantry is con-nected to the external high energy transformer and power supply isthrough the cables. The spiral or helical CT design does not involvecables.

Figure 16 illustrates the principle of spiral CT. There are two pos-sible scanning modes: single helical and cluster helical. In the singlehelical mode, the bed advances linearly while the gantry rotatesin sync for a period of time, say 30 seconds. In the cluster helical



Direction ofX-Ray Motion

X-Ray Tube

Stationary Scintillation Dectector Array

Fig. 15. Schematic of the rotation scanning mode with a stationary scintillationdetector array, only X-ray source rotates.

Fig. 16. Helical (spiral) CT scanning modes.



mode, the simultaneous rotation and translation lasts only 15 sec-onds, whereupon both motions stop for 7 seconds before resumingagain. The single helical mode is used for patients who can hold theirbreath for a longer period of time, while the cluster helical mode isfor patients who need to take a breath after 15 seconds.

The design of the helical XCT introduced in the late 1980s.It is based on three technological advances: the slip-ring gantry,improved detector efficiency, and greater X-ray tube cooling capa-bility. The slip-ring gantry contains a set of rings and electrical com-ponents that rotate, slide and make contact to generate both highenergy (to supply the X-ray tube and generator) and standard energy(to supply powers to other electrical and computer components).For this reason, no electrical cables are necessary to connect thegantry and external components. During the helical scanning, theterm “pitch” is used to define the relationship between the X-raybeam collimation and the velocity of the bed movement.

Pitch = Table movement in mm per gantry rotation/slice thickness.

Thus, a pitch equals to “1” means that the gantry rotates a complete360◦ as the bed advances 1.5 mm in one second which gives a slicethickness of 1.5 mm. During this time, raw data is collected cover-ing 360 degrees and 1.5 mm. Assuming one rotation takes one sec-ond, then for the single helical scan mode, 30 seconds of raw dataare continuously collected while the bed moves 45 mm. After thedata collection phase, the raw data are interpolated and/or extrapo-lated to sectional projections. These organized projections are used toreconstruct individual sectional images. In this case, they are 1.5 mmcontiguous slices. Reconstruction slice thickness can be from 1.5 mmto 1 cm, depending on the interpolation and extrapolation methodsused.

The advantages of the spiral CT scans are speed of scanning,allowing the user to select slices from continuous data to recon-struct slices with peak contrast medium, retrospective creation ofoverlapping or thin slices, and volumetric data collection. The disad-vantages are the helical reconstruction artifacts and potential objectboundary unsharpness.



3.6.2.3 Cine XCT

Cine XCT, introduced in early 1980s, uses a completely differentX-ray technology, namely, an electron beam X-ray tube: this scan-ner is fast enough to capture the motion of the heart. The detec-tor array of the system is based on the fourth-generation stationarydetector array (scintillator and photodiode). As shown schemati-cally in Fig. 17, an electron beam (1) is accelerated through the X-raytube and bent by the deflection coil (2) toward one of the four tar-get rings (3). Collimators at the exit of the tube restrict the X-raybeam to a 30◦ fan beam, which forms the energy source of scan-ning. Since there are four tungsten target rings, each of which hasa fairly large area (210◦ tungsten, 90 cm radius) for heat dissipa-tion, the X-ray fan beam can sustain the energy level required forscanning continuously for various scanning modes. In addition, thedetector and data collection technologies used in this system allowvery rapid data acquisition. Two detector rings (indicated by 4 inFig. 17) allow data acquisition for two consecutive sections simul-taneously. For example, in the slow acquisition mode with a 100 msscanning time, and a 8 ms interscan delay, cine XCT can provide9 scans/s, or in the fast acquisition mode with a 20 ms scanning time,34 scans/s.

The scanning can be done continuously on the same body section(to collect dynamic motion data of the section) or along the axis of thepatient (to observe the vascular motion). Because of its fast scanningspeed, cine XCT is used for cardiac motion and vascular studies and

Fig. 17. Schematic of the cine XCT. Source: Diagram adapted from a technicalbrochure of Imatron Inc.



emergency room scans. Until the availability of the multislice XCT,cine XCT is the fast scanner for dynamic studies.

3.6.2.4 Multislice XCT

In spiral XCT, the patient’s bed moves during scan, but the X-raybeam is a fan beam perpendicular to the patient axis, and the detec-tor system is built to collect data for the reconstruction of one slice. Ifthe X-ray beam is shaped to a three-dimensional cone beam with thez-axis parallel to the patient’s axis, and if a multiple detector array(in the z-direction) system is used to collect the data, then we havea multislice XCT scanner (see Fig. 18). Multislice XCT, in essence, isalso spiral scan except that the X-ray beam is shaped to a cone beamgeometry. Multislice XCT can obtain many images in one exami-nation with a very rapid acquisition time, for example, 160 imagesin 20 seconds, or 8 images/sec, or 4 MB/sec of raw data. Figure 18shows the schematic. It is seen from this figure that a full rotationof the cone beam is necessary to collect sufficient projection data toreconstruct the number of slides equal to the z-axis collimation ofthe detector system (see below for definition). Multislice XCT usesseveral new technologies:

(1) New detector: Ceramic type detector is used to replace tra-ditional crystal technology. Ceramic detector has the advan-tages of more light photons in the output, less afterglow time,higher resistance to radiation and mechanical damage, and canbe shaped much thinner (1/2) for equivalent amount of X-rayabsorption, compared with the crystal scintillators.

(2) Real-time dose modulation: A method to minimize dose deliv-ered to the patient using the cone beam geometry by modulatingthe mAs (milliampere-seconds) of the X-rays beam during thescan.

(3) Cone beam geometry image reconstruction algorithm: Efficientcollection and recombination of cone beam X-ray projections(raw data) for sectional reconstruction.



Fig. 18. Geometry of the multislice XCT. The patient axis is in the z-direction.The X-rays (X) shaped as a collimated cone beam rotates around the z-axis 360◦in sync continuously with the patient’s bed moving linearly in z-direction. Thedetector system (D) is a combination of detector arrays shaped in a concave surfacefacing the X-ray beam. The number of slices per 360◦ rotation are determined bytwo factors: the number of detector arrays in the z-direction, and the method usedto recombine the cone beam projection data into transverse sectional projections(Fig. 13). The reconstructed images are transverse view perpendicular to the z-axis.If the cone beam does not rotate while the patient’s bed is moving, the reconstructedimage is equivalent to a digital fluorographic image.

(4) High speed data output channel: During one examination, say,for 160 images, much more data have to be collected during thescanning. Fast I/O data channels from the detector system toimage reconstruction are necessary.



If the patient bed is moving linearly, but the gantry does notrotate, the result is a digital fluorographic image with better imagequality than that discussed in Sec. 3.2. Currently, 16-slice and32-slice detector CT scanners are in heavy clinical use with the64, 128, and 256-slice detector CT scanners on the near horizon.Figure 19 shows a 3D rendered volume image of acquisition datafrom a 64-slice detector CT scan of a human heart. With the adventof the 256-slice detector CT scanner, it will be feasible to acquireimage data for an entire organ such as the heart in a single scancycle as shown in the figure.

3.6.3 Some Standard Terminology Used in Multi-Slice XCT

Recall the term “pitch” defined in spiral XCT, with cone beam-multidetector scanning, because of the multi-detector arrays in thez-direction (Fig. 18) the table movement can be many times the thick-ness of an individual slice. For example, take a 16 mm × 1.5 mmdetector system (16 arrays with 1.5 mm thickness per array), and

Fig. 19. A 3D volume rendered image of the heart from data acquired by a 64-sliceCT scanner. Note that a 256-slice CT scanner would be able to scan the entire heartin one single rotation (courtesy of Toshiba Medical Systems).



with the slice thickness of an individual image being 1.5 mm, thenuse the definition of “pitch” in spiral scan:

Pitch = Table movement in mm per gantry rotation/Slice thickness

= (16 × 1.5 mm/rotation)/1.5 mm = (24 mm/rotation)/1.5 mm.

That means the table moves 24 mm/rotation with a reconstructedslice thickness of 1.5 mm would have a pitch of 16. (Sec. 3.6.2.2) Thiscase also represents contiguous scans. Comparing this example withthat shown in Section 3.6.2.2 for single slice scan, the definition of“pitch” shows some discrepancy due to the size of the multidetec-tor arrays. Since different manufacturers produce different sizes ofmultidetector arrays, the word “pitch” becomes confusing. For thisreason, the international electrotechnical commission (IEC) acceptsthe following definition of pitch (now often referred to as the IECpitch):

z-axis collimation (T) = the width of the tomographic section alongthe z-axis imaged by one data channel (array). In multidetector row(multislice) CT scanners, several detector elements may be groupedtogether to form one data channel (array).

Number of data channels (N) = the number of tomographic sectionsimaged in a single axial scan.

Table speed or increment (I) = the table increment per axial scan orthe table increment per rotation of the X-ray tube in a helical (spiral)scan.

Pitch (P) = Table speed (I mm/rotation)/(N · T)

Thus, for a 16 detector scanner in a 16 × 1.5 mm scan mode, N = 16and T = 1.5 mm, and if the table speed = 24 mm/rotation, then P= 1,a contiguous scan. If the table speed is 36 mm/rotation, then the pitchis 36/(16∗1.5) = 1.5.

3.6.4 Four-Dimensional (4D) XCT

Refer to Fig. 17, with the bed stationary, but the gantry continuouslyrotates, we would have a four-dimensional XCT, with the fourth



dimension as time. In this scanning mode, human body physiologi-cal dynamic can be visualized in 3D. Current multislice XCT with alimited size of detector arrays in z-direction, and data collection sys-tem of 100 MB/sec can only visualize a limited segment of the body.In order to realize the potential clinical applications of 4D XCT, sev-eral challenges are in order:

(1) Extend the cone beam X-ray and the length of the detector arrayin the z-direction. Currently, detector system with 256 arrayswith 912 detectors per array is available in some prototype 4DXCT systems.

(2) Improve the efficiency and performance of the A/D conversionat the detector.

(3) Increase the data transfer rate between the data acquisition sys-tem to the display system from the 100 MB/sec to 1 GB/sec.

(4) Revolutionize display method for 4D images.

4D XCT can produce images of gigabyte range per examination.Methods of archive, communication, and display become challeng-ing issues.

3.6.4.1 PET/XCT Fusion Scanner

XCT is excellent for anatomical delineation with fast scanning time,while positron emission tomography (PET) is slow in obtainingphysiological images of poorer resolution, but good for the differen-tiation between benign and malignant tumors. PET requires atten-uation correction in image reconstruction, and the fast CT scan timecan provide the anatomical tissue attenuation in seconds which canbe used as a base for PET data correction. Thus, the combination ofa CT and a PET scanner during a scan would give a very power-ful tool for improving the clinical diagnostic accuracy when neitheralone would be able to provide such result. Yet, the two scannershave to be combined as one system otherwise the misregistrationbetween CT and PET images would sometimes give misinforma-tion. CT/PET Fusion scanner is such a hybrid scanner which can



Fig. 20. Reconstructed image showing fusion of both CT image data and positronemission tomography (PET) image data into a single image. Note that PET imagedata shows physiological function while the CT image data shows the anatomicalfeatures. Tools allow the user to dynamically change how much PET or CT datais displayed in the fused image. Note the areas in the body such as the heart withhigh activity signal from the acquired PET data.

obtain the CT images as well as PET images during an examina-tion. The PET images so obtained actually have better resolutionthan that without using the CT attenuation correction. The outputof a PET/CT fusion scanner is two sets of images, CT and PET withthe same coordinate system for easy fusing of the images together.Figure 20 shows an example of CT/PET fused image data set show-ing both anatomical as well as physiological function.

3.6.4.2 Components and Data Flow of an XCT Scanner

The major components and data flow of an XCT include a gantryhousing the X-ray tube, the detector system, and signal pro-cessing/conditioning circuits; a front-end preprocessor unit for



cone/fan beam projection data corrections and recombination totransverse sectional projection data; a high-speed computationalprocessor; a hardware back-projector unit; and a video controller fordisplaying images. In XCT, its CT number, or pixel/voxel value, orHounsfield number, represents the relative X-ray attenuation coef-ficient of the tissue in the pixel/voxel, is defined as follows:

CT number = K(µ − µw)/µw,

where µ is the attenuation coefficient of the material under consid-eration, µw is the attenuation coefficient of water, and K is a constantset by the manufacturer.

3.6.5 XCT Image Data

3.6.5.1 Slice Thickness

Current multislice CT scanners can feature up to 32 detectors in anarray. In a spiral scan, multiple slices of data can be acquired simul-taneously for different detector sizes, and 0.75 mm, 1 mm, 2 mm,3 mm, 4 mm, 5 mm, 6 mm, 7 mm, 8 mm, and 10 mm slice thicknesscan be reconstructed.

3.6.5.2 Image Data Size

A standard Chest CT of coverage size between 300 mm–400 mm canyield image sets from 150–200 images all the way up to 600–800images depending on the slice thickness, or data sizes from 75 MBup to 400 MB. Performance-wise, that same standard chest CT canbe acquired in 0.75 mm slices in 10 seconds. A whole body CT canproduce up to 2 500 images or 1 250 MB (1.25 GB) of data. Each imageis 512 × 512 × 2 byte size.

3.6.5.3 Data Flow/Post-Processing

The fan/cone beam raw data are obtained by the acquisition hostcomputer. Slice thickness reconstructions are performed on the rawdata. Once the set of images is acquired in DICOM format, any post-processing is performed on the DICOM data. This includes sagittal,



coronal, and off-axis slice reformats as well as 3D post processing.Sometimes the cone beam raw data are saved for future reconstruc-tion of different slice thicknesses.

Some newer scanners feature a secondary computer, whichshares the same database as the acquisition host computer. This sec-ondary computer can perform the same post-processing functionswhile the scanner is acquiring new patient data. This secondarycomputer also can perform network send jobs to data storage oranother DICOM destination (e.g. highly specialized 3D processingworkstation) and maintains a send queue, thus alleviating the acqui-sition host computer from these functions and improving systemthroughput.

References

1. Cao X, Huang HK, Current status and future advances of digital radio-graphy and PACS, IEEE Eng Med & Bio 19(5): 80–88, 2000.

2. Feldkamp LA, Davis, LC, Kress JW, Practical cone-beam algorithm,J Optical Society Amer A 1: 612–619, 1984.

3. Huang HK, PACS and Imaging Informatics: Basic Principles and Applica-tions, Wiley & Sons, NY, 2004.

4. Stahl JN, Zhang J, Chou TM, Zellner C, Pomerantsev EV, Huang HK,Anew approach to tele-conferencing with intravascular ultrasound andcardiac angiography in a low-bandwidth environment, RadioGraphics20: 1495–1503, 2000.

5. Taguchi K, Aradate H, Algorithm for image reconstruction in multislicehelical CT, Medical Physics 25(4): 550–561, 1998.


CHAPTER 4

Principles of Nuclear MedicineImaging Modalities

Lionel S Zuckier

Nuclear medicine utilizes radioactive molecules (radiopharmaceuticals)for the diagnosis and treatment of disease. The diagnostic informationobtained from imaging the distribution of radiopharmaceuticals is fun-damentally functional and thus differs from other imaging disciplineswithin radiology, which are primarily anatomic in nature. Imaging usingradiopharmaceuticals can be subdivided into single- and dual-photonmodalities. A wide selection of radiopharmaceuticals is available forsingle-photon imaging designed to study numerous physiologic pro-cesses within the body. Static, dynamic, gated and tomographic modesof single-photon acquisition can be performed. Dual-photon imaging isthe principle underlying positron emission tomography (PET) and is fun-damentally tomographic. PET has expanded rapidly due to the clinicalimpact of the radiopharmaceutical 18F-fluorodeoxyglucose, a glucose ana-logue used for imaging of malignancy. The fusion of nuclear medicinetomographic images with anatomic CT is evolving into a dominant imag-ing technique. The current chapter will review physical, biological andtechnical concepts underlying nuclear medicine.

4.1 INTRODUCTION

4.1.1 Physical Basis of Nuclear Medicine

Nuclear medicine is a branch of medicine which utilizes moleculescontaining radioactive atoms (radiopharmaceuticals) for the diag-nosis and treatment of disease. Radioactive atoms have structurallyunstable nuclei and seek to achieve greater stability by the releaseof energy and/or particles in a process termed radioactive decay.1

63


64 Lionel S Zuckier

Atoms with unstable arrangements of protons and neutrons aretermed radionuclides. This stochastic process is governed by first-order kinetics such that for N atoms, the rate of decay dN/dt isequal to −λN, where t is time and λ is the physical decay constant.It follows that N(t) = N0e−λt where N0 is the number of radioactiveatoms present at time 0. The time for half of a sample of atoms todecay is a constant termed the physical half-life (T1/2) and is charac-teristic for each radionuclide. The physical decay constant λ can beexpressed as 0.693/T1/2. It is customary to quantify the amount ofa radioactive substance by its rate of decay, or activity. The S.I. unitBecquerel (Bq) is equal to 1 disintegration per second (dps), while thetraditional unit Curie (Ci) is equal to 3.7 × 1010 dps.

A second important feature in characterizing a radionuclide isthe nature, frequency, and energy of its emitted radiations. Varioustypes of radiation may be emitted from the atomic nucleus (Table 1).Alpha (α), beta (β−) and positron (β+) radiations are particulate andpenetrate relatively short distances in tissue. Gamma (γ) radiationis non-particulate and penetrating, making it useful for diagnosticimaging purposes, where it can be detected by instruments externalto the body. Other types of penetrating radiations which are imagedin nuclear medicine include X-rays that are emitted as a consequenceof rearrangement of shell electrons, and 511 keV photons (annihila-tion radiations) that result from positron-electron annihilation.

Table 1. Ionizing Radiations33

Type Rest Mass Charge Origin(Atomic Mass Units)

Alpha (α) 4.002 +2 NucleusBeta negative (β−) 5.486 × 10−4 −1 Nucleus

or electron (e−)Gamma (γ) 0 0 NucleusBeta + (β+) 5.486 × 10−4 (e−)+1 Nucleus

or positron (e+)X-ray 0 0 Shell electronsAnnihilation photons 0 0 Annihilation of

e+ and e−


Principles of Nuclear Medicine Imaging Modalities 65

Penetrating radiation is subject to attenuation in soft-tissues inan exponential manner. As photons travel a distance x through mat-ter, the intensity of radiation decreases as e−µx where µ is the linearattenuation coefficient dependent on the mass density of the atten-uator, its atomic number (Z), and the energy of the radiation. Forphotons typically used in nuclear medicine, the predominant inter-action with soft tissue is Compton scattering, potential degradingthe image by redirecting the photons. Attenuation is also a fun-damental foil to quantitative analysis, as the radiation measuredby detectors external to the body is reduced to a variable degreedepending on the nature and amount of intervening attenuator andis no longer directly proportional to the activity at the source beingimaged. Methods are available to estimate and compensate for atten-uation in nuclear medicine imaging and will be discussed whererelevant.

The types of radiation enumerated above are all ionizing; whenthey pass through tissues they deposit energy leading to potentialchemical and biologic effects. In addition to man-made radiationfrom medical, industrial and military causes, inevitable exposureto ionizing radiation also results from natural radiation emanatingfrom outer space and radionuclides in the earth’s crust. Mammaliancells posses repair mechanisms which, at least in part, repair suchdamage. The study of the interaction of radiation and living organ-isms comprises the discipline of radiation biology.2 Much has beenlearned regarding the potential toxicity of radiation and this hasinformed the field of radiation safety.3 Controversy persists to thepresent day as to whether there is a minimal amount of radiationnecessary to cause cell damage. However, for safety and regulatorypurposes we assume that exposure to even a small amount of radia-tion is associated with a finite risk (nonthreshold model). As a generalrule, particulate radiation is significantly more injurious than non-particulate, as the energy is deposited in a more concentrated track.Physicians must weigh the potential benefits of radiologic studieswith their inherent risks. Radiopharmaceuticals are therefore admin-istered in the smallest amounts necessary to effect a diagnosis ortherapy.


66 Lionel S Zuckier

4.1.2 Conceptual Basis of Nuclear Medicine

Nuclear medicine owes much of its approach to the Tracer Principle,developed by George de Hevesy, who was awarded the Nobel Prizein Chemistry in 1943 for his “work on the use of isotopes as tracer ele-ments in researches on chemical processes.”4 In de Hevesy’s method,a radioactive atom is introduced within a molecule under study,thereby allowing the newly formed radioactive moiety to serve asan easily identifiable version of the compound, a radioactive tracer.5

The use of radioactive molecules to elucidate physiologic pathwayshas been adopted by nuclear medicine. Radionuclides with appro-priate physical properties are substituted into molecules of biologi-cal interest thereby creating radiopharmaceuticals which, combinedwith selectivity of the body’s physiologic processes, can be used toidentify and target cells and organs of interest (Fig. 1). Techniques

L S

H

K

Br

Bl

K

Legend

M

M

M

M

T

Fig. 1. 18F-FDG PET scan demonstrates foci of malignant tumor within lymphnodes in the patient’s right neck (arrow). Malignant tumor, in contrast to mostother tissues, tends to preferentially metabolize glucose; FDG can therefore be usedto identify sites of malignancy, as in the present case. The distribution of FDG alsoreflects normal tissues of the body that avidly utilize glucose, including brain (Br),and to a lesser degree liver (L) and spleen (S). In the resting state, uptake of FDG bymuscles (M) is minimal. Uptake by the heart (H), tonsils (T) and bowel (arrowheads)noted in the current study tend to be variable in intensity. Unlike glucose, a largeproportion of FDG is excreted by the kidneys (K) into the urinary bladder (Bl).



within nuclear medicine are unique amongst the radiologic modal-ities in that they primarily yield information regarding the functionof tissues, rather than anatomic or structural detail.

Optimally, one of the atoms within a biologically relevantmolecule is substituted with a suitable radioactive isotope of thesame element. The difference in atomic mass of isotopes is due toa variation in the number of neutrons while the number of protonsis unchanged, the latter guaranteeing virtually identical chemicalbehavior of the moieties. For example, 123I or 131I can be substi-tuted for stable (nonradioactive) 127I within sodium iodide (NaI),which is still taken up by the thyroid gland in a manner identi-cal to the nonradioactive substrate. More commonly, because oflimitations in available radionuclides and their imaging proper-ties, an analog of the molecule of interest is created which sharescritical biochemical features, although its chemical structure differsand its biological fate is not identical to that of the original com-pound. 18F-Fluorodeoxyglucose (FDG) represents a radiopharma-ceutical which shares some, but not all, features of glucose, yet is ofimmense clinical utility (Fig. 2).

GlucoseO

OHOH

HOHO

HO

GLYCOLYSIS

XGlucose-6P

-------FDG-6P

Cell

Glucose-------FDG

O

OH18F

HOHO

HO

FDG

Glu

t 1, 3

Hex

okin

ase

Fig. 2. Structure of glucose and 18F-flouro-deoxyglucose (FDG) are illustratedon the left side of the diagram; in the latter, a hydroxyl group has been replacedwith radioactive 18F-flourine to form FDG. Although glucose and FDG are takenup similarly in cells by the glut1 and glut 3 glucose transporters and both arephosphorylated by the enzyme hexokinase, FDG-6-phosphate (FDG-6P), in contrastto glucose-6-phosphate (glucose-6P), can not proceed to glycolysis and in effectbecomes trapped within the cells.


68 Lionel S Zuckier

The methods of tracking distribution of radiopharmaceuticalsin clinical nuclear medicine are varied. Infrequently, radiophar-maceuticals are used in non-imaging quantitative assays wheresamples of blood or urine are measured in sensitive well type scin-tillation detectors as a means of deriving information regardingphysiologic function and metabolic clearance. Examples includethe measurements of the absorption and subsequent excretion of57Co-labeled vitamin B12 (used in the evaluation of vitamin B12

deficiency), and the determination of the rate of renal excretionof 51Cr-EDTA (used in the measurement of renal function). Non-imaging probe-type radiation detectors are used for organ-basedcounting such as in the measurement of thyroid uptake of radio-active iodine. While no image is generated by the thyroid probe, thedata are fundamentally spatial in that the detector interrogates adefined volume of tissue. The use of nonimaging probes has alsospread to the surgical suite, in order to identify lymph nodes whichhave been rendered radioactive by virtue of draining an anatomicregion where radiopharmaceutical has been injected into the subcu-taneous tissue. These collimated solid-state hand-held scintillationdetectors, used in sentinel lymph node biopsy, have a well-definedfield-of-view and can be slowly translated over the surgical bedto reveal the location of radioactive lymph nodes or other targets.A third and most common method of assaying the distribution ofradiopharmaceuticals used in the current practice of clinical nuclearmedicine, is radionuclide imaging. This noninvasive technique hasevolved as an integral component of medical imaging for over fivedecades, often serving as the proving ground for concepts that weresubsequently introduced into other radiologic modalities.6

An important facet of nuclear medicine is the administra-tion of therapeutic radiopharmaceuticals designed to destroy tar-geted cells.7 Therapeutic radionuclides in clinical use today emitβ− particles. For example, 131I-NaI is used for treatment of thy-roid cancer while 90Y or 131I labeled antibodies are administeredto destroy lymphoma cells. While these therapies, per se, are notimaging examinations, therapeutic applications are frequently pre-ceded by imaging using γ-emitting analogues in order to predict



efficacy and toxicity, a discipline within medical physics termeddosimetry.

4.1.3 Radiopharmaceuticals in Nuclear Medicine

Radiopharmaceuticals for diagnostic purposes are generally labeledwith γ emitting radionuclides; γ photons which readily exit thebody, are detectable by a variety of instruments discussed withinthis chapter, and pose the lowest radiation risk to the patient. Theγ-emitting radionuclides used emit photons with one or more princi-pal energies in the range of 69 keV–394 keV (Table 2). The most com-mon radionuclide used today is 99mTc which possesses the nearlyoptimal characteristics of 140 keV γ-radiation, physical T1/2 of sixhours, and the absence of energetic particulate emissions. Determi-nation of the spatial distribution of a radiopharmaceutical basedon the detection of individual photons emitted from the patient’sbody is termed single-photon imaging. A contrasting imaging pro-cess prevails in positron emission tomography (PET) where radio-pharmaceuticals incorporate radionuclides that emit positrons inthe course of their radioactive decay (Table 3). In PET, the spatialdistribution of the radiopharmaceutical is determined by detect-ing a pair of simultaneously-emitted photons resulting from the

Table 2. Radionuclides used in Single-Photon Radionuclide Imaging34

Radionuclide Principle PhotonEnergies (keV)

Half-Life(Hours)

CommonRadiopharmaceuticalForms

Clinical Application

Tc-99m 140 6.02 Numerous NumerousI-131 364 193 NaI, MIBG Thyroid, tumors (1)Ga-67 93, 185, 300, 394 78.3 Ga-citrate Inflammation,

infectionIn-111 171, 245 67.9 Labeled leukocytes,

octreotideInfection, tumors (2)

Tl-201 69–80 73.1 Thallous chloride Cardiac perfusion(Hg X-rays)

I-123 159 13.2 NaI, MIBG Thyroid, tumors (1)

Legend: MIBG — metaiodobenzylguanidine, octreotide, (1) tumor of the APUDfamily, (2) somatostatin receptor bearing tumors.


70 Lionel S Zuckier

Table 3. Radionuclides Commonly used in Positron Emission Tomography26

Radionuclide Method of Half-life Max β+ Energy Maximal Range inProduction (Minutes) (MeV) Water (mm)

C-11 Cyclotron 20.4 0.96 3.9N-13 Cyclotron 9.96 1.2 5.1O-15 Cyclotron 2.05 1.7 8.0F-18 Cyclotron 110 0.64 2.3Ru-82 Strontium 1.3 3.4 18

82 generator

annihilation of a positron and electron in a process termed dual-photon or coincidence imaging. In the discussion that follows, instru-ments for both single-photon and dual-photon imaging will bereviewed. Discussion of nonimaging detectors will serve as anintroduction to the principles of equipment used in nuclearmedicine imaging.

4.2 NUCLEAR MEDICINE EQUIPMENT8

4.2.1 Nonimaging

Basic to understanding the techniques and equipment used forimaging in nuclear medicine are the principles underlying the scin-tillation detector.9 When used for in vivo assay of a radiopharma-ceutical within an organ, such as the amount of radioactive iodineuptake within the thyroid gland, the scintillation detector is collo-quially termed a scintillation probe. Components of the scintillationprobe include a collimator, scintillation crystal, photomultiplier tube(PMT), and electronics (Fig. 3). The collimator restricts the field-of-view of the crystal to a finite region opposite the collimator opening(aperture). The scintillation crystal effectively shifts the wavelengthof photon energy from γ rays to visible light, in quanta proportionalto the energy of the incident photon. In clinical systems, crystalsof sodium iodide, purposely contaminated or “doped” with thal-lium ions (NaI(Tl)), are commonly used. The crystals are optically



Fig. 3. Scintillation crystal. In the illustration, a 159 keV photon, emitted from thedecay of 123I within the thyroid gland, enters the aperture of the collimator andis absorbed within the NaI(Tl) crystal, resulting in conversion to visible light (ascintillation). The light is incident on the photocathode within the photomultipliertube (PMT), which dislodges electrons that are subsequently amplified many foldsby a series of dynodes held at progressively greater voltages. Pulses produced bythe PMT for each photon absorbed are sorted by the pulse-height analyzer (PHA),resulting in an energy spectrum (actual 123I spectrum shown). Counts within adefined range of energies (i.e. the photopeak energy window) on the PHA areintegrated over time by the scaler/ratemeter. In a related design, the scintillationcrystal is fabricated with a well into which samples are placed for analysis andcounting with high geometric efficiency (lower right).

coupled to PMTs which convert the scintillations of light into currentwhich is amplified to detectable levels. PMTs consist of a photocath-ode, designed to emit photoelectrons upon being struck by incidentlight, multiple dynodes held at progressively increasing voltagesproducing to an amplified cascade of electrons, and an anode whichcollects the current. Voltage across PMTs is in the 1000 volt range.Each γ photon originally absorbed in the crystal results in a dis-crete pulse of current exiting the PMT; the amplitude of this pulseis proportional to the incident photon energy. A device similar tothe scintillation probe is used to characterize and count radioactive


72 Lionel S Zuckier

samples which are placed within a specialized scintillation crystalwith an indentation or well which serves to increase the efficiency ofcounting (Fig. 3).

Electronics in counting systems typically include a pulse-height analyzer (PHA), which can discriminate pulses of differingamplitudes originating from photons of differing energies. Scat-tered photons loose a portion of their initial energy and canthereby be differentiated from unscattered photons and excludedfrom counting if so desired. Photons emitted from multiple radio-nuclides can also be discriminated using the PHA. As a generalrule, counts are integrated over a fixed period of time and dis-played on a scaler/ratemeter. Modern devices can estimate andcompensate for the fraction of counts lost due to dead time (pulseresolving time).

4.2.2 The Rectilinear Scanner

When collimated, the scintillation probe effectively interrogates awell-defined volume, and therefore conveys spatial information.Early efforts to provide a distribution map of radiopharmaceuti-cals within the body were obtained by manually translating thescintillation crystal over a region of interest.5 A mechanical sys-tem of rectilinear scanning, developed by Benedict Cassen in theearly 1950s, incorporated a systematic method of measuring countrates in a raster pattern over a region of interest.10 Images wererecorded on paper or film as dots whose intensity was proportionalto the count rate sampled at the corresponding locations over thepatient. One of the advantages of the rectilinear scanner was thatit allowed for a simple method of mapping palpable abnormalitieson the patient to locations on the printed image. A disadvantageof rectilinear scanning was the relatively protracted time neededto scan an area of interest, since the data were collected serially.11

This precluded imaging of dynamic processes, such as the flow ofblood to an organ, leading to the development of alternate imagingmethods.



4.2.3 The Anger Gamma Camera

4.2.3.1 Background

The need to simultaneously, rather than sequentially, sample countsfrom within the volume of interest was a factor that led todevelopment of the gamma-camera (γ-camera) by Hal Anger in1952.12,13 Basic principles introduced by Anger remain operative innuclear medicine imaging systems used today, albeit with refine-ments in image acquisition, storage and retrieval made possible bywidespread availability of microprocessors. Elements of the Angerscintillation camera design include the collimator, crystal, PMTs, andelectronics (Fig. 4).

4.2.3.2 Collimation14,15

Purpose of collimation in γ-cameras is to map the distribution ofactivity onto the crystal surface in an orderly and interpretable man-ner. The majority of collimators used today are parallel hole, consist-ing of multiple lead septa (or partitions) arranged perpendicularly tothe crystal face so that they only permit passage of γ photons normalto the crystal. Collimators are designed to be specific to a particularrange of radionuclide energies. Collimators are also designed basedon preferences between the competing goals of count-rate sensitiv-ity and spatial resolution, as determined by the length, thickness,and spacing (aperture width) of the septa.16 Clinical systems soldtoday are typically equipped with a selection of low energy col-limators designated for “high-resolution,” “high-sensitivity,” and“all-purpose” applications. For nuclear imaging laboratories thatutilize 67Ga, 111In or 131I, collimators designed for medium-energy(67Ga, 111In) and-high-energy (131I) imaging are also required.

There is a relationship between the source-to-collimator distance(r), image spatial resolution and count-rate sensitivity for parallel-hole collimators. As r increases, spatial resolution worsens whilethe count-rate sensitivity remains constant (Fig. 5). While this latterobservation appears to contradict the so-called inverse square law,in fact, the intensity of radiation at each collimator aperture does


74 Lionel S Zuckier

Fig. 4. Basic principle of the Anger scintillation camera. In the current illustra-tion, an area of concentrated radiopharmaceutical within the patient’s body emitsγ photons in an isotropic manner. The fate of various emitted photons is illus-trated. Photon 1 exits the body but does not intersect the γ camera, while photon 2is completely absorbed within the patient. Photon 3 exits the body and intersectsthe γ camera, but the angle of incidence is such that the photon is absorbed by thelead septa partition of the parallel-hole collimator. Photon 4 travels in a directionsuch that it is able to pass through a collimator aperture, strike and be absorbed bythe Na(I) crystal. The energy of the photon is transformed to visible light emittedisotropically within the crystal which travels or is reflected towards the photomul-tiplier tubes (PMTs). These convert the light signal to an electronic pulse which isamplified and then analyzed by the positioning and summing circuits to determinethe apparent position of absorption. If the total energy, or Z signal, of the ampli-fied photon (as indicated on the illustrated 99mTc energy spectrum by the number4) falls within a 20% window centered on the 140 keV energy peak, the event isaccepted and x and y coordinates are stored within the image matrix. If a γ photonis scattered within the patient as in photon 5, the lower energy of its pulse will berejected by the PHA, and the erroneous projection of this photon will not be addedto the image matrix. Less commonly, diverging, converging or pinhole collimatorsmay be used in place of the parallel hole collimator.

decrease as 1/r2; however, the number of elements which admitphotons increases as r2, based on geometric considerations. Thisin turn explains the loss of resolution with increasing distance, withradiation emitted from a focal source passing through the collimatorto interact with ever-larger regions of the crystal.



Fig. 5. Four one-minute images taken anteriorly over the chest of a patient withthe collimator placed 2, 4, 8 and 16 inches from the subject. While the count rateremains relatively constant over these distances (total counts noted in the lowerright corner of each image), degradation of spatial resolution is readily apparent.

An additional collimator design is used to image small objectswith superior resolution. The pinhole collimator (Fig. 4) consistsof a single small aperture (3 mm–5 mm in diameter) within a leadhousing which is offset from the crystal, thereby restricting inci-dent photons to those that pass through the aperture. This createsan inverted and potentially magnified image on the crystal face in amanner analogous to that used in pinhole photography. The pinholecollimator has the capacity to increase resolution (through its magni-fying effect), especially at close distances, but does so at the expenseof field-of-view and parallax distortion. As source-to-collimator dis-tance increases with this collimator, the field of view of the cam-era increases while magnification and resolution less and count-rateefficiency markedly decreases.

Additional collimator designs include diverging and converg-ing collimators, the former allowing imaging of an area larger thanthe collimator face and the latter allowing the enlargement of small


76 Lionel S Zuckier

regions of interest (Fig. 4). These are infrequently used today butmay still find application in portable cameras with small field-of-view crystals.

4.2.3.3 Crystal

As in the non-imaging probe, γ-camera crystals are generally com-posed of NaI(Tl). Features that make this crystal desirable includehigh mass density and atomic number (Z), thereby effectivelystopping γ photons, and high efficiency of light output. Mostcurrent cameras incorporate large (50 cm × 60 cm) rectangular detec-tors. While expensive, the larger field of view results in increasedefficiency.17 In early designs, crystals were often 0.5 inches thick,which was well-suited for high energy γ photons. In more recentimplementations of the γ-camera, crystals only 3/8-inch or 1/4-inchthick are used, which is more than adequate for stopping the pre-dominantly low-energy photons in common use today and whichalso results in superior intrinsic spatial resolution. In the Anger cam-era design, the NaI(Tl) crystal is optically coupled to an array ofPMTs which is packed against the undersurface of the crystal. Light-pipes may be used to redirect light from the crystal into the PMTs.

4.2.3.4 Positioning Circuitry and Electronics

Early positioning circuitry in the Anger γ-camera was analog innature. γ photons incident on the NaI crystal resulted in productionof light which propagated throughout the crystal and was convertedto an amplified electrical pulse by the PMTs. Output of the PMTs wassummed to produce an aggregate X and Y signal which reflected thelocation of the scintillation event in the crystal, and which was usedto deflect the beam of a cathode ray tube (CRT) in order to produce asingle spot on the image. The sum of the PMT signals (Z signal) wasproportional to the γ photon energy and was used to exclude lowerenergy scattered photons. In order to accurately superimpose thedistribution of multiple γ photons of different energies emanatingfrom one or more radionuclides (such as 171 and 245 keV photons



of 111In and 93, 185, 300 and 394 keV photons of 67Ga), the Z pulseis also used to normalize the X and Y signals so that the imagesdescribed by each photopeak are superimposable and image size isenergy-invariant (Z pulse normalization).

As microcomputers became faster, less expensive, and morewidely available, successive versions of γ-cameras increasinglyincorporated microprocessors.18 Initially, the x and y signals ofthe γ-camera were first processed by analog means and subse-quently converted to digital signals by analog-to-digital convert-ers (ADCs). Computers were then used for image storage, viewing,image correction19 and various quantitative analyses. γ-cameraseventually became fully “digital” in that the output of each PMTwas digitized. Most current digital γ-cameras have ability to adjustthe gain of each PMT individually, leading to the improved overallcamera performance.20 Individual events, as detected by the PMTs,are then corrected for local differences in pulse-height spectra andfor positioning. These refinements have led to improvement in spa-tial resolution and image uniformity.

4.2.3.5 Modes of Acquisition

Prior to image acquisition, the operator must specify acquisitionparameters such as size of the image matrix, number of brightnesslevels, photopeak and window width. Typically, for a 99mTc acqui-sition, a 128 × 128 matrix is used with 28 or 216 levels of bright-ness, corresponding to maximum of 256 or 65 536 counts per pixel,respectively. The acquisition window refers to the range of photonenergies which will be accepted by the PHA. The peak and win-dow width selected for 99mTc are 140 keV and 20%, respectively. Asmentioned earlier, scattered photons have decreased energy, andthe energy window excludes most, though not all, of these lowerenergy photons. Modern cameras allow concurrent acquisition ofphotons in multiple energy windows, whether emanating from asingle radionuclide with multiple γ emissions (such as 67Ga), ormultiple radionuclides with one or more energy peaks each (Fig. 6).


78 Lionel S Zuckier

Fig. 6. Dual-isotope acquisition. 24 hours prior to imaging, 0.5 mCi of 111In-labeledautologous white blood cells (111In-WBCs) were injected intravenously into thepatient to localize sites of infection. 30 minutes prior to imaging, 10 mCi of 99mTc-sulfur colloid were injected intravenously to localize marrow. Coimaging of the99mTc window (140% ± 20% keV) and dual 111In windows (171% ± 15% keV and245% ± 15% keV) was performed, thereby producing simultaneous images of themarrow (left panel) and white blood cell distribution (right panel). In spite of differ-ing energy, Z-pulse normalization has resulted in superimposable images. Marrow,liver and spleen are visible on both marrow and WBC studies. The 99mTc study isused to estimate WBC activity which is due to normal marrow distribution andof no pathologic consequence. To signify infection, 111In-WBC activity must be inareas other than the visualized marrow distribution.

In the latter case, photons derived from each radionuclide can sub-sequently be separated into separate images, each reflecting the dis-tribution of a single radiopharmaceutical. Multi-isotope imaging isespecially helpful when the two sets of images are used for com-parison purposes. Depending on the relative activities of the radio-pharmaceuticals and other considerations, the images of the isotopeemitting the lower energy photons may have to be corrected fordown-scatter of higher energy photons into its energy window.



Current clinical γ-cameras typically acquire several types of data(Fig. 7). To acquire a static (or spot) view, the camera is placedover a region of the body and an acquisition is performed for apredetermined number of counts or interval of time. The latter isappropriate when intensities of different parts of the body are beingcompared. Dynamic imaging refers to the acquisition of multiplesequential images at defined intervals. These may be of short dura-tion, such as a series of two-second images to portray blood flow, or

Fig. 7. Images taken from a bone scan illustrate various modes of acquisition.Initial anterior images from the dynamic flow study (top panel) consist of sequential2-second images taken over the feet subsequent to injection of 25 mCi of 99mTc-labeled MDP. Static (spot) images were taken two hours thereafter in anterior, leftlateral, and plantar (sole of foot) projections for five minutes each. Sweep imageswere also taken at that time, in anterior and posterior projections, where the detectorand patient table move with respect to each other in order to produce an extendedfield-of-view image. Increased flow and bone uptake within the left foot are highlysuggestive of osteomyelitis (infected bone).


80 Lionel S Zuckier

of longer duration, such as multiple one-minute images to demon-strate uptake and excretion of radiopharmaceuticals by the kidneysor liver. Many cameras have the ability to acquire whole-body views,where the detector and patient bed move with respect to each otherduring the acquisition, allowing an elongated sweep image to beobtained. Gated images are obtained during the course of a cyclicalphysiologic process, where a series of dynamic images is acquiredbased on repetitive sampling synchronized by a physiologic trigger(Fig. 8). This is commonly used to obtain a series of images dur-ing phases of the cardiac cycle, thereby portraying the change inleft-ventricular volume during this period. In this method, the R-wave of the electrocardiograph (ECG) is used to repetitively triggeracquisition into a series of 16 brief frames into which counts areaccrued. When summed over the course of several hundred cardiacbeats, the limitation of statistical “noise” imposed by the few countscollected over each sub-second segment of the physiologic cycle isovercomed.

In general, two modes of data collection are possible, frame modeand list mode. In the former and more common method of imaging,events detected by the PHA that fall within predetermined param-eters are incremented into a specific image matrix. For example, indynamic or gated imaging, frame length is prescribed a priori, and thecounts are parsed into the appropriate matrices in real time. Framemode is an efficient method of memory utilization, and images areretrievable immediately following completion of acquisition. A dis-advantage of frame mode is that the acquisition parameters mustbe selected prior to the acquisition, and cannot be retrospectivelychanged. For example, if the patient’s heart rate changes during theacquisition, or if we wish to adjust the energy window, there is noway to alter the acquisition parameters. Alternatively, the time, loca-tion, and even energy of each scintillation event over the entire dura-tion of the acquisition, in addition to any relevant physiologic trig-gers, are stored when acquiring in list mode. At the conclusion of theacquisition, each event can be retrospectively sorted from the list intospecific time bins or energy windows. As the data list remains intact,this exercise can be repeated as many times as desired. List mode



A B C D E F G H I J A B C D E F G H I JK L M N O P A B C D E F G H I JK L M N O P K L M N O P

A B C D

E F G H

I J K L

M N O P

Time

% EDV

100

75

50

25

0

ECG

ROI

Time-Activity Curve

BK

Fig. 8. Gated cardiac study performed after labeling of the patient’s red blood cellswith 25 mCi of 99mTc. The electrocardiograph tracing (top) illustrates the divisionof the cardiac cycle into 16 frames, marked A-P for illustrative purposes. Countsfrom the γ-camera during each portion of the cycle are assigned to a correspondingimage matrix (labeled A-P). After counts from several hundred beats have beensummed, count statistics are adequate to portray the change in volume of bloodwithin the heart during a cardiac cycle. These can be shown as sequential images,as illustrated, or as a cine loop. Quantitative analysis can also be performed. Asillustrated, regions-of-interest (ROIs) are placed over the left ventricle (LV) at bothend diastole and end systole (black and white curves, respectively). Non-specificactivity overlying the heart is estimated from an adjacent background (BK) regionand subtracted from the LV regions. A time-activity curve (solid line) depicts thechange in LV ventricular volume during the course of an average heart-beat. Thedotted line illustrates the first derivative. In the current illustration, the percentchange in LV volume during contraction (ejection fraction) is 45%, slightly belownormal. RV and SP indicate locations of the right ventricle and spleen, respectively.

necessitate increased storage requirements, but is especially usefulin research applications where data may be analyzed in multipleways.

Tomographic imaging refers to the acquisition of 3D data. Theinitial development of tomography occurred in nuclear medicine6;


82 Lionel S Zuckier

subsequently this technique was extended to computed transmis-sion tomography (CT) and magnetic resonance imaging (MRI).Single- and dual-photon methods of tomography will be discussedseparately below.

4.2.3.6 Analysis of Data

A strength of nuclear medicine is quantitative analysis. Regions ofinterest (ROIs) can be defined on an image, or series of images, andused to extract areal count rates. When applied to a dynamic series ofimages, the result is called a time-activity curve (TAC). For example,a ROI over the left ventricle, in conjunction with gating by the elec-trocardiograph (ECG), can be applied to obtain a TAC of ventricularvolume during systole from which we can derive a left-ventricularejection fraction (Fig. 8). Routine applications in common use todaywhich utilize ROIs include studies of renal function, gastric empty-ing, gall-bladder ejection fraction, and cardiac ejection fraction.

Two factors complicate the analysis of ROIs in planar scintigra-phy. Attenuation of overlying soft tissues may vary across a singleimage, among multiple images of the same patient, and certainlyfrom patient to patient. Attempts to compare relative uptake byleft and right kidneys within a single image may therefore be con-founded by differences in attenuation of the overlying soft tissues.To a certain degree, the approximate depth of organs, estimated fromorthogonal views or other anatomic imaging modalities, can be usedto compensate for attenuation based on the assumption that soft tis-sue is equivalent to water as an attenuating medium.

The second factor which confounds quantitative analysis is theactivity residing within tissues above or below a structure of interest.With reference to the example cited above, attempts to compare leftand right renal activity may be confounded by activity in overlyingsoft tissues, such as the liver. To compensate, a background ROI istypically defined adjacent to the area of interest and is then used toestimate and correct for these non-specific counts. A similar methodis used to correct the left ventricular ROI for blood pool activity



originating in the overlying chest wall and lungs in calculation ofleft ventricular ejection fraction (Fig. 8).

4.2.4 Alternate Scintigraphic Camera Designs

By far, the Anger-style γ-camera has predominated in clinicalradionuclide imaging. However, other camera designs have beendeveloped and commercialized, especially for niche applications.One such camera, designed by Bender and Blau in the early 1960s,utilized 294 small NaI(Tl) crystals that were monitored by 35 PMTsassigned to 14 rows and 21 columns.21 In contrast to the Anger scin-tillation camera, in which imaging is predicated upon localizing theposition of scintillation events in a large crystal, this scintillationcamera decodes position based on the photon’s interaction withspecific crystals, each of which represents a finite spatial location.A major advantage of this design, which found application in firstpass cardiac studies, is a higher count-rate capability than that of theAnger gamma camera. Recently, development of solid state detec-tors has resulted in reemergence of multicrystal cameras, especiallyfor portable or dedicated cardiac applications (Fig. 9).

4.3 TOMOGRAPHY

4.3.1 Single-Photon

In current methods of single-photon tomography, the γ-cameradescribes a circular or elliptical orbit around the patient as it acquiresprojection images at multiple angles. Data are then reconstructedusing either filtered backprojection or iterative algorithms to esti-mate the original 3D distribution of activity22 (Fig. 10). Tomographydoes not increase spatial resolution. It is used to increase contrast byeliminating overlying activity and is helpful in improving anatomiclocalization. Tomographic images are also amenable to fusion withCT or MRI data.

In theory, projection images obtained subtending 180◦ aroundthe patient should be sufficient to reconstruct the three-dimensional


84 Lionel S Zuckier

Fig. 9. Pediatric bone scan image using a portable solid state multicrystal camera(Digirad, Poway, CA) to visualize detailed uptake of radiopharmaceutical in bonesof the hand. The detector consists of 4096 3 mm × 3 mm crystals of thallium-dopedcesium iodide [CsI(Tl)].

distribution of activity and this is the standard acquisition methodused in cardiac perfusion tomography. The heart is situated antero-laterally in the left chest and a 180◦ acquisition, centered on theheart, is obtained extending from right anterior oblique to left pos-terior oblique projections. Cameras with multiple detector heads areuseful for tomographic acquisitions because they reduce the acqui-sition time required. An efficient method of using two detectors forcardiac imaging is to arrange the detectors 90◦ to one another, in a socalled “L” configuration. In this way, the assembly need rotate only90◦ to complete the entire 180◦ data acquisition.

Attenuation of photons in tissue and loss of resolution with dis-tance from the collimator act to degrade photons originating on thefar side of the body. As a result, in most noncardiac tomographicapplications, acquisitions are performed using projections over a



Fig. 10. SPECT imaging. In this example, red blood cells from the patient havebeen labeled with 99mTc and reinjected into the patient. An acquisition is performedconsisting of 64 projections taken about the upper abdomen. Eight representativeprojection images are displayed in panel A. The 3D distribution of activity has beeniteratively reconstructed from the projection data; selected axial, saggital and coro-nal images are shown (panel B). Note the relative greater intensity of the peripheryof the liver as compared to its center due to effect of soft-tissue attenuation. Leg-end: A aorta; H heart; I inferior vena cava; K kidney; L liver; S spleen; V vertebralbodies.

full 360◦ rotation. In this case, dual-headed cameras are configuredwith the heads opposing each other at 180◦. The assembly rotates180◦ to complete the entire 360◦ data acquisition. For a three-headedcamera, heads are spaced 120◦ apart and a complete acquisition cantake place following a 120◦ rotation.

Attenuation interferes with quantitative analysis of images inSPECT. This problem is manifested in cardiac perfusion imaging,where variable soft-tissue attenuation leads to apparent regionaldefects in the myocardium which simulate lack of perfusion. In theimaging of the brain, it has been possible to correct for attenuation


86 Lionel S Zuckier

by assuming the cranial contour conforms to an oval and consists ofwater density; however, this method cannot be generalized to morecomplicated and heterogeneous parts of the body. Attenuation cor-rection can also be based on actual attenuation measurements usingradioactive sources which are rotated around the patient, therebyobtaining a crude transmission-based attenuation map. The mea-sured attenuation map is then typically segmented, to minimizestochastic noise, and used to derive an energy-appropriate correc-tion for the emission scan.23 Most recently, SPECT cameras have beenmanufactured with integrated inline CT scanners.24 Using lookuptables, it is then possible to translate the attenuation of the low energyX-ray beam to the energy-appropriate attenuation correction. At thepresent time, a minority of clinical SPECT is performed with atten-uation correction however its use appears to be increasing.

4.3.2 Dual-Photon

Radionuclides for dual-photon imaging emit positrons.25 As dis-cussed earlier, these particles do not exit the body and thereforecannot be directly imaged. Each positron travels only several mil-limeters or less within the patient (depending on its kinetic energyand location), comes to rest, and combines with an electron. Inthis process, the electron and positron annihilate each other andtheir rest masses are converted into energy, resulting in creationof two 511 keV photons which travel in nearly opposite direc-tions (Fig. 11). Imaging systems in positron emission tomography(PET) are designed to identify these paired photons and deter-mine their line of origin, the line-of-response.26 In some early clinicalsystems, modified dual-headed γ-cameras with rectangular detec-tors were used to detect coincident photons in PET imaging. How-ever, because these detectors only subtended a small fraction ofthe angles surrounding the patient, count rate sensitivity was poorand use of this method has declined. Currently, clinical systemsutilize multiple dedicated rings of detectors which surround thepatient in a 360◦ geometry. A typical clinical PET system consistsof 3–4 rings of detectors, each subdivided into 6–8 planes and with1000 elements per transaxial plane. The large number of detector



511KeV

511KeVβ−

β+

*

*

FDG

18F

Detector Ring

Fig. 11. Principle of PET imaging. A patient with lymphoma is noted to haveintense 18F-FDG concentration in an axillary lymph node. 18F-FDG localizes in thetumor, presumably due to the increased rate of glycolysis (glucose metabolism)in malignant tissue. With radioactive decay of the 18F, a positron (β+) is emitted,travels on the order of 1 mm, comes to rest, and combines with a ubiquitous elec-tron (β−) to produce a pair of nearly-opposed 511 keV photons (dotted lines). In theillustration, the pair of annihilation radiation photons nearly simultaneously inter-sect two elements within the ring of detectors (asterisks). The line that is defined bythe two detectors is termed the line-of-response (dashed line). Millions of such coin-cidences are used to reconstruct the original distribution of 18F, thereby identifyingthe location of the tumor.

elements in a PET scanner is expensive and difficult to design.A method which has been used to simplify the scanner is to score alarge block detector crystal in such a way as to have it function as upto 64 individual detector elements, backed by only 4 PMTs. In somedesigns, each PMT is shared among four adjacent detectors, furtherreducing their number and the overall cost. The exact location ofeach photon interaction within the block detector is encoded by theintensity of light recorded at the PMTs, which is unique for eachcrystal element.

Optimal scintillators for PET are different than those for single-photon imaging. The 511 keV photons are difficult to stop, and lack


88 Lionel S Zuckier

Table 4. Scintillators Used in PET After Zanzonico26

Scintillator Mass Density (ρ) Effective Light Output Scintillation(gm/cm3) Atomic (Photons/ Decay Time

Number, Z MeV) (µsec)

NaI(Tl) 3.7 51 41 000 230Bismuth germanate, 7.1 75 9 000 300

BGOLutecium ortho- 7.4 66 30 000 40

oxysilicate, LSOGermanium ortho- 6.7 59 8 000 60

oxysilicate, GSO

of absorptive collimation in PET leads to high count rates and poten-tially high dead-times. Crystals with high densities, high atomicnumbers, and rapid light output are favored (Table 4). High lightoutput is also desirable in that it reduces statistical uncertainty andtherefore improves scatter rejection.

In contrast to single-photon imaging, where collimation isrequired to relate a photon to its line of origin, no absorptive collima-tion is required in PET. This allows for far greater count-rate sensitiv-ity than in single-photon systems. Furthermore, no degradation ofresolution occurs with increasing distance from the detectors. Detec-tor elements are usually operated in coincidence with only a subsetof all the other remaining detector elements, eliminating the consid-eration of adjacent elements where the lines of response would lieoutside of the patient’s body. When two photons are detected by thescanner within a finite time interval τ, typically only 6 ns–12 ns, thedetectors involved define a line-of-response for subsequent recon-struction of the in vivo distribution of radionuclide. The energies ofthe photons are usually windowed to reduce scattering. Millions oflines-of-response are then used to calculate the distribution of radio-pharmaceutical within the patient.

The paired 511 keV annihilation photons may interact withthe PET camera in several different ways (Fig. 13). Dependingon the orientation of the 511 keV photons, some positron annihi-lations are missed completely. In a large number of others, only



one of the pair of annihilation photons interacts with a detector,resulting in an unpaired single. Of the pairs of photons detected bythe camera within the timing window τ and which therefore definea line-of-response, some reflect accurate or true events, correspond-ing to absorption of non-scattered 511 keV photons that originatefrom a single positron-electron annihilation. Other pairs of pho-tons detected occur following the scattering of one or both photonsthrough shallow angles, thereby remaining within an acceptableenergy window but defining an erroneous line-of-response (scatteredcoincidences).Athird category of paired photons is designated as ran-doms, due to the mistaken pairing of two independent 511 keV pho-tons which are incident on the detectors within the specified timingwindow τ despite originating from two separate positron-electronannihilations. The narrower the timing window τ, the smaller thenumber of random coincidences accepted. However, as τ is madetoo short, true coincidences are also lost because of the nontriv-ial time required for photon flight, scintillation within the crystal,and electronic processing. Analogously, a narrower energy win-dow will decrease scattered photons but at a cost of decreasedtrue coincidences due to limitations in energy resolution of thesystem.

In the past, a major limitation of PET systems has been the highcount rate, and demands placed upon the electronics and proces-sors. Additionally, the true count rate increases proportional to theactivity present within the patient while the random coincidence rateincreases as the square of the activity, and becomes critical at highcount rates. In order to decrease the number of randoms and scat-tered coincidences and to reduce the huge computational demandwhich increases as the square of the number of detectors, many sys-tems introduce lead or tungsten septa between the rings of detectors,which prevents oblique lines-of-response. These systems are termed2D in that only events within rings or between adjacent rings areacquired (Fig. 12). The transaxial images so derived are stacked toconstitute a three-dimensional volume of distribution from whichcoronal, saggital and maximum-intensity projection (MIP) imagesare derived.


90 Lionel S Zuckier

2-D Acquisition

Septa Extended

-3 -1-2 n +1 +2 +3

-3 -1-2 n +1 +2 +3 -3 -1-2 n +1 +2 +3

-3 -1-2 n +1 +2 +3

Septa Retracted

3-D Acquisition

Fig. 12. 2D versus 3D PET. In 2D acquisition, lead or tungsten septa are extendedbetween adjacent rings of scintillation detectors. Only lines-of-response within asingle ring or possibly adjacent rings are permitted (solid line) while paired pho-tons with greater obliquity are absorbed by the septa (broken lines). In 3D acquisi-tion, the septa are retracted and each detector element may be in coincidence withopposite detector elements in any of the rings. This results in increased sensitivitybut markedly increased random coincidences and potentially dead time. Sequen-tial scans in a patient with previously treated tumor in the frontal lobe demonstratethe improved quality of 3D scan relative to 2D scans, which is achievable whenimaging small body parts such as the head where scatter is minimal.

By removing the lead or tungsten septa between adjacent ringsof detectors, it is possible to perform an acquisition where lines-of-response between all the detector rings are potentially available todetect coincident pairs of photons, an approach termed 3D (Fig. 12).3D-acquisition has been facilitated by faster computer processorsand improvements in detectors which allow better temporal andenergy resolution. Because of the deleterious effect of high countrate, the administered activities are decreased in 3D acquisitions.Imaging times may also be significantly reduced due to the greatercount-rate sensitivity.



511KeV

511KeV

1b

1a

4a

3a

5b

5a

*

A

B

C

2a

Fig. 13. Variety of photon interactions in PET imaging. Positrons emitted from thetumor in the patient’s left axilla result in emission of pairs of annihilation photons.Representative interactions are illustrated. (1) A pair of annihilation photons isdetected by the ring of detectors, resulting in a “true” coincidence which is recordedas the line-of-response “A” (dashed line). (2) Only one of the pair of photons isrecorded by the ring of detectors, a “single.” The second photon escapes by passingthrough the ring of detectors, or by passing out of plane or into a lead septum. (3a)and (4a) Two unrelated single photons are recorded within the finite coincidencetiming window τ, resulting in false line-of-response “B” (random). (5) One of the511 keV photons is directly detected by a detector (5a) while the second undergoesCompton scattering within the patient (5b). Depending on the scattering angle, ifthis latter photon remains within the acceptable energy window, there will result amalpositioned coincidence and erroneous line-of-response “C.”

A number of sources of error impact upon the accuracy of dual-photon imaging. Depending on its kinetic energy, the positron maytravel up to several millimeters prior to coming to rest, whichdisplaces the line-of-response from the actual site of the annihila-tion event. Secondly, because positrons may actually have nonzeromomenta immediately prior to annihilation, the emitted 511 keVphotons may not be exactly collinear (at 180◦ to each other). This,


92 Lionel S Zuckier

too, leads to errors in the designated line-of-response. Each detectorelement also has a finite size, which introduces further uncertaintyin the line-of-response. As mentioned above, random and scatteredcoincidences can lead to erroneous lines of response as well. ModernPET scanners in clinical use have spatial resolution on the order of3 mm–4 mm.

In PET imaging, attenuation correction is routinely performedfor clinical interpretation and is required for quantitative analy-sis. In uniform and geometrically simple regions of the body suchas the skull, attenuation correction may be based on the assump-tion that the body contour conforms to a water-equivalent regu-lar geometric shape, as discussed with regard to SPECT scanners.Measurement-based attenuation correction in PET utilizes radioac-tive sources such as 68Ge (energy 511 keV) and 137Cs (energy 662 keV)which are rotated around the patient and used to yield a crude atten-uation map for correcting the emission data. Often, the 68Ge and137Cs attenuation maps are segmented into lung, bone, and soft tis-sue densities to overcome the degradative effects of the low-count(“noisy”) transmission data. These are then used to derive a 511 keVappropriate attenuation map with which to correct the emissionscans. Most recently, PET scanners have been manufactured withintegrated inline CT scanners.27 Using segmentation and lookuptables, it is also possible to translate the energy-specific attenuationof the X-ray beam to that of the appropriate 511 keV photon energy(Fig. 14).

A widely used measure of radiopharmaceutical uptake in clin-ical PET imaging is the standardized uptake ratio (SUV), which isa dimensionless ratio of radiotracer concentration in the structureof interest to the average concentration in the body (administeredactivity divided by patient mass). Quantitation is most useful whenvalues are compared in a single subject before and after therapy.Accurate measurement of SUV is dependent on accurate attenua-tion correction, and its clinical utility also requires standardizationof the interval between FDG administration and imaging, fasting ofthe patient before FDG administration and other biologically rele-vant variables.28 Most studies have shown that a semiquantitative



NAC CT

AC

Fused

Fig. 14. Non-attenuation corrected (NAC) and attenuation corrected (AC) scansin same patient as Fig. 12. Original data from a PET-CT scan consists of NAC and CTimages. The CT scan is used to create an energy-appropriate attenuation correctionmap which is applied to createAC images. CT scan is often fused with theAC imagesin order to improve anatomic localization of abnormalities. Note the distribution ofactivity in the NAC image where central structures are relatively lower in activitythan peripheral structures; this has been corrected in the AC PET images.

visual grading system is equally effective as SUV-based diagnosticcriteria in differentiating malignant from benign tissue.

4.3.3 Fusion Imaging in Nuclear Medicine

A development in nuclear medicine which continues to evolve isthe combination or fusion of scintigraphic images with anatomicradiologic modalities, chiefly CT. To a certain degree, this hasbeen facilitated by development of standards for image storageand retrieval (Digital Imaging and Communications in Medicine[DICOM]) and the interface of multiple modalities to common data


94 Lionel S Zuckier

and storage systems (Picture Archive and Communication Sys-tems or PACS).29,30 Nuclear medicine images contain unique func-tional information but are often limited in anatomic content. CT andanatomic MR imaging have superior spatial resolution, but gener-ally lack functional information. By fusing nuclear medicine andanatomic images, it is possible to correlate changes in function withparticular anatomic structures. Initially, this was attempted by usingsoftware to retrospectively register31 and then fuse (overlay) studiesperformed on separate scanners at different times (Fig. 15). Methods

CT

RBC

Fusion

Axial Saggital Coronal

Fig. 15. Retrospective SPECT Fusion. Contrast enhanced CT scan, performed sev-eral days prior to the blood pool study illustrated in image 10, has been regis-tered and fused to the nuclear medicine study using a mutual information method(MIMvista version 4.0, MIMvista Corp, Cleveland, OH). Note excellent registrationof the inferior vena cava (arrows) and spleen (arrowhead) on blood pool and CTimages. The study was done to evaluate the blood volume corresponding to anarea of contrast enhancement in the periphery of the left lobe of the liver on CT(dashed circle). No corresponding increase in blood volume is noted and the lesiontherefore does not represent a hemangioma.



of retrospective registration were frequently hampered by inevitabledifferences in patient positioning, bowel preparation and other vari-able factors. To minimize this problem, a hardware approach wasdeveloped where SPECT and PET scanners were combined inlinewith CT scanners, thereby permitting both studies to be sequentiallyacquired on the same gantry with little or no time and patient motionbetween them (Fig. 16). An additional benefit of combined deviceshas been the ability to correct for attenuation as described above.Currently, the vast majority of PET scanners sold for clinical useincorporate CT; while less common on SPECT scanners, this featureis increasing in frequency. Indeed, the success of CT fusion in clinical

Stress →

Rest →

Stress →

Rest →

Ver

tica

l Lo

ng

Axi

s

Sh

ort

Axi

s

A

Stress

Rest

LAD

RCA

B

Fig. 16. Hardware fusion of SPECT and multislice CT coronary angiography(CTA) in a 50 year old man with chest pain performed on an inline dedicatedscanner (Research SPECT/16-CT Infinia LS, General Electric Healthcare Technolo-gies). (A) Selected tomographic perfusion images of the heart are displayed in shortaxis, and vertical long axis. An area of diminished perfusion at stress (odd rows,arrows) exhibits improved perfusion at rest (even rows, arrowheads). (B) FusedSPECT/CTAdata combining an epicardial display of myocardial perfusion at stress(upper image) and rest (lower image) with the coronary tree derived from the CTAstudy illustrates the relationship of the ischemic territory and the right coronaryartery (RCA). The course of the left anterior descending (LAD) artery is also demon-strated. CTA images illustrate luminal narrowing in the RCA (not shown). (Imagescourtesy of Dr R Bar-Shalom, Rambam Health Care Campus, Haifa, Israel.)


96 Lionel S Zuckier

single- and dual-photon tomography has led to development of par-allel techniques for small animal imaging and research.32


In nuclear medicine, molecules containing radioactive atoms,termed radiopharmaceuticals, are used to diagnose and treat dis-ease; the interaction of the radiopharmaceuticals with physiologicprocesses within the body reveals unique functional information.Methods of diagnosis in nuclear medicine include imaging of sin-gle photons originating from the radiopharmaceuticals, and in thecase of PET, imaging of dual photons that derive from annihilation ofpositrons. Current developments in nuclear medicine include fusionof SPECT and PET with anatomic modalities such as CT.

References

1. Groch MW, Radioactive decay, Radiographics 18(5): 1245–1246, 1998.2. Hall EJ, Giaccia AJ, Radiobiology for the Radiologist, 6th edn., Lippincott

Williams & Wilkins, Philadelphia, PA, 2006.3. Brodsky A, Kathren RL, Historical development of radiation safety

practices in radiology, Radiographics 9(6): 1267–1275, 1989.4. Schuck HSR, Osterling A et al., Nobel. The Man and his Prizes, 2nd edn.,

Elsevier Publishing Company, Amsterdam, 1962.5. Graham LS et al., Nuclear medicine from Becquerel to the present,

Radiographics 9(6): 1189–1202, 1989.6. Kuhl DE, Edwards RQ, Reorganizing data from transverse section

scans of the brain using digital processing, Radiology 91(5): 975–983,1968.

7. Kassis AI, Adelstein SJ, Radiobiologic principles in radionuclide ther-apy, Journal of Nuclear Medicine 46 (Suppl 1): 4S–12S, 2005.

8. Budinger TF, Rollo FD, Physics and instrumentation, Prog CardiovascDis 20(1): 19–53, 1977.

9. Ranger NT, Radiation detectors in nuclear medicine, Radiographics19(2): 481–502, 1999.

10. Blahd WH, Ben Cassen and the development of the rectilinear scanner,Semin Nucl Med 26(3): 165–170, 1996.

11. Gottschalk A, Anger HO, Use of the Scintillation camera to reduceradioisotope scanning time, Jama 192: 448–452, 1965.



12. Anger HO, Scintillation Camera, The Review of Scientific Instruments29(1): 27–33, 1958.

13. Collica CJ, Robinson T, Hayt DB, Comparative study of the gammacamera and rectilinear scanner, Am J Roentgenol Radium Ther Nucl Med100(4): 761–779, 1967.

14. Formiconi AR, Collimators, Q J Nucl Med 46(1): 8–15, 2002.15. Swann S et al., Optimized collimators for scintillation cameras, J Nucl

Med 17(1): 50–53, 1976.16. Anger HO, Scintillation camera with multichannel collimators, J Nucl

Med 5: 515–531, 1964.17. Murphy PH, Burdine JA, Large-field-of-view (LFOV) scintillation cam-

eras, Semin Nucl Med 7(4): 305–313, 1977.18. Todd-Pokropek A, Advances in computers and image processing with

applications in nuclear medicine, Q J Nucl Med 46(1): 62–69, 2002.19. Muehllehner G, Colsher JG, Stoub EW, Correction for field nonunifor-

mity in scintillation cameras through removal of spastial distortion,J Nucl Med 21(8): 771–776, 1980.

20. Genna S, Pang SC, Smith A, Digital scintigraphy: Principles, design,and performance, J Nucl Med 22(4): 365–371, 1981.

21. Bender MA, Blau M, The autofluoroscope, Nucleonics 21(10): 52–56,1963.

22. Larsson SA, Gamma camera emission tomography. Development andproperties of a multisectional emission computed tomography system,Acta Radiol Suppl 363: 1–75, 1980.

23. Xu EZ et al., A segmented attenuation correction for PET, J Nucl Med32(1): 161–165, 1991.

24. Bocher M et al., Gamma camera-mounted anatomical X-ray tomogra-phy: Technology, system characteristics and first images, Eur J NuclMed 27(6): 619–627, 2000.

25. Votaw JR, The AAPM/RSNA physics tutorial for residents. Physics ofPET, Radiographics 15(5): 1179–1190, 1995.

26. Zanzonico P, Positron emission tomography: A review of basic prin-ciples, scanner design and performance, and current systems, SeminNucl Med 34(2): 87–111, 2004.

27. Beyer T et al., A combined PET/CT scanner for clinical oncology, J NuclMed 41(8): 1369–1379, 2000.

28. Keyes JW, Jr, SUV: Standard uptake or silly useless value? J Nucl Med36(10): 1836–1839, 1995.

29. Alyafei S et al., Image fusion system using PACS for MRI, CT, and PETimages, Clin Positron Imaging 2(3): 137–143, 1999.


98 Lionel S Zuckier

30. Graham RN, Perriss RW, ScarsbrookAF, DICOM demystified:Areviewof digital file formats and their use in radiological practice, Clin Radiol60(11): 1133–1340, 2005.

31. Zitova B, Flusser J, Image registration methods: A survey, Image andVision Computing 21(11): 977–1000, 2003.

32. Lewis JS et al., Small animal imaging. Current technology and perspec-tives for oncological imaging, Eur J Cancer 38(16): 2173–2188, 2002.

33. Technology NI.o.S.a. [cited February 11, 2007]; Available from:http://physics.nist.gov/cuu/index.html.

34. Cherry SR, Sorenson JA, Phelps ME, Physics in Nuclear Medicine, 3rdedn., Saunders, Philalephia, PA, 2003.


CHAPTER 5

Principles of Magnetic ResonanceImaging

Itamar Ronen and Dae-Shik Kim

The phenomenon of nuclear magnetic resonance was first described byFelix Bloch1 and independently by EM Purcell in 1946.2 Both scien-tists shared the Nobel Prize in 1952 for this pivotal discovery. The phe-nomenon is tightly linked to the broader field of interaction betweenmatter and radiation commonly known as spectroscopy. It is within theframe of nuclear magnetic resonance (NMR) spectroscopy that the fielddeveloped in leaps and bounces, leading to the discovery of FourierTransform NMR by RR Ernst (Nobel Prize in Chemistry, 1991) and toan astonishingly wide range of applications, from the structural deter-mination of protein structure in solution to the investigation of metabolicprocesses in live organisms, from solid state research to myriad appli-cations in organic chemistry. The unexpected paradigm shift in NMRresearch came in the early 1970s, when independent research by twoingenious scientists, Paul Lauterbur, then at SUNY Stony Brook and PeterMansfield at Nottingham University, UK, has raised the possibility ofobtaining images based on the signal generated by nuclear mag-netic resonance.3−5 The humble beginnings, namely the projection-reconstruction maps of two water-filled test tubes shown by Lauterburin his first publication on the matter in the journal Nature, were soonfollowed by the first applications of this new technique to obtainimages of the human body, and this spawned a new field — thefield of magnetic resonance imaging (MRI). Both scientists shared theNobel Prize in Physiology or Medicine in 2003. MRI effectively revolu-tionized the biomedical sciences, allowing the noninvasive imaging ofpractically every organ of the human body in health and disease.MRI methodology began covering a broad range of diagnostic tools,and invaded an astonishing variety of basic research fields, includingthe ability to visualize brain function, characterize tissue microscopic

99


100 Itamar Ronen and Dae-Shik Kim

structure, reconstruct neural connections, and more — all in a nonin-vasive and harmless manner. In this chapter, we will explore the basicprinciples of nuclear magnetic resonance — the way in which the NMRsignal is generated and detected, the properties of the NMR signaland the way in which this signal is manipulated to provide us withimages.

5.1 PHYSICAL AND CHEMICAL FOUNDATIONS OF MRI

5.1.1 Angular Momentum of Atomic Nuclei

Atomic nuclei posses intrinsic angular momentum, associated withprecession of the nucleus about its own axis (spin). In classicalphysics, angular momentum is a vector associated with a body rotat-ing or orbiting around an axis of rotation, and is given by �L = �r × �Pwhere �L is the angular momentum vector, �r is the radius vector fromthe center of rotation and �P is the linear momentum vector. The vec-tor multiplication operator, ×, generates the angular momentum tobe perpendicular to the plane defined by �P and �r, as can be seen inFig. 1.

In quantum mechanics, the angular momentum for particlessuch as electrons, protons or atomic nuclei is given by L =√

I(I + 1)h, where L stands for the total angular momentum of the

Fig. 1. The relationship between the angular momentum L, the linear momentumP and the radius vector r.


Principles of Magnetic Resonance Imaging 101

particle, I is the spin number, or simply the spin of the particle, andh is Planck’s constant in the appropriate units. The spin quantumnumber I is characteristic of every nuclear species. I can be zero orcan take positive integer or half integer values. The differences in thevalue of the spin quantum number reflect differences in the nuclearcomposition and charge distribution. For instance, the nucleus of 1H(I = 1/2) consists of one proton only, while the nucleus of 2H (I = 1)consists of a proton and a neutron. For 12C and 16O I = 0, and thesenuclei have zero angular momentum. Table 1 lists some of the stableisotopes of common elements in the periodic table together withtheir spin numbers.

5.1.2 Energy States of a Nucleus with a Spin I

Anucleus with a spin = I possesses 2I + 1 possible states, defined bya quantum number mI . m can obtain the values −I, −I + 1,…,I −1, I.These states are associated with the different projections of L on the(arbitrarily chosen) z-axis. The projection is then given by Lz(m) =mh. The relationship between L and the different Lz for the case ofI = 3/2 is given in Fig. 2. It should be noted that the projection ofthe angular momentum is well defined on one axis only, while as aresult of Heisenberg’s uncertainty principle, Lx and Ly are not welldefined, causing L to lie on an uncertainty cone. In the absence of an

Table 1.

Isotope Natural Spin (I) Magnetic Gyromagnetic RatioAbundance (%) Moment (µ)† (γ)∗

1H 99.9844 1/2 2.7927 26.7532H 0.0156 1 0.8574 4,10711B 81.17 3/2 2.6880 —13C 1.108 1/2 0.7022 6,72817O 0.037 5/2 −1.8930 −3,62819F 100.0 1/2 2.6273 25,17929Si 4.700 1/2 −0.5555 −5,31931P 100.0 1/2 1.1305 10,840

∗γ in units of 107 rad T−1 sec−1

†µ in units of nuclear magnetons = 5.05078 · 10−27 JT−1



Fig. 2. Quantization of the angular momentum for I = 3/2. The four possible mstates represent four projections on the z-axis.

external magnetic field, states with different m have the same energy —they are degenerate states.

5.1.3 Nuclear Magnetic Moment

The overall spin of the (charged) nucleus generates a magnetic dipolemoment, or a magnetic moment, along the spin axis. The nuclear mag-netic moment, µ is again an intrinsic property of the specific nucleus.The magnetic moment µ results from a motion of a charged parti-cle, similar to a generation of magnetic moment as a result of a loopcurrent. The magnetic moment µ is also a vector, and it is propor-tional to the angular momentum L through the gyromagnetic ratio γ :µ = γL µz = γLz = γmh. The gyromagnetic ratio is one of the mostuseful constants in MR physics, and we will encounter it on several



occasions in our discussion. Table 1 lists the gyromagnetic ratio forseveral nuclei of stable isotopes, as well as for the electron.

5.1.4 The Interaction with an External Magnetic Field

The nuclear magnetic moment can interact with an external magneticfield B0. The energy of this interaction is given by: E = −µ · B0 =−(µxB0,x + µyB0,y + µzB0,z), the scalar product between the magneticfield and the magnetic dipole. If the field is oriented solely along thez-axis, the energy of this interaction is thus given by: E = −µ · B0 =−µzB0 = −γImIhB0 mI = −I, −I + 1, . . . , I − 1, I where γI is thegyromagnetic ratio of the nucleus I. As can be appreciated, the effectof the external magnetic field is the removal of the degeneracy of theenergy levels. Each m state is now characterized by a distinct energy. Inthe case of spin 1/2, m is equal to either −1/2 or +1/2, and the energylevels associated with these two states are: Em=−1/2 = E0+ 1

2γhB0 andEm=+1/2 = E0− 1

2γhB0. Figure 3 describes the energy level diagram ofa spin 1/2 particle in the presence of an external magnetic field. The

Fig. 3. The effect of B0 on the two degenerate m states for I = 1/2 particle.



energy gap between the two levels is given by �E = E−1/2 − E1/2 =γhB0. This energy expressed in frequency, using the Planck relation�E = hυ where υ is the Larmor frequency:

�ω0 = 2πν0 = γB0.

The removal of energy level degeneracy can be viewed as a resultof break of spherical symmetry introduced by the external magneticfield B0. As a result of the interaction with B0, L is now confined to2I + 1 orientations with respect to B0, dictated by the possible 2I + 1Lz values, which now reflect the projections of L on the axis definedby the direction of B0. In the case of I = 1/2, the state m = 1/2 has Lz

lie on the positive z, and m = −1/2 on the negative z-axis, similarlyto what is seen on Fig. 2 for the case I = 3/2.

5.1.5 The Classical Picture

For spin = 1/2, the same result can be obtained solely from classi-cal considerations. One can envision the nucleus as a small mag-netic dipole with a magnetic moment µ. When placed in an externalmagnetic field B0, the nuclei will experience a torque in a plane per-pendicular to B0 and precess along the magnetic field lines with aprecession frequency ω0, proportional to the external magnetic fieldthrough the gyromagnetic ratio: ω0 = γB0. The dipole precessesalong a cone that forms an angle θ with the z-axis. This is an equiva-lent to the uncertainty cone described for Lx and Ly of the quantumparticle. It should be noted that the classical picture converges withthe quantum-mechanical picture only for I = 1/2.

5.1.6 Distribution Among m States

The removal of energy degeneracy thus creates a uniformly spacedenergy ladder with 2I+1 distinct states, separated by �ω = γB0. Theenergy of the states increases as m decreases. In the case of I = 1/2,the state m = 1/2 has the lower energy and is typically designatedas the α state, and the state with m = −1/2 is higher in energy and isdesignated as the β state. Most importantly for MRI, the energy gapis linearly proportional to the external magnetic field, and so is the



Larmor frequency. In room temperature, one can use the Maxwell-Boltzmann distribution to estimate the distribution of the particles,in our case — nuclei, among the various energy levels. For I = 1/2,the MB distribution among the states α and β is given by:

nα,β

N=

exp(−E0 ± 1

2 hν0

kT

)

exp(−E0 − 1

2 hν0

kT

)+ exp

(−E0+ 1

2 hν0

kT

) ,

where k is the Boltzmann constant, υ0 is the Larmor frequency andT is the temperature. For T = 300 k, and a magnetic field of 3 Tesla,the Larmor frequency is roughly 127.7 MHz, and the α state is morepopulated than the β state such that:

nβ

nα

=exp

(−E0+ 1

2 hv0

kT

)

exp(−E0− 1

2 hv0

kT

) = exp(−hv0

kT

)= 0.99998.

5.1.7 Macroscopic (Bulk) Magnetization

In a macroscopic sample, the total magnetic moment of the sample iscalled the macroscopic or bulk magnetization. The bulk magnetiza-tion, or simply the magnetization at thermal equilibrium is denotedby M(eq) or simply by M, and it is the sum of the individual momentsin the sample. The sum of all moments with state α or β are givenby Mα,β, where Mα,β(x) = Mα,β(y) = 0 and Mα,β(z) > 0. The x andy projections of Mα,β vanish because of the uncertainty cone (in thequantum mechanical picture) or the lack of phase coherence amongthe individual precessions (in the classical picture). As a result of thethermal distribution among states, Mα > Mβ, and the total magne-tization M is given by M = Mα − Mβ. It can be easily shown that athigh temperatures, M = 1

4N(γh)2B0/kT, known also as Curie’s law.

5.1.8 The Interaction with Radiofrequency Radiation —the Resonance Phenomenon

At this point, an important point has been reached — the creation ofan energy gap between two unevenly populated states. A radiofre-quency (RF) radiation at a frequency equal to the frequency gap



between the states will result in transitions of particles from α to theβ states and in absorption of energy quanta. This is the nuclear mag-netic resonance phenomenon, and thus the resonance condition isωRF = ω0. The simplest experimental setting that can be envisionedis that of a magnet that generates a static homogeneous magneticfield B0 and a radiofrequency source that generates RF radiationωRF. If the sample inside the homogeneous B0 contains nuclei withI > 0 (e.g. a water sample where the hydrogen atoms have nucleiwith I = 1/2), by slowly varying either the external magnetic fieldB0 or ωRF, the resonance condition will be met at some point, result-ing in absorption of RF. This absorption can be detected by a RFdetector. One of the first NMR spectra ever obtained was of ethanol(CH3CH2OH). The three resonances that were visible on the spec-trum were those of the three hydrogen “types” (the CH3 group, theCH2 group and the OH group) and the slight variations in reso-nance frequencies among the three stem from slight differences inthe electron shielding around the different 1H nuclei.

5.2 THE BLOCH EQUATIONS

A phenomenological description of the equations of motion for M,the bulk magnetization, was given by Felix Bloch, and it is knownas the Bloch equations. The Bloch equations are extremely use-ful for the understanding of the various effects that experimentalmanipulation of M have, and in particular — the effects of radiofre-quency radiation. The Bloch equations for the three components ofthe magnetization are:

dMx

dt= −γ(B0,yMz − B0,zMy) − Mx

T2,

dMy

dt= −γ(B0,zMx − B0,xMz) − My

T2,

dMz

dt= −γ(B0,xMy − B0,yMx) − Meq,z − Mz

T1,

where the first term in each equation represents the torque exertedon each magnetization component by the components of B0 perpen-dicular to it. The second term in each equation is a relaxation term



that allows for the magnetization to regain its equilibrium value.Since we assume B0 along the z-axis, the relaxation along the z-axisis called the longitudinal relaxation, whether the relaxation on thexy plane is called the transverse relaxation. The sources of theserelaxation processes are different, and will be discussed later. If themagnetic field B0 is aligned along the z-axis, and taking into accountthe relation ω0 = γB0, the Bloch equations in the presence of a staticmagnetic field take the form:

dMx

dt= ω0My − Mx

T2,

dMy

dt= −ω0Mx − My

T2,

dMz

dt= −Mz,eq − Mz

T1.

The solution of the Bloch equations for the transverse (xy) compo-nents of the magnetization is a clockwise precession accompaniedby decay at a rate 1/T2 until Mxy → 0. The longitudinal componentof M decays according to 1/T1, approaching Mz,eq:

Mx(t) = Mxy(0) cos (ωt) · exp(

− tT2

),

My(t) = Mxy(0) sin (ωt) · exp(

− tT2

),

Mz(t) = Mz(0) +(Mz,eq − Mz(0) · exp

(− t

T1

)).

5.2.1 The Inclusion of the RF Field in the Bloch Equations

As mentioned earlier, the actual MR experiment involves the pertur-bation of Meq with a radiofrequency irradiation. The RF irradiationgenerates an EM field oscillating at a frequency which we denoteby ω1. The magnetic part of the electromagnetic field can thus begiven by B1 = γω1. In order to drive M out of equilibrium, B1 mustoperate perpendicular to the z-axis. For simplicity reasons, in ourdiscussion, we will assume B1,x > 0, B1,y,z = 0, or in other words,B1 exerts torque on the yz plane, tilting the equilibrium magneti-zation away from the z-axis toward the y-axis. Typically we apply



a linearly polarized oscillating field. The linear polarization can bedecomposed into two circularly polarized counter-rotating fields witha frequency difference of 2ω:

BRF = [B1,x cos (ωt) + B1,y sin (ωt)] + [B1,x cos (ωt) − B1,y sin (ωt)].The first component is a counter-clockwise rotating component. Weare interested in irradiation frequencies close to resonance, and atresonance ω = ω0. Since M is precessing clockwise under B0, it is 2ω0

away from resonance, and its influence on M can be neglected. ThusBRF can be viewed as a circularly polarized field, where the polar-ization rotates at a frequency ω : BRF = B1,x cos (ωt) − B1,y sin (ω, t).

When a BRF field is applied, the total magnetic field B is thus:

�B = B1 cos (ωt)

−B1 sin (ωt)B0

The Bloch equations thus assume the following form:

dMx

dt= −ω1 sin (ωt)Mz + ω0My − Mx

T2,

dMy

dt= ω1 cos (ωt)Mz + ω0Mx − My

T2,

dMz

dt= −ω1 cos (ωt)My + ω1 sin (ωt)Mx − Mx,eq − Mz

T1,

where ω1 = γB1.

5.2.2 The Rotating Frame of Reference

This is a rather complicated picture, since the effective field B is acombination of a static magnetic field B0 and a rotating field B1. Inorder to simplify the picture, we move from the laboratory frame ofreference to a frame of reference that moves along with the rotatingfield B1, i.e. rotates clockwise at a frequency ω. Since B1 is perpendic-ular to B0, and the frame of reference rotates at exactly the frequencyof rotation of the RF field, both fields now appear to be static in thisreference. Expressing the components of the transverse magnetiza-tion in the rotating frame Mx′,y′ in terms of the components in the



laboratory frame yields:

Mx′ = Mx cos (ωt) − My sin (ωt); My′ = Mx sin (ωt) + My cos (ωt).

Rewriting the Bloch equations for Mx′ and My′ gives:

dMx′

dt= (ω0 − ω)My′ − Mx

T2,

dMy′

dt= ω1Mz − (ω0 − ω)Mx′ − My

T2,

dMz

dt= −ω1My′ − Mz,eq − Mz

T1.

In this frame, the effective magnetic field is now �Beff =(

B10

B0−ω/γ

).

This is a static magnetic field that is a sum of the RF field B1, whichoperates along the x′-axis, and a reduced static magnetic field B0 −ω/γ that operates along the z-axis. The magnetic field along the z-axis seems reduced since now rotating frame follows at a frequencyω the magnetization, which precesses at a frequency ω0. The relativefrequency among them is thus ω0 − ω, and thus the static magneticfield seems “reduced.” The precession of M about the axis definedby the effective magnetic field is depicted in Fig. 4(a).

Fig. 4. The effective field in the rotating frame of reference (A) off resonance and(B) on resonance.



At resonance, ω = ω0 and the z component of Beff vanishes.This creates a particularly simple picture, where motion of the mag-netization is solely dictated by B1. This is an extremely importantachievement in our discussion, because it makes the description ofthe effects of pulse sequences on the magnetization extremely intu-itive. In Fig. 4(b), the resonance condition in the rotating frame isdescribed. As can be seen, with the application of B1,x, M will pre-cess about the axis defined by B1,x, i.e. on the zx plane, moving fromthe positive x-axis towards the positive y-axis and so on.

5.2.3 RF Pulses

RF can be applied in a constant manner (CW) or a transient one (pulses).An RF pulse will cause M to precess around B1, but only for the periodof its duration. After the pulse had ended, M will obey the Blochequations where the field consists only of B0. When an RF pulse isapplied to M, the angle between M and the positive z-axis achievedat the end of the pulse is defined as the flip, or tilt angle. A simpleformula for the tilt angle is: θ = γB1τ, where θ is the tilt angle, B1 isthe amplitude of the RF field, and τ is the pulse duration. Figure 5describes a 90◦ tilt angle for B1 applied on the x-axis, or 90◦

x, and a180◦ pulse when B1 is applied on the y-axis, or 180◦

y.

Fig. 5. The effect of (A) a 90 degree RF pulse when B1 is along the x-axis; (B) a 180degree RF pulse when B1 is along the y-axis.



5.3 THE FREE INDUCTION DECAY

The simplest pulse-NMR experiment involves an excitation of themagnetization by a RF pulse, and the detection of the precessionof the magnetization about the B0 axis in the absence of a RF field.The signal picked up by the RF receiver coil (which may, or may notbe the one used for the RF transmission) is the one induced by theoscillations of the x and y components of M. This signal is calledthe free induction decay, or the FID. The FID should be identicalto the solution of the Bloch equations given previously. However,since the signal undergoes demodulation, or in other words — theRF frequency is subtracted from the actual frequency of the detectedsignal, the FID is analogous to the solution of the Bloch equations inthe rotating frame. The solution is given by:

Sx′(t) = S(0) cos[(ω0 − ωref)t] · exp(

− tT2

),

Sy′(t) = −S(0) sin[(ω0 − ωref)t] · exp(

− tT2

),

where S(0) is proportional to the projection of M on the XY planeimmediately after the RF pulse is given and is thus proportionalto Mzsinθ, where θ is the flip angle. The reference or demodulationfrequency ωref is essentially the rotation frequency of the rotatingframe. The projection is of course maximized when θ = 90◦. TheFID is in fact a complex signal, and with a quadrature detection coilboth the real and imaginary parts of the signal, separated by a phaseof π/2, are detected. The FID for a simple signal that consists of oneresonance at ω0 is thus a damped oscillation at a frequency ω0 −ωref

and a decay time constant of T2, as can be seen in Fig. 6.

5.3.1 The NMR spectrum

The typical NMR experiment, and as we will see later — the MRIimage, carries the FID data onto the frequency domain via theFourier transformation. The Fourier transformation of the FID yieldsthe real and imaginary parts of the NMR spectrum, also known asthe absorption and dispersion modes. It should be noted that the phase



-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

0 0.1 0.2 0.3 0.4 0.5

t

Fig. 6. The real part of the FID (off resonance).

associated with the detection of the FID is arbitrary, and thus can bemodified in the processing phase to yield a “pure” absorption (in-phase) spectrum, or any desired phase. The explicit expressions forthe real and imaginary parts of the NMR spectrum are given by:

Sy′(ω) =∫ ∞

0Sy′(t) exp (iωt)dt = S(0)T2

1 + T22(�ω − ω)2

,

Sx′(ω) =∫ ∞

0Sx′(t) exp (iωt)dt = S(0)T2

2(�ω − ω)

1 + T22(�ω − ω)2

,

where �ω = ω0 −ωref. The real and imaginary parts of the spectrumare seen on Fig. 7. The real part is a spectral line with a Lorenzianline shape. The full width at half maximum is inversely proportionalto the characteristic decay time of the FID. Here, it is given by therelaxation constant T2, and the relation between the full width athalf maximum (FWHM) and T2 is �υ1/2 = 1/πT2. Later on we willsee that relaxation is enhanced by experimental factors that are notnecessarily intrinsic to the sample, and thus a new term will be addedto the apparent transverse relaxation — T ∗

2 .



0

0.02

0.04

0.06

0.08

0.1

0.12

-500 -300 -100 100 300 500

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

-500 -300 -100 100 300 500

Fig. 7. The real part or absorption mode (left) and the imaginary part or dispersionmode (right) of the NMR spectrum of a single resonance.

5.3.2 Relaxation in NMR

The NMR signal is governed by two distinct relaxation times. T1 =1/R1 is the longitudinal relaxation time, which is dictated by the energyexchange between the system and the “lattice,” which is contains allother degrees of freedom to which the spin system is coupled. T1

describes the rate of the return of Mz to its equilibrium value, Mz,eq.T2 = 1/R2 is the transverse relaxation time. T2 is associated with loss of



coherent spin motion on the xy plane, which results in a net decreaseof Mxy and ultimately in its vanishing. Both relaxation times are inti-mately related to molecular motion and the interactions between thespins with neighboring spins and their surroundings. Relaxationis induced by randomly fluctuating magnetic fields, typically associ-ated with the modulation of nuclear interactions by the random, orstochastic molecular motion.

5.3.2.1 T1 Relaxation

The magnetization at equilibrium, Meq is governed by the distribu-tion of the spins among the two magnetic states, α and β: Meq = Mα−Mβ. At equilibrium — this distribution is given by the Boltzmanndistribution. When the magnetization is out of equilibrium — whatwill drive it back to equilibrium are fluctuations in the magneticfield, whose frequency is somewhat represented by ω0 and thus allowfor energy exchange. In the case of T1, which operates on Mz, the fluc-tuations have to induce changes in Mz, thus will be generated byfluctuations in Bx,y. Fluctuations in the magnetic field will be gener-ated by interactions that are modulated by, e.g., molecular motion.Many of those mechanisms include interactions between two neigh-boring spins. If the interaction between two spins is dependent ontheir orientation in the magnetic field, this interaction will be modu-lated, for example, by rotation of the molecule in which these spinsare incorporated.

5.3.2.2 Example — the dipole-dipole interaction

Two neighboring magnetic dipoles interact with each other (thinktwo magnets). The interaction strength when the two are in an exter-nal magnetic field B0 depends, among other things, on the angle θ

between the axis that connects the two dipoles and the magneticfield. Specifically, this interaction is proportional to 1

r3ij(3cos2θij − 1),

where rij is the internuclear distance between nuclei i and j, andθij is the angle described above. In liquids, random motion will



modulate both r and θ, and if the two nuclei belong to the samemolecule (e.g. the two hydrogen atoms in a water molecule), θ isprimarily modulated by molecular rotational motion. Rotation is arandom motion, but a typical rotation time will be closely relatedto, e.g. the size of the molecule at hand: the larger the molecule,the slower its characteristic rotation time. The characteristic time forsuch random motion is given by a correlation time, τc. If the char-acteristic motional/rotational time constant τc is characterized bya frequency, 2π/τc that is similar to ω0, a new kind of resonance isachieved — between the Larmor precession, and a random process(e.g. molecular rotation). This allows for energy exchange betweenthe spin system and the “lattice,” here characterized only by its rota-tional degree of freedom. This energy exchange is irreversible andleads to loss of energy in the spin system that eventually returns tothermal equilibrium where M = Meq. It is thus the rapport betweenτc, governed among other things by the molecular size, and ω0 thatprimarily defines T1 in most situations.

5.3.2.3 T2 Relaxation

T2 is the decay of the x and y components of the magnetization.For argument’s sake, let’s assume M = Mx. The decay will resultfrom random fluctuations on By and Bz. Fluctuations on By induceenergy level changes since they act on Mz, similarly to what wepreviously saw. Only that this time, we do not have the contribu-tion from Bx, thus the energy-exchange component in T2 is 1/2 ofthat of T1. Fluctuations in Bz are tantamount to randomly varyingthe Larmor frequency ω0. This broadening of the resonance froma single frequency ω0 to a distribution of frequencies will result inloss of coherence in the precession of the transverse componentsof M about the z-axis. This irreversible phase loss will graduallydecrease the magnitude of Mxy until phase coherence is completelylost and Mxy → 0. This effect increases with B0, and thus T2 decreasesmonotonously with B0. T2 is referred to as the transverse relaxationtime or as the spin-spin relaxation time.



5.3.2.4 T ∗2 — The Effects of Field Inhomogeneity

As we saw earlier, T2 contributes to the decay on the XY plane evenwhen the external field is completely homogeneous. Inhomogeneityof B0 will contribute further to the loss of coherence, or dephasing ofMxy — simply because different parts of the sample “feel” a differentfield B0, resulting in a spread of frequencies. The total amount ofdecay is given by an additional decay constant — T ∗

2 . The relaxationrate due to T ∗

2 combines contributions from the “pure” T2 relaxationand those that stem from B0 inhomogeneity: 1/T ∗

2 = 1/T2 + 1/T′2,

where T′2 denotes the contribution to transverse relaxation from B0

inhomogeneity.

5.3.2.5 Refocusing the Effects of Static B0Inhomogeneity — The Spin Echo

Erwin Hahn first noticed that if an excitation pulse is followed byanother pulse after a time period τ, a FID is regenerated after anotherτ period elapsed from the second pulse, even though the original FIDhas completely vanished.2 This phenomenon was to be later calledthe spin echo, and it became a staple feature in numerous pulsesequences, the most basic of which is used to measure T2. The spinecho is best explained using the concept of two isochromats, or twospin populations with distinctly different resonance frequencies, ωs

and ωf , i.e. a “slow” and a “fast” frequencies stemming from dif-ferent B0 felt by these populations. A diagrammatic description ofthe sequence of events in a spin echo is given in Fig. 8. Followingthe first 90◦(x) pulse, both isochromats create an initial transversemagnetization (a). After a period τ, as a result of the frequency dif-ference between the two, ωs is lagging behind ωf , as seen in (b). If a180◦(x) pulse is given, the isochromats are flipped around the x-axis,and the “mirror image” of the two isochromats is such that now ωs

is in the lead and ωf is lagging behind (c). After the same periodτ, the two isochromats will converge, or refocus, on the negative y-axis (d). The phase between the two isochromats that was created bythe inhomogeneity of B0 is now restored to 0. By generalization —the spin echo sequence refocuses phase loss that is due to static B0



Fig. 8. Spin dynamics of two isochromats during a Hahn spin echo sequence: (A)immediately after excitation; (B) following a delay τ; (C) after the 180◦ (refocusing)pulse and (D) after a second delay τ.

inhomogeneities. One should note that phase losses due to T2 relax-ation are not restored, and neither are losses due to spin motion inan inhomogeneous B0. Figure 9 shows the FID and the echo thatis generated by a spin echo sequence. It should be noted thatalthough the intensity of the echo is weighted by T2, the envelopesof both the original FID and the echo are still decaying as a functionof T∗

2 .

5.3.2.6 The Effect of T1

Since T1 operates on the z-axis, its effects are not directly visible onthe FID, or the NMR spectrum for that matter. Since the amount of

Fig. 9. The FID and echo formation for a Hahn spin echo.



magnetization available for detection is dictated by Mz, the inten-sity of the signal detected will depend on how large was Mz prior tothe excitation pulse. If the time between subsequent excitations, alsoknown as TR (time-to-repetition) is too short to let Mz from previ-ous excitation to reach its equilibrium value Mz,eq, then a reductionin signal intensity occurs. This reduction is more severe for spinpopulations with a longer T1, and this is the basis for obtaining T1-based contrast in MR images. Since T1 affects Mz, an inversion pulse(a 180◦ pulse) applied first to the sample inverts the magnetizationto yield M(0) = −Mz,eq. From this point and on, the magnetization,which does not possess transverse components, will relax along thez-axis according to Mz(t) = Mz,eq(1 − 2 exp (− t

T1)) (see the solution

for Mz in the chapter “the Bloch equations”). To make the magneti-zation detectable, another pulse, a detection pulse, is needed, to flipthe magnetization to the xy plane. This is the inversion-recoverysequence, used both for measuring T1 in a sample as well as for gen-erating contrast based on T1 and on other mechanisms that will bebriefly mentioned later.

5.4 SPATIAL ENCODING — DIFFERENTIATING THE NMRSIGNAL ACCORDING TO ITS SPATIAL ORIGINS

Let us revise some of the simplest principles we know so far througha simple example: in a homogeneous field, a couple of test tubes filledwith water, set apart from each other on the x-axis will generate asingle peak, whose frequency is dictated by B0 and γ : ω0 = γ B0. Orin other words: The FID (and thus the spectrum) will have one char-acteristic frequency, defined by the chemical species in our sample(e.g. water protons). As long as the field B0 is homogeneous — ω0 isconstant across the sample.

If variability is introduce in B0 along a certain axis, e.g. the x-axis,the same variability will be expressed in ω0, and each point in spacewith the same x coordinate will have the same ω0, only that nowω0 = ω0(x). The simplest such variability is one that is linear withdistance from an arbitrary point — a linear magnetic field gradient



(MFG). It should be emphasized that in the MR convention, the B0

field is always oriented along the z-axis, but the variation in B0 is linearalong the axis of choice. The resulting magnetic field in the presenceof a MFG, e.g. along the x-axis is then:

B0(x) = B0(0) + �B0(x) = B0(0) + dB0

dxx = B0(0) + gx · x,

gx has thus units of g · cm−1 (cgs), and it is the slope of the variationof B0 with x.

5.4.1 Acquisition in the Presence of a MFG

As can be seen on Fig. 10, application a MFG on the x-axis assignsa resonance frequency to each position on that axis. In other words,the FID consists now of a range of resonance frequencies and thisrange is a result of the variability of B0 across the sample. Acqui-sition of a FID in the presence of the MFG, and a following FTyields a spectrum on which frequency is proportional to position on thex-axis. The intensity of the “peak” is proportional to the total M(0)at that specific location on the x-axis. This 1D-image is thus a pro-jection of the 3-D spin distribution in our sample on the x-axis.

Fig. 10. The sample in the presence of a magnetic field gradient. Each point in thesample along the x-axis feels a different magnetic field (left). The result (right) is afrequency encoded one-dimensional image (projection) of the object.



Historically speaking, Paul Lauterbur (Nature, 1973) first suggestedthe use of MFG for spatial encoding. His idea was to measurethe projections in different radial directions, and reconstruct theobject from them (projection-reconstruction). He called his methodZeugmatography.

5.4.2 MFG, Spectral Width and Field-of View

The range of frequencies or the spectral width SW that is spannedby the MFG is related to the spatial range D on the axis of inter-est through the gradient strength and the gyromagnetic ratio.In the case where the gradient was applied along the x-axis,SW(x) = γ·g(x)·D(x). The implication is that the stronger is the gra-dient, the broader is the frequency span for a specific desiredfield-of-view (FOV) on a desired axis. Or, conversely, increasing theFOV in a specific axis increases the SW on that axis.

5.4.3 Another Way to Look at the Effect of MFG

If two locations a and b on an object along the gradient axis are desig-nated the locations xa and xb, respectively, then in the rotating framethe frequencies generated at those two locations in the presence ofa MFG are two non-identical frequencies, ωa and ωb, respectively.By doubling the gradient strength twice, the frequencies are alsodoubled to become 2ωa and 2ωb, respectively. This means that theevolution of the FID in the presence of a magnetic field gradient,or more specifically, of the phase of each frequency component ofthe FID is a function of both time (t) and the gradient strength (g).Twice the time — twice the phase for the same g; twice the gradientstrength — twice the phase for the same t. Thus the detected signalS is S(γ , t, g). A new variable k can be introduced, which has theunits of cm−1 (inverse space). k is defined as: k = γ

∫g(t)dt and if the

gradient is constant with time, then S = S(k) = S(γ · g · t). In a similarway in which time and frequency domains are related to each othervia the Fourier transformation, so are the coordinate vector r and



the vector k:

f (r) = 2π

∫ ∫ ∫F(k) exp (ik · r)dk F(k) = 2π

∫ ∫ ∫f (r) exp (ik · r)dr,

or in other words, k-space and coordinate space are Fourier-conjugates.

The effect that gradients and RF pulses have on the phase ofthe transverse magnetization can thus be efficiently described as atrajectory in k-space. It is instructive to consider the case where aslice-selective excitation pulse (e.g. a 90x pulse) is applied to theequilibrium magnetization, and only manipulations of the magne-tization on the XY plane are considered. This covers the commonsituation of trajectories in a 2D k-space, encountered in all multisliceschemes. At t = 0 (right after the excitation pulse), Mxy is in-phaseand thus kx = ky = 0.

This is all very similar to spectroscopy only that instead of havingdifferent resonance frequencies that originate from different chemi-cal shifts, the different frequencies originate from different positionsin a nonhomogeneous field!

5.4.4 Flexibility in Collecting Data in K-Space

Since s = s(k) = s(γ · t · g), the signal (encoded for position on, e.g.,the x-axis) can be acquired in two different ways:

Keep the gradient constant: let the signal evolve with time, andsample the FID at different time points. This is typically referred toas frequency encoding.

Keep the time constant: sample the FID following application ofshort gradients of the same duration, but with different gradientstrength. This is referred to as phase encoding.

Thus, if a rectangular portion of k-space needs to be sampled,a logical way to achieve this goal is to acquire the data follow-ing sequential excitations of the spin system, where each excitation



is frequency encoded in one direction (say, the x-axis) and phaseencoded in the perpendicular direction (e.g. the y-axis).

5.4.5 The Gradient Echo

Typically, in order to allow for efficient time management of otherpulse sequence elements and the acquisition of a full echo signal,the magnetization is first dephased along the frequency encodingdirection, and the rephased using a gradient of opposite polarity. Ifthe dephasing gradient amplitude equals the rephrasing gradientamplitude, then magnetization components that gained phase −�

created by a local field B0 − �B for a time period t, will now recoverthe same phase, if the local field at the same point is B0 + �B forthe same time t. More generally, the refocusing condition is that thearea of the dephasing gradient be equal to that of the rephrasinggradient:

∫ t(end)t(start) gdeph.dt = − ∫ t(end)

t(start) greph.dt. This allows for flexibilityin choosing the time it takes to the magnetization to refocus. Thetime between excitation and refocusing is the gradient echo time(TE). The principle of the gradient echo is illustrated in Fig. 11.

Fig. 11. The gradient echo.



5.4.6 Encoding for the Third Dimension: Slice Selection orAdditional Phase Encoding

There are two main options for spatially encoding the out-of-planedimension. One is to add a phase-encoding loop on the third dimen-sion. This choice is popular with imaging modalities that aim forhigh spatial resolution. The other option is to combine frequencyselective RF pulses with magnetic field gradients for a spatiallyselective excitation. For example, for a RF pulse with a sinc-shapeenvelope sin τ

τ, the frequency response function is rectangular with a

bandwidth of 1/τ. In the presence of a gradient, the external mag-netic field is given by B(�x) = B(0) + �B = B(0) + g · �x. Therange of the magnetic field �B can be expressed in terms of rangeof frequencies �ω/γ and it is this range of frequencies that are inresonance with those contained in the sinc pulse bandwidth. Thisprovides a simple relationship between the bandwidth of the pulse,the gradient applied in conjunction with the pulse and the spatialextent of the excitation, or slice thickness: BW = γg�x, where BWis the bandwidth of the RF pulse, γ is the gyromagnetic ratio, g isthe gradient strength and �x is the slice thickness. The carrier fre-quency of the RF pulse can be monitored to modify the location ofthe center of the slice. In a typical multislice MRI experiment, the slicelocation is varied cyclically, and in order to avoid artifacts associatedwith slice overlaps, the cycle is performed on odd and even slicessequentially.

5.4.7 Intraslice Phase Dispersion

One problem associated with slice selection stems from the fact thatdue to the presence of the gradient during the application of the RFpulse, a range of frequencies is being created. This means that exceptfor the one frequency ω0, i.e. the reference frequency for that partic-ular nucleus, all other frequencies excited by the pulse are off reso-nance. Off resonance effects are marginal when the pulse duration isshort with respect to the frequency offset caused by the gradients, butthis is typically not the case. The result is that frequency componentswithin the slice gain a phase component that is proportional to their



distance from the point in the slice excited by ω0 (typically the cen-ter of the slice for symmetric frequency responses). This in turncauses signal loss, which can be at times quite severe. In order torefocus this phase dispersion, a gradient with the opposite polarityto that of the slice selection gradient is applied immediately at theend of the pulse. It can be shown that for complete refocusing thefollowing condition has to be met: S(grefocusing) = S(gslice selection)/2,where S is the “area”, or the integral over time of the givengradient.

5.4.8 A Complete Pulse Sequence

The first MRI pulse sequence that incorporated all three elementsof spatial encoding: frequency encoding, phase encoding and sliceselection was the “spin warp” suggested by W. Edelstein in 1980. Theschematics of the spin warp are given in Fig. 12. Many of the MRIpulse sequences that were subsequently developed are conceptuallysimilar to the spin warp. Notable exceptions are sequences that arebased on a single excitation, such as echo planar imaging (EPI) andmultiple spin-echo sequences.

Fig. 12. The spin warp pulse sequence.



5.4.9 Contrast in MRI Sequences

Contrast is the visual differentiation between different parts of theimage. In MRI, contrast is typically based on a physical propertyrelated to a specific spin population. The physical property is thuscalled a contrast mechanism. Contrast can be based on relaxationproperties: T1, T2, T∗

2. Additionally, contrast based on spin mobility —flow (e.g. blood flow), perfusion (mobility of water through the cap-illary bed into tissue), and self-diffusion (random motion of watermolecules). A different type of contrast is based on chemical envi-ronment effects — proton or water chemical exchange between dif-ferent environments (e.g. binding sites on heavy macromolecules)gives rise to contrast through magnetization transfer mechanism.Relaxation-based contrast is the most basic way to obtain contrastin MRI. The contrast is achieved by sensitizing the image to one(or more) of the relaxation mechanisms previously mentioned. Byexamining the simple spin-warp sequence, it is already possible toget a sense of how contrast is achieved. First, since this is a gradientecho sequence, the image will be primarily T∗

2 weighted: the inten-sity of the echo is given by S(TE) = S(0)∗exp(−TE/T∗

2). This is notentirely correct, since there are two other main factors that influ-ence the contrast. One is explicitly present in the equation above —S(0), or spin density. The other is caused by the finite time betweenconsecutive excitations (TR) — which affects the amount of longitu-dinal magnetization available for the next excitation. This contrast isbased on T1 and becomes more pronounced as TR becomes shorteror as the flip angle is closer to 90◦. The possibility of obtaining con-trast based on T2 is based on the introduction of a spin-echo ele-ment in the pulse sequence. This can be easily done by inserting an180◦ pulse between the excitation and the center of the acquisition,and accounting for the polarity of the gradient echo gradients. Thismodification converts the spin-warp sequence into a T2-weightedsequence. Other mechanisms will be described in detail in otherchapters of this book.



5.4.10 Echo Planar Imaging (EPI)

In our review of pulse sequence principles, we assumed that eachk-space line requires a separate excitation. This puts a severe limiton the minimum time required for obtaining an image — the needto introduce a delay between excitations (TR) is the one most timeconsuming element in the entire pulse sequence. P Mansfield5 sug-gested the possibility of obtaining an image with a single excitation.The trick is to find a trajectory in k-space that will cover the portionof k-space we are interested in. The way it is done is demonstrated inFig. 13. Following the excitation pulse, pre-encoding gradients areapplied in the phase encoding and read-out (frequency encoding)directions (a). From now on, the read-out gradients switch polar-ity back and forth to allow for “zig-zag” scan of k-space. Phase-encoding “blips” are introduced between read-out lines to bring themagnetization to consecutive k-space lines. Since the read-out timeis typically very short (on the order of 1 ms–2 ms), the acquisition ofan entire image of a single slice takes typically less than 100 ms. Anentire volume that consists of several slices can thus be acquired ina couple of seconds. EPI is thus the natural sequence to be used inapplications where temporal (and not spatial) resolution is required.

Fig. 13. The EPI pulse sequence.



The most notable application of this class is that of functional MRI.In the version of EPI shown above, the signal is T∗

2 weighted. Anintroduction of a 180◦ pulse, similar to what has been previouslymentioned, will result in a T2-weighted EPI image.

5.5 CONCLUSION

In this chapter, we developed the theoretical foundations of NMRand subsequently of MRI. The principles of nuclear magnetic reso-nance, nuclear relaxation, spatial encoding and MRI image contrasthave been discussed and amply illustrated. This basis should givethe reader a strong tool for understanding the sophisticated applica-tions of MRI in the biomedical sciences, such as functional MRI of thebrain using the blood oxygenation level dependent (BOLD) effect,diffusion weighted and diffusion tensor imaging (DTI) and more.

References

1. Bloch F, Nuclear induction, Phys Rev 70(7–8): 460, 1946.2. Hahn EL, Spin echoes, Phys Rev 80: 580–594, 1950.3. Edelstein WA, Hutchison JMS, Johnson G, Redpath T, Spin warp NMR

imaging and applications to human whole body imaging, Phys Med Biol25: 751–756, 1980.

4. Lauterbur PC, Image formation by induced local interaction: Examplesemploying nuclear magnetic resonance, Nature 242: 190–191, 1970.

5. Mansfield P, Multiplanar image formation using NMR spin echoes,J Phys C 10: 55–58, 1977.

6. Purcell EM, Torrey HC, Pound RV, Resonance absorption by nuclearmagnetic moments in a solid, Phys Rev 69(1–2): 37, 1946.


CHAPTER 6

Principles of Ultrasound ImagingModalities

Elisa Konofagou

Despite the fact that medical ultrasound preceded MRI and PET, ongo-ing advances have allowed it to continuously expand as a field in itsnumerous applications. In the past decade, with the advent of faster pro-cessing, specialized contrast agents, a better understanding of nonlinearwave propagation, novel and real-time signal and image processing andcomplex ultrasound transducer manufacturing, ultrasound imaging andultrasound therapy have enjoyed a multitude of new features and clinicalapplications. Those have added to the higher quality and wider applica-tions of diagnostic ultrasound images. Due to these developments, ultra-sound has become a very powerful imaging modality mainly due to itsunique temporal resolution, low cost, nonionizing radiation and portabil-ity. Lately, unique features such as harmonic imaging, coded excitation,3D visualization and elastic imaging, have added to higher quality andwider range of applications of diagnostic ultrasound images. In this chap-ter, a short overview of the fundamentals of diagnostic ultrasound and abrief summary of its many applications and methods are provided. Thefirst part of this chapter will provide a short background on the ultrasoundphysics and the second part will constitute a short overview on ultrasoundimaging and image formation.

6.1 INTRODUCTION

Sounds with a frequency above 20 kHz are called ultrasonic, sincethey occur at frequencies inaudible to the human ear. When emittedat short bursts, propagating through media, such as water, with lowreflection coefficients and reflected by obstacles along their propa-gation path, the detection of the reflection, or echo, of the ultrasonic

129


130 Elisa Konofagou

wave can help localize the obstacle. This principle has been usedby sonar (SOund NAvigation and Ranging) and inherently usedby marine mammals, such as dolphins and whales, to help themlocalize prey, obstacles or predators. In fact, the frequencies used for“imaging” vary significantly dependent upon the application: fromunderwater sonar (up to 300 kHz), diagnostic ultrasound (1 MHz–40 MHz), therapeutic ultrasound (0.8 MHz–4 MHz) and industrialnondestructive testing (0.8 MHz–20 MHz) to acoustic microscopy(up to 2 GHz).

6.2 BACKGROUND

6.2.1 The Wave Equation

As the ultrasonic wave propagates through the tissue, its energy andmomentum are transferred to the tissue. No net transfer of massoccurs at any particular point in the medium unless this is inducedby the momentum transfer. As the ultrasonic wave passes throughthe medium, the peak local pressure in the medium increases. Theoscillations of the particles result to harmonic pressure variationswithin the medium and to a pressure wave that propagates through

Particledistribution

Particledisplacement



Direction of propagation


λ






Direction of propagation

Fig. 1. Particle displacement and particle distribution for a traveling longitudinalwave. The direction of propagation is from left to right, namely the longitudinal(or, axial) direction. A shear wave can be created in the perpendicular direction,in which case the particles would also be moving in a direction orthogonal to thedirection of propagation (not shown here).1


Principles of Ultrasound Imaging Modalities 131

S

S

(1)

(2)

z z+ z

u u+ u

F

SS

S

(1)

(2)

z z+δz

u u+δu

F

Fig. 2. A small volume of the medium of impedance Z (1) at equilibrium and(2) undergoing oscillatory motion when an oscillatory force F is applied.

the medium as neighboring particles move with respect to oneanother (Fig. 1). The particles of the medium can move back andforth in a direction parallel (longitudinal wave) or perpendicular(transverse wave) to the traveling direction of the wave.

Let’s consider the first case.Assuming that a small volume of the medium that can be mod-

eled as a nonviscous fluid (no shear waves can be generated) isshown on Fig. 2, an applied force δF produces a displacement ofu + δu in the x-position on the right-hand side of the small vol-ume. A gradient of force ∂F

∂z is thus generated across the element inquestion, and, assuming that the element is small enough so thatthe measured quantities within the medium are constant, it can beassumed as being linear, or:

δF = ∂F∂z

δz, (1)

and according to Hooke’s Law,

F = KS∂u∂z

, (2)

where K is the adiabatic bulk modulus of the liquid and S is the areaof the region on which the force is exerted. By taking the derivative ofboth sides of Eq. 2 with respect to z and following Newton’s Second


132 Elisa Konofagou

Law, from Eq. 1 we obtain the so-called “wave equation”:

∂2u∂z2 − 1

c2

∂2u∂t2 = 0 (3)

where c is the speed of sound given by c =√

Kρ

=√

1ρκ

, where ρ is thedensity of the medium and κ is the compressibility of the medium.Eq. 3 relates the second differential of the participle displacementwith respect to distance to the acceleration of a simple harmonicoscillator. Note that the average speed of sound in most soft tissuesis about 1540 m/s with a total range of ±6%. For the shear wavederivation of this equation please refer to Wells1 or Kinsler and Frey2

among others.The solution of the wave equation is given by a function u, where:

u = u(ct − z). (4)

An appropriate choice of function for u in Eq. 4 can be:

u(t, z) = u0 exp[jk(ct − z)], (5)

where k is the wavenumber and equal to 2π/λ with λ denoting thewavelength (Fig. 1).

6.2.1.1 Impedance, Power and Reflection

The pressure wave that results from the displacement generated andgiven by Eq. 5 is given by:

p(t, z) = p0 exp[jk(ct − z)], (6)

where p0 is the pressure wave amplitude and j is equal to√−1. The

particle speed and the resulting pressure wave are related throughthe following relationship:

u = pZ

, (7)

where Z is the acoustic impedance defined as the ratio of the acousticpressure wave at a point in the medium to the speed of the particle at



the same point. The impedance is thus characteristic of the mediumand given by:

Z = ρc. (8)

The acoustic wave intensity is defined as the average flow ofenergy through a unit area in the medium perpendicular to the direc-tion of propagation.2 By following that definition, the intensity canbe found equal to3:

I = p20

2Z, (9)

and usually measured in units of mW/cm2 in diagnostic ultrasound.A first step into understanding the generation of ultrasound

images is to follow the interaction of the propagating wave withthe tissue. Thanks to the varying mean acoustic properties of tis-sues, a wave transmitted into the tissue will get partly reflected atareas where the properties of the tissue and, thus its impendance,are changing. These areas constitute a so-called “impedance mis-match” (Fig. 3).

ReflectedIncident

Transmitted

rurp

tu

ip

tp

tu

t

i r

ReflectedIncident

Transmitted

ru rurp rp

tu tu

ip ip

tp tp

tu tu

ttϑ

ii rr

Medium 1

Medium 2

Interface

ReflectedIncident

Transmitted

ru rurp rp

tu tu

ip ip

tp tp

tu tu

tt

ii rr

ReflectedIncident

Transmitted

ru rurp rp

tu tu

ip ip

tp tp

tu tu

tt

iiϑ

rrϑ

Medium 1

Medium 2

Interface

Fig. 3. An incident wave at an impedance mismatch (interface): A reflected anda transmitted wave with certain velocities and pressure amplitudes are createdensuring continuity at the boundary.


134 Elisa Konofagou

The reflection coefficient R of the pressure wave at an incidenceangle of ϑi is given by:

R = pr

pi= Z2cosϑt − Z1cosϑi

Z2cosϑt + Z1cosϑi, (10)

where ϑt is the angle of the transmitted wave (Fig. 3) also related tothe incidence angle through Snell’s Law:

λ1 cos ϑi = λ2 cos ϑt, (11)

where λ1 and λ2 are the wavelengths of the waves in medium 1 and2, respectively, and related to the speeds in the two media through:

c = λf , (12)

where f is the frequency of the propagating wave.As Fig. (3) also shows, the wave impingent upon the impedance

mismatch also generates a transmitted wave, i.e. a wave that prop-agates through. The transmission coefficient is defined as:

T = pt

pi= 2Z2 cos ϑi

Z2 cos ϑi + Z1 cos ϑt. (13)

According to the parameters reported by Jensen3 on impedanceand speed of sound of air, water and certain tissues, the reflectioncoefficient at a fat-air interface is equal to −99.94% showing thatvirtually all of the energy incident on the interface is reflected backin tissues such as the lung. A more realistic example found in thehuman body is the muscle-bone interface, where the reflection coef-ficient is 49.25%, demonstrating the challenges encountered whenusing ultrasound for the investigation of bone structure. On the otherhand, given the overall similar acoustic properties between differentsoft tissues, the reflection coefficient is too low when used to differ-entiate between different soft tissue structures ranging only between−10% and 0.

The values mentioned above determine both the interpretationof ultrasound images, or sonograms, as well as the design of trans-ducers, as discussed in the sections below.



6.2.1.2 Tissue Scattering

In the previous section, the notions of reflection, transmission andpropagation were discussed in the simplistic scenario of plane wavepropagation and its impingment on plane boundaries. In tissues,however, such a situation is rarely encountered. In fact, tissues areconstituted by cells and groups of cells that serve as complex bound-aries to the propagating wave. As the wave propagates through allthese complex structures, reflected and transmitted waves are gen-erated at each one of these interfaces dependent on the local density,compressibility and absorption of the tissue. The groups of cellsare called “scatterers” as they scatter acoustic energy. The backscat-tered field, or what is “scattered back” to the transducer, is used togenerate the ultrasound image. In fact, the backscattered echoes areusually coherent and can be used as “signatures” of tissues that aree.g. in motion or under compression, as applied in elasticity imagingmethods.

An example of such an ultrasound image can be seen in Fig. 4.The capsule of the prostate is shown to have a strong echo, mainlydue to the high impedance mismatch between the surroundingmedium, gel in this case, and the prostate capsule. However, theremaining area of the prostate is depicted as a grainy region sur-rounding the fluid filled area of the urethra (dark, or low scattering,area in the middle of the prostate). This grainy appearance is

Urethralcrest

Central zone

Peripheral zone

Verumontanum

Fibrous connective tissue

Urethralcrest

Central zone

Peripheral zone

Verumontanum

Fibrous connective tissue

(A) (B)

Fig. 4. Sonogram of (A) an in vitro canine prostate and (B) its correspondinganatomy at the same plane as that scanned.


136 Elisa Konofagou

called “speckle,” a term borrowed from the laser literature.4 Speckleis produced by the constructive and destructive interference ofthe scattered signals from structures smaller than the wavelength;hence, the appearance of bright and dark echoes, respectively. So,speckle does not necessarily relate to a particular structure in thetissue.

Given its statistical significance, in its simplest representation,the amplitude of speckle has been represented as having a Gaus-sian distribution with a certain mean and variance.5 In fact, thesesame parameters have been used to indicate that the signal-to-noise ratio of an ultrasound image is fundamentally limitedto only 1.91.5 As a result, in the past, several authors havetried different speckle cancellation techniques6 in an effort toincrease the image quality of diagnostic ultrasound. However,speckle offers one important advantage that has rendered itvital in the current applications of ultrasound (Sec. 5). Despite itbeing described solely by statistics, speckle is not a random signal.As mentioned earlier, speckle is coherent, i.e. it preserves its char-acteristics when shifting from position. Consequently, motion esti-mation techniques that can determine anything from blood flow totissue elasticity are made possible in a field that is widely known as“speckle tracking.”

6.2.1.3 Attenuation

As the ultrasound wave propagates inside the tissue, it undergoes aloss of power dependent on the distance traveled in the tissue.Atten-uation of the ultrasonic signal can be attributed to a variety of factors,such as divergence of the wavefront, reflection at planar interfaces,scattering from irregularities or point scatterers and absorption ofthe wave energy.7 In this section, we will concentrate on the latter,being the strongest factor in soft (other than lung) tissues. In this case,the absorption of the wave’s energy leads to heat increase. The actualcause of absorption is still relatively unknown but simple modelshave been developed to demonstrate the dependence of the resultingwave pressure amplitude decrease in conjunction with the viscosityof tissues.8



By not going into detail concerning the derivations of such arelationship, an explanation of the phenomenon is provided here.Let’s consider a fluid with a certain viscosity that provides a certainresistance to a wave propagating through its different layers. In orderto overcome the resistance, a certain force per unit area or, pressure,needs to be applied that is proportional to the shear viscosity of thefluid η as well as the spatial gradient of the velocity,7 or:

p ∝ η∂u∂z

. (14)

Equation 14 shows that a fluid with higher viscosity will requirehigher force to experience the same velocity gradient compared toa less viscous fluid. By considering Eqs. 2 and 14, an extra term canbe added to the wave equation that includes both the viscosity andcompressibility of the medium,7 or:

∂2u∂z2 +

(4η

3+ ξ

)k

∂3u∂z2∂t

− 1c2

∂2u∂t2 = 0, (15)

where ξ denotes the dynamic coefficient of compressional viscosity.The solution to this equation is given by:

u(t, z) = u0 exp (−αz) exp[jk(ct − z)], (16)

where α is the attenuation coefficient also given by (for α � k):

α =(

4η

3 + ξ)

k2

2ρc. (17)

From Eq. 16, the effect of attenuation on the amplitude of the waveis clearly depicted (Fig. 5). An exponential decay on the envelope ofthe pressure wave highly dependent on the distance results from thetissue attenuation. The intensity of the wave will decrease at twicethe rate, given that from Eq. 9:

I(t, z) = p20

Zexp (−2αz) exp[2jk(ct − z)] (18)

or, the average intensity is equal to:

〈I〉 = I0 exp (−2αz). (19)


138 Elisa Konofagou

z

u(t,z)

z

u(t,z)

Fig. 5. This is the attenuated wave of Fig. 1. Note that the envelope of the wave isdependent on the attenuation of the medium.

Another important effect that the tissue attenuation can have onthe propagating wave is a frequency shift. This is because a morecomplex form for the attenuation α is:

α = β0 + β1f , (20)

where β0 and β1 are the frequency-independent and frequency-dependent attenuation coefficients. In fact, the frequency-dependentterm is the largest source of attenuation and increases linearly withfrequency. As a result, the spectrum of the received signal changesas the pulse propagates through the tissue in such a way that a shiftto smaller frequencies, or downshift, occurs. In addition, the down-shift is dependent on the bandwidth of the pulse propagating in thetissue and the mean frequency of a spectrum (in this case Gaussian3)can be given by:

〈f 〉 = f0 − (β1B2f 20 )z, (21)

where f0 and B denote the center frequency and bandwidth of thepulse. Thus, according to Eq. 21, the downshift due to attenuationdepends on the tissue frequency-dependent attenuation coefficient,and the pulse center frequency and bandwidth. A graph showingthe typical values of frequency-dependent attenuation coefficients(measured in dB/cm/MHz) in the biological tissue is given in Fig. 6.



10-3

10-2

10-1

100

101

102

Attenuation (dB/cm/MHz)

Plasma

Blood

Spleen

Bone

Fat

Liver

Kidney

10-3

10-2

10-1

100

101

102

Attenuation (dB/cm/MHz)

Plasma

Blood

Spleen

Bone

Fat

Liver

Kidney

Fig. 6. Attenuation values of certain fluids and soft tissues.9

6.3 KEY TOPICS WITH RESULTS AND FINDINGS

6.3.1 Transducers

The pressure wave that was discussed in the previous section isgenerated using an ultrasound transducer, which is typically apiezoelectric material. “Piezoelectric” denotes the particular prop-erty of certain crystal polymers of transmitting a pressure (“piezo”means “to press” in Greek) wave generated when an electricalpotential is applied across the material. Most importantly, sincethis piezoelectric effect is reversible, i.e. a piezoelectric crystalwill convert an impinging pressure wave to an electric poten-tial, the same transducer can also be used as a receiver. Suchcrystalline or semicrystalline polymers are the poly-vinylidenefluoride (PVDF), quartz, barium titanate and lead zirconiumtitanate (PZT).


140 Elisa Konofagou

A single-element ultrasound transducer is shown in Fig. 2.Dependent upon its thickness (l) and propagation speed (c), thepiezoelectric material has a resonance frequency given by:

f0 = c2l

. (22)

The speed in the PZT material is around 4 000 ms−1, so for a 5 MHztransducer, the thickness should be 0.4 mm thick. The matchinglayer is usually coated onto the piezoelectric crystal in order to min-imize the impedance mismatch between the crystal and the skinsurface and, thus, maximize the transmission coefficient (Eq. 13). Inorder to overcome the aforementioned impedance mismatch, theideal impedance Zm and thickness dm of the matching layer arerespectively given by:

Zm =√

ZTZ (23)

and

dm = λ

4, (24)

with ZT denoting the transducer impedance and Z the impedanceof the medium.

The backing layers behind the piezoelectric crystal are used inorder to increase the bandwidth and the energy output. If the back-ing layer contains air, then the air-crystal interface yields a max-imum reflection coefficient given the high impedance mismatch.Another by-product of an air-backed crystal element is that thecrystal remains relatively undamped, i.e. the signal transmitted willhave a low bandwidth and a longer duration. On the other hand,the axial resolution of the transducer depends on the signal dura-tion, or pulse width, transmitted. As a result, there is a tradeoffbetween transmitted power and resolution of an ultrasound sys-tem. Depending on the application, different backing layers aretherefore used. Air-backed transducers are used in continuous-wave and ultrasound therapy applications. Heavily-backed trans-ducers are utilized in order to obtain high resolution, e.g. for highquality imaging at the expense of lower sensitivity and reduced



Fig. 7. Typical construction of a single-element transducer.3

penetration. Coded-excitation techniques have recently beensuccessfully applied to circumvent such tradeoffs.

For imaging purposes, an assembly of elements such as that inFig. 7 is usually used and called an “array” of such elements. In anarray, the elements are stacked next to each other at a distance equalto less than a wavelength for the minimum interference and reducedgrating lobes. The linear array has the simplest geometry. It selectsthe region of interest by firing elements above that region. The beamcan then be moved on a line by firing groups of adjacent elementsand then the rectangular image obtained is formed by combining thereceived signals by all the elements. A curved array is used whenthe transducer is smaller than the area scanned. A phased array canbe used to change the “phase” or delay between the fired elementsand thus achieve steering of the beam. The phased array is usuallythe choice for cardiovascular exams, when the window between theribs allows for a very small transducer to image the whole heart.Focusing and steering can both be achieved by modifying the profileof firing delays between elements (Fig. 8).

6.3.2 Ultrasonic Instrumentation

Figure 9 shows a block diagram of the different steps that are usedin order to acquire, process and display the received signal from thetissue.


142 Elisa Konofagou

Array of elements

Beam wavefront

Beam direction

Group 1 Group 2 Group 3 Group 1 Group 2 Group 3 x x x x

x x

Group 1 Group 2 Group 3

xx

x

… … … … … …

τ τ τ

(A) (B) (C)

Fig. 8. Electronic (A) beam forming, (B) focusing and (C) focusing and beam steer-ing as achieved in phased arrays. The time delay between the firings of differentelements is denoted here by τ.

Fig. 9. Block diagram of a pulsed-wave system and the resulting signal or imageat three different steps.



6.3.2.1 Transducer Frequency

In ultrasound imaging, a pulse of a given duration, frequency andbandwidth is first transmitted. As mentioned before, a tradeoffbetween penetration (or, low attenuation) and resolution exists.Therefore, the chosen frequency will depend on the application.Usually, for deeper-organs, such as the heart, the uterus and theliver, the frequencies are restricted in the range of 3 MHz–5 MHzwhile for more superficial structures, such as the thyroid, thebreast, the testis and applications on infants, a wider range of4 MHz–10 MHz is applied. Finally, for ocular applications, a rangeof 7 MHz–25 MHz is determined by the low attenuation, low depthand higher resolution required.

The pulse is usually a few cycles of that frequency long (usually3–4 cycles) so as to ensure high resolution, and is generated by thetransmitter through a voltage step sinusoidal function at a voltageamplitude (100 V–500 V) and a frequency equal to that of the res-onance frequency of the transducer elements. For static structures,a single pulse or multiple pulses (usually used for averaging later)could be used at an arbitrary frequency. However, for moving struc-tures, such as blood, liver and the heart, a fundamental limit on themaximum pulse repetition frequency (PRF) is set by the maximumdepth of the structure, or PRF (kHz) = c/2Dmax. Typically, the PRFis in the range of 1 kHz–3 kHz.

6.3.2.2 RF Amplifier

The received signal needs to be initially amplified so as to guaranteea good signal-to-noise ratio.At the same time, the input of the ampli-fier should be devoid of the high voltage pulse in order to protectthe circuits but also maintain its low noise and high gain. A typicaldynamic range expected at the output is on the order of 70 dB–80 dB.

6.3.2.3 Time-Gain Compensation (TGC)

As indicated above, attenuation is unavoidable as the wave trav-els through the medium and it increases with depth. In order to


144 Elisa Konofagou

avoid artificial darkening of deeper structures as a result, a voltage-controlled attenuator is usually employed, where a control voltage isutilized to manually adjust the system gain accordingly after recep-tion of an initial scan. A logarithmic voltage ramp is usually appliedthat compensates for a mean attenuation level with depth.6 Thedynamic range becomes further reduced to 40 dB–50 dB.

6.3.2.4 Compression Amplifier

The signals will ultimately be displayed as a greyscale on a cathoderay tube (CRT), where the dynamic range is typically only 20 dB–30 dB. To this purpose, an amplifier with a logarithmic response isutilized.

6.3.3 Ultrasonic Imaging

Ultrasonic imaging is usually known as echography or sonography,depending on which side of the Atlantic ocean one is scanning from.As mentioned earlier, the signal acquired by the scanner can be pro-cessed and displayed in several different fashions. In this section,the most typical and routinely used ones are discussed.

6.3.3.1 A-Mode

Since the image is a grayscale picture, the amplitude of the signalis displayed. For this, the envelope of the RF signal needs to becalculated. This is, for example, achieved by applying the Hilberttransforms. The resulting signal is called a detected A-scan, A-lineor A-mode scan (A- for Amplitude). An example of that is shown onFig. 8.

6.3.3.2 B-Mode

When the received A-scans are spatially combined after acquisitionusing either a mechanically moved transducer or the previouslymentioned arrays and used to brightness-modulate the display ina 2D format, the brightness or B-mode is created, which has a true



Fig. 10. Top: B-scan of an abdominal aorta in a mouse at 30 MHz; Bottom: M-modeimage over several cardiac cycles taken along the dashed line in the B-scan.

image format and is by far the most widely used diagnostic ultra-sound mode. By default, sonogram or echogram refers to B-mode.Figure 10 shows a longitudinal B-mode image of an abdominal aorta.One of the biggest advantages of ultrasound scanning is real-timescanning and this is achieved due to the shallow depth of scanning inmost tissues and the high speed of sound. The frame rate is usuallyon the order of 30 Hz–100 Hz (while in the M-mode version it canbe as fast the PRF itself, see below). The frame rate is limited by thenumber of A-mode scans acquired, NA, and the maximum depth,i.e. the maximum frame rate is given by PRFF = c/2Dmax/NA.

6.3.3.3 M-Mode

Another way of displaying the A-scans is in function of time, espe-cially in cases where tissue motion needs to be monitored and ana-lyzed such as the case of heart valves or other cardiac structures. Inthe case of Fig. 10, only one A-scan from a particular tissue struc-ture is displayed in brightness mode and followed in time, called


146 Elisa Konofagou

motion-, or M-mode scan. A depth-time display is then generated. Atypical application of the M-mode display is used in the examinationof heart valve leaflets motion and Doppler displays.

6.4 DISCUSSION

One of the main problems with the standard use of ultrasound arisesfrom high attenuation in some tissues and especially small vesselsand blood cavities. In order to overcome this limitation, contrastagents are routinely used. Contrast agents are typically microspheresof encapsulated gas or liquid coated by a shell, usually albumin.Due to the high impedance mismatch created by the gas or liquidcontained, the resulting backscatter generated by the contrast agentsis a lot higher than that of the blood echoes.

An alternative method to generating higher backscatter due tothe increased impedance mismatch is based on the harmonics gen-erated by the bubble’s interaction with the ultrasonic wave. Thebubble vibration also generates harmonics above and below the fun-damental frequency, with the second harmonic possibly exceedingthe first harmonic. In other words, the contrast agent introducesnonlinear backscattering properties into the medium where it lies.Several processes of filtering out undesired echoes from station-ary media surrounding the region, where flow characteristics areassessed, result to weakening of the overall signal at the fundamen-tal frequency. Therefore, since residual harmonics will result frommoving scatterers, motion characteristics can all be obtained fromthe higher harmonic echoes, after using a high-pass filter and fil-tering out the fundamental frequency spectrum that also containsthe undesired stationary echoes. Another method for distilling theharmonic echo information is the more widely used phase or pulseinversion method, in which two pulses (instead of one) are sequen-tially transmitted with their phases reversed. Upon reception, theechoes resulting from the two pulses are then added and only thehigher harmonics remain.

Despite the fact that the idea of contrast agent use originatedfor blood flow measurements, the same type of approach can be



applied in the case of soft tissues as well. After being injected intothe bloodstream, the contrast agents can also appear and remain onthe tissues and offer the same advantages of motion detection andcharacterization as in the case of blood flow. However, it turns outthat contrast agents are not always needed for imaging of tissuesat higher harmonics, especially since tissue scattering can be up totwo orders of magnitude higher than blood scattering. The nonlin-ear wave characteristic of the tissues themselves is, thus, sufficientin itself to allow imaging of tissues, despite the resulting higherattenuation at those frequencies. The avoidance of patient discom-fort following contrast agent injection is one of the major advantagesof this approach in tissues. Imaging using the harmonic approach(whether with or without contrast agents) is generally known asharmonic imaging. Compared to the standard approach, harmonicimaging in tissues offers the ability to distinguish between noiseand fluid-filled structures, e.g. cysts and the gall bladder. In addi-tion, harmonic imaging allows for better edge definition in structuresand, thus, is generally known to increase image clarity, mainly due tothe much smaller influence of the transmitted pulse to the receivedspectrum. Harmonic imaging is now available in most commerciallyavailable ultrasound systems. One of the main requirements for har-monic imaging is the large bandwidth of the transducer at receiveso as to allow reception of the higher frequency components. Thiscomes into very good agreement with the higher resolution require-ment for diagnostic imaging.

Another field that has emerged out of ultrasonic imaging in thepast decade is elasticity imaging. Its premise is built on two provenfacts (1) that significant differences between mechanical propertiesof several tissue components exist and (2) that the information con-tained in the coherent scattering, or speckle, is sufficient to depictthese differences following an external or internal mechanical stimu-lus. For example, in the breast, not only is the hardness of fat differentthan that of glandular tissue, but, most importantly, the hardness ofnormal glandular tissue is different than tumorous tissue (benign ormalignant) by up to one order of magnitude. This is also the reason


148 Elisa Konofagou

why palpation has been proven a crucial tool in the detection ofcancer.

The second observation is based on the fact that coherent echoescan be tracked while or after the tissue in question undergoes motionand/or deformation caused by the mechanical stimulus, e.g. anexternal vibration or a quasi-static compression in a method calledElastography. Speckle tracking techniques are also employed herefor the motion estimation. In fact, Doppler techniques, such as thoseused for blood velocity estimation, were initially applied in orderto track motion during vibration (Sonoelasticity imaging or Sonoelas-tography). Parameters, such as velocity and strain, are estimated andimaged in conjunction with the mechanical property of the under-lying tissue. The higher the velocity or strain estimated the softerthe material and vice versa. Numerous applications ranging fromthe breast to the thyroid and the heart have been implimented inclinical applications.


Despite the fact that diagnostic ultrasound is an older imagingmodality compared to MRI and PET, it is very intriguing to seethat it continues to expand as a field offering numerous and diverseapplications. In this chapter, we have described some of the funda-mental aspects of ultrasound physics and ultrasonic imaging as wellas referred to examples of more recent methods and applications.

References

1. Wells PNT, Biomedical Ultrasonics, Medical Physics Series, AcademicPress, London NW1, 1977.

2. Kinsler LE, Frey AR, Fundamentals of Acoustics, 2nd edn., John Wiley &Sons, NY, 1962.

3. Jensen JA, Estimation of Blood Velocities Using Ultrasound, CambridgeUniversity Press, Cambridge, U.K., 1996.

4. Burckhardt CB, Speckle in ultrasound B-mode scans, IEEE Trans on Sonand Ultras SU-25: 1–6, 1978.

5. Wagner RF, Smith SW, Sandrik JM, Lopez H, Statistics of speckle inultrasound B-scans, IEEE Trans on Son and Ultras 30: 156–163, 1983.



6. Bamber JC, Tristam M, in Webb S (ed.), Diagnostic Ultrasound, IOPPublishing Ltd., pp. 319–386, 1988.

7. Christensen PA, Ultrasonic Bioinstrumentation, 1st edn., John Wiley &Sons, 1988.

8. Morse, Ingard, Theoretical Acoustics, New York, McGraw-Hill, 1968.9. Haney MJ, O’Brien Jr, WD, Temperature dependence of ultrasonic

propagation in biological materials, in Greenleaf JF (ed.), TissueCharacterization with Ultrasound, CRC Press Boca Raton FL, pp. 15–55,1986.


CHAPTER 7

Principles of Image ReconstructionMethods

Atam P Dhawan

Multidimensional medical imaging in most radiological applicationsinvolves three major tasks: (1) raw data acquisition using imaginginstrumentation; (2) image reconstruction from the raw data; and (3)image display and processing operations as needed. Image recon-struction in multidimensional space is generally an ill posed prob-lem, where a unique solution representing an ideal reconstructionof the true object from the acquired raw data may not be possibledue to limitations on data acquisition. However, using specific fil-tering operations on the acquired raw data along with appropriateassumptions and constraints in the reconstruction methods, a feasi-ble solution for image reconstruction can be obtained. Radon trans-form has been most extensively used in image reconstruction fromacquired projection data in medical imaging applications such asX-ray computed tomography. Fourier transform is directly applied tothe raw data for reconstructing images in medical imaging applica-tions, such as magnetic resonance imaging (MRI) where the raw datais acquired in frequency domain. Statistical estimation and optimiza-tion methods often show advantages in obtaining better results inimage reconstruction dealing with the ill posed problems of imaging.This chapter describes principles of image reconstruction in multidi-mensional space from raw data using basic transform and estimationmethods.

7.1 INTRODUCTION

Diagnostic radiology has evolved into multidimensional imagingin the second half of the twentieth century in terms of X-ray

151


152 Atam P Dhawan

computed tomography (CT), nuclear magnetic resonance imaging(NMRI/MRI), nuclear medicine: single photon emission computedtomography (SPECT) and positron emission tomography (PET),ultrasound computed tomography, and optical tomographic imag-ing. The foundation of such and many other multidimensionaltomographic imaging techniques started from a basic theory ofimage reconstruction from projections that was first published byJ Radon in 19171 and later explored by a number of researchersincluding Cramer and Wold,2 Renyi,3 Gilbert,4 Bracewell,5

Cormack6 and Hounsfield7,8 and many others for imaging appli-cations in many areas including medicine, astronomy, microscopyand geophysics.9–11 The implementation of the Radon transform forreconstructing medical images from the data collected from imag-ing instrumentation was only realized in the 1960s. Cormack in19636 showed the radiological applications of Radon’s work forimage reconstruction from projections using a set of measurementsdefining line integrals. In 1972, GN Hounsfield developed the firstcommercial X-ray computed tomography (CT) scanner that used acomputerized image reconstruction algorithm based on the Radontransform. GN Hounsfield and AM Cormack jointly received the1979 Nobel Prize for their contributions to the development of com-puterized tomography for radiological applications.6–8

Image reconstruction algorithms have been continuously devel-oped to reconstruct the true structural characteristics such as shape,density, etc. of an object in the image. Image reconstruction fromprojections or data collected from a scanner is an ill posed problembecause of the finite amount of data used to reconstruct the char-acteristics of the object. Furthermore, the acquired data is severelydegraded because of occlusion, detector noise, radiation scatteringand inhomogeneities of the medium.

The classical image reconstruction from projection method basedon the Radon transform is popularly known as the “backprojection”method. The backprojection method has been modified to incorpo-rate specific data collection schemes and to improve quality. Fouriertransform and iterative series expansion based methods havebeen developed for reconstructing images from projections. With


Principles of Image Reconstruction Methods 153

the fast developments in computer technology, advanced imagereconstruction algorithms using statistical and estimation meth-ods were developed and implemented for several medical imagingmodalities.

7.2 RADON TRANSFORM

Radon transform first defines ray- or line-integrals to form projec-tions from an unknown object and then uses infinite number of pro-jections to reconstruct an image of the object. It should be noted thatthough the early evolution in computed tomography was based onimage reconstruction using parallel beam geometry for data acqui-sition, more sophisticated geometrical configuration and scanninginstrumentation are used today for faster data collection and imagereconstruction. New computed tomography (CT) image scanners(often called as fourth generation CT scanners) utilize a cone-beam ofX-ray radiation and multiple rings of detectors for fast 3D multislicescanning. Also, the basic Radon transform that established the foun-dation of image reconstruction from projections has been extendedto a spectrum of exciting applications of image reconstructions inmultidimensional space using a variety of imaging modalities. How-ever, the discussion in this chapter is focused on two-dimensionalrepresentation of Radon transform only for image reconstructionfrom projections that are obtained through parallel beam scanninggeometry in computed tomography.

Let us define a two-dimensional object function f (x, y) and itsRadon transform by R{f (x, y)}. Let us use the rectangular coordinatesystem (x, y) in the spatial domain. The Radon transform is definedby the line integral

∫L along the path L such that:

R{f (x, y)} = Jθ(p) =∫

Lf (x, y)dl, (1)

where the projection Jθ(p) acquired at angle θ in the polar coordinatesystem is a one-dimensional symmetric and periodic function witha period of 2π. The polar coordinate system (p, θ) can be expressedinto rectangular coordinates in the Radon domain by using a rotated


154 Atam P Dhawan

x

y

q

p

θ

θ

p

f(x,y)

Jθ(p)

Fig. 1. Line integral projection Jθ(p) of a two-dimensional object f (x, y) at an angle θ.

coordinate system (p, q) that is obtained by rotating the (x, y) coor-dinate system (Fig. 1) by an angle θ as:

x cos θ + y sin θ = p,

−x sin θ + y cos θ = q. (2)

A set of line integrals or projections can be obtained for different θ

angles as:

R{f (x, y)} = Jθ(p) =∫ ∞

−∞f (p cos θ − q sin θ, p sin θ + q cos θ)dq. (3)

A higher-dimensional Radon transform can be defined in a simi-lar way. For example, the projection space for a three-dimensionalRadon transform would be defined by 2D planes instead oflines.

The significance of using the Radon transform for computingprojections in medical imaging is that an image of a human organ canbe reconstructed by back projecting the projections acquired throughthe imaging scanner. Figure 2 shows an illustration of the backprojection method for image reconstruction using projections.



Projection J (p)

Projection J (p)

Projection J (p)

A

B Projection 4θ

θ3

θ2

θ1

(p)J

ReconstructionSpace

Fig. 2. A schematic diagram for reconstructing images from projections. Threeprojections are back projected to reconstruct objects A and B.

Three simulated projections of two objects A and B are back pro-jected into the reconstruction space. Each projection has two seg-ments of values corresponding to the objects A and B. When theprojections are back projected, the areas of higher values becauseof the intersection of back projected projection data represents tworeconstructed objects. It should be noted that the reconstructedobjects may have geometrical or aliasing artifacts because of thelimited number of projections used in the imaging and reconstruc-tion processes. In the early development of first and second gener-ations of CT scanners, only parallel beam scanning geometry wasused for direct implementation of Radon transform for image recon-structions from projections. To improve the geometrical shape andaccuracy of the reconstructed objects, a large number of projec-tions is needed that must be acquired in a fast and efficient way.Today, fourth generation CT scanners utilize a cone-beam of X-ray radiation and multiple rings of detectors for fast 3D multislicescanning. More advanced imaging protocols, such as spiral CT useeven faster scanning and data manipulation techniques. Figure 3


156 Atam P Dhawan

Ring of Detectors

Source

Source Rotation Path

X-rays

Object

Fig. 3. An advanced X-ray CT scanner geometry with rotating source and ring ofdetectors.

shows a fourth generation an X-ray CT scanner to obtain projec-tions using a divergent cone-beam X-ray beam source that is rotatedto produce multiple projections at various angles for multislice 3Dscanning. Modern CT scanners are used in many biomedical, indus-trial, and other commercial applications using a large spectrumof imaging modalities in multidimensional image reconstructionspace.

To establish a fundamental understanding of Radon transformand image reconstruction from projections, only 2D representationof Radon transform with image reconstruction from projectionsdefined through a parallel beam scanning geometry is discussedbelow.

7.2.1 Reconstruction with Fourier Transform

The projection theorem, also called the central slice theorem, pro-vides a relationship between the Fourier transform of the objectfunction and the Fourier transform of its Radon transform orprojection.



The Fourier transform of the Radon transform of the object func-tion f (x, y) can be written as1,9–13:

F{R{f (x, y)}} = F{Jθ(p)}=

∫ ∞

−∞

∫ ∞

−∞f (p cos θ − q sin θ, p sin θ + q cos θ)e−j2πωpdqdp,

(4)

where ω represents the frequency component in the Fourier domain.The Fourier transform, Sθ(ω) of the projection Jθ(p) can also be

expressed as:

Sθ(ω) =∫ ∞

−∞

∫ ∞

−∞Jθ(p)e−j2πωpdp. (5)

From Eqs. 4–5, the Fourier transform of the Radon transform of theobject function can be written as:

Sθ(ω) =∫ ∞

−∞

∫ ∞

−∞f (x, y)e−j2πω(x cos θ+y sin θ)dxdy = F(ω, θ). (6)

Equation 6 can be considered as the two-dimensional Fouriertransform of the object function f (x, y) and can be represented asF(u, v) with:

u = ω cos θ,

v = ω sin θ,(7)

where u and v represents frequency components along the x- andy-directions in a rectangular coordinate system.

It should be noted that Sθ(ω) represents the Fourier transform ofthe projection Jθ(p) that is taken at an angle θ in the space domain witha rotated coordinate system (p, q). The frequency spectrum Sθ(ω) isplaced along a line or slice at an angle θ in the frequency domainof F(u, v).

If several projections are obtained using different values of theangle θ, their Fourier transform can be computed and placed alongthe respective radial lines in the frequency domain of the Fouriertransform, F(u, v) of the object function f (x, y).Additional projectionsacquired in the space domain provide more spectral informationin the frequency domain leading to filling up the entire frequency


158 Atam P Dhawan

domain. Now the object function can be reconstructed using two-dimensional inverse Fourier transform of the spectrum F(u, v).

7.2.2 Reconstruction using Inverse Radon Transform

The forward Radon transform is used to obtain projections of anobject function at different viewing angles. Using the central slicetheorem, an object function can be reconstructed by taking theinverse Fourier transform of the spectral information in the fre-quency domain that is assembled with the Fourier transform ofthe individual projections. Thus, the reconstructed object function,f (x, y) can be obtained by taking the two-dimensional inverse Fouriertransform of F(u, v) as:

f (x, y) = F−1{F(u, v)} =∫ ∞

−∞

∫ ∞

−∞F(u, v)ej2π(xu+vy)dudv. (8)

With the change of variables:{u = ω cos θ,v = ω sin θ.

Equation 8 can be rewritten with the change of variables as:

f (x, y) =∫ π

0

∫ ∞

−∞F(ω, θ)ej2πw(x cos θ+y sin θ)|ω|dωdθ. (9)

In Eq. 9, the frequency variable ω appears because of the Jacobiandue to change of variables. Replacing F(ω, θ) with Sθ(ω), the recon-struction image f (x, y) can be expressed as the backprojected integral(sum) of the modified projections J∗

θ (p) as:

f (x, y) =∫ π

0

∫ ∞

−∞|ω|Sθ(ω)ej2πω(x cos θ+y sin θ)dωdθ

=∫ π

0

∫ ∞

−∞|ω|Sθ(ω)ej2πωpdωdθ =

∫ π

0J∗θ (p)dθ,

where

J∗θ (p) =

∫ ∞

−∞|ω| Sθ(ω)ej2πω(x cos θ+y sin θ)dω. (10)



7.3 BACKPROJECTION METHOD FOR IMAGERECONSTRUCTION

The classical image reconstruction from projection method basedon the Radon transform is popularly known as the backprojectionmethod. The backprojection method has been modified by a numberof investigators to incorporate specific data collection schemes andto improve quality of reconstructed images.

Though the object function can be reconstructed using theinverse Fourier transform of the spectral information of thefrequency domain F(u, v) obtained using the central slice theorem,an easier implementation of the Eq. 10 can be obtained by its realiza-tion through the modified projections, J∗

θ (p). This realization leadsto the convolution backprojection, also known as filtered backpro-jection method for image reconstruction from projections.

The modified projection J∗θ (p) can be expressed in terms of a

convolution of:

J∗θ (p) =

∫ ∞

−∞|ω|Sθ(ω)ej2πωpdω

= F−1{|ω|Sθ(ω)}= F−1{|ω|} ⊗ Jθ(p), (11)

where ⊗ represents the convolution operator.Equation 11 presents some interesting challenges for imple-

mentation. The integration over the spatial frequency variable ω

should be carried out from −∞ to ∞. But in practice, the projec-tions are considered to be bandlimited. This means that any spectralenergy beyond a spatial frequency, say �, must be ignored. UsingEqs. 10–11, it can be shown that the reconstruction function or image,f (x, y) can be computed as:

f (x, y) = 1π

∫ π

0dθ

∫ ∞

−∞dp′Jθ(p′)h(p − p′), (12)

where h(p) is a filter function that is convolved with the projectionfunction.


160 Atam P Dhawan

Ramakrishnan and Lakshiminarayanan9 computed the filterfunction h(p) strictly from Eq. 11 in the Fourier domain as

HR−L ={|ω| if |ω| ≤ �

0 otherwise

}, (13)

where HR−L is the Fourier transform of the filter kernel functionhR−L(p) in the spatial domain and is bandlimited.

In general, H(ω) a bandlimited filter function in the frequencydomain (Fig. 4) can be expressed as:

H(ω) = |ω|B(ω),

where B(ω) denotes the bandlimiting function,

B(ω) ={

1 if |ω| ≤ �

0 otherwise

}. (14)

For the convolution operation with the projection function in thespatial domain (Eqs. 10–11), the filter kernel function, H(ω) can beobtained from h(p) by taking the inverse Fourier transform as:

h(p) =∫ ∞

−∞H(ω)ej2πωpdω. (15)

If the projections are sampled with a time interval of τ, the projec-tions can be represented as Jθ(kτ) where k is an integer. Using the

H(ω)

1/2τ-1/2τ

1/2τ

ω

Fig. 4. A bandlimited filter function H(ω).



sampling theorem and the bandlimited constraint, all spatial fre-quency components beyond � are ignored such that:

� = 12τ

. (16)

For the bandlimited projections with a sampling interval of t, Eq. 15can be expressed with some simplification as:

h(p) = 12τ2

sin (πp/τ)πp/τ

− 14τ2

(sin (πp/2τ)

πp/2τ

)2

. (17)

Thus the modified projection J∗θ (p′) and the reconstruction image can

be computed as:

J∗θ (p) =

∫ ∞

−∞Jθ(p′)h(p − p′)dp′,

f (x, y) = π

L

L∑i=1

Jθi(p), (18)

where L is the total number of projections acquired during the imag-ing process at viewing angles θi; for i = 1, . . . , L.

The quality of the reconstructed image depends heavily onthe number of projections and the spatial sampling interval of theacquired projection. For better quality images to be reconstructed,it is essential to acquire a large number of projections covering theentire range of viewing angles around the object. Higher resolu-tion images with fine details can only be reconstructed if the projec-tions are acquired with a high spatial sampling rate satisfying thebasic principle of the sampling theorem. If the raw projection datais acquired at a sampling rate lower than the Nyquist sampling rate,aliasing artifacts would occur in the reconstructed image because ofthe overlapping spectra in the frequency domain. The fine details inthe reconstructed images represent high frequency components.

The maximum frequency component that can be reconstructedin the image is thus limited by the detector size and the scanning pro-cedure used in the acquisition of raw projection data. To reconstructimages of higher resolution and quality, the detector size should besmall. On the other hand, the projection data may suffer from poor


162 Atam P Dhawan

signal-to-noise ratio if there is an insufficient number of photonscollected by the detector due to its smaller size.

There are several variations in the design of the filter functionH(ω) investigated in the literature. The acquired projection data isdiscrete in the spatial domain. To implement convolution backpro-jection method in the spatial domain, the filter function has to berealized as discrete in the spatial domain. The major problem of theRamachandaran-Lakshiminarayanan filter9 is that it has sharp cut-offs in the frequency domain at ω = 1/2τ and ω = −1/2τ as shown inFig. 4. The sharp cut-offs based function provides sinc functions forthe filter in the spatial domain as shown in Eq. 5.16 causing mod-ulated ringing artifacts in the reconstructed image. To avoid suchartifacts, the filter function must have smooth cut offs such as thoseobtained from Hamming window function. A bandlimited general-ized Hamming window can be represented as:

HHam min g(ω) = |ω|[α + (1 − α) cos (2πωτ)]B(ω), for 0 ≤ α ≤ 1(19)

where the parameter α can be adjusted to provide appropriate char-acteristic shape of the function.

The Hamming window based filter kernel function providessmoother cutoffs as shown in Fig. 5. The Hamming window basedconvolution function provides smoother function in the spatial

H(ω)

1/2τ-1/2τω

Fig. 5. A Hamming window based filter kernel function in the frequency domain.



Fig. 6. (A) A reconstructed image of a cross sectional slice of chest of a cadaverusing the Radon transform based backprojection method; (B) The actual patholog-ically stained slice of the respective cross section.

domain that reduces the ringing artifacts and improves signal-to-noise ratio in the reconstructed image. Other smoothing functionscan be used for reducing ringing artifacts and improving the qualityof the reconstructed image.12–13

Figure 6(A) shows a reconstructed image of a cross sectional sliceof chest of a cadaver using the Radon transform based backprojec-tion method. The actual pathologically stained slice of the respectivecross section is shown in Fig. 6(B).

7.4 ITERATIVE ALGEBRAIC RECONSTRUCTIONTECHNIQUES (ART)

The iterative reconstruction methods are based on optimizationstrategies incorporating specific constraints about the object domainand the reconstruction process. Algebraic reconstruction tech-niques (ART)11–14 are popular algorithms used in iterative imagereconstruction. In the algebraic reconstruction methods, the raw


164 Atam P Dhawan

projection data from the scanner is distributed over a prespec-ified image reconstruction grid such that the error between thecomputed projections from the reconstructed image and the actualacquired projections is minimized. Such methods provide a mecha-nism to incorporate additional specific optimization criteria such assmoothing and entropy maximization in the reconstruction processto improve the image quality and signal-to-noise ratio. The alge-braic reconstruction methods are based on the series expansion rep-resentation of a function and were used by Gordon and Herman formedical image reconstruction.12–14

Let us assume a two-dimensional image reconstruction grid ofN pixels. Let us define pi representing the projection data as a setof ray sums that are collected by M scanning rays passing throughthe image at specific angles (rays as defined in Fig. 1). Let fj be thevalue of j-th pixel of the image that is weighted by wi,j to meet theprojection measurements. Thus the ray sum pi in the projection datacan be expressed as:

pi =N∑

j=1

wi,jfj for i = 1, . . . , M. (20)

The representation in Eq. 5.19 provides M equations of N unknownvariables to be determined. The weight wi,j represents the contri-bution of the pixel value in determining the ray sum and can bedetermined by geometrical consideration as the ratio of the areaoverlapping with the scanning ray to the total area of the pixel. Theproblem of determining fj for image reconstruction can be solvediteratively using the ART algorithm. Alternately, it can be solvedthrough matrix inversion since the measured projection data pi isknown. The set of equations can also be solved using dynamicprogramming methods.12

In algebraic reconstruction methods, each pixel is assigned a pre-determined value such as the average of the raw projection data perpixel to start the iterative process. Any time during the reconstruc-tion process, a computed ray sum from the image under reconstruc-tion is obtained by passing a ray. In each iteration, an error between



the measured projection ray sum and the computed ray sum is eval-uated and distributed on the corresponding pixels in a weightedmanner. The correction to the pixel values can be obtained in anadditive or multiplicative manner, i.e. the correction value is eitheradded to the current pixel value or multiplied with it to obtain thenext value. The iterative process continues until the error betweenthe measured and computed ray sums is minimized or meets a pre-specified criterion. The fj values from the last iteration provide thefinal reconstructed image.

Let qkj be the computed ray sum in the k-th iteration that is pro-

jected over the reconstruction grid in the next iteration. The iterativeprocedure can then be expressed as:

qki =

N∑l=1

f k−1l wi,l for all i = 1, . . . , M

fk+1j = f k

j +[

pi − qki∑N

l=1 w2i,l

]wi,j. (21)

Gordon14 used an easier way to avoid large computation of theweight matrix by replacing the weight by 1 or 0. If the center of thepixel passes through the ray, the corresponding weight is assignedas 1, otherwise 0. This simplification provides an efficient imple-mentation of the algorithm and is known as additive ART. Otherversions of ART including multiplicative ART have been developedto improve the reconstruction efficacy and quality.12

Iterative ART methods offer an attractive alternative to the fil-tered backprojection method because of their abilities to deal withthe noise and random fluctuations in the projection data causedby detector inefficiency and scattering. These methods are par-ticularly suitable for limited view image reconstruction as moreconstraints defining the imaging geometry and prior informationabout the object can easily be incorporated into the reconstructionprocess.


166 Atam P Dhawan

7.5 ESTIMATION METHODS

Though the filtered backprojection methods are most commonlyused in medical imaging, in practice, a significant number ofapproaches using statistical estimation methods have been investi-gated for image reconstruction for transmission as well as emissioncomputed tomography.15–26 These methods assume a certain distri-bution of the measured photons and then find the parameters forattenuation function (in the case of transmission scans such as X-rayCT) or emitter density (in the case of emission scans such as PET).

The photon detection statistics of a detector is usually charac-terized by Poisson distribution. Let us define a measurement vector�J = [J1, J2, . . . , JN] with Ji to be the random variable representing thenumber of photons collected by the detector for the i-th ray suchthat17:

E[Ji] = mie− ∫L µ(x,y,z)dl for i = 1, 2, . . . , N, (22)

where L defines the ray along which the photons with monochro-matic energy have been attenuated with the attenuation coefficientsdenoted by µs and mi is the mean number of photons collected bythe detector for the i-th ray position. Also, in the above formulation,the noise, scattering and random coincidence effects are ignored.

The attenuation parameter vector �µ can be expressed in termsof a series expansion as a weighted sum of individual attenuationcoefficients of corresponding pixels (for 2D reconstruction) or voxels(for 3D reconstruction). If the parameter vector �µ has Np number ofindividual elements (pixels or voxels), it can be represented as:

�µ =Np∑j=1

µjwj, (23)

where wj is the basis function that is the weight associated with theindividual µj belonging to the corresponding pixel or voxel.

One simple solution to obtain wj is to assign it a value 1 if theray contributing to the corresponding photon measurement vectorpasses through the pixel (or voxel) and 0 otherwise. It can be shown



that a line integral or ray sum for i-th ray is given by:

∫Li

µ(x, y, z)dl =Np∑

k=1

aikµk, (24)

where aik = ∫Li

wk(�x) with �x representing the position vector for(x, y, z) coordinate system.

The weight matrix A = {aik} is defined to rewrite the measure-ment vector as:

Ji(�µ) = mie−[A�µ]i ,

where

[A�µ]i =Np∑

k=1

aikµk. (25)

The reconstruction problem is to estimate �µ from a measured setof detector counts realizing the random variable �J. The maximumlikelihood (ML) estimate can be expressed as17–19:

�µ = arg max�µ≥�0

L(�µ),

L(�µ) = log P[�J = �j; �µ),(26)

where L(�µ) is the likelihood function defined as the logarithmic ofthe probability function P[�J = �j; �µ). The ML reconstruction methodsare developed to obtain an estimate of the parameter vector �µ thatmaximizes the probability of observing the measured data (photoncounts).

Using the Poisson distribution model for the photon counts,the measurement joint probability function P[ �J =�j; �µ) can beexpressed as:

P[�J = �j; �µ) =N∏

i=1

P[Ji = ji; �µ) =N∏

i=1

e−ji(µ)[ji(µ)]jiji! . (27)

If the measurements are obtained independently through definingray sums, the log likelihood function can be expressed combining


168 Atam P Dhawan

Eqs. 5.21, 5.25 and 5.26 as:

L(�µ) =N∑

i=1

hi([A�µ]i),

where

hi(l) = ji log (mie−l) − mie−l. (28)

Let us consider an additive nonnegative function ri representing thebackground photon count for the i-th detector due to the scatter-ing and random coincidences, the likelihood function can then beexpressed as17:

L(�µ) =N∑

i=1

hi([A�µ]i),

where

hi(l) = ji log (mie−l + ri) − (mie−l + ri). (29)

Several algorithms have been investigated in the literature to obtainan estimate of the parameter vector that maximizes the log likeli-hood function given in Eq. 5.27. However, it is unlikely that thereis a unique solution to this problem. There may be several solu-tions of the parameter vector that can maximize the likelihood func-tion. All solutions may not be appropriate or even feasible for imagereconstruction. To improve quality reconstructed images, a num-ber of methods imposing additional constraints such as smoothnessare applied by incorporating the penalty functions in the optimiza-tion process. Several iterative optimization processes incorporatingroughness penalty function for the neighborhood values of the esti-mated parameter vector have been investigated in the literature.17–19

Let us represent a general roughness penalty function R(µ)17–19

such that:

R(�µ) =K∑

k=1

ψ([C�µ]k),



where

[C�µ]k =Np∑l=1

cklµl. (30)

where ψk’s are potential functions working as a norm on the smooth-ness constraints Cµ ≈ 0 and K is the number of such constraints. Thematrix C is a K × Np penalty matrix. It should be noted that ψk’s areconvex, symmetric, nonnegative and differentiable functions.17 Apotential choice for a quadratic penalty function could be by defin-ing ψk(t) = wkt2/2 with non-negative weights, i.e. wk≥0. Thus theroughness penalty function R(�µ) is given by:

R(�µ) =K∑

k=1

wk12

([C�µ]k)2. (31)

The objective function for optimization using the penalized MLapproach can now be revised as:

�µ = arg max �(�µ)

where

�(�µ) = L(�µ) − βR(�µ). (32)

The parameter β controls the level of smoothness in the final recon-structed image.

Several methods for obtaining the ML estimate have been inves-tigated in the literature. These optimization methods include expec-tation maximization (EM), complex conjugate gradient, gradientdescent optimization, grouped coordinated ascent, fast gradientbased Bayesian reconstruction and ordered subsets algorithms.28–30

Such iterative algorithms have been applied to obtain a solution forthe parameter vector for reconstructing an image from both trans-mission and emission scans. In addition, multigrid EM methodshave also been applied for image reconstruction in positron emis-sion tomography (PET).23–24 Figure 7(A) shows axial PET images ofthe brain reconstructed using filtered backprojection methods whileFig. 7(B) shows same cross sectional images reconstructed using amultigrid EM method.


170 Atam P Dhawan

(A)

(B)

Fig. 7. (A) Axial PET images of the brain reconstructed using filtered backpro-jection methods; (B) cross sectional images reconstructed using a multigrid EMmethod.


Image reconstruction is an integral and probably the most importantpart of medical imaging. Utilizing more information about imaginggeometry and physics of imaging, quality of reconstruction can beimproved. Furthermore, a priori and model based information canbe used with constrained optimization methods for better recon-struction. In this chapter, basic image reconstruction approachesare presented that are based on Radon transform, Fourier trans-form, filtered backprojection, iterative ART, and statistical esti-mation and optimization methods. More details and advancedimage reconstruction methods are presented in Chapter 15 inthis book.



References

1. Radon J, Uber die Bestimmung von Funktionen durch ihre Integral-werte langs gewisser Mannigfaltigkeiten, Ber Verb Saechs AKAD Wiss,Match Phys, Kl 69: 262–277, 1917.

2. Cramer H, Wold H, Some theorems on distribution functions, J LondonMath Soc 11: 290–294, 1936.

3. Renyi A, On projections of probability distributions, Acta Math AcadSci Budapest 3: 131–141, 1952.

4. Gilbert WM, Projections of probability distributions, Acta Math AcadSci Budapest 6: 195–198, 1955.

5. Bracewell RN, Strip integration radio astronomy, Aust J Physcis 9:198–217, 1956.

6. Cormack AM, Representation of a function by its line integrals withsome radiological applications, J Appl Phys 34: 2722–2727, 1963.

7. Hounsfield GN, Computerized transverse axial scanning tomography:Part-1, description of the system, Br J Radiol 46: 1016–1022, 1973.

8. Hounsfield GN, A method and apparatus for examination of a bodyby radiation such as X or gamma radiation, Patent 1283915, The patentOffice, London, England, 1972.

9. Ramachandran GN, Lakshminaryanan AV, Three-dimensional recon-struction from radiographs and electron micrographs, Proc Nat AcadSci USA 68: 2236–2240, 1971.

10. Deans SR, The Radon Transform and Some of Its Applications, John Wiley &Sons, 1983.

11. Dhawan AP, Medical Image Analysis, John Wiley and Sons, 2003.12. Herman GT, Image Reconstruction from Projections, Academic Press,

1980.13. Rosenfeld, Kak AC, Digital Picture Processing, Vol 1, Academic Press,

1982.14. Gordon R, A tutorial on ART (Algebraic Reconstruction Techniques),

IEEE Trans Nucl Sci 21: 78–93, 1974.15. Dempster AP, Laird NM, Rubin DB, Maximum likelihood from incom-

plete data via the EM algorithm, J R Stat Soc Ser B 9: 1–38, 1977.16. Shepp LA, Vardi Y, Maximum likelihood reconstruction for emission

tomography, IEEE Trans Med Imag 1: 113–121, 1982.17. Fessler JA, Statistical image reconstruction methods for transmission

tomography, in Sonka, M, Fitzpatrick, JM (eds.), Handbook of Medi-cal Imaging, Vol. 2, Medical Image Processing and Analysis, SPIE Press,pp. 1–70, 2000.

18. Erdogen H, Fessler J, Monotonic algorithms for transmission tomog-raphy, IEEE Trans Med Imag 18: 801–814, 1999.


172 Atam P Dhawan

19. Yu DF, Fessler JA, Ficaro EP, Maximum likelihood transmission imagereconstruction for overlapping transmission beams, IEEE Trans MedImag 19: 1094–1105, 2000.

20. Lange K, Carson R, EM reconstruction algorithms for emission andtransmission tomography, J Comp Asst Tomogr 8: 306–316, 1984.

21. Olinger JM, Maximum likelihood reconstruction of transmissionimages in emission computed tomography via the EM algorithm, IEEETrans Med Imag 13: 89–101, 1994.

22. Welch A, Clack R, Natterer F, Gullberg G, Toward accurate attenuationcorrection in SPECT without transmission measurements, IEEE TransMed Imag 16: 532–541, 1997.

23. Ranganath MV, Atam Dhawan P, Mullani N, A multigrid expectationmaximization reconstruction algorithm for positron emission tomog-raphy, IEEE Trans on Med 7: 273–278, 1988.

24. RahejaA, DhawanAP, Wavelet based multiresolution expectation max-imization reconstruction algorithm for emission tomography, CompMed Imag And Graph 24: 87–98, 2000.

25. Solo V, Purdon P, Weisskoff R, Brown E, A signal estimation approachto functional MRI, IEEE Trans Med Imag 20: 26–35, 2001.

26. Basu S, Bresler Y, O(N3log N)Backprojection algorithm for the 3-DRadon transform, IEEE Trans Med Imag 21: 76–88. 2002.

27. Bouman CA, Saur K, A unified approach to statistical tomographyusing coordinate descent optimization, IEEE Trans Image Process 5: 480–492, 1996.

28. Erdogen H, Gualtiere G, Fessler JA, Ordered subsets algorithms fortransmission tomography, Phys Med Biol 44: 2835–2851, 1999.

29. Mumcuoglu EU, Leahy R, Cherry SR, Zhou Z, Fast gradient-basedmethods for Bayesian Reconstruction of transmission and emissionPET images, IEEE Trans Med Imag 13: 687–701, 1994.

30. Green PJ, Bayesian reconstructions from emission tomography datausing a modified EM algorithm, IEEE Trans Med Imag 9: 84–93, 1990.


CHAPTER 8

Principles of Image ProcessingMethods

Atam P Dhawan

Medical image processing methods including image restoration andenhancement methods are very useful for effective visual examinationand computerized analysis. Image processing methods enhance featuresof interest for better analysis and characterization. Though there havebeen more advanced model-based image processing methods investi-gated and developed recently, this chapter presents the principles ofselected basic image processing methods. Advanced image process-ing and reconstruction methods are described in other chapters in thisbook.

8.1 INTRODUCTION

Medical images are examined through visual inspection by expertphysicians or analyzed through computerized methods for spe-cific feature extraction, classification, and statistical analysis. In bothof these approaches, image processing operations such as imagerestoration (such as smoothing operations for noise removal) andenhancement for better feature representation, extraction and anal-ysis are very useful. The principles of some of the most commonlyused basic image processing methods for noise removal, imagesmoothing, and feature enhancement are described in this chap-ter. These methods are usually available in any image processingsoftware such as MATLAB through the image processing toolbox.

173


174 Atam P Dhawan

Medical images show characteristic information about thephysiological properties of the structures and tissues. However,the quality and visibility of information depend on the imagingmodality and the response functions (such as point spread func-tion) of the imaging scanner. Medical images from specific modal-ities need to be processed using methods suitable to enhancethe features of interest. For example, a chest X-ray radiographicimage shows the anatomical structure of the chest based on thetotal attenuation coefficients. If the radiograph is being examinedfor a possible fracture in the ribs, an image enhancement methodis required to improve the visibility of hard bony structure. But, ifan X-ray mammogram is obtained for the examination of potentialbreast cancer, an image processing method is required to enhancevisibility of microcalcifications, speculated masses and soft tis-sue structures such as parenchyma. A single image enhancementmethod may not serve both of these applications. Image enhance-ment methods for improving the soft tissue contrast in MR brainimages may be entirely different than those used for PET brainimages. Thus, image enhancement tasks and methods are very muchapplication dependent.

Image enhancement methods may also include image restora-tion methods which are generally based on minimum mean-squarederror operations, such as Wiener filtering and other constraineddeconvolution methods incorporating some a priori knowledge ofdegradation.1–5 Since the main objective is to enhance features ofinterest, a suitable combination of both restoration and contrastenhancement algorithms is the integral part of pre-processing inimage analysis. The selection of a specific restoration algorithm fornoise removal is highly dependent on the image acquisition system.For example, in the filtered-backprojection method for reconstruct-ing images in computed tomography (CT), the raw data obtainedfrom the scanner is first deconvolved with a specific filter. Filterfunctions such as Hamming window, as described in chapter 7, mayalso be used to reduce noise in the projection data. On the otherhand, several image enhancement methods, such as neighborhoodbased operations, frequency filtering operations, etc., implicitly de-emphasize noise for feature enhancement.


Principles of Image Processing Methods 175

Image processing methods are usually performed in one of thetwo domains: (1) spatial domain; (2) spectral domain. Image or spa-tial domain provides a distribution of an image feature such asbrightness over the spatial grid of samples. Spectral or frequencydomain provides spectral information in a transformed domain suchas the one obtained through Fourier transform. In addition, specifictransform based methods such Hough transform, neural networksand model-based methods have also been used for image processingoperations.1–7

8.2 IMAGE PROCESSING IN SPATIAL DOMAIN

Spatial domain methods process an image with pixel-by-pixeltransformation based on the histogram statistics or neighborhoodoperations. These methods are usually faster in computer imple-mentation as compared to frequency filtering methods that requirecomputation of Fourier transform for frequency domain represen-tation. However, frequency filtering methods may provide betterresults in some applications if a priori information about the char-acteristic frequency components of the noise and features of inter-est is available. For example, specific spike based degradationsdue to mechanical stress and vibration on the gradient coils inthe raw signal often cause striation artifacts in fast MR imagingtechniques. The spike degradations based noise in the MR signalcan be modeled with their characteristic frequency componentsand can be removed by selective filtering and wavelet processingmethods.7 Wiener filtering methods have been applied for signalenhancement to remove frequency components related to the unde-sired resonance effects of the nuclei and noise suppression in MRimaging.8–10

8.2.1 Image Histogram Representation

A histogram of an image provides information about the intensitydistribution of pixels in the image. The simplest form of a histogramis the plot of occurrence of specific gray-level values of the pixels inthe image. The occurrence of gray levels can be provided in terms of


176 Atam P Dhawan

the absolute values, i.e. the number of times a specific gray-level hasoccurred in the image, or the probability values, i.e. the probabilityof occurrence of a specific gray-level in the image. In mathematicalterms, a histogram h(ri) is expressed as:

h(ri) = ni for i = 0, 1, . . . , L − 1, (1)

where ri is the ith gray-level in the image for a total of L grayvalues and ni is the number of occurrences of gray-level ri in theimage.

If a histogram is expressed in terms of the probability of occur-rence of gray-levels, it can be expressed as:

p(ri) = ni

n, (2)

where n is the total number of pixels.Thus, a histogram is a plot of h(ri) or p(ri) versus ri. Figure 1(A)

shows an X-ray mammogram image while 1(B) shows its gray-level histogram.

(A) (B)

Fig. 1. An X-ray (A) mammogram image on the left with its (B) histogram at theright.



8.2.2 Histogram Equalization

A popular general-purpose method of image enhancement is his-togram equalization. In this method, a monotonically increasingtransformation function, T(r) is used to map the original gray values,ri of the input image into new gray values, si of the output imagesuch that:

si = T(ri) =i∑

j=0

pr(rj)

=i∑

j=0

nj

nfor i = 0, 1, . . . , L − 1, (3)

where pr(ri) is the probability based histogram of the input imagethat is transformed into the output image with the histogram ps(si).

The transformation function T(ri) in Eq. (6.3) stretches the his-togram of the input image such that the gray values occur in theoutput image with equal probability of occurrence. It should benoted that the uniform distribution of the histogram of the out-put image is limited by discrete computation of the gray-leveltransformation. The histogram equalization method forces imageintensity levels to be redistributed with an equal probability ofoccurrence.

Figure 2 shows the original mammogram image and its his-togram equalized image with their respective histograms. Imagesaturation around the middle of the image can be noticed in thehistogram equalized image.

8.2.3 Histogram Modification

The histogram equalization method stretches the contrast of animage by redistributing the gray values to achieve a uniform dis-tribution. This general method may not provide good results inmany applications. It can be noted from Fig. 2 that the histogramequalization method can cause saturation in some regions of theimage resulting in loss of details and high-frequency informationthat may be necessary for interpretation. Sometimes, local histogram


178 Atam P Dhawan

Fig. 2. Top left: original X-ray mammogram image; Bottom left: histogram ofthe original image; Top right: the histogram equalized image; Bottom right:histogram of the equalized image.

equalization is applied separately on predefined local neighborhoodregions, such as 7 × 7 pixels, to provide better results.1

If a desired distribution of gray values is known a priori, a his-togram modification method is used to apply a transformation thatchanges the gray values to match the desired distribution. The tar-get distribution can be obtained from a good contrast image that isobtained under similar imaging conditions. Alternatively, an orig-inal image from a scanner can be interactively modified throughregional scaling of gray values to achieve the desired contrast. Thisimage can now provide a target distribution to the rest of the images,obtained under similar imaging conditions, for automatic enhance-ment using the histogram modification method.



The conventional scaling method of changing gray values fromthe range [a, b] to [c, d] can be given by a linear transformation as:

znew = d − cb − a

(z − a) + c, (4)

where z and znew are, respectively, the original and new gray valuesof a pixel in the image.

Let us assume that pz(zi) is the target histogram expressed, andpr(ri) and ps(si) are, respectively, the histograms of the input andoutput image.Atransformation is needed such that the output imageps(si) should have the desired histogram of pz(zi). The first step inthis process is to equalize pr(ri) using the Eq. 3 such that1,6:

ui = T(ri) =i∑

j=0

pr(rj) for i = 0, 1, . . . , L − 1, (5)

where is ui represents the equalized gray values of the input image.A new transformation V can be defined to equalize the target

histogram such that:

vi = V(zi) =i∑

k=0

pz(zk) for i = 0, 1, . . . , L − 1. (6)

Putting V(zi) = T(ri) = ui to achieve the target distribution, newgray values si for the output image are computed from the inversetransformation V−1 as:

si = V−1[T(ri)] = V−1(ui). (7)

With the transformation defined in Eq. 7, the histogram distributionof the output image ps(si) would become similar to that of pz.

8.2.4 Image Averaging

Signal averaging is a well known method for enhancing signal-to-noise ratio. In medical imaging, data from the detector is often aver-aged over time or space for signal enhancement. However, suchsignal enhancement is achieved at the cost of some loss of temporalor spatial resolution. Sequence images, if properly registered and


180 Atam P Dhawan

acquired in non-dynamic applications, can be averaged for noisereduction leading to smoothing effects. Selective weighted averag-ing can also be performed over a specified neighborhood of pixelsin the image.

Let us assume that an ideal image f (x, y) suffers through anadditive noise n(x, y). The acquired image g(x, y) then can be rep-resented as:

g(x, y) = f (x, y) + n(x, y). (8)

In a general imaging process, the noise is assumed to be uncorrelatedand random with a zero average value. If a sequence of K imagesis acquired for the same object under the same imaging conditions,the average image g(x, y) can be obtained as:

g(x, y) = 1K

K∑i=1

gi(x, y), (9)

where gi(x, y); i = 1, 2, . . . , K represents the sequence of images tobe averaged.

As the number of images K increases, the expected value of theaverage image g(x, y) approaches to f (x, y) reducing the noise perpixel in the averaged image as:

E{g(x, y)} = f (x, y)

σg(x,y) = 1√K

σn(x,y), (10)

where σ represents the standard deviation of the respective randomfield.

8.2.4.1 Neighborhood Operations

The spatial filtering methods using neighborhood operationsinvolve the convolution of the input image with a specific mask(such as Laplacian based high frequency emphasis filtering mask)to enhance an image. The gray value of each pixel is replaced by thenew value computed according to the mask applied in the neigh-borhood of the pixel. The neighborhood of a pixel may be defined



in any appropriate manner based on a simple connectedness or anyother adaptive criterion.13

Let us assume a general weight mask of (2p+1)× (2p+1) pixelswhere p can take integer values, such as 1, 2, . . . , depending uponthe size of the mask. For p = 1, the size of the weight mask is 3 × 3pixels. A discrete convolution of an image f (x, y) with a spatial filterrepresented by a weight mask w(x, y) is given by:

g(x, y) = 1∑p

x′=−p

∑p

y′=−pw(x′, y′)

p∑x′=−p

p∑y′=−p

w(x′, y′)f (x+x′, y+y′),

(11)where the convolution is performed for all values of x and y in theimage. In other words, the weight mask of the filter is translated andconvolved over the entire extent of the input image to provide theoutput image.

The values of the weight mask are derived from a discrete repre-sentation of the selected filter. Based on the filter, the characteristicsof the input image are changed in the output image. For example,Fig. 3 shows a weighted averaging mask that can be used for imagesmoothing and noise reduction. In this mask, the pixels in the4-connected neighborhood are weighted twice than other pixels asthey are closer than others to the central pixel. The mask is used witha scaling factor of 1/16 that is multiplied to the values obtained byconvolution of the mask with the image Eq. 11.

Figure 4 shows an X-ray mammogram image smoothed by spa-tial filtering using the weighted averaging mask shown in Fig. 3.Some loss of details can be noted in the smoothed image because of

1 2 1

2 4 2

1 2 1

Fig. 3. A weighted averaging mask for image smoothing.


182 Atam P Dhawan

Fig. 4. Left: an original X-ray mammogram image; right: a smoothed image usingthe weight mask shown in Fig. 3.

the averaging operation. In order to minimize the loss of details, anadaptive median filtering may be applied.1–4

8.2.4.2 Median Filter

Median filter is a well known order-statistics filter that replaces theoriginal gray value of a pixel by the median of gray values of pixels inthe specified neighborhood. For example, for 3×3 pixels based fixedneighborhood, the gray value of the central pixel f (0, 0) is replacedby the median of gray values of all nine pixels in the neighborhood.Instead of replacing the gray value of the central pixel by the medianoperation of the neighborhood pixels, other operations such as mid-point, arithmetic mean, and geometric mean, can also be used inorder-statistics filtering methods.1–5 A median filter operation for asmoothed image f (x, y) computed from the acquired image g(x, y) is



defined as:

f (x, y) = median(i, j) ∈ N {g(i, j)}, (12)

where N is the prespecified neighborhood of the pixel (x, y).

8.2.4.3 Adaptive Arithmetic Mean Filter

Adaptive local noise reduction filtering can be applied using thevariance information of the selected neighborhood and an estimateof the overall variance of noise in the image. If the noise variance ofthe image is similar to the variance of gray values in the specifiedneighborhood of pixels, the filter provides an arithmetic mean valueof the neighborhood. Letσ2

n be an estimate of the variance of the noisein the image and σ2

s be the variance of gray values of pixels in thespecified neighborhood, an adaptive local noise reduction filteringcan be implemented as:

f (x, y) = g(x, y) − σ2n

σ2s

[g(x, y) − gms(x, y)], (13)

where gms(x, y) is the mean of the gray values of pixels in the specifiedneighborhood. This should be noted that if the noise variance is zeroin the image, the resultant image is the same as the input image. Ifan edge were present in the neighborhood, the local variance wouldbe higher than the noise variance of the image. In such cases, theabove estimate in Eq. 13 would return the value close to the originalgray value of the central pixel.

8.2.4.4 Image Sharpening and Edge Enhancement

Edges in an image are basically defined by the change in gray valuesof pixels in the neighborhood. The change of gray values of adjacentpixels in the image can be expressed by a derivative (in continuousdomain) or a difference (in discrete domain) operation.

A first-order derivative operator, such as Sobel, computes thegradient information in a specific direction. The derivative operatorcan be encoded into a weight mask. Figure 5 shows two Sobel weight


184 Atam P Dhawan

-1 -2 -1

0 0 0

1 2 1

-1 0 1

-2 0 2

-1 0 1

Fig. 5. Weight masks for first derivative operator known as Sobel. The mask atthe left is for computing gradient in the x-direction while the mask at the rightcomputes the gradient in the y-direction.

masks that are used, respectively, in computing the first-order gra-dient in x- and y-directions (defined by δ f (x, y)/δx and δ f (x, y)/δy).These weight masks of 3 × 3 pixels each are used for convolution tocompute respective gradient images. For spatial image enhancementbased on the first-order gradient information, the resultant gradientimage can simply be added to the original image and rescaled usingthe full dynamic range of gray values.

A second-order derivative operator, known as Laplacian, can bedefined as:

∇2f (x, y) = δ2f (x, y)δx2 + δ2f (x, y)

δy2

= [f (x + 1, y) + f (x − 1, y) + f (x, y + 1)

+ f (x, y − 1) − 4f (x, y)], (14)

where ∇2f (x, y) represents the second-order derivative or Laplacianof the image f (x, y).

An image can be sharpened with enhanced edge information byadding the Laplacian of the image to the original image itself. Such amask with Laplacian added to the image is shown in Fig. 6. Figure 7shows the enhanced version of the original mammographic imageshown in Fig. 4.

8.3 FREQUENCY DOMAIN FILTERING

Frequency domain filtering methods process an acquired image inthe Fourier domain to emphasize or de-emphasize specified fre-quency components. In general, the frequency components can be



-1 -1 -1

-1 9 -1

-1 -1 -1

Fig. 6. Weight masks for image enhancement through addition of Laplacian gra-dient information to the image.

Fig. 7. The original mammogram image on the left, also shown in Fig. 4(left), withthe Laplacian gradient based image enhancement shown in at the right.

expressed in low and high ranges. The low frequency range compo-nents usually represent shapes and blurred structures in the imagewhile high frequency information belongs to sharp details, edgesand noise. Thus, a low-pass filter with attenuation to high frequencycomponents would provide image smoothing and noise removal.A high-pass filtering with attenuation to low frequency extractsedges and sharp details for image enhancement and sharpeningeffects.


186 Atam P Dhawan

8.3.1 Inverse Filtering

As presented in Chapter 2, an acquired image g(x, y) can be expressedas a convolution of the object f (x, y) with a point spread function(PSF) h(x, y) of a linear spatially invariant imaging system with addi-tive noise n(x, y) as:

g(x, y) = h(x, y) ⊗ f (x, y) + n(x, y). (17)

The Fourier transform of Eq. 17, provides a multiplicative relation-ship of F(u, v), the Fourier transform of the object and H(u, v), theFourier transform of the PSF:

G(u, v) = H(u, v)F(u, v) + N(u, v), (18)

where u and v represents frequency domain along x- andy-directions, and G(u, v) and N(u, v) are respectively the Fouriertransforms of the acquired image g(x, y) and the noise n(x, y).

The object information in the Fourier domain can be recoveredby inverse filtering as:

F(u, v) = G(u, v)H(u, v)

− N(u, v)H(u, v)

, (19)

where F(u, v) is the restored image in the frequency domain.The inverse filtering operation represented in Eq. 19 provides

a basis for image restoration in the frequency domain. InverseFourier transform of F(u, v) provides the restored image in the spa-tial domain. The PSF of the imaging system can be experimentallydetermined or statistically estimated.1

8.3.2 Wiener Filtering

The image restoration approach presented in Eq. 19 appears to besimple but poses a number of challenges in practical implementa-tion. Besides the difficulties associated with the determination of thePSF, low-values or zeros in H(u, v) cause computational problems.Constrained deconvolution approaches and weighted filtering havebeen used to avoid the “division by zero” problem in Eq. 19.1–3



Wiener filtering is a well known and effective method for imagerestoration to perform weighted inverse filtering as:

F(u, v) =(

1H(u, v)

) |H(u, v)|2

|H(u, v)|2 + Sn(u,v)Sf (u,v)

G(u, v), (20)

where Sf (u, v) and Sn(u, v) are, respectively, the power spectrum ofthe signal and noise.

The Wiener filter, also known as the minimum square error fil-ter, provides an estimate determined by exact inverse filtering if thenoise spectrum is zero. In cases of non-zero signal-to-noise spec-trum ratio, the division is appropriately weighted. If the noise canbe assumed to be spectrally white, Eq. 20 reduces to a simple para-metric filter with a constant K as:

F(u, v) =[(

1H(u, v)

) ( |H(u, v)|2|H(u, v)|2 + K

)]G(u, v). (21)

In implementing inverse filtering based methods for image restora-tion, the major issue is the estimation of the PSF and noise spectra.The estimation of PSF is dependent on the instrumentation andparameters of the imaging modality. For example, in the EPI methodof MR imaging, an image formation process can be described in adiscrete representation by16:

g(x, y) =M−1∑x′=0

N−1∑y′=0

f (x′, y′)H(x′, y′; x, y), (22)

where g(x, y) is the reconstructed image of M×N pixels, f (x, y) is theideal image of the object and H(x′, y′; x, y) is the PSF of the imageformation process in EPI. The MR signal s(k, l) at a location (k, l) inthe k-space for the EPI method can be represented as:

s(k, l) =M−1∑x=0

N−1∑y=0

f (x, y)A(x, y; k, l), (23)

where

A(x, y; k, l) = e−2πj((kx/M)+(ly/N)−(γ/2π)�Bx,yTk,l), (24)


188 Atam P Dhawan

where �Bx,y is spatially variant field inhomogeneity and tk,l is thetime between the sampling of the k-space location (k, l) and the RFexcitation.

With the above representation, the PSF H(x′, y′; x, y) can beobtained from the 2D inverse FFT of the function A(x, y; k, l) as:

H(x′, y′; x, y) =M−1∑k=0

N−1∑l=0

A(x, y; k, l)e2πj((kx/M)+ly/N)

=∑ ∑

e2πj((k(x′−x)/M)+(l(y′−y)/N)−(γ/2π)�Bx,yTk,l). (25)

8.4 CONSTRAINED LEAST SQUARE FILTERING

The constrained least square filtering method uses optimizationtechniques on a set of equations representing the image formationprocess. The Eq. 6.18 can be rewritten in the matrix form as:

g = Hf + n, (26)

where g is a column vector representing the reconstructed imageg(x, y), f is a column vector of MN × 1 dimension, representing theideal image f (x, y), and n represents the noise vector. The PSF isrepresented by the matrix H of MN × MN elements.

For image restoration using the above equation, an estimate fneeds to be computed such that the mean-square error betweenthe ideal image and the estimated image is minimized. The over-all problem may not have a unique solution. Also, small variationsin the matrix H may have significant impact on the noise contentof the restored image. To overcome these problems regularizationmethods involving constrained optimization techniques are used.Thus, the optimization process is subjected to the specific constraintssuch as smoothness to avoid noisy solutions for the vector f. Thesmoothness constraint can be derived from the Laplacian for theestimated image. Using the theory of random variables, the opti-mization process is defined to estimate f such that the mean square



error, e2 given by:

e2 = Trace E{(f − f)ft},is minimized subject to the smoothness constraint involving the min-imization of the roughness or Laplacian of the estimated image asmin{ft[C][C]f},

where [C] =

1−2 11 −2 1

1 −21 ·

· 1· −2

1

. (27)

It can be shown that the estimated image f can be expressed as4:

f = ([H]t[H] + 1λ[C]t[C])−1[H]tg, (28)

where λ is a Lagrange multiplier.

8.4.1 Low-Pass Filtering

The ideal low-pass filter suppresses noise and high-frequencyinformation providing a smoothing effect to the image. A two-dimensional low-pass filter function H(u, v) is multiplied with theFourier transform G(u, v) of the image to provide a smoothedimage as:

F(u, v) = H(u, v)G(u, v), (29)

where F(u, v) is the Fourier transform of the filtered image f (x, y) thatcan be obtained by taking an inverse Fourier transform.

An ideal low-pass filter can be designed by assigning a frequencycut-off value ω0. The frequency cut-off value can also be expressed


190 Atam P Dhawan

as the distance D0 from the origin in the Fourier (frequency) domain:

H(u, v) ={

1 if D(u, v) ≤ D0

0 otherwise, (30)

where D(u, v) is the distance of a point in the Fourier domain fromthe origin representing the dc value.

An ideal low-pass filter has sharp cut-off characteristics in theFourier domain causing a rectangular window for the pass band.From Chapter 2, it can be shown that a rectangular function in thefrequency domain provides a sinc function in the spatial domain.Also, the multiplicative relationship of the filter model in Eq. 29leads to a convolution operation in the spatial domain. The rectan-gular pass-band window in the ideal low-pass filter causes ringingartifacts in the spatial domain. To reduce ringing artifacts the passband should have a smooth fall-off characteristic. A Butterworthlow-pass filter of nth order can be used to provide smoother fall-offcharacteristics and is defined as:

H(u, v) = 11 + [D(u, v)/D0]2n . (31)

As the order n increases, the fall off characteristics of the pass bandbecome sharper. Thus, a first-order Butterworth filter provides theleast amount of ringing artifacts in the filtered image.

AGaussian function is also commonly used for low-pass filteringto provide smoother fall-off characteristics of the pass band and isdefined by:

H(u, v) = e−D2(u,v)/2σ2, (32)

where D(u, v) is the distance from the origin in the frequency domainand σ represents the standard deviation of the Gaussian functionthat can be set to the cut off distance D0 in the frequency domain.In this case, the gain of the filter is down to 0.607 of its maximumvalue at the cut off frequency. Figure 8 shows a CT axial image of thechest cavity with its Fourier transform. The image was processedwith a low-pass filter with the frequency response shown in themiddle column of Fig. 8. The resultant low-pass filtered image withits Fourier transform is shown in the right column. It can be seen

January22,2008

12:2W

SPC/

SPI-B540:Principles

andR

ecentAd

vancesch08

FA

Principles

ofImage

Processing

Methods

191

Fig. 8. Left column: the original CT image with its Fourier transform; middle column: frequency response of the desired andactual low-pass filter; right column: the resultant low-pass filtered image with its Fourier transform.


192 Atam P Dhawan

that low-frequencyinformation is preserved while some of the high-frequency information is removed from the filtered image. The fil-tered image appears to be smoother.

8.4.2 High-Pass Filtering

High-pass filtering is used for image sharpening and extraction ofhigh-frequency information such as edges. The low-frequency infor-mation is attenuated or blocked depending on the design of the filter.An ideal high-pass filter has a rectangular window function for thehigh-frequency pass-band. Since the noise in the image usually car-ries high-frequency components, high-pass filtering also shows thenoise along with edge information. An ideal 2D high-pass filter witha cut-off frequency at a distance D0 from the origin in the frequencydomain is defined as:

H(u, v) ={

1 if D(u, v) ≥ D0

0 otherwise. (33)

As described above for an ideal low-pass filter, the sharp cut-offcharacteristic of the rectangular window function in the frequencydomain as defined in Eq. 33 causes the ringing artifacts in the fil-tered image in the spatial domain. To avoid ringing artifacts filterfunctions with smoother fall-off characteristics such as Butterworthand Gaussian are used. A Butterworth high-pass filter of n-th orderis defined in the frequency domain as:

H(u, v) = 11 + [D0/D(u, v)]2n . (34)

Figure 9 shows a CT axial image of the chest cavity with itsFourier transform. The image was processed with a high-pass filterwith the frequency response shown in the middle column of Fig. 9.The resultant high-pass filtered image with its Fourier transformis shown in the right column. It can be seen that the low-frequencyinformation is attenuated or de-emphasized in the high-pass filteredimage. High-frequency information belonging to the edges can beseen in the filtered image.

January22,2008

12:2W

SPC/

SPI-B540:Principles

andR

ecentAd

vancesch08

FA

Principles

ofImage

Processing

Methods

193

Fig. 9. Left column: the original CT image with its Fourier transform; middle column: frequency response of the desired andactual high-pass filter; right column: the resultant high-pass filtered image with its Fourier transform.


194 Atam P Dhawan


Image processing operations such as noise removal, averaging,filtering and feature enhancement are critically important in com-puterized image analysis for feature characterization, analysisand classification. These operations are also important to helpvisual examination and diagnostic evaluation for medical applica-tions. Though the basic image processing operations as describedin this chapter are quite efficient and effective, more sophis-ticated model-based methods have been developed for image-specific feature enhancement operations. These methods utilizea priori information about the statistical distribution of gray-levelfeatures in the context of a specific application. Such methodsare useful in enhancing the signal-to-noise ratio of the acquiredimage for better analysis and classification of medical images.Some of the recently developed image processing methods aredescribed in various chapters in the second and third part ofthis book.

References

1. Jain AK, Fundamentals of Digital Image Processing, Prentice Hall, 1989.2. Gonzalez RC, Woods RE, Digital Image Processing, Prentice Hall, 2002.3. Jain R , Kasturi R, Schunck BG, Machine Vision, McGraw-Hill, 1995.4. Rosenfeld A, Kak AV, Digital Picture Processing, 1 & 2, 2nd edn.

Academic Press, 1982.5. Russ JC, The Image Processing Handbook, 2nd edn., CRC Press, 1995.6. Schalkoff RJ, Digital Image Processing and Computer Vision,

John Wiley & Sons, 1989.7. Kao Y H, MacFall JR, Correction of MR-k space data corrupted by

spike noise, IEEE Trans Med Imag 19: 671–680, 2000.8. Ahmed OA, Fahmy MM, NMR signal enhancement via a new time

frequency transform, IEEE Trans Med Imag 20: 1018–1025, 2001.9. Goutte C, Nielson FA, Hansen LK, Modeling of Hemodynamic

response in fMRI using smooth FIR filters, IEEE Trans Med Imag 19:1188–1201, 2000.

10. Zaroubi S, Goelman G, Complex denoising of MR data via waveletanalysis: Applications for functional MRI, Mag Reson Imag 18: 59–68,2000.



11. Davis GW, Wallenslager ST, Improvement of chest region CTimages through automated gray-level remapping, IEEE Trans MedImaging 1–5: 30–35, 1986.

12. Pizer SM, Zimmerman JB, Staab EV, Adaptive gray-level assignmentin CT scan display, J Comput Assist Tomog 8: 300–306, 1984.

13. Dhawan AP, LeRoyer E, Mammographic feature enhancement bycomputerized image processing, Comp Methods & Programs in Biomed27: 23–29, 1988.

14. Kim JK, Park JM, Song KS, Park HW, Adaptive mammographicimage enhancement using first derivative and local statistics, IEEETrans Med Imag 16: 495–502, 1997.

15. Chen G, Avram H, Kaufman L, Hale J, et al., T2 restoration andnoise suppression of hybrid MR images using Wiener and linearprediction techniques, IEEE Trans Med Imag 13: 667–676, 1994.

16. Munger P, Crelier GR, Peters TM, Pike GB, An inverse problemapproach to the correction of distortion in EPI images, IEEE TransMed Imag 19: 681–689, 2000.

17. DhawanAP, Medical Image Analysis, Wiley Interscience, John Wiley andSons, Hoboken, NJ, 2003.


CHAPTER 9

Image Segmentation and FeatureExtraction

Atam P Dhawan

Medical image segmentation tasks are important to visualize features ofinterests such as lesions with boundary and volume information. Simi-lar information is required in the computerized quantitative analysis andclassification for diagnostic evaluation and characterization. This chap-ter presents some of the most effective and commonly used edge andregion segmentation methods. Statistical quantitative features from graylevel distribution, segmented regions, and texture in the image are alsopresented.

9.1 INTRODUCTION

After an image is processed for noise removal, restoration and fea-ture enhancement as needed, it is important to analyze the imagefor extraction of features of interest involving edges, regions, texture,etc. for further analysis. This goal is accomplished by image segmen-tation task. Image segmentation refers to the process of partitioningan image into distinct regions by grouping together neighborhoodpixels based on a predefined similarity criterion. The similarity cri-terion can be determined using specific properties or features ofpixels representing objects in the image. Thus, image segmentationcan also be considered as a pixel classification technique that allowsan edge or region based representation towards the formation ofregions of similarities in the image. Once the regions are defined, sta-tistical and other features can be computed to represent regions for

197


198 Atam P Dhawan

characterization, analysis and classification. This chapter describesmajor image segmentation methods for medical image analysis andclassification.

9.2 EDGE-BASED IMAGE SEGMENTATION

Edge-based approaches use spatial filtering methods to compute thefirst-order or second-order gradient information of the image. Thereare a number of gradient operators that can be used for edge-basedsegmentation. These operators include Roberts, Sobel, Laplacian,Canny and others.1–5 Some involve directional derivative masks thatare used to compute gradient information. The Laplacian mask canbe used to compute second-order gradient information of the image.For segmentation purposes, after edges are extracted, an edge link-ing algorithm is applied to form closed regions.1–3 Gradient infor-mation of the image can be used to track and link relevant edges.This step is usually very tedious for it needs to deal with the noiseand irregularities in the gradient information.

9.2.1 Edge Detection Operations

The gradient magnitude and directional information from the Sobelhorizontal and vertical direction masks can be obtained by convolv-ing the respective Gx and Gy masks with the image as1–2:

Gx =−1 0 1

−2 0 2−1 0 1

Gy = 1 2 1

0 0 0−1 −2 −1

(1)

M =√

G2x + G2

Y ≈ |Gx| + |GY|,where M represents the magnitude of the gradient that can beapproximated as the sum of the absolute values of the horizontal


Image Segmentation and Feature Extraction 199

and vertical gradient images obtained by convolving the image withthe horizontal and vertical masks, Gx and Gy.

The second order gradient operator Laplacian can be computedby convolving one of the following masks, Gl(4) and GL(8), which,respectively, use a 4- and 8-connected neighborhood.

GL(4) = 0 −1 0

−1 4 −10 −1 0

or

GL(8) =−1 −1 −1

−1 8 −1−1 −1 −1

(2)

The second-order derivative, Laplacian is very sensitive to noiseas it can be seen from the distribution of weights in the masks in Eq. 2.The Laplacian mask provides a non-zero output even for a singlepixel based speckle noise in the image. Therefore, it is usually bene-ficial to apply a smoothing filter first before taking a Laplacian of theimage. The image can be smoothed using a Gaussian weighted spa-tial averaging as the first step. The second step then uses a Laplacianmask to determine edge information. Marr and Hildreth3 combinedthese two steps into a single Laplacian of Gaussian function as:

h(x, y) = ∇2[g(x, y) ⊗ f (x, y)]= ∇2[g(x, y)] ⊗ f (x, y), (3)

where ∇2[g(x, y)] is the Laplacian of the Gaussian function thatis used for spatial averaging and is commonly expressed as theMexican Hat operator:

∇2[g(x, y)] =(

x2 + y2 − 2σ2

σ4

)e

(x2+y2)2σ2 , (4)

where σ2 is the variance of the Gaussian function.A Laplacian of Gaussian (LOG) mask for computing the second-

order gradient information of the smoothed image can be computed


200 Atam P Dhawan

from Eq. 4. With σ = 2, the LOG mask GLOG of 5 × 5 pixels isgiven by:

GLOG =

0 0 −1 0 00 −1 −2 −1 0

−1 −2 16 −2 −10 −1 −2 −1 00 0 −1 0 0

. (5)

The image obtained by convolving the LOG mask with the orig-inal image is analyzed for zero crossing to detect edges since theoutput image provides values from negative to positive values. Onesimple method to detect zero crossing is to threshold the outputimage for zero value. This operation provides a new binary imagesuch that a “0” gray value is assigned to the binary image if theoutput image has a negative or zero value for the correspondingpixel. Otherwise, a high gray value (such as “255” for an 8 bitimage) is assigned to the binary image. The zero crossing of theoutput image can now be easily determined by tracking the pixelswith a transition from black ( “0” gray value) to white (“255” grayvalue).

9.2.1.1 Boundary Tracking

Edge detection operations are usually followed up by the edge-linking procedures to assemble meaningful edges to form closedregions. Edge-linking procedures are based on pixel-by-pixel searchto find connectivity among the edge segments. The connectivity canbe defined using a similarity criterion among edge pixels. In addi-tion, geometrical proximity or topographical properties are usedto improve edge-linking operations for pixels that are affected bynoise, artifacts or geometrical occlusion. Estimation methods basedon probabilistic approaches, graphs and rule-based methods formodel-based segmentation have also been used.4–12

In the neighborhood search methods, the simplest method isto follow the edge detection operation by a boundary-tracking



algorithm. Let us assume that the edge detection operationproduces an edge magnitude e(x, y) and an edge orientation φ(x, y)information. The edge orientation information can be directlyobtained from the directional masks, as described in chapter 6, orcomputed from the horizontal and vertical gradient masks. Let usstart with a list of edge pixels that can be selected from scanning thegradient image obtained from the edge detection operation. Assum-ing the first edge pixel as a boundary pixel bj, a successor boundarypixel bj+1 can be found in the 4- or 8-connected neighborhood if thefollowing conditions are satisfied:

|e(bj)| > T1

|e(bj+1)| > T1

|e(bj) − e(bj+1)| < T2

|φ(bj) − φ(bj+1)|mod 2π < T3,

(6)

where T1, T2, and T3 are pre-determined thresholds.If there is more than one neighboring pixel satisfying these con-

ditions, the pixel that minimizes the differences is selected as thenext boundary pixel. The algorithm is recursively applied until allneighbors are searched. If no neighbor is found satisfying these con-ditions, the boundary search for the striating edge pixel is stoppedand a new edge pixel is selected. It can be noted that such a boundarytracking algorithm may leave many edge pixels and partial bound-aries unconnected. Some a priori knowledge about the object bound-aries is often needed to form regions with closed boundaries. Also,relational tree structures or graphs can be used to help the formationof closed regions.13–14

A graph-based search method attempts to find paths betweenthe start and end nodes minimizing a cost function that may beestablished based on the distance and transition probabilities. Thestart and end nodes are determined from scanning the edge pixelsbased on some heuristic criterion. For example, an initial search maylabel the first edge pixel in the image as the start node and all theother edge pixels in the image or a part of the image as potential


202 Atam P Dhawan

end nodes. Among several graph-based search algorithms, the A*algorithm is widely used.13–15

9.3 PIXEL-BASED DIRECT CLASSIFICATION METHODS

The pixel-based direct classification methods use histogram statisticsto define single or multiple thresholds to classify an image pixel-by-pixel. The threshold for classifying pixels into classes is obtainedfrom the analysis of the histogram of the image.Asimple approach isto examine the histogram for bimodal distribution. If the histogramis bimodal, the threshold can be set to the gray value correspondingto the deepest point in the histogram valley. If not, the image canbe partitioned into two or more regions using some heuristics aboutthe properties of the image. The histogram of each partition can thenbe used for determining thresholds. By comparing the gray value ofeach pixel to the selected threshold, a pixel can be classified into oneof the two classes.

Let us assume that an image or a part of the image has a bimodalhistogram of gray values. The image f (x, y) can be segmented intotwo classes using a gray value threshold T such that:

g(x, y) ={

1 if f (x, y) > T0 if f (x, y) ≤ T

(7)

where g(x, y) is the segmented image with two classes of binary grayvalues “1” and “0” and T is the threshold selected at the valley pointfrom the histogram. A simple approach to determine the gray valuethreshold T is by analyzing the histogram for the peak values andthen finding the deepest valley point between the two consecutivemajor peaks.

9.3.1 Optimal Global Thresholding

To determine an optimal global gray value threshold for image seg-mentation, parametric distribution based methods can be applied tothe histogram of an image.1,2,5,15 Let us assume that the histogram ofan image to be segmented has two Gaussian distributions belong-ing to two respective classes such as background and object. Thus,



the histogram can be represented by a mixture probability densityfunction p(z) as:

p(z) = P1p1(z) + P2p2(z), (8)

where p1(z) and p2(z) are the Gaussian distributions of class 1 and 2,respectively, with the class probabilities of P1 and P2 such that:

P1 + P2 = 1. (9)

Using a gray value threshold T, a pixel in the image f (x, y) can beclassified to class 1 or class 2 in the segmented image g(x, y) as:

g(x, y) ={

Class1 if f (x, y) > TClass2 if f (x, y) ≤ T

. (10)

Let us define the error probabilities of misclassifying a pixel as:

E1(T) =∫ T

−∞p2(z)dz

and

E2(T) =∫ T

−∞p1(z)dz, (11)

where E1(T) and E2(T) are, respectively, the probability of erro-neously classifying a class 1 pixel to class 2 and a class 2 pixel toclass 1.

The overall probability of error in pixel classification using thethreshold T is then expressed as:

E(T) = P2(T)E1(T) + P1(T)E2(T). (12)

For image segmentation, the objective is to find an optimalthreshold T that minimizes the overall probability of error in pixelclassification. The optimization process requires the parameteriza-tion of the probability density distributions and likelihood of bothclasses. These parameters can be determined from a model or set oftraining images.1,2,15,19,24

Let us assume σi and µi to be the standard deviation and meanof the Gaussian probability density function of the class i (i = 1, 2


204 Atam P Dhawan

for two classes) such that:

p(z) = P1√2πσ1

e−(z−µ1)2/2σ21 + P2√

2πσ2e−(z−µ2)2/2σ2

2 . (13)

The optimal global threshold T can be determined by finding a gen-eral solution that minimizes Eq. 12 with the mixture distribution inEq. 13 and thus satisfies the following quadratic expression2:

AT2 + BT + C = 0,

where

A = σ21 − σ2

2

B = 2(µ1σ22 − µ2σ

21 )

C = σ21µ2

2 − σ22µ2

1 + 2σ21σ2

2 ln (σ2P1/σ1P2). (14)

If the variances of both classes can be assumed to be equal to σ2, theoptimal threshold T can be determined as:

T = µ1 + µ2

2+ σ2

µ1 − µ2ln

(P2

P1

). (15)

It should be noted that in case of equal likelihood of classes, the aboveexpression for determining the optimal threshold is simply reducedto the average of the mean values of two classes. Figure 1 shows the

Fig. 1. Segmentation of a T-2 weighted MR brain image (shown at the left) usingoptimal thresholding method at T = 54 yielding the binary segmented image shownat the right.



results of the optimal thresholding method applied to a T-2 weightedMR brain image. It can be seen that most of such a segmentationmethod is quite effective in determining the intercranial volume.

9.3.2 Pixel Classification Through Clustering

In histogram based pixel classification method for image segmen-tation, the gray values are partitioned into two or more clustersdepending on the peaks in the histogram to obtain thresholds. Thebasic concept of segmentation by pixel classification can be extendedto clustering the gray values or feature vector of pixels in the image.This approach is particularly useful when images with pixels repre-senting a feature vector consisting of multiple parameters of interestare to be segmented. For example, a feature vector may consist ofgray value, contrast and local texture measures for each pixel in theimage. A color image may have additional color components in aspecific representation such as red, green and blue components inthe R-G-B color coordinate system that can be added to the featurevector. Magnetic Resonance (MR) or multimodality medical imagesmay also require segmentation using a multidimensional featurespace with multiple parameters of interest.

Images can be segmented by pixel classification through cluster-ing of all features of interest. The number of clusters in the multi-dimensional feature space thus represents the number of classes inthe image. As the image is classified into cluster classes, segmentedregions are obtained by checking the neighborhood pixels for thesame class label. However, clustering may produce disjoint regionswith holes or regions with single pixel. After the image data is clus-tered and pixels are classified, a post-processing algorithm such asregion growing, pixel connectivity or rule-based algorithm is usu-ally applied to obtain the final segmented regions.21,37 There are anumber of algorithms developed for clustering in the literature andused for a wide range of applications.15,20,21,36–41

Clustering is the process of grouping data points with similarfeature vectors together in a single cluster while data points withdissimilar feature vectors are placed in different clusters. Thus, the


206 Atam P Dhawan

data points that are close to each other in the feature space are clus-tered together. The similarity of feature vectors can be represented byan appropriate distance measure such as Euclidean or Mahalanobisdistance.42 Each cluster is represented by its mean (centeroid) andvariance (spread) associated with the distribution of the correspond-ing feature vectors of the data points in the cluster. The formation ofclusters is optimized with respect to an objective function involvingprespecified distance and similarity measures along with additionalconstraints such as smoothness.

9.3.2.1 k-Means Clustering

The k-means clustering is a popular approach to partitiond-dimensional data into k clusters such that an objective functionproviding the desired properties of the distribution of feature vectorsof clusters in terms of similarity and distance measures is optimized.Ageneralized k-means clustering algorithm initially places k clustersat arbitrarily selected cluster centroids vi; i = 1, . . . 2, k and modifiescentroids for the formation of new cluster shapes optimizing theobjective function. The k-means clustering algorithm includes thefollowing steps:

(1) Select the number of clusters k with initial cluster centroids vi;i = 1, . . . 2, k.

(2) Partition the input data points into k clusters by assigning eachdata point xj to the closest cluster centroid vi using the selecteddistance measure, e.g. Euclidean distance, defined as:

dij = ‖xj − vi‖, (16)

where X = {x1, x2, . . . , xn} is the input data set.(3) Compute a cluster assignment matrix U representing the parti-

tion of the data points with the binary membership value of thej-th data point to the i-th cluster such that:

U = �uij,



where

uij ∈ {0, 1} for all i, jk∑

i=1

uij = 1 for all j and 0 <

n∑j=1

uij < n for all i. (17)

(4) Recompute the centroids using the membership values as:

vi =∑n

j=1 uijxj∑nj=1 uij

for all i. (18)

(5) If cluster centroids or the assignment matrix does not changefrom the previous iteration, stop; otherwise go to step 2.

The k-means clustering method optimizes the sum-of-squared-errorbased objective function Jw(U, v) such that:

Jw(U, v) =k∑

i=1

n∑j=1

‖xj − vi‖2. (19)

It can be noted from the above algorithm that the k-means clus-tering method is quite sensitive to the initial cluster assignmentand the choice of the distance measure. Additional criterion suchas within-cluster and between-cluster variances can be included inthe objective function as constraints to force the algorithm to adaptthe number of clusters k (as needed for optimization of the objectivefunction).

9.3.2.2 Fuzzy c-Means Clustering

The k-means clustering method utilizes the hard binary values forthe membership of a data point to the cluster. The fuzzy c-meansclustering method utilizes an adaptable membership value that canbe updated based using the distribution statistics of the data pointsassigned to the cluster minimizing the following objective function


208 Atam P Dhawan

Jm(U, v):

Jm(U, v) =c∑

i=1

n∑j=1

umij d2

ij =c∑

i=1

n∑j=1

umij ‖xj − vi‖, (20)

where c is the number of clusters, n the number of data vectors,uij is the fuzzy membership and m is the fuzziness index. Basedon the constraints defined on the distribution statistics of the datapoints in the clusters, fuzziness index can be defined between 1 and avery large value for the highest level of fuzziness (maximum allow-able variance within a cluster). The membership values in the fuzzyc-means algorithm can be defined as36:

0 ≤ uij ≤ 1 for all i, jc∑

i=1

uij = 1 for all j and 0 <

n∑j=1

uij < n for all i. (21)

The algorithm described for k-means clustering can be used for fuzzyc-means clustering with the update of the fuzzy membership valuesas defined in Eq. 21 minimizing the objective function as defined inEq. 20.

Figure 2 shows the results of k-means clustering on a T-2weighted MR brain image with k = 9. Different regions segmentedfrom selected clusters are shown in Fig. 2.

Fig. 2(A). A T-2 weighted MR brain image used for segmentation in Fig. 2(B).



Fig. 2(B). Results of segmentation of the image shown in Fig. 2(A) using k-meansclustering algorithm with k = 9; top left: all segmented regions belonging to all9 clusters; top middle: regions segmented from cluster k = 1; top right: regionssegmented from cluster k = 4; bottom left: regions segmented from cluster k = 5;bottom middle: regions segmented from cluster k = 6; bottom right: regions seg-mented from cluster k = 9. (Courtesy DonAdams,Arwa Gheith and Valerie Rafalkofrom their class project.)

9.4 REGION-BASED SEGMENTATION

Region-growing based segmentation algorithms examine pixels inthe neighborhood based on a predefined similarity criterion andthen assign pixels into groups to form regions. The neighborhoodpixels with similar properties are merged to form closed regionsfor segmentation. The region growing approach can be extended tomerging regions instead of merging pixels to form larger meaning-ful regions of similar properties. Such a region merging approach isquite effective when the original image is segmented into a largenumber of regions in the preprocessing phase. Large meaning-ful regions may provide a better correspondence and matching tothe object models for recognition and interpretation. An alternateapproach is region splitting in which either the entire image or large


210 Atam P Dhawan

regions are split into two or more regions based on a heterogene-ity or dissimilarity criterion. For example, if a region has a bimodaldistribution of gray value histogram, it can be split into two regionsof connected pixels with gray values falling in their respective dis-tributions. The basic difference between the region and threshold-ing based segmentation approaches is that region-growing methodsguarantee the segmented regions of connected pixels. On the otherhand, pixel thresholding-based segmentation methods as defined inthe previous section may yield regions with holes and disconnectedpixels.

9.4.1 Region-growing

Region-growing methods merge pixels of similar properties byexamining the neighborhood pixels. The process of merging pixelscontinues with the growth of region adapting a new shape and sizeuntil there is insufficient number of neighborhood pixels to be addedin the current region. Thus, the region-growing process requires asimilarity criterion that defines the basis for inclusion of pixels inthe growth of the region; and a stopping criterion that stops thegrowth of the region. The stopping criterion is usually based on theminimum number or percentage of neighborhood pixels requiredto satisfy the similarity criterion for inclusion in the growth of theregion.

In the region-merging algorithms, an image may be partitionedinto a large number of potential homogeneous regions. For exam-ple, an image of 1024 × 1024 pixels can be portioned into regions of8 × 8 pixels. Each region of 8 × 8 pixels can now be examined forhomogeneity of predefined property such as gray values, contrast,texture, etc. If the histogram of the predefined property for the regionis unimodal, the region is said to be homogeneous. Two neighbor-hood regions can be merged if they are homogeneous and satisfya predefined similarity criterion. The similarity criterion imposesconstraints on the value of the property with respect to its meanand variance values. For example, two homogeneous regions canbe merged if the difference in their mean gray values is within 10%



of the entire dynamic range and the difference in their variances iswithin 10% of the variance in the image. These thresholds may beselected heuristically or through probabilistic models.10,15 It is inter-esting to note that the above criterion can be easily implemented asa conditional rule in a knowledge-based system. Region-merging orregion-splitting (described in the next section) methods have beenimplemented using a rule based system for image segmentation.25

Model-based systems typically encode knowledge of anatomyand image acquisition parameters. Anatomical knowledge can bemodeled symbolically, describing the properties and relationshipsof individual structures, or geometrically either as masks or tem-plates of anatomy, or using an atlas.18,19,25–26 Figure 3 shows a MRbrain image and the segmented regions for ventricles. The knowl-edge of anatomical locations of ventricles was used to establish initialseed points for region growing. A feature adaptive region growingmethod was used for segmentation.

9.4.2 Region-splitting

Region-splitting methods examine the heterogeneity of a predefinedproperty of the entire region in terms of its distribution and the mean,

Fig. 3(A). A T-2 weighted MR brain image used for ventricle segmentation usinga region growing approach.26


212 Atam P Dhawan

Fig. 3(B). Segmented ventricle regions of image shown in Fig. 3(A) using a mod-elbased region growing algorithm.26

variance, minimum and maximum values. If the region is evaluatedas heterogeneous, that is it fails the similarity or homogeneity crite-rion, the original region is split into two or more regions. The region-splitting process continues until all regions satisfy the homogeneitycriterion individually. In the region-splitting process, the originalregion R is split into R1, R2, . . . ., Rn subregions such that the follow-ing conditions are met2,5:

(1) Each region, Ri; i = 1, 2, . . . , n is connected.

(2)n⋃

i=1Ri = R

(3) Ri⋂

Rj = O for all i, j; i �= j(4) H(Ri) = TRUE for i = 1, 2, . . . , n.(5) H(Ri

⋃Rj) = FALSE for i �= j,

where H(Ri) is a logical predicate for the homogeneity criterion onthe region Ri.

Region-splitting methods can also be implemented by rule-based systems and quad-trees. In the quad-tree based region-splitting method, the image is partitioned into four regions that arerepresented by nodes in a quad tree. Each region is checked for thehomogeneity and evaluated for the logical predicate H(Ri). If the



region is homogeneous, no further action is taken for the respectivenode. If the region is not homogeneous, it is further split into fourregions.

9.5 RECENT ADVANCES IN SEGMENTATION

The problem of segmenting medical images into anatomically andpathologically meaningful regions has been addressed using variousapproaches including model-based estimation methods and rule-based systems.17–27 Nevertheless, automatic (or semi-automaticwith minimal operator interaction) segmentation methods for spe-cific applications are still current topics of research. This is due to thelarge variability in anatomical structures and challenging needs ofa reliable, accurate, and diagnostically useful segmentation. A rulebased low-level segmentation system for automatic identificationof brain structures from MR images has been described by Raya.17

Neural network based classification approaches have also beenapplied for medical image segmentation.10,28 A multi-level adap-tive segmentation method (MAS) was used to segment and classifymultiparameter MR brain images into a large number of classes ofphysiological and pathological interest.24 The MAS method is basedon estimation of signatures for each segmentation class for pixel-by-pixel classification.

9.6 IMAGE SEGMENTATION USING NEURAL NETWORKS

Neural networks provide another pixel classification paradigm thatcan be used for image segmentation.10,28–29 Neural networks do notrequire underlying class probability distribution for accurate clas-sification. Rather, the decision boundaries for pixel classificationare adapted through an iterative training process. Neural networkbased segmentation approaches may provide good results for med-ical images with considerable variance in structures of interest. Forexample, angiographic images show a significant variation in arte-rial structures and therefore are difficult to segment. The variation


214 Atam P Dhawan

in image quality among various angiograms and the introductionof noise in the course of image acquisition emphasizes the impor-tance of an adaptive non-parametric segmentation method. Neuralnetwork paradigms such as Backpropagation, Radial Basis Func-tion and Self-Organizing Feature Maps have been used to segmentmedical images.10,28–35

Neural networks learn from examples in the training set inwhich the pixel classification task has already been performed usingmanual methods. A non-linear mapping function between the inputfeatures and the desired output for labeled examples is learnedby neural networks without using any parameterization. After thelearning process, a pixel in a new image can be classified for seg-mentation by the neural network.

It is important to select a meaningful set of features to provideas input to the neural network for classification. The selection oftraining examples is also very important, as they should representa reasonably complete statistical distribution of the input data. Thearchitecture of the network and the distribution of training examplesplay a major role in determining its performance for accuracy, gen-eralization and robustness. In its simplest form, the input to a neuralnetwork can be the gray values of pixels in a predefined neighbor-hood in the image. Thus, the network can classify the center pixel ofthe neighborhood based on the information of the entire set of pixelsin the corresponding neighborhood. As the neighborhood windowis translated in the image, the pixels in the central locations of thetranslated neighborhoods are classified. Neural network architec-ture and learning methods are described in Chapter 10 for patternclassification that can be used for pixel-based classification for imagesegmentation.28–35

9.7 FEATURE EXTRACTION AND REPRESENTATION

Gray-level statistics of the image, gray-level statistics and shape ofthe segmented regions, and texture can be used in feature represen-tation of the image for characterization, analysis and classification.Selection of correlated features for a specific classification task is veryimportant. Details about clustering and classification are provided



in Chapter 10 of this book. Various commonly used features in imageanalysis and classification are briefly described below.

9.7.1 Statistical Pixel-Level Image Features

Once the regions are segmented in the image, gray values of pixelswithin the region can be used for computing the following statisticalpixel-level (SPL) features1–2:

(1) The histogram of the gray values of pixels in the image as:

p(ri) = n(ri)n

, (22)

where p(ri) and n(ri) are, respectively, the probability and num-ber of occurrence of a gray value ri in the region and n is the totalnumber of pixels in the region.

(2) Mean m of the gray values of the pixels in the image can becomputed as:

m = 1n

L−1∑i=0

rip(ri), (23)

where L is the total number gray values in the image with0, 1, . . . , L − 1.

(3) Variance and central moments in the region can be computed as:

µn =L−1∑i=0

p(ri)(ri − m)n, (24)

where the second central momentµ2 is the variance of the region.The third and fourth central moments can be computed, respec-tively, for n = 3 and n = 4. The third central moment is a measureof non-centrality while the fourth central moment is a measureof flatness of the histogram.

(4) Energy: Total energy E of the gray-values of pixels in the regionis given by:

E =L−1∑i=0

[p(ri)]2. (25)


216 Atam P Dhawan

(5) Entropy: The entropy Ent as a measure of information repre-sented by the distribution of gray-values in the region is given by:

Ent =L−1∑i=0

p(ri) log2(ri). (26)

(6) Local contrast corresponding to each pixel can be computed bythe difference of the gray-value of the center pixel and the meanof the gray values of the neighborhood pixels. The normalizedlocal contrast C(x, y) for the center pixel can also be computed as:

C(x, y) = |Pc(x, y) − Ps(x, y)|max{Pc(x, y), Ps(x, y)} , (27)

where Pc(x, y) and Ps(x, y) are the average gray-level values of thepixels corresponding to the “center” and the “surround” regionsthat are grown around the centered pixel through a region grow-ing method.5,45

(7) Additional features such as maximum and minimum gray val-ues can also be used for representing regions.

(8) The features based on the statistical distribution of local contrastvalues in the region also provide useful characteristics informa-tion about the regions representing objects.

(9) Features based on the gradient information for the boundarypixels of the region are also an important consideration in defin-ing the nature of edges. For example, the fading edges with lowgradient form a characteristic feature of malignant melanomaand must be included in the classification analysis of images ofskin lesions.9

9.7.2 Shape Features

Shape features of the segmented region can also be used in classifica-tion analysis. The shape of a region is basically defined by the spatialdistribution of boundary pixels. A simple approach for computingshape features for a 2D region is representing circularity, compact-ness, elongatedness through the minimum bounded rectangle thatcovers the region.1–5



Several shape features using the boundary pixels of the seg-mented region can be computed as:

(1) Longest axis.(2) Shortest axis.(3) Perimeter and area of the minimum bounded rectangle.(4) Elongation ratio.(5) Perimeter p and area A of the segmented region.(6) Hough transform of the region using the gradient information

of the boundary pixels of the region1−5 [also described later inthis chapter].

(7) Circularity (C = 1 for a circle) of the region computed as:

C = 4πAp2 . (28)

(8) Compactness Cp of the region computed as:

Cp = p2

A. (29)

(9) Chain code for boundary contour as obtained using a set oforientation primitives on the boundary segments derived froma piecewise linear approximation.

(10) Fourier descriptor of boundary contours as obtained usingthe Fourier transform of the sequence of boundary segmentsderived from a piecewise linear approximation.

(11) Central moments based shape features for the segmentedregion.

(12) Morphological shape descriptors as obtained though the mor-phological processing on the segmented region.46–51

9.7.3 Moments for Shape Description

The shape of a boundary or contour can be represented quantita-tively by the central moments for matching. The central momentsrepresent specific geometrical properties of the shape and are invari-ant to the translation, rotation and scaling. The central moments µpq


218 Atam P Dhawan

of a segmented region or binary image f (x, y) are given by1,2,5,52:

µpq =L∑

i=1

L∑j=1

(xi − x)p(yj − y)qf (x, y)

where

x =L∑

i=1

L∑j=1

xif (xi, yj),

y =L∑

i=1

L∑j=1

yjf (xi, yj). (30)

For example, the central moment µ21 represents the vertical diver-gence of the shape of the region indicating the relative extent of thebottom of the region compared to the top. The normalized centralmoments can be computed as:

ηpq = µpq

(µ00)γ,

where

γ = p + q2

+ 1. (31)

There are seven invariant moments φ1– φ7 for shape matching aredefined as52:

φ1 = η20 + η02

φ2 = (η20 − η02)2 + 4η211

φ3 = (η30 − 3η12)2 + (3η21 − η03)2

φ4 = (η30 + η12)2 + (η21 + η03)2

φ5 = (η30 − 3η12)(η30 + η12)[(η30 + η12)2 − 3(η21 + η03)2]+ (3η21 − η03)(η21 + η03)[3(η30 + η12)2 − (η21 + η03)

φ6 = (η20 − η02)[(η30 + η12)2 − (η21 + η03)2]+ 4η11(η30 + η12)(η21 + η03)

φ7 = (3η21 − η03)(η30 + η12)[(η30 + η12)2 − 3(η21 − η03)2]+ (3η12 − η30)(η21 + η03)[3(η30 + η12)2 − (η21 + η03). (32)



The invariant moments are used extensively in the literature forshape matching and pattern recognition.1,52

9.7.4 Texture Features

Texture is an important spatial property that can be used inregion segmentation as well as description. There are three majorapproaches to represent texture: statistical, structural and spec-tral. Since texture is a property of the spatial arrangements ofthe gray values of pixels, the first order histogram of gray val-ues provide no information about the texture. Statistical methodsrepresenting the higher order distribution of gray values in theimage are used for texture representation. The second approachuses structural methods such as arrangements of prespecified prim-itives in texture representation. For example, a repetitive arrange-ment of square and triangular shapes can produce a specifictexture. The third approach is based on spectral analysis meth-ods such as Fourier and wavelet transforms. Using spectral anal-ysis, texture is represented by a group of specific spatiofrequencycomponents.53,54

The gray-level co-occurrence matrix (GLCM) exploits the higherorder distribution of gray values of pixels that are defined with aspecific distance or neighborhood criterion. In the simplest form,the GLCM P(i, j) is the distribution of the number of occurrenceof a pair of gray values i and j separated by a distance vectord = [dx,dy].

The GLCM can be normalized by dividing each value in thematrix by the total number of occurrences providing the probabilityof occurrence of a pair of gray values separated by a distance vec-tor. Statistical texture features are computed from the normalizedGLCM as the second order histogram H(yq, yr,d) representing theprobability of occurrence of a pair of gray values yq and yr separatedby a distance vector d. Texture features can also be described by adifference histogram, Hd(ys,d), where ys = |yq − yr|. Hd(ys,d) indi-cates the probability that a difference in gray-levels exists between


220 Atam P Dhawan

two distinct pixels. Commonly used texture features based on thesecond order histogram statistics are as follows:

(1) Entropy of H(yq, yr, d), SH :

SH = −yt∑

yq=y1

yt∑yr=y1

H(yq, yr, d)log10[H(yq, yr, d)]. (33)

The entropy is a measure of texture nonuniformity. Lowerentropy values indicate greater structural variation among theimage regions.

(2) Angular Second Moment of H(yq, yr, d), ASMH :

ASMH =yt∑

yq=y1

yt∑yr=y1

[H(yq, yr, d)]2. (34)

The ASMH indicates the degree of homogeneity among tex-tures, and is also representative of the energy in the image (11).A lower value of ASMH is indicative of finer textures.

(3) Contrast of H(yq, yr, d):

Contrast =yt∑

yq=y1

yt∑yr=y1

∂(yq, yr)H(yq, yr, d), (35)

where ∂(yq, yr) is a measure of intensity similarity and is definedby ∂ = (yq − yr)2. Thus the contrast characterizes the extent ofvariation in pixel intensity.

(4) Inverse Difference Moment of H(yq, yr, d), IDMH :

IDMH =yt∑

yq=y1

yt∑yr=y1

H(yq, yr, d)1 + ∂(yq, yr)

, (36)

where δ is defined as before. The IDMH provides a measure ofthe local homogeneity among textures.

(5) Correlation of H(yq, yr, d):

CorH = 1σyqσyr

yt∑yq=y1

yt∑yr=y1

(yq − µyq)(yr − µyr)H(yq, yr, d), (37)

where µyq , µyr , σyq , σyr and are the respective means and stan-dard deviations of yq and yr. The correlation can also be



expanded and written in terms of the marginal distributionsof the second order histogram, which are defined as:

Hm(yq, d) =yt∑

yr=y1

H(yq, yr, d), and

Hm(yr, d) =yt∑

yq=y1

H(yq, yr, d).

(38)

The correlation attribute is large for similar elements of thesecond order histogram.

(6) Mean of H(yq, yr, d), µHm:

µHm =yt∑

yq=y1

yqHm(yq, d). (39)

The mean characterizes the nature of the gray-level distribu-tion. Its value is typically small if the distribution is localizedaround yq = y1.

(7) Deviation of Hm(yq, d), σHm:

σHm =

√√√√√yt∑

yq=y1

yq −

yt∑yr=y1

yrHm(yr, d)

2

Hm(yq,d). (40)

The deviation indicates the amount of spread around the meanof the marginal distribution. The deviation is small if the his-togram is densely clustered about the mean.

(8) Entropy of Hd(ys, d), SHd(ys,d):

SHd(ys,d) = −yt∑

ys=y1

Hd(ys, d)log10[Hd(ys, d)]. (41)

(9) Angular second moment of Hd(ys, d), ASMHd(ys,d):

ASMHd(ys,d) =yt∑

ys=y1

[Hd(ys, d)]2. (42)


222 Atam P Dhawan

(10) Mean of Hd(ys, d), µHd(ys,d):

µHd(ys,d) =yt∑

ys=y1

ys[Hd(ys, d)]. (43)

The features computed using the difference histogram,Hd(ys, d), have the same significance as those attributes deter-mined by the second order statistics.

9.7.5 Hough Transform

The Hough transform is used to detect straight lines and other para-metric curves such as circles, ellipses, etc.1,2,5 It can also be used todetect boundaries of an arbitrarily shaped object if the parametersof the object are known. The basic concept of the Generalized Houghtransform is that an analytical function such as straight line, circleor a closed shape, represented in the image space (spatial domain)has a dual representation in the parameter space. For example, thegeneral equation of a straight line can be given as:

y = mx + c, (44)

where m is the slope and c is the y-intercept.As can be seen from Eq. 44, the locus of points is described by

two parameters, slope and y-intercept. Therefore, a line in the imagespace forms a point (m, c) in the parameter space. Likewise, a pointin the image space forms a line in the parameter space. Therefore,a locus of points forming a line in the image space will form a setof lines in the parameter space, whose intersection represents theparameters of the line in the image space. If a gradient image isthreshold to provide edge pixels, each edge pixel can be mappedto the parameter space. The mapping can be implemented usingthe bins of points in the parameter space. For each edge pixel of thestraight line in the image space, the corresponding bin in the param-eter space is updated. At the end, the bin with the maximum countrepresents the parameters of the straight line detected in the image.The concept can be extended to map and detect boundaries of apredefined curve. In general, the points in the image space become



hyperplanes in the N-dimensional parameter space and the parame-ters of the object function in the image space can be found by search-ing the peaks in the parameter space caused by the intersection ofthe hyperplanes.

To detect object boundaries using the Hough transform, it is nec-essary to create a parameter model of the object. The object modelis transferred into a table called an R-table. The R-table can be con-sidered as a one-dimensional array where each entry of the arrayis a list of vectors. For each point in the model description (MD),a gradient along with the corresponding vector extending from theboundary point to the centroid is computed. The gradient acts as anindex into the R-table.

For object recognition, a 2D parameter space of possible x-y coor-dinate centers is initialized with accumulator values associated witheach location set to zero. An edge pixel from the gradient image isselected. The gradient information is indexed into the R-table. Eachvector in the corresponding list is added to the location of the edgepixel. The endpoint of the vector should now point to a new edgepixel in the gradient image. The accumulator of the correspondinglocation in the parameter space is then incremented by one count.As each edge pixel is examined, the accumulator of the correspond-ing location receives the highest count. If the model object is con-sidered to be translated in the image, the accumulator of the correcttranslation location would receive the highest count. To deal withrotation and scaling, the process must be repeated for all possiblerotations and scales. Thus, the complete process could become verytedious if a large number of rotations and scales are examined. Toavoid this complexity, simple transformations can be made in theR-table of the transformation.16


Segmenting image into regions of interest and extracting featuresfrom the image and segmentation are essential for analyzing andclassifying the information represented in the image. In this chap-ter, commonly used edge and region segmentation methods are


224 Atam P Dhawan

described. Image and region statistics based features extracted thegray-level distributions along with shape and texture features arealso presented. Depending on the contextual knowledge, a numberof relational features with adjacencies graph and relational attributescan also be included in the analysis and classification of medi-cal images. Model-based methods representing anatomical knowl-edge from standardized atlases can be introduced in the segmen-tation and feature analysis to help computerized classification andinterpretation of medical images. Pattern classification methods aredescribed in Chapter 10 while the model-based registration andmedical image analysis methods are described in various chaptersin the second and third part of this book. Recent developmentsin model-based medical image analysis include probabilistic andknowledge based approaches and can be found in detail in the pub-lished literature.22–24,53–62 This trend of using multifeature analysisincorporating a priori and model-based knowledge is expected tocontinue in medical image analysis for diagnostic applications, aswell as for understanding of physiological processes linked withcritical diseases and designing better treatment intervention proto-cols for better healthcare.

References

1. Jain R, Kasturi R, Schunck BG, Machine Vision, McGRaw Hill Inc, 1995.2. Gonzalez RC, Woods RE, Digital Image Processing, Prentice Hall, 2nd

edn., 2002.3. Marr D, Hildreth EC, Theory of edge detection, Proc R Soc Lond B 207:

187–217, 1980.4. Haralick RM, Shapiro LG, Image segmentation techniques, Comp Vis

Graph Imag Process 7: 100–132, 1985.5. DhawanAP, Medical Image Analysis, Wiley Interscience, John Wiley and

Sons, Hoboken, NJ, 2003.6. Stansfield SA, ANGY: A rule-based expert system for automatic seg-

mentation of coronary vessels from digital subtracted angiograms,IEEE Trans Patt Anal Mach Intel 8: 188–199, 1986.

7. Ohlander R, Price K, Reddy DR, Picture segmentation using a recursiveregion splitting method, Comp Vis Graph Imag Process 8: 313–333, 1978.

8. Zucker S, Region growing: Childhood and adolescence, Comp Vis GraphImag Process 5: 382–399, 1976.



9. Dhawan AP, Sicsu A, Segmentation of images of skin lesions usingcolor and texture information of surface pigmentation, Comp Med ImagGraph 16: 163–177, 1992.

10. Dhawan AP, Arata L, Segmentation of medical images through com-petitive learning, Comp Methods and Prog In Biomed 40: 203–215, 1993.

11. Raya SR, Low-level segmentation of 3D magnetic resonance brainimages, IEEE Trans Med Imag 9: 327–337, 1990.

12. Liang Z, Tissue classification and segmentation of MR images, IEEEEng Med Biol Mag 12: 81–85, 1993.

13. Nilson NJ, Principles of Artificial Intelligence, Springer Verlag, 1982.14. Winston PH, Artificial Intelligence, Addison Wesley, 3rd edn., 1992.15. Dawant BM, Zijdenbos AP, Image segmentation, in Sonka M,

Fitzpatrick JM (eds.), Handbook of Medical Imaging, Vol 2: Medical ImageProcessing and Analysis, SPIE Press, 2000.

16. Ballard DH, Generalizing the Hough transform to detect arbitraryshapes, Pattern Recognition 13: 111–122, 1981.

17. Bomans M, Hohne KH, Tiede U, Riemer M, 3D segmentation of MRimages of the head for 3D display, IEEE Trans Medical Imaging 9: 177–183, 1990.

18. Raya SR, Low-level segmentation of 3D magnetic resonance brainimages: A rule based system, IEEE Trans Med Imaging 9(1): 327–337,1990.

19. Cline HE, Lorensen WE, Kikinis R, Jolesz F, Three-dimensional seg-mentation of MR images of the head using probability and connectiv-ity, Journal of Computer Assisted Tomography 14: 1037–1045, 1990.

20. Clarke L, Velthuizen R, Phuphanich S, Schellenberg J, et al., MRI: Sta-bility of three supervised segmentation techniques, Magnetic ResonanceImaging 11: 95–106, 1993.

21. Hall LO, Bensaid AM, Clarke LP, Velthuizen RP, et al., A comparison ofneural network and fuzzy clustering techniques in segmenting mag-netic resonance images of the brain, IEEE Trans on Neural Networks 3:672–682, 1992.

22. Vannier M, Pilgram T, Speidel C, Neumann L, et al., Validation ofmagnetic resonance imaging (MRI) multispectral tissue classification,Computerized Medical Imaging and Graphics 15: 217–223, 1991.

23. Choi HS, Haynor DR, KimY, Partial volume tissue classification of mul-tichannel magnetic resonance images — A mixed model, IEEE Trans-actions on Medical Imaging 10: 395–407, 1991.

24. Zavaljevski A, Dhawan AP, Holland S, Ball W, et al., Multispectral MRbrain image classification, Computerized Medical Imaging, Graphics andImage Processing 24: 87–98, 2000.

25. Nazif AM, Levine MD, Low-level image segmentation: An expert sys-tem, IEEE Trans Pattern Anal Mach Intell 6: 555–577, 1984.


226 Atam P Dhawan

26. Arata LK, Dhawan AP, Levy AV, Broderick J, et al., Three dimensionalanatomical model based segmentation of MR brain images throughprinicpal axes registration, IEEE Trans Biomed Eng 42: 1069–1078, 1995.

27. Xu L, Jackowski M, Goshtasby A, Yu C, et al., Segmentation of skincancer images, Image and Vision Computing 17: 65–74, 1999.

28. SarwalA, DhawanAP, Segmentation of coronary arteriograms throughRadial Basis Function neural network, Journal of Computing and Infor-mation Technology, 135–148, 1998.

29. Ozkan B, Dawant RJ, Maciunas RJ, Neural-Network-Based Segmenta-tion of Multi-Modal Medical Images: A Comparative and ProspectiveStudy, IEEE Trans on Medical Imaging 12: 1993.

30. Xuanli C, Beni G, A Validity Measure for Fuzzy Clustering, IEEE Transon Pattern Anal Mach Intell 133: 1991.

31. Bezdek A, Pattern Recognition with Fuzzy Objective Function Algorithms,Plenum, New York, 1981.

32. Chen C, Cowan CFN, Grant PM, Orthogonal least squares learningfor radial basis function networks, IEEE Trans On Neural Networks 2(2):302–309, 1991.

33. Poggio T, Girosi F, Networks for approximation and learning, Proceed-ings of the IEEE 78(9): 1481–1497, 1990.

34. Jacobson IRH, Radial basis functions: a survey and new results, inHandscomb DC (ed.), The Mathematics of Surfaces III, pp. 115–133,Clarendon Press, 1989.

35. SarwalA, DhawanAP, Segmentation of coronary arteriograms throughRadial Basis Function neural network, Journal of Computing and Infor-mation Technology, 135–148, 1998.

36. Xuanli G, Beni A, Validity Measure for Fuzzy Clustering, IEEE Transon Pattern Anal Mach Intell 133(8): 1991.

37. Loncaric S, Dhawan AP, Brott T, Broderick J, 3D image analysis ofintracerebral brain hemorrhage, Computer Methods and Programs inBiomed 46: 207–216, 1995.

38. Broderick J, Narayan S, Dhawan AP, Gaskil M, et al., Ventricular mea-surement of multifocal brain lesions: Implications for treatment trialsof vascular dementia and multiple sclerosis, Neuroimaging 6: 36–43,1996.

39. Schmid P, Segmentation of digitized dermatoscopic images by two-dimensional color clustering, IEEE Trans Med Imag 18: 164–171, 1999.

40. Pham DL, Prince JL, Adaptive fuzzy segmentation of magnetic reso-nance images, IEEE Trans Med Imag 18: 737–752, 1999.

41. Kanungo T, Mount DM, Netanvahu NS, Piatko CD, et al., An efficientk-means algorithm: analysis and implementation, IEEE Trans onPattern Anal Mach Intell 24: 881–892, 2002.



42. Duda RO, Hart PE, Pattern Classification and Scene Analysis, Wiley, 1973.43. Zurada JM, Introduction to Artificial Neural Systems, West Publishing Co,

1992.44. Fahlman SE, Lebeire C, The cascade-correlation learning architecture,

Tech Report, School of Computer Science, Carnegie Mellon University,1990.

45. DhawanAP, LeRoyer E, Mammographic feature enhancement by com-puterized image processing, Comp Methods & Programs in Biomed 27:23–29, 1988.

46. Serra J, Image Analysis and Mathematical Morphology, Academic Press,1982.

47. Sternberg S, Shapiro L, MacDonald R, Ordered structural shape match-ing with primitive extraction by mathematical morphology, PatternRecognition 20: 75–90, 1987.

48. Maragos P, Pattern spectrum and multiscale shape representation, IEEETrans on Pattern Anal Mach Intell 11: 701–716, 1989.

49. Loncaric S, DhawanAP,Amorphological signature transform for shapedescription, Pattern Recognition 26(7): 1029–1037, 1993.

50. Loncaric S, Dhawan AP, Brott T, Broderick J, 3D image analysis ofintracerebral brain hemorrhage, Computer Methods and Programs inBiomed 46: 207–216, 1995.

51. Loncaric S, Dhawan AP, Optimal MST-based shape description viagenetic algorithms, Pattern Recognition 28: 571–579, 1995.

52. Flusser J, Suk T, Pattern recognition by affine moments invariants,Pattern Recognition 26: 167–174, 1993.

53. Loew MH, Feature extraction, in Sonka M, Fitzpatrick JM, Handbookof Medical Imaging, Vol. 2: Medical Image Processing and Analysis, SPIEPress, 2000.

54. Dhawan AP, Chitre Y, Kaiser-Bonassoand, Moskowitz M, Analysis ofmammographic microcalcifications using gray levels image structurefeatures, IEEE Trans Med Imaging 15: 246–259, 1996.

55. Xu L, Jackowski M, Goshtasby A, Yu C, et al., Segmentation of skincancer images, Image and Vision Computing 17: 65–74, 1999.

56. Dhawan AP, Sicsu A, Segmentation of images of skin lesions usingcolor and texture information of surface pigmentation, Comp Med ImagGraph 16: 163–177, 1992.

57. Staib LH, Duncan JS, Boundary finding with parametricallydeformable models, IEEE Trans Pattern Anal Mach Intel 14: 1061–1075,1992.

58. Fan Y, Shen D, Gur RC, Gur RE, et al., COMPARE: Classification of Mor-phological Patterns using Adaptive Regional Elements, IEEE Transac-tions on Medical Imaging, 2006.


228 Atam P Dhawan

59. Grosbras MH, Laird AR, Paus T, Cortical regions involved in gazeproduction, attention shifts and gaze perception, Hum Brain Mapp 25:140–154, 2005.

60. Laird AR, Fox PM, Price CJ, Glahn DC, et al., ALE meta-analysis: Con-trolling the false discovery rate and performing statistical contrasts,Hum Brain Mapp 25: 155–164, 2005.

61. Zhang Y, Brady M, Smith S, Segmentation of brain MR imagesthrough a hidden Markov random field model and the expectation-maximization algorithm, IEEE Trans Med Imaging 20(1): 45–57, 2001.

62. Scherfler C, Schocke MF, Seppi K, Esterhammer R, et al., Voxel-wiseanalysis of diffusion weighted imaging reveals disruption of the olfac-tory tract in Parkinson’s disease, Brain 129(Pt 2): 538–42, 2006.


CHAPTER 10

Clustering and Pattern Classification

Atam P Dhawan and Shuangshuang Dai

Clustering is a method to arrange data points into groups or clus-ters based on a predefined similarity criterion. Classification mapsthe data points or their representative features into predefined classesto help the interpretation of the input data. There are severalmethods available for clustering and classification for computer-aided diagnostic or decision making systems for medical applica-tions. This chapter reviews some of the clustering and classificationmethods using deterministic as well as fuzzy approaches for dataanalysis.

10.1 INTRODUCTION

Image classification is an important task in computer-aided diag-nosis. An image after any preprocessing as needed to enhancefeatures of interest is processed to extract features for further analy-sis. Computed features are then arranged as a feature vector. Sincefeatures may utilize different dynamic ranges of values, normal-ization may be required before they are analyzed for classificationinto various categories. For example, a mammography image maybe processed to extract features related to microclacifications, e.g.number of microcalcification clusters, number of microcalcificationsin each cluster, size and shape of microcalcifications, spatial distri-bution of microcalcifications, spatial-frequency and texture informa-tion, mean and variance of gray-level values of microcalcifications,etc. These features are then used in a classification method such as

229


230 AP Dhawan and Shuangshuang Dai

statistical pattern classifier, Bayesian classifier, or neural network toclassify the image into two classes: benign and malignant.

Let us review some terms commonly used in pattern classi-fication.

Pattern: A pattern (feature vector, observation, or datum) χ is a vec-tor of measurements used by the clustering algorithm. It typicallyconsists of a vector of d measurements: χ = (x1, . . . xd).

Feature: A feature is defined as an individual scalar components xi

of a pattern χ.

Dimensionality: The dimensionality d usually refers to the numberof variables in the pattern or feature vector.

Pattern Set: A pattern set is denoted ℵ = {χ1, . . . χn}. The i-th patternin ℵ is denoted χi = (xi,1, . . . xi,d). In many cases, a pattern set to beclustered can be viewed as an n × d pattern matrix.

Class: A class, in the abstract, refers to a state of nature that gov-erns the pattern generation process. More concretely, a class can beviewed as a source of patterns whose distribution in feature spaceis governed by a probability density specific to the class.

Clustering: Clustering is a specific method that attempts to grouppatterns into various classes on the basis of a similarity criterion.

Hard clustering: Hard clustering techniques assign a class label lito each pattern χi, using a deterministic similarity criterion or crispmembership function.

Fuzzy clustering: Fuzzy clustering methods assign a class to eachinput pattern χi based on a fuzzy membership criterion with a frac-tional degree of membership fij for each cluster j.

Distance measure: A distance measure is a metric on the featurespace used to quantify the similarity of patterns.

A traditional pattern classification system can be viewed as amapping from input variables representing the raw data or a featureset to an output variable representing one of the categories or classes.To obtain a reasonable dimensionality, it is usually advantageous to


Clustering and Pattern Classification 231

Feature Selection/ Extraction

InterpatternSimilarity

Clustering and/or Classifier

ClassificationInput Data

Fig. 1. A typical classification system.

apply preprocessing transformations to the raw data before it is fedinto a classification system. Preprocessing usually involves featureextraction and/or feature selection to reduce the dimensionality toa reasonable number. Feature selection is the process of identifyingthe most effective subsets of the original features to be used in theclustering. The selected features are expected to be correlated withthe classification task for better results.

After the preprocessing and pattern (feature) representation areestablished, interpattern similarity should be defined on pairs ofpatterns and it is often measured by a distance function. Finally, theoutput of the clustering task is a set of clusters and it can be hard(a deterministic partition of the data into clusters) or fuzzy whereeach pattern has a variable degree of membership in each of theoutput clusters. Figure 1 shows a schematic diagram of a typicalclassification system.

10.2 DATA CLUSTERING

Clustering is assigning data points or patterns (usually representedas a vector of measurements in a multidimensional space) intogroups or clusters based on a predefined similarity measure. Intu-itively, patterns within a valid cluster are more similar to each otherthan they are to a pattern belonging to a different cluster. Data clus-tering is an efficient method to organize a large set of data for sub-sequent classification. Except in certain advanced fuzzy clusteringtechniques, each data point should belong to a single cluster, and nopoint should be excluded from membership in the complete set ofclusters.

Since similarity is fundamental to the definition of a cluster, ameasure of the similarity between two patterns drawn from the



same feature space is essential to most clustering procedures.1–10

Because of the variety of feature types and scales, the proper choiceof distance measure is of great importance. It is common to calcu-late dissimilarity between two patterns using a distance measuredefined on the feature space. Euclidean distance is the most popularmetric1,2 and it is defined as:

d2(xi, xj) =(

d∑k=1

(xi,k − xj,k

)2

)1/2

= ∥∥xi − xj∥∥

2 . (1)

It is noted that Euclidean distance is actually a special case (p = 2)of the Minkowski metric as1,2:

dp(xi, xj) =(

d∑k=1

(xi,k − xj,k

)p

)1/p

= ∥∥xi − xj∥∥

p . (2)

The Euclidean distance has an intuitive appeal as it is commonlyused to evaluate the proximity of objects in two or three-dimensionalspace. It works well when a data set has “compact” or “isolated”clusters.11 The drawback to the direct use of the Minkowski metricsis the tendency of the largest-scaled feature to dominate all others.Solutions to this problem include normalization of the continuousfeatures or other weighting schemes. Linear correlation among fea-tures can also distort distance measures. This distortion can be alle-viated by applying a whitening transformation to the data or byusing the squared Mahalanobis distance:

dM(xi, xj) = (xi − xj)A−1(xi − xj)T , (3)

where A is the sample covariance matrix of the patterns.In this process, dM(xi, xj) assigns different weights to different

features based on their variances and pairwise linear correlations.It is implicitly assumed here that class conditional densities are uni-modal and characterized by multidimensional spread, i.e. that thedensities are multivariate Gaussian. The regularized Mahalanobisdistance was used in11 to extract hyperellipsoidal clusters.



Traditional clustering algorithms can be classified into two maincategories1,2: hierarchical and partitional. In hierarchical clustering,the number of clusters need not be specified a priori, and problemsdue to initialization and local minimum do not arise. However, sincehierarchical methods consider only local neighbors in each step, theycannot incorporate a prior knowledge about the global shape or sizeof clusters. As a result, they cannot always separate overlappingclusters. Moreover, hierarchical clustering is static, and points com-mitted to a given cluster in the early stages cannot move to a differentcluster.

Partitional clustering obtains a single partition of the data insteadof a clustering structure by optimizing a criterion function definedeither locally (on a subset of the patterns) or globally (defined overall of the patterns). Partitional clustering can be further divided intotwo classes: crisp clustering and fuzzy clustering. In crisp clustering,every data point belong to only one cluster, while in fuzzy cluster-ing every data point belongs to every cluster to a certain degree asdetermined by the membership function.3 Partitional algorithms aredynamic, and points can move from one cluster to another. They canincorporate knowledge about the shape or size of clusters by usingappropriate prototypes and distance measures.

Hierarchical clustering is inflexible due to its greedy approach:after a merge or a split is selected it is not refined. Fisher4 studied iter-ative hierarchical cluster redistribution to improve once constructeddendrograms. Karypis et al.5 also researched refinements for hier-archical clustering. The problem with partitional algorithms is theinitial guess of the number of clusters. A simple way to mitigatethe effects of clusters initialization was suggested by Bradley andFayyad.6 First, k-means is performed on several small samples ofdata with a random initial guess. Each of these constructed systemsis then used as a potential initialization for a union of all the samples.Centroids of the best system constructed this way are suggested asan intelligent initial guesses to ignite the k-means algorithm on thefull data. Zhang7 suggested another way to rectify the optimiza-tion process by soft assignment of points to different clusters withappropriate weights, rather than by moving them decisively from



one cluster to another. Nowadays, probabilistic models have beenproposed as a basis for cluster analysis. In this approach, the data areviewed as coming from a mixture of probability distributions, eachrepresenting a different cluster. Methods of this type have shownpromise in a number of practical applications.8–10

10.2.1 Hierarchical Clustering with the Agglomerative Method

In hierarchical clustering, the number of clusters may not be spec-ified in advance. It builds a cluster hierarchy or, in other words, atree of clusters. Every cluster node contains child clusters; siblingclusters partition the points covered by their common parent. Suchan approach allows exploring data on different level of granularity.Hierarchical clustering methods are divided into agglomerative anddivisive.2,10,11 An agglomerative clustering method may start withone-point (singleton) based clusters and recursively merges two ormore appropriate clusters. Adivisive clustering starts with one clus-ter of all data points and recursively splits the most appropriatecluster. The process continues until a stopping criterion is satisfiedproviding a reasonable number of clusters.

Hierarchical methods of cluster analysis permit a convenientgraphical display in which the entire sequence of merging (or split-ting) is shown. Because of its tree-like nature, the display has thename of dendrogram. The agglomerative method is usually chosenbecause it is more important and more widely used. One reason forthe popularity of agglomerative method is that during the mergingprocess, the choice of threshold is not a big concern which will beillustrated in the details of the algorithm shown below. In contrast,divisive methods are more computationally intensive and the diffi-culty of choosing potential allocations to clusters during the splittingstages.

To merge or split subsets of points rather than individual points,the distance between individual points has to be generalized tothe distance between subsets. Such a derived proximity measureis called a linkage metric. The type of the linkage metric usedsignificantly affects hierarchical algorithms, since it reflects the



particular concept of closeness and connectivity. Major interclus-ter linkage metrics include single link, average link and completelink.2,10–13 The underlying dissimilarity measure (usually distance)is computed for every pair of points with one point in the first setand another point in the second set.Aspecific operation such as min-imum (single link), average (average link), or maximum (completelink) is applied to pair-wise dissimilarity measures:

d(C1, C2) = operation {d(x, y)| x ∈ C1, y ∈ C2}. (4)

For example, the SLINK algorithm,12 based on the single-link metricrepresentation provides the Euclidean minimal spanning tree withO(N2) computational complexity.

As described above, the agglomerative methods are based onmeasures of distance between clusters. From a representation of sin-gle point based clusters, two clusters that are nearest and satisfy sim-ilarity criterion are merged to form a reduced number of clusters.This is repeated until just one cluster is obtained. Let us supposethat “n” sample (data) points are to be clustered, the initial num-ber of clusters will then be equal to n as well. Let us represent thedata vector D with n data points as D = {x(1),…,x(n)} and a functionD(Ci, Cj) as distance measure between two clusters Ci and Cj. Anagglomerative algorithm for clustering can be defined as follows:

Algorithm (agglomerative hierarchical clustering)

Step 1: for i = 1, . . ., n let Ci = {X(i)};Loop: While there is more than one cluster left do

Minimizing the distance D(Ck, Ch) between any twoclustersLet Ci and Cj be the clusters with minimum distanceCi = Ci ∪ Cj;Remove cluster Cj;

End

In the above algorithm, a distance measure should be carefullychosen. Normally, Euclidean distance is employed which assumesome degree of commensurability between the different variables.It makes less sense if the variables are non-commensurate, that is,



Fig. 2. A sample dendrogram.

variables are measured in different units. A common strategy is tostandardize the data by dividing the sample value of each of thevariables by its sample standard deviation, so that they are equallyimportant. Figure 2 shows a sample dendrogram produced by theagglomerative hierarchical clustering method for a given data.

Linkage metrics-based hierarchical clustering suffers from timecomplexity. Under reasonable assumptions, such as reducibility con-dition, linkage metrics methods have O(N2) complexity.10–14 Chiuet al.15 proposed another hierarchical clustering algorithm usinga model-based approach in which maximum likelihood estimateswere introduced.

Traditional hierarchical clustering is inflexible due to its greedyapproach: after a merge or a split is selected, it is not refined. Inaddition, since they consider only local neighbors in each step, it isdifficult to incorporate a prior knowledge about the global shape orsize of clusters. Moreover, hierarchical clustering is static in a sense



that points assigned to a cluster in the early stages cannot be movedto a different cluster in later stages.

10.2.2 Non-hierarchical or Partitional Clustering

A non-hierarchical or partitional clustering algorithm obtains a sin-gle partition of the data instead of a hierarchical clustering represen-tation such as the dendrogram. Partitional methods have advantagesin applications involving large data sets for which the construction ofa dendrogram is computationally problematic. The partitional tech-niques usually produce clusters by optimizing an objective functiondefined either locally (on a subset of the patterns) or globally (overall of the patterns).

10.2.2.1 K-Means Clustering Approach

K-means2 is the simplest and most commonly used algorithmemploying a squared error criterion which is defined as:

e2 (ℵ, �) =K∑

j=1

nj∑i=1

∥∥∥x(j)i − cj

∥∥∥2. (5)

The K-means algorithm starts with a random initial partition andkeeps reassigning the patterns to clusters based on the similaritybetween the pattern and the cluster centers until a convergence cri-terion is met, e.g. there is no reassignment of any pattern from onecluster to another, or the squared error ceases to decrease signifi-cantly after some number of iterations. The k-means algorithm ispopular because it is easy to implement with a computational com-plexity of O(N), where N is the number of patterns.

A major problem with this algorithm is that it is sensitive to theselection of the initial partition and may converge to a local min-imum of the criterion function value if the initial partition is notproperly chosen. Bradley and Fayyad6 suggested a way to mitigatethe effects of cluster initialization. One variation to the k-means algo-rithm is to permit the splitting and merging of the resulting clusters.Typically, a cluster is split when its variance is above a prespecified



threshold and two clusters are merged when the distance betweentheir centroids is below another prespecified threshold. Under such ascheme, it is possible to obtain the optimal partition starting from anyarbitrary initial partition, provided proper threshold values are spec-ified. Another variation of the k-means algorithm involves select-ing a different criterion function altogether. Diday16 and Symon17

described a dynamic clustering approach obtained by formulatingthe clustering problem in the framework of maximum-likelihoodestimation. The regularized Mahakanobis distance was used in Maoand Jain11 to obtain hyperellipsoidal clusters.

Partitioning clustering algorithms can be divided into twoclasses: crisp (or hard) clustering and fuzzy clustering. Hard cluster-ing is the traditional approach in which each pattern belongs to oneand only one cluster. Hence, the clusters are disjoint. Fuzzy cluster-ing extends this notion to associate each pattern with every clusterusing a membership function.2 Fuzzy set theory was initially appliedto clustering in Ruspini.28 The most popular fuzzy clustering algo-rithm is the fuzzy k-means (FCM) algorithm. A generalization ofthe FCM algorithm was proposed by Bezdek18 through a family ofobjective functions. A fuzzy c-shell algorithm and an adaptive vari-ant for detecting circular and elliptical boundaries was presented inDave.19 It was also extended in medical image analysis to segmentmagnetic resonance images.20 Even though it is better than the hardk-means algorithm at avoiding local minima, FCM can still convergeto local minima of the squared error criterion. The design of the mem-bership function is the most important problem in fuzzy clustering;different choices include those based on similarity decompositionand centroids of clusters.

10.2.3 Fuzzy Clustering

Conventional clustering and classification approaches assign a datapoint to a cluster or class with a well defined metric. In other words,the membership of a data point for a cluster is deterministic and canbe represented by a crisp membership function. In many real worldapplications, setting up a crisp membership function for clustering



or classification often makes the result intuitively unreasonable.Using a less deterministic approach with probabilistic membershipfunctions providing fuzzy overlapping boundaries in the featurespace have provided very useful results in many applications.18–20

10.2.3.1 Fuzzy Membership Function

A fuzzy set is a set without a crisp boundary for its membership. IfX is a space of input data points denoted generically by x, then afuzzy set A in X is defined as a set of ordered pairs:

A = {(x, µA(x))| x ∈ X} , (6)

where µA(x) is called the membership function (MF) for the fuzzyset A and its value ranges from 0 to 1. In other words, a membershipfunction can be represented as a mapping function that provideseach point in the input space with a membership value (or degreeof membership) between 0 and 1. For example, the age of a personcan be defined into some predefined deterministic groups such asin the interval of 10, e.g. 21–30; 31, 40; 41–50, etc. However, defininga “middle-aged” group of people is quite subjective to individualperception. If we consider a range, say, between 40 and 50, as“middle-aged,” a probabilistic membership function can be deter-mined to represent the degree of belongingness to the group of mid-dle aged people.

A membership function may be expressed as the generalizedCauchy distribution18 as:

µA(x) = bell (x; a, b, c) = 1

1 + ∣∣ x−ca

∣∣2b, (7)

where c is the median value of the range (for example, it is 45 in themiddle age group as described above), and a and b are parametersto adjust the width and sharpness of the curve. The membershipfunction for a = 15, b = 3, and c = 45 is shown in Fig. 3 as µA(x) =bell(x; 15, 3, 45).

It can be noted that the definition of “middle aged” as rep-resented by a membership function becomes more reasonable as



Fig. 3. A plot of the “bell-shape” membership function bell(x; 15, 3, 45).

against to a crisp representation. If a person is between 40 and 50, themembership function value is 1 which is considered middle-aged.Extending this concept to three groups: “young,” “middle-aged,”and “old” with three membership functions (MF) based representa-tion, a probabilistic interpretation of the age group can be obtainedas shown in Fig. 4. A person with 35 years of age is more likely con-sidered to be middle-aged than young because the correspondingMF value is around 0.8 for the middle-age versus 0.2 for the younggroup. Therefore, a particular age has three corresponding MF val-ues in different categories.As mentioned above, the three MFs totallycover the value range of X and the transition from one MF to anotheris smooth and gradual.

10.2.3.2 Membership Function Formulation

The parameterized functions can be used to define membershipfunctions (MF) with different transition properties. For example,



Fig. 4. A plot of three Bell MFs for “young,” “middle aged” and “old.”

triangular, trapezoidal, Gaussian and bell shape functions havedifferent transition curves and therefore the corresponding prob-ability function provides different mappings to the data distribu-tion. Further, multidimensional MFs with desired shape (Triangle,Gaussian, Bell, etc.) may be needed to deal with multidimensionaldata. A multidimensional Gaussian MF can be represented as:

µA(X) = gaussian (X; M, K) = exp

{−1

2(X − M)TK−1(X − M)

},

(8)where X and M are column vectors defined by: X = [x1, x2, . . . , xn]Tand M = [m1, m2, . . . , mn]T = [E(x1), E(x2), . . . , E(xn)]T , mi is themean value of variable xi, and K is covariance matrix of variables xi

defined as:

K = var(x1) cov(x1, x2) . . . . cov(x1, xn)

cov(x2, x1) var(x2) . . . . cov(x2, xn)cov(xn, x1) cov(xn, x2) . . . . var(xn)

. (9)

10.2.3.3 Fuzzy k-Means Clustering

The fuzzy k-means algorithm18 is based on the minimization ofan appropriate objective function J, with respect to U, a fuzzy



K-partition of the dataset, and to V, a set of K prototypes as:

Jq(U, V) =N∑

j=1

K∑i=1

(uij)qd2(Xj, Vi); K ≤ N (10)

where q is any real number greater than 1, Xj is the j-th m-dimensional feature vector, Vj is the centroid of the i-th cluster, uij

is the degree of membership of Xj in the i-th cluster. d2(Xj, Vi) is anyinner product metric (distance between Xj and Vj), N is the num-ber of data points. K is the number of clusters. The parameter q isthe weighting exponent for uij and controls the “fuzziness” of theresulting clusters.18 Fuzzy partition may be carried out through aniterative optimization of the above objective function as the follow-ing algorithm.

Step 1: Choose primary centroid Vi (prototypes);

Step 2: Compute the degree of membership of all feature vectors inall the clusters:

uij = (1/d2(Xj, Vi)1/(q−1)∑Kk=1 (1/d2(Xj, Vi)1/(q−1)

; (11)

Step 3: Compute new centroids∧Vi:

∧Vi =

∑Nj=1 (uij)qXj∑N

j=1 (uij)q, (12)

and update the degree of membership, uij to∧uij, according to Eq. 11.

Step 4: If max[|uij − ∧uij|] < ε stop, otherwise go to Step 3

where ε is a termination criterion between 0 and 1.

Computation of the degree of membership uij depends on thedefinition of the distance measure, d2(Xj, Vi)18 as:

d2(Xj, Vi) = (Xj − Vi)TA(Xj − Vi). (13)

The inclusion of A (an m × m positive-definite matrix) in thedistance measure results in weighting according to the statisticalproperties.2,18



10.3 NEAREST NEIGHBORED CLASSIFIER

Apopular statistical method for classification is the nearest neighborclassifier, which assigns a data point to the nearest class model in thefeature space. It is apparent that the nearest neighbor classifier is asupervised method as it uses labeled clusters of training samples inthe feature space as models of classes. Let us assume that there areC number of classes represented by cj; j = 1, 2, . . . , C. An unknownfeature vector f is to be assigned to the class that is closest to the classmodel developed from clustering the labeled feature vectors duringthe training. A distance measure Dj(f) is defined by the Euclideandistance in the feature space as2:

Dj(f) = ‖f − uj‖, (14)

where

uj = 1Nj

∑f∈cj

fj j = 1, 2, . . . , C

is the mean of the feature vectors for the class cj and Nj is the totalnumber of feature vectors in the class cj.

The unknown feature vector is assigned to the class ci if:

Di(f) = minCj=1[Dj(f)]. (15)

A probabilistic approach can be applied to the task of classifica-tion to incorporate a priori knowledge to improve performance.Bayesian and maximum likelihood methods have been widely usedin object recognition and classification for different applications. Letus assume that the probability of a feature vector f belonging to theclass ci is denoted by p(ci/f). Let an average risk of wrong classifica-tion for assigning the feature vector to the class cj be expressed byrj(f) as:

rj(f) =C∑

k=1

Zkj p(ck/f), (16)

where Zkj is the penalty of classifying a feature vector to the class cj

when it belongs to the class ck.



It can be shown that:

rj(f) =C∑

k=1

Zkj p(f/ck)P(ck), (17)

where P(ck) is the probability of occurrence of class ck.A Bayes classifier assigns an unknown feature vector to the class

ci if:

ri(f) < rj(f)

orC∑

k=1

Zkip(f/ck)P(ck) <

C∑q=1

Zqjp(f/cq)P(cq) for j = 1, 2, . . . , C. (18)

Other versions of the Bayesian classification as applied to medicalimage classification can be found in many papers20–25 for radiolog-ical image analysis and computer-aided diagnosis.

10.4 DIMENSIONALITY REDUCTION

As described above, the goal of clustering is to group the data orfeature vector into some meaningful categories for better classifica-tion and decision making without making errors of assigning a datavector to a wrong class. For example, for computer-aided analysisof mammograms, mammography image feature vectors may needto be classified into “benign” or “malignant” classes by a patternclassification system. The error in classification may assign a nor-mal patient to “malignant” class (therefore creating a false positive)or may assign a cancer patient to “benign” class (therefore miss-ing a cancer). If the data (or features) are assumed to be statisticallyindependent, the probability of classification error decreases as thedistance between the classes increases. This distance is defined asRef. [14]:

d2 =n∑

i=1

µi1 − µi2

σ2i

, (19)

where µi1 and µi2 are the mean of each feature for the two classes.Thus, the most useful features are those with large differences in



mean as compared to their standard deviation. The performanceshould continue to improve by the addition of new features as longas the means for the two classes differ (thereby increasing d) andthe numbers of observations are increased accordingly. The classi-fier performance may be affected by unnecessary or noisy observa-tions or features that are not well correlated to the required classes.Therefore, it is useful to reduce the number of features to those thatcan provide maximum separation in the feature space among therequired classes. In addition, by reducing the number of features,significant gain many be achieved in computational efficiency. Thisprocess is usually called dimensionality reduction. Though there area number of approaches investigated for dimensionality reductionand improving the performance of a classifier in the feature space,two useful approaches using principal component analysis (PCA)and genetic algorithms (GA) are described here.

10.4.1 Principal Component Analysis

Principal component analysis (PCA) is an efficient method to reducethe dimensionality of a data set which consists of a large numberof interrelated variables while retaining as much as possible of thevariation present in the data set.2 The goal here is to map vectorsXd in a d-dimensional space (x1, x2, . . . , xd) onto vectors ZM in anM-dimensional space (z1, z2, . . . , zM) where M < d. Without loss ofgenerality, we express vector X as a linear combination of a set of dorthonormal vectors ui:

X =d∑

i=1

xiui, (20)

where the vectors ui satisfy the orthonormality relation:

uTi uj = δij. (21)

Therefore the coefficient in (20) can be expressed as:

xi = uTi X. (22)

Let us suppose that only a subset of M < d of the basis vectors ui areto be retained, so that only M coefficients xi are used. In general, PCA



does not retain a subset of the original set of basis vectors. It finds anew set of basis vectors that spans the original d-dimensional spacesuch that the data can be well represented by a subset of these newbasis vectors. Here, vi is used to denote the new basis vectors whichmeet the orthonormality requirement. As above, only M coefficientsxi are used and the remaining coefficients will be replaced by con-stants bi. Now each vector x is approximated by an expression of theform:

X =M∑

i=1

xivi +d∑

i=M+1

bivi (23)

xi = vTi X. (24)

We need to choose the basis vectors vi and the coefficients bi to bemade such that an approximation given by Eq. 23, with the valuesof xi determined by Eq. 24, provides the best approximation to theoriginal vector X on average for the whole set data set.

The next step is to minimizes the sum of squares of errors over thewhole data set. The sum-of-square error can be written as follows:

EM = 12

d∑i=M+1

vTi Avi, (25)

where A is the covariance matrix of the set of vectors Xn, which isdefined as follows:

A =∑

(xn − x)(xn − x)T . (26)

Now the problem is converted to minimizing EM with respect to thechoice of basis vectors vi. A minimum value is obtained when thebasis vectors satisfy the following condition:

Avi = βivi (27)

Thus, vi (i = M + 1· · · d) are the eigenvectors if the covariancematrix. Note that, since the covariance matrix is real and symmetric,



its eigenvectors can indeed be chosen to be orthonormal. Finally, theminimum of error is in the form:

EM = 12

d∑i=M+1

βi. (28)

Therefore, the minimum error is achieved by rejecting the (d-M)smallest eigenvalues and their corresponding eigenvectors. The firstM largest eigenvalues are then retained. Each of the associated eigen-vectors vi is called a principal component.

With matrix representation, singular value decomposition (SVD)algorithm can be employed to calculate the eigenvalues and its cor-responding eigenvectors. The use of SVD has two important impli-cations. First, it is computationally efficient and second, it providesadditional insight into what a PCA actually does. It also provides away to represent the results of PCA graphically and analytically.

10.4.2 Genetic Algorithms Based Optimization

In nature, the features that characterize an organism determine itsability to endure in a competition for limited resources. These fea-tures are fixed by the building block of genetics, or the gene. Thesegenes form chromosomes, the genetic structures which ultimatelydefine the survival capability of an organism. Thus, the most supe-rior organisms survive and pass their genes on to future generations,while the genes of less fit individuals are eventually eliminated fromthe population.

Reproduction introduces diversity into a population of individ-uals through the exchange of genetic material. Repeated selectionof the fittest individuals and recombination of chromosomes pro-motes evolution in the gene pool of a species which creates evenbetter population members.

A genetic algorithm (GA) is a robust optimization and searchmethod based on the natural selection principles outlined above.Genetic algorithms provide improved performance by exploitingpast information and promoting competition for survival. GAs gen-erate a population of individuals through selection, and search for



the fittest individuals through crossover and mutation.Afundamen-tal feature of GAs is that they operate on a representation of problemparameters, rather than manipulating the parameters themselves.These parameters are typically encoded as binary strings that areassociated with a measure of goodness, or fitness value. As in nat-ural evolution, GAs encourage the survival of the fittest throughselection and recombination. Through the process of reproduction,individual strings are copied according to their degree of fitness.In crossover, strings are probabilistically mated by swapping allcharacters located after a randomly chosen bit position. Mutationis a secondary genetic operator that randomly changes the value ofa string position to introduce variation in the population and recoverlost genetic information.31,32

GAs maintain a population of structures that are potentialsolutions to an objective function. Let us assume that featuresare encoded into binary strings that can be represented as A =a1, a2, . . . , aL, where L is the specified string length, or the numberof representative bits. A simple genetic algorithm operates on thesestrings according to the following iterative procedure:

(1) Initialize a population of binary strings.(2) Evaluate the strings in the population.(3) Select candidate solutions for the next population and apply

mutation and crossover operators to the parent strings.(4) Allocate space for new strings by removing members from the

population.(5) Evaluate the new strings and add them to the population.(6) Repeat steps 3–5 until the stopping criterion is satisfied.

Detailed knowledge of the encoding mechanism, the objec-tive function, the selection procedure, and the genetic operators,crossover and mutation, is essential for a firm understanding of theabove procedure as applied to a specific problem. These componentsare considered below.

The structure of the GA is based on the encoding mechanismused to represent the variables in the given optimization problem.



The candidate solutions may encode any number of variable types,including continuous, discrete, and boolean variables. Althoughalternate string codings exist,31,32 a simple binary encoding mech-anism is considered within the scope of this thesis. Thus, the alleleof a gene in the chromosome indicates whether or not a feature issignificant in microcalcification description. The objective functionevaluates each chromosome in a population to provide a measureof the fitness of a given string. Since the value of the objective func-tion can vary widely between problems, a fitness function is usedto normalize the objective function within the range of 0 to 1. Theselection scheme uses this normalized value, or fitness, to evaluatea string.

One of the most basic reproduction techniques is proportionateselection, which is carried out by the roulette wheel selection scheme.In roulette wheel selection, each chromosome is given a segment of aroulette wheel whose size is proportionate to the chromosome’s fit-ness. A chromosome is reproduced if a randomly generated numberfalls in the chromosome’s corresponding roulette wheel slot. Thussince more fit chromosomes are allocated larger wheel portions, theyare more likely to generate offspring after a spin of the wheel. Theprocess is repeated until the population for the next generation iscompletely filled. However, due to sampling errors the populationmust be very large in order for the actual number of offspring pro-duced for an individual chromosome to approach the expected valuefor that chromosome.

In proportionate selection, a string is reproduced according tohow its fitness compares to the population average, in other words,as fi/f , where fi is the fitness of the string and f is the average fit-ness of the population. This proportionate expression is also knownas the selective pressure on an individual. The mechanics of pro-portionate selection can be expressed as: Ai receives more than oneoffspring on average if fi > f ; otherwise, Ai receives less than oneoffspring on average. Since the result of applying the proportionatefitness expression will always be a fraction, this value represents theexpected number of offspring allocated to each string, not the actualnumber.



Once the parent population is selected through reproduction,the offspring population is created after application of genetic oper-ators. The purpose of recombination, also referred to as crossover, isto discover new regions of the search space, rather than relying on thesame population of strings. In recombination, strings are randomlypaired and selected for crossover. If the crossover probability condi-tion is satisfied, then a crossover point along the length of the stringpair is randomly chosen. The offspring are generated by exchangingthe portion of the parent strings beyond the crossover position. Fora string of length l, the l−1 possible crossover positions are chosenwith equal probability.

Mutation is a secondary genetic operator that preserves the ran-dom nature of the search process and regenerates fit strings thatmay have been destroyed or lost during crossover or reproduction.The mutation rate controls the probability that a bit value will bechanged. If the mutation probability condition is exceeded, then theselected bit is inverted.

An example of a complete cycle for the simple genetic algorithmis shown in Table 1.31 The initial population contains four stringscomposed of ten bits. The objective function determines the numberof 1’s in a chromosome and the fitness function normalizes the valueto lie in the range of 0 to 1.

The proportional selection scheme allocates 0, 1, 1, and 2 off-spring to the initial offspring in their respective order. After selec-tion, the offspring are randomly paired for crossover so that strings1 and 3 and strings 2 and 4 are mated. However, since the crossoverrate is 0.5, only strings 1 and 3 are selected for crossover. The otherstrings are left intact. The pair of chromosomes then exchange theirgenetic material after the fifth bit position, which is the randomlyselected crossover point. The final step in the cycle is mutation. Sincethe mutation rate is selected to be 0.05, only two bits out of the fortypresent in the population are mutated. The second bit of string 2and the fourth bit of string 4 are randomly selected for mutation. Ascan be seen from the figure, the average fitness of population P4 issignificantly better than the initial fitness after only one generationalcycle.



Table 1. A Sample Generational Cycle of the Simple Genetic Algorithm

Chromosome Fitness Value Average Fitness

Population P1 0001000010 0.2 0.50(initial population) 0110011001 0.5

1010100110 0.51110111011 0.8

Population P2 0110011001 0.5 0.65(after selection) 1010100110 0.5

1110111011 0.81110111011 0.8

Population P3 01100|11011 0.6 0.65(after crossover) 1010100110 0.5

11101|11001 0.71110111011 0.8

Population P4 0110011011 0.6 0.70(after mutation) 1110100110 0.6

1110111001 0.71111111011 0.9

Although roulette wheel selection is the simplest method toimplement proportionate reproduction, it is highly inefficient sinceit requires n spins of the wheel to fill a population with n mem-bers. Stochastic universal selection (SUS) is an efficient alternativeto roulette wheel selection. SUS also uses a weighted roulette wheel,but adds equally spaced markers along the outside rim of the wheel.The wheel is spun only once, and each individual receives as manycopies of itself as there are markers in its slot.32

The average fitness value in the initial stages of a GA is typ-ically low. Thus, during the first few generations the proportion-ate selection scheme may assign a large number of copies to a fewstrings with relatively superior fitness, known as super individuals.These strings will eventually dominate the population and cause theGA to converge prematurely. The proportionate selection procedurealso suffers from decreasing selective pressure during the last gen-erations when the average fitness value is high. Scaling techniquesand ranking selection can help alleviate the problems of inconsistentselective pressure and domination by superior individuals.



In linear scaling, the fitness value is adjusted by:

f ′ = af + b, (29)

where f is the original fitness value and f ′ is the scaled fitness value.The coefficients a and b are chosen so that the fittest individualsdo not receive too many copies, and average individuals typicallyreceive one copy. These coefficients should also be adjusted to avoidnegative fitness values.

Ranking selection techniques assign offspring to individuals byqualitatively comparing levels of fitness. The population is sortedaccording to their fitness values and allotted offspring based on theirrank. In ranking selection, subsequent populations are not influ-enced by the balance of the current fitness distributions so that selec-tive pressure is uniform. Each cycle of the simple GA produces acompletely new population of offspring from the previous gener-ation, known as generational replacement. Thus, the simple GA isnaturally slower in manipulating useful areas of the search spacefor a large population. Steady-state replacement is an alternativemethod which typically replaces one or more of the worst membersof the population each generation. Steady-state replacement can becombined with an elitist strategy, which retains the best strings inthe population.32

GAs are efficient global optimization techniques which arehighly suited to searching in nonlinear, multidimensional prob-lem spaces.32 The most widely accepted theory on the operationof the GA search mechanism in global optimization is the SchemaTheorem. This theorem states that the search for the fittest individ-uals is guided by exploiting similarities among the superior stringsin a population. These similarities are described by schemata, whichare composed of strings with identical alleles at the same position oneach string. The order of a particular schema is the number of fixedpositions among the strings, and the defining length is the distancebetween the first and last fixed positions on a string. The schematawith superior fitness, low order and small defining length increasewith each passing generation.



From a set of coded parameters, GAs use a population of pointsto search for the optimal solution, not just a single point in the searchspace. The GAthus has a high probability of discovering the optimalglobal solution in a multimodal search space since it is less likelyto be troubled by false optima. This ability becomes a tremendousadvantage over traditional methods in more complex problems.

10.5 NON-PARAMETRIC CLASSIFIERS

Artificial neural network based classifiers have been explored exten-sively in the literature for non-parametric classification using a setof training vectors providing relationships between input featuresor measurements to output classes. Such classification methodsthat do not require any prior probabilistic model of class distribu-tions of input vectors; they learn this relationship during training.Though there are a number of such classifiers have been used fordifferent applications, more common networks such as Backprop-agation and Radial-Basis Function neural networks are describedhere.33–34

10.5.1 Backpropagation Neural Network for Classification

The backpropagation network is the most commonly used neuralnetwork in signal processing and classification applications. It usesa set of interconnected neural elements that process the informationin a layered manner. A computational neural element, also calledas perceptron, provides an output as a thresholded weighed sumof all inputs. The basic function of the neural element, as shown inFig. 5, is analogous to the synaptic activities of a biological neuron.In a layered network structure, the neural element may receive itsinput from an input vector or other neural elements. A weightedsum of these inputs constitutes the argument of a non-linear activa-tion function such as a sigmoidal function. The resulting thresholdedvalue of the activation function is the output of the neural element.The output is distributed along weighted connections to otherneural elements.



X f(ϕ)Σ

1

w2ϕ

wn+1

w1

wd

f(ϕ):

Y

Fig. 5. A computational neuron model with linear synapses.

In order to learn a specific pattern of input vectors for classifi-cation, an iterative learning algorithm, such as the LMS algorithm,often called the Widrow-Hoff Delta Rule34 is used with a set of pre-classified training examples that are labeled with the input vectorsand their respective class outputs. For example, if there are two out-put classes for classification of input vectors, the weighted sum ofall input vectors may be thresholded to a binary value, 0 or 1. Theoutput 0 represents class 1, while the output 1 represents class 2. Thelearning algorithm repeatedly presents input vectors of the trainingset to the network and forces the network output to produce therespective classification output. Once the network converges on alltraining examples to produce the respective desired classificationoutputs, the network is used to classify new input vectors into thelearned classes.

The computational output of a neural element can beexpressed as:

y = F

(n∑

i=1

wixi + wn+1

), (30)

where F is a non-linear activation function that is used to thresholdthe weighted sum of inputs xi and wi is the respective weight. A biasis added to the element as wn+1, as shown in Fig. 5.

Let us assume a multilayer feed-forward neural network with Llayers of N neural elements (Perceptrons) in each layer, as shown in



Hidden Layer Neurons

Output Layer Neurons

Ly1

x1 x2 x3 xn 1

Ly2Lny

Fig. 6. A feedforward Backpropagation neural network.

Fig. 6, such that:

y(k) = F(Wky(k−1)

)for k = 1, 2, . . . L, (31)

where y(k) is the output of the k-th layer neural elements with k = 0representing the input layer and W (k) is the weight matrix for thek-th layer such that:

y(0) =

x1

x2

·xn

1

; y(k) =

y(k)1

y(k)2

·y(k)

n

y(k)(n+1)k



and

W(k) =

w(k)11 w(k)

12 · w(k)1n w(k)

1(n+1)

w(k)21 w(k)

22 · w(k)2n wk

2(n+1)

· · · · ·w(k)

n1 w(k)n2 · w(k)

nn w(k)n(n+1)

w(k)(n+1)1 w(k)

(n+1)2 · w(k)(n+1)n w(k)

(n+1)(n+1)

. (32)

The neural network is trained by presenting classified exam-ples of input and ouput patterns. Each example consists of theinput and output vectors {y(0), yL} or {x, yL} that are encoded for thedesired classes. The objective of the training is to determine a weightmatrix that would provide the desired output, respectively for eachinput vector in the training set. The least mean squared (LMS) erroralgorithm43,44 can be implemented to train a feed forward neuralnetwork using the following steps:

(1) Assign random weights in the range of [−1,+1] to all weightswk

ij.(2) For each classified pattern pair {y(0), yL} in the training set, do

the following steps:

a. Compute the output values of each neural element using thecurrent weight matrix.

b. Find the error e(k) between the computed output vector andthe desired output vector for the classified pattern pair.

c. Adjust the weight matrix using the change �W(k) computedas �W(k) = αe(k)[y(k−1)] for all layers k = 1, . . . , L,where α is the learning rate that can set between 0 and 1.

(3) Repeat step 2 for all classified pattern pairs in the training setuntil the error vector for each training example is sufficientlylow or zero.

The non-linear activation function is an important consideration incomputing the error vector for each classified pattern pair in the



training set. A sigmoidal activation function can be used as:

F(y) = 11 + e−y . (33)

The above described gradient descent algorithm for training a feed-forward neural network also called as backpropagation neural net-work (BPNN) is sensitive to the selection of initial weights and noisein the training set that can cause the algorithm to get stuck in localminima in the solution pace. This causes a poor generalization per-formance of the network when it is used to classify new patterns.Another problem with the BPNN is to find optimal network archi-tecture with the consideration of optimal number of hidden layersand neural elements in each of the hidden layers. Several solutionsto find the best architecture and generalization performance havebeen explored in the literature.34

10.5.2 Classification Using Radial Basis Functions

Radial basis function (RBF) classifiers are useful interpolation meth-ods for multidimensional tasks. One major advantage of RBFs istheir structural simplicity, as seen from Fig. 7. The response of eachnode in the single hidden layer is weighted and linearly summed atthe output.

The RBF network was configured by finding the centers andwidths of the basis functions and then determining the weights at

x1

x2 w1

x3 . y

. .

.

. w

ϕ

ϕ Σ

ϕm

xn

Fig. 7. The radial basis function neural network representation.



the output of each node. The goal in selecting the unit width, orvariance, is to minimize the overlap between nearest neighbors andto maximize the network’s generalization ability. For good gener-alization, the eigen values of the covariance matrix of each basisare chosen as large as possible. Typically, the kernel function is aGaussian with unit normalization as given by34:

ϕ(x) = exp

[−

∥∥�x − �ci∥∥

2σ2i

], (34)

where ci is the center of a given kernel, and σ2i is the corresponding

variance. Basis functions with less than exponential decay shouldbe avoided because of inferior local response.

The network output can be written in terms of the aboveGaussian basis function and the hidden-to-output connectionweights, wi, as:

f (�x) =K∑

i=1

wiϕ(�x). (35)

To account for large variances among the nodal outputs, the networkoutput is usually normalized. The normalized result is specified as:

f (�x) =∑K

i=1 wiϕ(�x)∑Ki=1 ϕ(�x)

, (36)

where K is the total number of basis functions.After the centers and widths of the basis functions are deter-

mined, the network weights can be computed from the following:

�y = Fnxp �w, (37)

where the elements of Fnxp are the activation functions, ϕij, whichare found by evaluating the j-th Gaussian function at the i-th inputvector. Typically Fnxp is rectangular with more rows than columns sothat �w is overdetermined and no exact solution exists. Thus, insteadof solving for the weights by matrix inversion, �w is determined by



solving a sum-of-squared-error functional as:

FTF �w = FT�y, such that�w = (FTF)−1FT�y

= F�y,, (38)

where F is called the pseudoinverse of F (4). In order to guarantee aunique solution to Eq. 19, F is better expressed as:

F = (FTF + εI)−1FT , (39)

where ε is a small constant known as the regularization parameter,and I is the identity matrix. If F is square and nonsingular, then sim-ple matrix inversion could be used to solve for the network weights.

When the amount of data is insufficient for complete approx-imation and that data is inherently noisy, it becomes necessary toimpose additional a priori constraints in order to manage the prob-lem of learning by approximation. The typical a priori supposition isthat of smoothness, or at least piecewise smoothness. The smooth-ness condition assumes that the response to an unknown data pointshould be similar to the response from its neighboring points. With-out the smoothness criterion, it would be infeasible to approximateany function because of the large number of examples required.34

Standard regularization is the method in learning for approxi-mation that utilizes a smoothness criterion. A regularization func-tion accomplishes two separate tasks: it minimizes the distancebetween the actual data and the desired solution, and it minimizesthe deviation from the learning constraint, which can be piecewisesmoothness in classification problems. The general functional to beminimized is as follows:

H[f ] =N∑

i=1

(f (�xi) − yi

)2 + ε∥∥Df

∥∥2, ε ∈ + (40)

where N is the dimension of the regularization solution, ε is thepositive regularization parameter, yi is the actual solution, f (�xi) isthe desired solution, and ‖Df‖2 is a stabilizer term with D as a first-order differential operator.



The solution to the above regularization functional is given by:

f (�x) =N∑

i=1

biG(�x; �ci), (41)

where G is the basis for the solution to the regularization problemcentered at ci, and bi = yi−f (�xi)

ε. Under conditions of rotational and

translational invariance, the solution can be written as:

f (�x) =N∑

i=1

biG∥∥�x − �ci

∥∥ , (42)

10.6 EXAMPLE CLASSIFICATION ANALYSIS USING FUZZYMEMBERSHIP FUNCTION

Skin lesion images obtained using the Nevoscope were classifiedusing different techniques into two classes, melanoma and dys-plastic nevus.36 The combined set of epi-illuminance and multi-spectral transilluminance images were classified using a waveletdecomposition based ADWAT method37 and Fuzzy MembershipFunction based classification.36 Wavelet transform based bimodalchannel energy features obtained from the images were usedin the analysis. Methods using both crisp and fuzzy member-ship based partitioning of the feature space were evaluated. Forthis purpose, the ADWAT classification method using crisp par-titioning was extended to handle multispectral image data. Also,multidimensional fuzzy membership functions with gaussian andbell profiles were used for classification. Results show that the fuzzymembership functions with bell profile are more effective than theextended ADWAT method in discriminating melanoma from dys-plastic nevus. The sensitivity and specificity of melanoma diagnosiscan be improved by adding the lesion depth and structure infor-mation obtained from the multispectral, transillumination imagesto the surface characteristic information obtained from the epi-illumination images.

Bimodal features were obtained from the epi-illuminationimages and the multispectral transillumination images using



wavelet decomposition and statistical analysis of the channel energyand energy ratios for the extended ADWAT classification method.All these features were combined to form a composite feature set.In this composite feature set, the dynamic range of the channelenergy ratio features is far less compared to the dynamic range ofthe channel energy features. For classification, it is necessary to nor-malize the feature set so that all the features have similar dynamicrange. Using linear transformations, all the features in the compos-ite feature set were normalized so that they have a dynamic rangebetween zero and one. Using the covariance information as obtainedfrom the feature distribution of the learning data set, the values ofthe dysplastic and melanoma membership functions were calcu-lated. Decision as to whether the unknown image set belongs to themelanoma or dysplastic nevus class was taken based on the “winnertakes all” criteria. The unknown image set was assigned to the classwith maximum membership function value. Although the member-ship functions can be thought of as multivariate conditional densi-ties similar to those used in the Bayes classifier, making the decisionbased on the probabilities of all the image classes for the candidate,gives the classifier its fuzzy nature.

Out of the 60 unknown images (15 melanoma and 45 dysplas-tic nevus cases) used in the classification phase, 52 cases were cor-rectly classified using the Gaussian membership function.36 All thecases of melanoma and 37 cases of dysplastic nevus were identifiedgiving a true positive fraction of 100 percent with a false positivefraction of 17.77 percent. For the eight dysplastic nevus cases thatwere misclassified, the values of both the melanoma and dysplasticnevus membership functions were equal to zero. These cases wereassigned to the melanoma category, since no decision about the classcan be taken if both the membership function values are the same.Classification results were obtained for the Bell membership func-tion using different values of the weighing constant W. Out of allthe values of W used, best classification results are obtained for avalue of 0.6, with a true positive fraction of 100 percent with a falsepositive fraction of 4.44 percent. The results obtained from all theseclassification techniques are summarized in Table 2.



Table 2. Results of Classification of Optical Images Using DifferentClassification Methods for Detection of Melanoma36

Type ofImages Used

Method Images CorrectlyClassified

TruePositive

FalsePositive

Melanoma Dysplastic

Epi-illuminance

Neural Network 13/15 34/45 86.66% 24.44%

Images BayesianClassifier

13/15 40/45 86.66% 11.11%

Multispectraland Epi-illuminanceImages

Fuzzy ClassifierwithGaussianMembershipFunction

15/15 37/45 100% 17.77%

Fuzzy Classifierwith BellMembershipFunctions

15/15 43/45 100% 4.44%


Clustering and image classification methods are critically impor-tant in medical imaging for computer-aided analysis and diagnosis.Though there is a wide spectrum of pattern analysis and classi-fication methods has been explored for medical image analysis,clustering and classification methods have to be customized andcarefully implemented for specific medical image analysis and deci-sion making applications.Agood understanding of the involvementof features and the contextual information may be incorporated inmodel-based approaches utilizing deterministic or fuzzy classifica-tion approaches.

References

1. Jain AK, Dubes RC, Algorithms for Clustering Data, Prentice Hall,Englewood Cliffs, NJ, 1998.

2. Duda RO, Hart PE, Stork DG, Pattern Classification (2nd edn.),John Wiley & Sons, 2001.



3. Zadeh LA, Fuzzy sets as a basis for a theory of possibility, Fuzzy Setsand Systems 1: 3–28, 1978.

4. Fisher D, Iterative optimization and simplification of hierarchical clus-tering, Journal of Artificial Intelligence Research 4: 147–179, 1996.

5. Karypis G, Han EH, Multilevel refinement for hierarchical clustering,Technical Report #99–020, 1999.

6. Bradley P, Fayyad U, Refining initial points for k-means clustering, inProceedings of the 15th ICML, pp. 91–99, Madison, WI, 1998.

7. Zhang B, Generalized k-harmonic means — Dynamic weighting of datain unsupervised learning, in Proceedings of the 1st SIAMICDM, Chicago,IL, 2001.

8. Campbell JG, Frakey C, Murtagh F, RafteryAE, Linear flaw detection inwoven textiles using model-based clustering, Pattern Recognition Letters18: 1539–1548, 1997.

9. Celeux G, Govaert G, Gaussian parsimonious clustering models,Pattern Recognition 28: 781–793, 1995 .

10. Olson C, Parallel algorithms for hierarchical clustering, Parallel Com-puting 21: 1313–1325, 1995.

11. Mao J, Jain AK, A self-organizing network for hyperellipsoidal cluster-ing (HEC), IEEE Trans Neural Network 7: 16–29, 1996.

12. Sibson R, SLINK: An optimally efficient algorithm for the single linkcluster method, Computer Journal 16: 30–34, 1973.

13. Voorhees EM, Implementing agglomerative hierarchical clusteringalgorithms for use in document retrieval, Information Processing andManagement 22(6): 465–476, 1986.

14. Dai S, Adaptive learning for event modeling and pattern classification,PhD dissertation, New Jersey Institute of Technology, Jan 2004.

15. Chiu T, Fang D, Chen J, Wang Y, A Robust and scalable clusteringalgorithm for mixed type attributes in large database environments, inProceedings of the 7th ACM SIGKDD, pp. 263–268, San Francisco, CA,2001.

16. Diday E, The dynamic cluster method in non-hierarchical clustering,J Comput Inf Sci 2: 61–88, 1973.

17. Symon MJ, Clustering criterion and multivariate normal mixture,Biometrics 77: 35–43, 1977.

18. Bezdek JC, Pattern Recognition With Fuzzy Objective Function Algorithms,Plenum Press, New York, NY, 1981.

19. Dave RN, Generalized fuzzy C-shells clustering and detection of cir-cular and elliptic boundaries, Pattern Recogn 25: 713–722, 1992.

20. Pham DL, Prince JL, Adaptive fuzzy segmentation of magnetic reso-nance images, IEEE Trans on Med Imaging 18(9): 737–752, 1999.



21. Vannier M, Pilgram T, Speidel C, et al., Validation of magnetic resonanceimaging (MRI) multispectral tissue classification, Computerized MedicalImaging and Graphics 15: 217–223, 1991.

22. Choi HS, Haynor DR, KimY, Partial volume tissue classification of mul-tichannel magnetic resonance images — A mixed model, IEEE Trans-actions on Medical Imaging 10: 395–407, 1991.

23. Zavaljevski A, Dhawan AP, Holland S, et al., Multispectral MR brainimage classification, Computerized Medical Imaging, Graphics and ImageProcessing 24: 87–98, 2000.

24. Nazif AM, Levine MD, Low-level image segmentation: An expert sys-tem, IEEE Trans Pattern Anal Mach Intell 6: 555–577, 1984.

25. Arata LK, Dhawan AP, Levy AV, et al., Three-dimensional anatomicalmodel based segmentation of MR brain images through prinicpal axesregistration, IEEE Trans Biomed Eng 42: 1069–1078, 1995.

26. Dhawan AP, Chitre Y, Kaiser-Bonassoand M Moskowitz, Analysis ofmammographic microcalcifications using gray levels image structurefeatures, IEEE Trans Med Imaging 15: 246–259, 1996.

27. Hall LO, Bensaid AM, Clarke LP, Velthuizen RP, et al., A comparison ofneural network and fuzzy clustering techniques in segmenting mag-netic resonance images of the brain, IEEE Trans on Neural Networks 3:672–682, 1992.

28. Xu L, Jackowski M, Goshtasby A, et al., Segmentation of skin cancerimages, Image and Vision Computing 17: 65–74, 1999.

29. Huo Z, Giger ML, Vyborny CJ, Computerized analysis of multiplemammographic views: Potential usefulness of special view mam-mograms in computer aided diagnosis, IEEE Trans Med Imaging 20:1285–1292, 2001.

30. Grohman W, Dhawan AP, Fuzzy convex set based pattern classifica-tion of mammographic microcalcifications, Pattern Recognition 34(7):119–132, 2001.

31. Bonasso C, GA based selection of mammographic microcalcifica-tion features for detection of breast cancer, MS Thesis, University ofCincinnati, 1995.

32. Peck C, Dhawan AP, A review and critique of genetic algorithm theo-ries, J of Evolutionary Computing, MIT Press 3(1): 39–80, 1995.

33. Dhawan AP, Medical Image Analysis, John Wiley Publications and IEEEPress June 2003, Reprint, 2004.

34. Zurada JM, Introduction to Artificial Neural Systems, West PublishingCo., 1992.

35. Mitra S, Pal SK, Fuzzy Multi-layer perceptron, inferencing and rulegeneration, IEEE Trans Neural Networks 6(1): 51–63, 1995.



36. Patwardhan S, Dai S, Dhawan AP, Multispectral image analysis andclassification of melanoma using fuzzy membership based partitions,Computerized Medical Imaging and Graphics 29: 287–296, 2005.

37. Patwardhan SV, Dhawan AP, Relue PA, Classification of melanomausing tree-structured wavelet transforms, Computer Methods and Pro-grams in Biomedicine 72: 223–239, 2003.


CHAPTER 11

Recent Advances in FunctionalMagnetic Resonance Imaging

Dae-Shik Kim

While functional imaging of the brain function using magnetic resonanceimaging (fMRI) has gained a wide acceptance as a useful tool in basic andclinical neurosciences, its ultimately utility remains elusive due to our lackof understanding of its basic physiological processes and limitations. In thepresent chapter, we will discuss recent advances that are shedding light onthe relationship between the observable blood oxygenation level depen-dent (BOLD) fMRI contrast and the underlying neuroelectrical activities.Finally, we will discuss topical issues that remain to be solved in future.

11.1 INTRODUCTION

The rapid progress of blood oxygenation level dependent (BOLD)functional magnetic resonance imaging (fMRI) in recent years1−3

has raised the hope that — unlike most existing neuroimaging tech-niques — the functional architecture of the human brain can bestudied directly in a noninvasive manner. The BOLD technique isbased on the use of deoxyhemoglobin as nature’s own intravas-cular paramagnetic contrast agent.4−6 When placed in a magneticfield, deoxyhemoglobin alters the magnetic field in its vicinity, par-ticularly when it is compartmentalized as it is within red bloodcells and vasculature. The effect increases as the concentration ofdeoxyhemoglobin increases. At concentrations found in venousblood vessels, a detectable local distortion of the magnetic field

267


268 Dae-Shik Kim

surrounding the red blood cells and surrounding blood vessel isproduced. This affects the magnetic resonance behavior of the waterproton nuclei within and surrounding the vessels, which in turnresults in decreases in the transverse relaxation times T2 and T∗

2.4,6

During the activation of the brain, this process is reduced: increase inneuronal and metabolic activity results in a reduction of the relativedeoxyhemoglobin concentration due to an increase of blood flow(and hence increased supply of fresh oxyhemoglobin) that follows.Consequently, in conventional BOLD fMRI, brain “activity” can bemeasured as an increase in T2 or T∗

2 weighted MR signals.1−3 Sinceits introduction about 10 years ago, BOLD fMRI was successfullyapplied — among numerous other examples — to precisely local-ize the cognitive,7 motor,8 and perceptual9−11 function of the humancortex cerebri (Figs. 1 and 2). The explanatory power of BOLD fMRI isbeing further strengthened in recent years through the introductionof high (∼3T) and ultrahigh (∼7T) MRI scanners.12 This is based onthe fact the stronger magnetic field will not only increase the fMRIsignal per se, but in addition, it will specifically enhance the sig-nal components originating from parenchymal capillary tissue. Onthe other hand, conventional, low-field magnets can be expected to“over-represent” macrovascular signals.

11.2 NEURAL CORRELATE OF fMRI

BOLD fMRI contrast does not measure neuronal activity per se.Rather, it reflects a complex convolution of changes ranging fromcerebral metabolic rate of oxygen (CMRO2), cerebral blood flow(CBF), and cerebral blood volume (CBV) following focal neuronalactivity (Fig. 1). This poses a fundamental problem for the accu-racy and validity of BOLD fMRI for clinical and basic neurosciences:while the greatest body of existing neurophysiological data providespiking and/or subthreshold measurements from a small numberof neurons (100–102), fMRI on the other hand labels the local hemo-dynamics from the parenchymal lattice consisting millions of neu-rons (106–108) and a dense network of microcapillaries. How can we


Recent Advances in Functional Magnetic Resonance Imaging 269

Fig. 1. Hemodynamic basis of functional MRI. Note that fMRI is an indirect mea-sure of the neuronal activity elicited by an external stimulus (“visual stimulation”)mediated through hemodynamic processes occurring in the dense network of veins(“V”), arteries (“A”) and capillaries.

bridge this gap from micron-scale neuronal receptive field proper-ties to millimeter scale voxel behaviors? The problem of bridgingthis conceptual gap is greatly hindered by the presence substan-tial differences between neuronal and fMRI voxel properties: smallnumber (100–102) versus large number (106–108) of neurons under-lying the observed activation; point-like individual neurons versusneurovascular lattice grid; largely spiking versus largely subthresh-old activities; excitatory or inihibitory versus excitatory and/orinhibitory (see Fig. 3 for differences in time scale between fMRIand electrophysiological signals). The crucial questions we need toaddress are discussed below.

11.2.1 Do BOLD Signal Changes Reflect the Magnitudeof Neural Activity Change Linearly?

Amplitude of the fMRI signal intensity change has been employedby itself to obtain information beyond simple identification of spa-tial compartmentalization of brain function by correlating variations


270 Dae-Shik Kim

Fig. 2. Functional MRI of the human visual cortex using BOLD contrast at 3T.Here, the receptive field properties for isoeccentricity was mapped using the stan-dard stimuli. Color-coded activation areas were responding to eccentricities repre-sented by the colored rings in the upper right corner. Regions of activity were weresuperimposed on the reconstructed and inflated brain surfaces.

in this amplitude with behavioral (e.g. Refs. 13–15) or the elec-troencephelography (EEG) response.16 However, extracting suchinformation requires the deconvolution of the compounded fMRIresponse,17 assuming that fMRI signals are additive. This assump-tion, however, appears not to be generally valid (e.g. Refs. 18–20).Tight and highly quantitative coupling between the EEG and T∗

2BOLD signals in the rat model was reported where the frequency offorepaw stimulation rate was varied under steady state conditions.21

A linear relationship between the BOLD response and somatosen-sory evoked potentials was demonstrated for brief stimuli but thenature of the relationship depended on the stimulation duration andultimately became nonlinear;22 in this study, the linearity was usedin a novel way to extract temporal information in the millisecondtime scale. More recently, local field potentials and spiking activ-ity was recorded for the first time simultaneously with T∗

2 BOLDfMRI signals in the monkey cortex, showing a linear relationship



between local field potentials and spiking rate, but displaying bet-ter correlation with the former.23 In a recent study, recording frommultiple sites for the first time, spiking activity was shown to be lin-early correlated with the T∗

2 BOLD response in the cat visual cortexusing a single orientation of a moving grid but with different spa-tial frequency of the grid lines.24 However, the correlation variedfrom point to point on the cortical surface and was generally validonly when the data were averaged at least over 4 mm–5 mm spa-tial scale,24 demonstrating the fact that T∗

2 BOLD responses are notspatially accurate at the level of orientation columns in the visualsystem, as discussed previously. A detailed set of studies were per-formed asking the same type of questions and using laser Dopplertechniques to measure cerebral blood flow (CBF)25,26,28; these stud-ies concluded that linear domains exist between CBF increases andaspects of electrical activity and that hemodynamic changes evokedby neuronal activity depend on the afferent input function but thatthey do not necessarily reflect output level of activity of a region.

11.2.2 Small Versus Large Number

Given the nominal voxel size of most fMRI scans (several milli-meters at best), it is safe to conclude that BOLD reflects the activ-ity of many neurons (let’s say, for a voxel of 1 mm3–2 mm3 around105 neurons).28 The overwhelming body of existing electrophysi-ological data, however, is based on electrode recordings from sin-gle (single unit recording, SUA) or a handful of neurons (multiunitrecording, MUA). The real question is hence to ask how accuratelythe responses of single cells (our “gold standard” given the existingbody of data) are reflected by a population response, such as in BOLDfMRI. Theoretically, if each neuron would “fire” independently of itsneighbor’s behavior, this would be an ill posed problem, as fMRI willnot be able to distinguish small activity changes in a large cellularpopulation from large changes in a small population. Fortunately,however, neurons are embedded in tight local circuitries, formingfunctional clusters with similar receptive field properties ragingfrom “micro-columns,” “columns,” to “hyper-columns.” Both the


272 Dae-Shik Kim

neuronal firing rate and phase are correlated between neighboringneurons (Singer, 1999), and in most sensory areas there is a goodcorrelation between local field potentials (LFP), which are assumedto reflect the average activity of a large number of neurons, andthe responses of individual spiking neurons. In fact, it is difficultto imagine how BOLD contrast could be detectable at all, if it weresensitized to the behavior of uncorrelated individual neurons, as themetabolic demand of a single neuron would be hardly sufficient toinitiate the chain of hemodynamic events giving rise to BOLD.

11.2.3 Relationship between Voxel Size and NeuralCorrespondence

Clearly, the MRI voxel size is a key element in determining the spa-tial dependence of the correlation between the BOLD and electrodedata. A large voxel will improve the relationship to the neuronalevent, since a voxel that displays BOLD signal changes will havea much higher probability of including the site of the electricallyactive column when its size increases, for example to sizes that areoften used in human studies (e.g. 3 mm3 × 3 mm3 × 3 mm3). How-ever, such a large voxel will provide only limited information aboutthe pattern of activation, due to its low spatial resolution. Smallervoxels (i.e. at the size of individual single unit recording sites) whichcould potentially yield a much better spatial resolution will result ina large variability in neuronal correspondence and the BOLD signaland a large number of “active” voxels will actually originate frompositions beyond the site of electrical activity (Fig. 4).

11.2.4 Spiking or Subthreshold?

According to the standard “integrate-and-fire” model of neurons,action potential is generated when the membrane potential reachesthreshold by depolarization, which in turn is determined by theintegration of incoming excitatory (EPSP) and inhibitory (IPSP)post-synaptic potentials. Action potentials are usually generatedonly around the axon hillock, while synaptic potentials can be


Recent Advances in Functional Magnetic Resonance Imaging 273S

pike

s/se

c

BO

LD [

%]

Time after stimulus onset [sec]

0 10 20

25

0 0.0

1.0

30

Rec

tifie

d vo

lts[µ

V] 0

-50

BOLD

Low frequency analogelectrode signals (100-300Hz)

Spike rate

Bin size = TR = 0.5 secstimulus

Fig. 3. Time course of BOLD and single unit recordings from the same corticallocation. Identical visual stimuli were used for fMRI and subsequent single unitrecording sessions. Blue trace: peristimulus histogram of the spike activity. Binsize for the histogram = 0.5 sec = TR for fMRI. Red trace: BOLD percent changesduring visual stimulation. X-axis: time after stimulus onset. Left Y-axis: Spikes persecond. RightY-axis: BOLD percent changes. Gray box: stimulus duration. The blacktrace above indicates the original low-frequency analog signals (100 Hz–300 Hz)underlying the depicted spike counts.

generated all across the dendritic tree (mostly on dendritic spines)and cell soma. The threshold-dependent action potential firingmeans that much more sub- and suprathreshold synaptic activitythan action potential activity is likely at any one time. And themuch larger neural surface area associated with synaptic activitymeans that the total metabolic demand (i.e. number of Na+/K+pumps involved etc.) for synaptic activity ought to be signifi-cantly higher than those required for generating action poten-tials. It seems therefore likely to be the case that BOLD con-trast — like other methods based on cortical metabolism, such as2-DG (14C-2-deoxyglucose)49 and optical imaging — is dominatedby the synaptic subthreshold activity. However, the precise


274 Dae-Shik Kim

contributions of synaptic and spiking activities are hard to quantify,since with conventional stimuli one would expect synaptic input andspiking output activity to be roughly correlated with each other.29−31

Indeed, it is not trivial to imagine an experiment where input andoutput activities would not correlate with each other. One way thishas been proposed in the past, is to look in a visual area at spa-tial activity resulting from the edge of a visual stimulus.32−34 Since“extra-classical” receptive fields across such an edge are by defini-tion subthreshold activity, it follows that a stimulus with an edgein it creates regions of cortex where activity is only subthresholdin origin. Existing optical imaging studies32,35 have concluded thatsubthreshold activity does indeed contribute significantly to the opti-cal signal, suggesting that it might contribute to the BOLD signalas well. The results of our combined BOLD and single unit studiessuggest that both local field potential (LFP) and single-unit correlatewell with the BOLD signal (see Figs. 3 and 4). We have used LFP on

1 70

0.5

1.0

Neural modulation [∆ spikes/sec]

BO

LD

mod

ulat

ion

[ %

]∆

R = 0.852

y = 0.12x + .0857.95 spikes per 1% BOLD

Fig. 4. Results of direct comparison between BOLD and single unit recordingsacross all sites (n = 58). X-axis: neural modulation for the single unit response inspikes per seconds. Y-axis: % BOLD modulation. The six data points indicate theBOLD/single unit responses for six different spatial frequencies used for this study.The thick black line is the regression line for the depicted data points. Coefficientof determination of the regression line, R2 = 0.85.



the assumption that it represents the average activity of thousandsof neurons. In agreement with previous findings,36 LFP signals mayprovide a better estimate of BOLD responses than suprathresholdspike rate. However, whether intracellular or extracellular activity isbetter correlated with BOLD is harder to address, since with a gratingstimulus (and in fact with many types of visual stimuli), one wouldexpect intracellular and extracellular activity to be roughly corre-lated with each other.29−31 Separating intracellular and extracellu-lar activity would have to be accomplished using a visual stimulusknown to do so. One imaging experiment presumptively showinga large contribution of intracellular activity to the optical imagingsignal uses focal iontophoresis of GABA-A antagonist bicucullinemethiodide37,38 to generate a mismatch between intracellular andextracellular activity. This is a rare case where a blood-dependentsignal could be reversibly altered by an artificial manipulation ofneural activity. We are currently repeating these studies using fMRItechniques to elucidate the spatial contribution of the intracellularand extracellular activity in BOLD functional MRI signals.

11.2.5 Excitatory or Inhibitory Activity?

Although the neuro- and cognitive-science communities haveembraced fMRI with exuberance, numerous issues remain poorlyunderstood regarding this technique. Because fMRI maps are basedon secondary metabolic and hemodynamic events that follow neu-ronal activity, and not the electrical activity itself, it remains mostlyunclear what the spatial specificity of fMRI is (i.e. how accurate arethe maps generated by fMRI compared to actual sites of neuronalactivity?). In addition, the nature of the link between the magnitudesof neuronal activity versus fMRI signals is not well understood (i.e.what does a change of particular magnitude in fMRI signals meanwith respect to the change in magnitude of processes that defineneuronal signaling, such as action potentials or neurotransmitterrelease?). fMRI is often used without considering these unknowns.For example, modulating the intensity of fMRI signals by meansof different paradigms and interpreting the intensity changes as


276 Dae-Shik Kim

changes in neuronal activity of corresponding magnitude is a com-mon practice that is not fully justified under most circumstances. Tothe best of our knowledge, there is currently no evidence that themetabolic demands differ greatly between excitatory and inhibitorysynapses. Therefore, fundamentally, both the excitatory (EPSP) andinhibitory (IPSP) synaptic inputs can be expected to cause simi-lar metabolic and hemodynamic events ultimately giving rise tosimilar BOLD contrasts. On the site of the spiking output activ-ity, however, they have an opposite effect: accumulation of EPSPswill increase the probability for spike generation (and thereforealso the metabolic demand), while IPSPs will decrease it. Assumingthat the BOLD response predominantly reflects changes in synap-tic subthreshold activity, it remains elusive whether excitatory andinhibitory cortical events can be differentiated using the BOLDresponse in any single region. Recently, one group proposed thatinhibition, unlike excitation, elicits no measurable change in theBOLD signal.39 They hypothesized that because of the lower num-ber of inhibitory synapses,40 their strategically superior location(inhibitory receptors: basal cell body; excitatory receptors: distaldendrites), and increased efficiency41 there could be lower metabolicdemand during inhibition compared to excitation. The validity ofthis claim notwithstanding, both empirical and theoretical studiessuggest that excitatory and inhibitory neurons in the cortex are sotightly interconnected in local circuits (see e.g. Ref. 42 for details ofthe local circuitry in cat primary visual cortex; see also Ref. 43 forthe anatomy of local inhibitory circuits in cats) that one is unlikely toobserve an increase in excitation without an increase in inhibition.After all, for an inhibitory neuron to increase its firing rate, it mustbe receiving more excitatory input, and most of the excitatory inputcomes from the local cortical neighborhood (see Refs. 42 and 44 foroverview). Naturally, excitation and inhibition would not occur intemporal unison, as otherwise no cell would reach threshold. Onthe temporal scale of several hundred milliseconds to seconds dur-ing which BOLD contrast emerges,3 however, such potential tem-poral differences would most likely be rendered indistinguishable.One viable hypothesis is therefore that BOLD contrast reflects a



steady-state balance of local excitation and inhibition. In particularif BOLD is more sensitive to subthreshold than to spiking activity.

11.3 NON-CONVENTIONAL fMRI

BOLD fMRI at conventional low magnetic field of 1.5T can com-monly achieve a spatial resolution of up to 3–5 millimeters. Thisis sufficient for labeling cortical organization at hypercolumn (sev-eral millimeters) or area (several centimeters) scales. But functionalimages at this resolution fail to accurately label the columnar orga-nization of the brain. Studies at higher magnetic fields (such as 3 or7T) can produce significant enhancement of the spatial resolutionand specificity of fMRI. Theoretical and experimental studies haveshown at least a linear increase in signal-to-noise ratio (SNR) withmagnetic field strength. The increase of the static MR signal can beused to reduce the volume needed for signal averaging. Further-more, as the field strength increases, the field gradient around thecapillaries becomes larger and extends further into the parenchymathus increasing the participation of the brain tissue in functional sig-nal. Concurrently, the shortened T∗

2 of the blood at high B0 reducesthe relative contribution from the large veins.

While these results suggest that stronger magnetic field per sewill specifically enhance the signal components originating fromparenchymal capillary tissue, recent optical spectroscopy and func-tional MRI data45−48 suggest that the spatial specificity of BOLDcould be further and more dramatically improved if an — hypoth-esized — initial decrease of MR signals can be utilized for functionalimaging formation. To this end, it is suggested that the first eventfollowing focal neuronal activity is a prolonged increase in oxygenconsumption, caused by an elevation in oxidative metabolism ofactive neurons. Based on 2-DG data,49 one can assume the increasein oxidative metabolism in mammalian cortex to be colocalized withthe site of electrical activity. The increase in oxidative metabolismwill naturally elevate the local deoxyhemoglobin content in theparenchyma of active neurons, assuming there is no immedi-ate commensurate change in cerebral blood flow.50 In T2 or T∗

2


278 Dae-Shik Kim

weighted BOLD fMRI images, such increase in paramagnetic deoxy-hemoglobin should therefore be detectable as a transient decreasein observable MR signals. Such an initial deoxygenation of thelocal cortical tissue will last only for a brief period, as fresh blood(fresh oxyhemoglobin) will rush into capillaries in response to theincreased metabolism, thus reversing the local ratio of hemoglobinin favor of oxyhemoglobin, and hence resulting in a delayed increasein observable MR signals (i.e. the conventional BOLD signal). Thecrucial question here is the “where” of the above described “bipha-sic” hemodynamic processes. Grinvald and coauthors51,46 hypoth-esized a fundamentally distinct functional specificity for these twoevents: The initial deoxygenation, as a consequence of an increasein oxidative metabolism, should be coregistered with the site ofelectrical activity up to the level of individual cortical columns (infact, the well established “optical imaging of intrinsic signals”,52,53

which has been cross validated with single unit techniques,54,55 issimilarly based on measuring the local transient increase of deoxy-hemoglobin). The delayed oxygenation of the cortical tissue on theother hand, is suggested to be far less specific due to the spreadof hemodynamic activity beyond the site of original neural activ-ity. Both the existence of “biphasic” BOLD response per se, and thesuggested differences in functional specificity has been the subjectof heated controversies in recent years (see Ref. 56 for a comprehen-sive update of this saga). While the initial deoxygenation signal infMRI (termed “initial dip”) has been reported in awake behavinghumans57,58 and anesthetized monkeys,59 studies in rodents failedto detect any significant initial decrease in BOLD signal followingsensory stimulation,60−62 but see Ref. 63. The question of whetherthe use of initial dip would indeed improve the spatial specificityof BOLD has been far more difficult to address experimentally. Thisis largely because most fMRI studies examining this phenomenonso far have been conducted in humans (e.g. Refs. 57 and 64), andtherefore, by necessity have used relatively coarse nominal spatialresolution above the level of the individual cortical columns. In ani-mal studies using ultra-high magnetic fields (e.g. 9.4T), in whichfunctional images at submillimeter scale can be acquired, the results



of our own group45 (Fig. 6) suggest that indeed the use of the “initialdip” can significantly improve the spatial specificity of BOLD. Thisresult has been questioned afterwards65; see Ref. 66 for our reply.On the other hand, in a recent pioneering study, preoperative func-tional MRI and intraoperative optical imaging were performed inthe same human subject.67 While the spatial overlap between opti-cal imaging and conventional (positive) fMRI was poor, there wasa dramatic improvement in spatial correspondence between thetwo dataset when the initial dip portion of the MRI signal wasused. Furthermore, combined single unit and oxygen tension probemeasurements68 convincingly demonstrated both the presence aswell as the functional significance of the initial deoxygenation sig-nal component.

Alternative to the initial deoxygenation signals, the spatial speci-ficity of T∗

2 based fMRI can be further improved if only the arterialcontribution and/or to attenuate the draining vessel artifacts areutilized for functional image construction. For example, perfusionweighted images based on arterial spin labeling can be made sensi-tive to the cerebral blood flow (CBF) changes from upstream arterialnetworks to the capillaries, thus providing better spatial localizationability69,70 than T∗

2 BOLD imaging methods.

11.4 CONCLUSIONS AND FUTURE PROBLEMS OF fMRI

In less than a decade since the first noninvasive measurements offunctional blood oxygenation level signals from the human brain,fMRI has developed into an indispensable neuroimaging tool thatis ubiquitous in both clinical and basic neuroscience settings. Theexplanatory power of fMRI however, is currently limited due to pres-ence of major theoretical and practical shortcoming. These include(but not limited to): (a) lack of the detailed understanding of itsneural correlate; (b) limited spatial resolution; and (c) the difficultyin combining fMRI with other imaging/measurement techniques.Furthermore, it is important to note that conventional functionalMRI data analysis techniques (e.g. General Linear Model, t-test,


280 Dae-Shik Kim

Fig. 5. Figure 4 shows the neuronal correspondence (R2 between BOLD andsingle unit responses) as a function of the reshuffled voxel sizes. For eachvoxel size, the distribution of the neuronal qualities is indicated by the stan-dard deviation. The red curve marks the mean neuronal correspondence foreach voxel size. For curve fitting, conventional sigmoidal fitting was used. Theresults depicted in Fig. 8 predict that the neuronal correspondence saturatesaround R2 = 0.7 at the voxel size of around 4.7 × 4.7 mm2. Larger voxel sizesare suggested to be ineffective in further improving the level of neuronal cor-respondence. That is, the maximum amount of variance in the underlying neu-ronal modulation that can be explained with the variance of conventional T∗

2based positive BOLD is about 70%. Once the voxel size has been reduced tobe smaller than ∼2.8 × 2.8 mm2, only less than 50% of the variance in theunderlying neuronal modulation can be explained through the observed BOLDresponses.

cross-correlation etc.) implicitly assume a modularity of corticalfunctions: parametric statistical methods test the hypothesis thatcertain areas of the brain are significantly more active than otherswith non-vanishing residual false positive detection error (repre-sented as p-value). However, such techniques assume that the brainconsists of individual computational modules (similar to “Phreno-logical” ideas) that are spatially distinct from each other. Interest-ingly, increasing number of evidences in recent years suggest an



alternative representation model: that the information in the brainis represented in a more distributed fashion.71,72 In the latter case, theconventional statistical techniques may fail to detect the correct pat-tern of neuronal activation, because they attempt to detect the areasof “strongest” activation, while the information may representedinformation using a much larger area of cortical tissue than conven-tionally assumed. In their original works, Haxby and colleagues71

have used simple voxel-to-voxel comparison methods to look for

Fig. 6. Improvement of BOLD spatial specificity by using nonconventional func-tional MRI signals. Time course on the left side shows biphasic evolution of MRsignals, resulting the early deoxygenation contrast. If used, such deoxygenationsignals produce high-resolution images of exceedingly high functional specificity(termed BOLD−) that contrasts with conventional BOLD fMRI signals (termedBOLD+).


282 Dae-Shik Kim

activity pattern in the human brain. Linear pattern discriminationtechniques, such as support vector machines (SVM) or fisher’s lin-ear discriminators (FLD) are inherently better suited for classifyingobserved activation pattern into separable categories. For example,when applied for discriminating orientation tuning behavior of vox-els from primary visual areas, SVM was able to detect minute differ-ences in orientation selectivity of individual voxels in human V1.73

Finally, while fMRI provides detailed information about the“where” of the brain’s functional architecture non-invasively, suchlocalization information alone, must leave pivotal questions aboutthe brain’s information processing (the “how” of the processing)unanswered, as long as the underlying pattern of neuronal connec-tivity cannot be mapped in an equally non-invasive manner. FuturefMRI studies in cognitive neuroimaging studies will have to embracea significantly more multimodal approach. For example, combin-ing fMRI with diffusion tensor imaging74,75 will label the pattern ofstructural connectivity between functionally active areas. The direc-tion of the flow of functional flow of information within this meshof neural networks could then be elucidated by performing time-resolved fMRI, effective connectivities, and possibly also repetitivetranscranial magnetic stimulations (rTMS) together with high reso-lution fMRI experiments.

11.5 ACKNOWLEDGMENTS

We thank Drs Louis Toth, Itamar Ronen, Mina Kim and KamilUgurbil for their help during the studies. This work was supportedby grants from NIH (MH67530, NS44820).

References

1. Bandettini PA, Wong EC, Hinks RS, Tikofsky RS, et al., Time course EPIof human brain function during task activation, Magn Reson Med 25:390–397, 1992.

2. Kwong KK, Belliveau J, Chesler DA, Goldberg IE, et al., Dynamicmagnetic resonance imaging of human brain acrivity during primarysensory stimulation, Proc Natl Acad Sci USA 89: 5675–5679, 1992.



3. Ogawa S, Tank DW, Menon R, Ellermann JM, et al., Intrinsic signalchanges accompanying sensory stimulation: Functional brain map-ping with magnetic resonance imaging, Proc Natl Acad Sci USA 89:5951–5955, 1992.

4. Ogawa S, Lee TM, Nayak AS, Glynn P, Oxygenation-sensitive contrastin magnetic resonance image of rodent brain at high magnetic fields,Magn Reson Med 14: 68–78, 1990.

5. Pauling L, Coryell CD, The magnetic properties and structures ofhemoglobin, oxyhemoglobin, and carbonmonoxyhemoglobin, ProcNatl Acad Sci USA 22: 210–216, 1936.

6. Thulborn KR, Waterton JC, Mattews PM, Radda GK, Dependence ofthe transverse relaxation time of water protons in whole blood at highfield, Biochem Biophys Acta 714: 1982.

7. Wagner AD, Schacter DL, Rotte M, Koutstaal W, et al., Building mem-ories: Remembering and forgetting of verbal experiences as predictedby brain activity, Science 281: 1188–1191, 1998.

8. Kim SG, Ashe J, Hendrich K, Ellermann JM, et al., Functional mag-netic resonance imaging of motor cortex: Hemispheric asymmetry andhandedness, Science 261: 615–617, 1993.

9. Engel SA, Glover GH, Wandell BA, Retinotopic organization in humanvisual cortex and the spatial precision of functional MRI, Cereb Cortex7: 181–192, 1997.

10. Sereno MI, Dale AM, Reppas JB, Kwong KK, et al., Borders of multi-ple visual areas in humans revealed by functional magnetic resonanceimaging, Science 268: 889–893, 1995.

11. Tootell RB, Mendola JD, Hadjikhani NK, Ledden PJ, et al., Functionalanalysis of V3A and related areas in human visual cortex, J Neurosci 17:7060–7078, 1997.

12. Ugurbil K, Toth L, Kim DS, How accurate is magnetic resonance imag-ing of brain function? Trends Neurosci 26: 108–114, 2003.

13. Gandhi SP, Heeger DJ, Boynton GM, Spatial attention affects brainactivity in human primary visual cortex, Proc Natl Acad Sci USA 96:3314–3319, 1999.

14. Salmelin R, Schnitzler A, Parkkonen L, Biermann K, et al., Native lan-guage, gender, and functional organization of the auditory cortex, ProcNatl Acad Sci USA 96: 10460–10465, 1999.

15. Tagaris GA, Kim, SG, Strupp JP,Andersen P, et al., Mental rotation stud-ied by functional magnetic resonance imaging at high field (4 tesla):Performance and cortical activation, J of Cogn Neurosci 9: 419–432, 1997.

16. Dehaene S, Spelke E, Pinel P, Stanescu R, et al., Sources of mathematicalthinking: Behavioral and brain-imaging evidence, Science 284: 970–974,1999.


284 Dae-Shik Kim

17. Glover GH, Deconvolution of impulse response in event-related BOLDfMRI, Neuroimage 9: 416–429, 1999.

18. Boynton GM, Engel SA, Glover GH, Heeger DJ, Linear systems analysisof functional magnetic resonance imaging in human V1, J Neurosci 16:4207–4221, 1996.

19. Sidtis JJ, Strother SC, Anderson JR, Rottenberg DA, Are brain functionsreally additive? Neuroimage 9: 490–496, 1999.

20. Vazquez AL, Noll DC, Nonlinear aspects of the BOLD response infunctional MRI, Neuroimage 7: 108–118, 1998.

21. Brinker G, Bock C, Busch E, Krep H, et al., Simultaneous recording ofevoked potentials and T�

2-weighted MR images during somatosensorystimulation of rat, Magn Reson Med 41: 469–473, 1999.

22. Ogawa S, Lee TM, Stepnoski R, Chen W, et al., An approach to probesome neural systems interaction by functional MRI at neural time scaledown to milliseconds (In Process Citation), Proc Natl Acad Sci USA 97:11026–11031, 2000.

23. Logothetis NK, Pauls J, Augath M, Trinath T, et al., Neurophysiologi-cal investigation of the basis of the fMRI signal, Nature 412: 150–157,2001.

24. Toth LJ, Ronen I, Olman C, Ugurbil K, et al., Spatial correlation of BOLDactivity with neuronal responses, Paper presented at Soc. Neurosci,Abstracts, 2001.

25. Lauritzen M, Relationship of spikes, synaptic activity, and localchanges of cerebral blood flow, J Cereb Blood Flow Metab 21: 1367–1383, 2001.

26. Mathiesen C, Caesar K,Akgoren N, Lauritzen M, Modification of activ-itydependent increases of cerebral blood flow by excitatory synap-tic activity and spikes in rat cerebellar cortex, J Physiol 512(Pt 2):555–566, 1998.

27. Mathiesen C, Caesar K, Lauritzen M, Temporal coupling betweenneuronal activity and blood flow in rat cerebellar cortex as indi-cated by field potential analysis, J Physiol 523(Pt 1): 235–246,2000.

28. Braitenberg V, Brain size and number of neurons: An exercise in syn-thetic neuroanatomy, J Comput Neurosci 10: 71–77, 2001.

29. Ferster D, Linearity of synaptic interactions in the assembly of receptivefields in cat visual cortex, Curr Opin Neurobiol 4: 563–568, 1994.

30. Jagadeesh B, Wheat HS, Ferster D, Linearity of summation of synapticpotentials underlying direction selectivity in simple cells of the catvisual cortex, Science 262: 1901–1904, 1993.

31. Nelson S, Toth L, Sheth B, Sur M, Orientation selectivity of corti-cal neurons during intracellular blockade of inhibition, Science 265:774–777, 1994.



32. Grinvald A, Lieke EE, Frostig RD, Hildesheim R, Cortical point-spreadfunction and long-range lateral interactions revealed by real-time opti-cal imaging of macaque monkey primary visual cortex, J Neurosci 14:2545–2568, 1994.

33. Gulyas B, Orban GA, Duysens J, Maes H, The suppressive influenceof moving textured backgrounds on responses of cat striate neurons tomoving bars, J Neurophysiol 57: 1767–1791, 1987.

34. Knierim JJ, van Essen DC, Neuronal responses to static texture patternsin area V1 of the alert macaque monkey, J Neurophysiol 67: 961–980, 1992.

35. Toth LJ, Rao SC, Kim DS, Somers D, et al., Subthreshold facilitationand suppression in primary visual cortex revealed by intrinsic signalimaging, Proc Natl Acad Sci USA 93: 9869–9874, 1996.

36. Logothetis NK, Pauls J, Augath M, Trinath T, et al., Neurophysiologicalinvestigation of the basis of the fMRI signal, Nature 412: 150–157, 2000.

37. Ajima A, Matsuda Y, Ohki K, Kim DS, et al., GABA-mediated repre-sentation of temporal information in rat barrel cortex, Neuroreport 10:1973–1979, 1999.

38. Toth LJ, Kim DS, Rao SC, Sur M, Integration of local inputs in visualcortex, Cereb Cortex 7: 703–710, 1997.

39. Waldvogel D, van Gelderen P, Muellbacher W, Ziemann U, et al., Therelative metabolic demand of inhibition and excitation, Nature 406:995–998, 2000.

40. Beaulieu C, Colonnier, M, A laminar analysis of the number ofroundasymmetrical and flat-symmetrical synapses on spines, dendritictrunks, and cell bodies in area 17 of the cat, J Comp Neurol 231: 180–189,1985.

41. Koos T, Tepper JM, Inhibitory control of neostriatal projection neuronsby GABAergic interneurons, Nat Neurosci 2: 467–472, 1999.

42. Payne BR, Peters A, The cat primary visual cortex, Academic Press,San Diego, 2001.

43. Kisvarday ZF, Kim DS, Eysel UT, Bonhoeffer T, Relationship betweenlateral inhibitory connections and the topography of the orientationmap in cat visual cortex, Eur J Neurosci 6: 1619–1632, 1994.

44. Sherpherd GM, The synaptic organization of the brain, Oxford Univer-sity Press, Oxford, 1990.

45. Kim DS, Duong TQ, Kim SG, High-resolution mapping of isoorienta-tion columns by fMRI, Nat Neurosci 3: 164–169, 2000.

46. Malonek D, Grinvald A, Interactions between electrical activity andcortical microcirculation revealed by imaging spectroscopy: Implica-tions for functional brain mapping, Science 272: 551–554, 1996.

47. Malonek D, Grinvald A, Vascular regulation at sub millimeter range.Sources of intrinsic signals for high resolution optical imaging, Adv ExpMed Biol 413: 215–220, 1997.


286 Dae-Shik Kim

48. Vanzetta I, Grinvald A, Increased cortical oxidative metabolism due tosensory stimulation: Implications for functional brain imaging, Science286: 1555–1558, 1999.

49. Sokoloff L, Reivich M, Kennedy C, Des Rosiers MH, et al., The[14C]deoxyglucose method for the measurement of local cerebral glu-cose utilization: Theory, procedure, and normal values in the consciousand anesthetized albino rat, J Neurochem 28: 897–916, 1977.

50. Fox PT, Raichle ME, Focal physiological uncoupling of cerebral bloodflow and oxidative metabolism during somatosensory stimulation inhuman subjects, Proc Natl Acad Sci USA 83: 1140–1144, 1986.

51. Malonek D, Dirnagl U, Lindauer U, Yamada K, et al., Vascular imprintsof neuronal activity: Relationships between the dynamics of corticalblood flow, oxygenation, and volume changes following sensory stim-ulation, Proc Natl Acad Sci USA 94: 14826–14831, 1997.

52. Frostig RD, Lieke EE, Ts’o DY, Grinvald A, Cortical functional architec-ture and local coupling between neuronal activity and the microcircu-lation revealed by in vivo high-resolution optical imaging of intrinsicsignals, Proc Natl Acad Sci USA 87: 6082–6086, 1990.

53. Grinvald A, Lieke E, Frostig RD, Gilbert CD, et al., Functional architec-ture of cortex revealed by optical imaging of intrinsic signals, Nature324: 361–364, 1986.

54. Crair MC, Gillespie DC, Stryker MP, The role of visual experience inthe development of columns in cat visual cortex, Science 279: 566–570,1998.

55. Shmuel A, Grinvald A, Functional organization for direction of motionand its relationship to orientation maps in cat area 18, J Neurosci 16:6945–6964, 1996.

56. Buxton RB, The elusive initial dip, Neuroimage 13: 953–958, 2001.57. Hu X, Le TH, Ugurbil K, Evaluation of the early response in fMRI in

individual subjects using short stimulus duration, Magn Reson Med 37:877–884, 1997.

58. Menon RS, Ogawa S, Hu X, Strupp JP, et al., BOLD based functional MRIat 4 Tesla includes a capillary bed contribution: Echo-planar imagingcorrelates with previous optical imaging using intrinsic signals, MagnReson Med 33: 453–459, 1995.

59. Logothetis NK, Guggenberger H, Peled S, Pauls J, Functional imagingof the monkey brain, Nat Neurosci 2: 555–562, 1999.

60. Lindauer U, Royl G, Leithner C, Kuhl M, et al., No evidence for earlydecrease in blood oxygenation in rat whisker cortex in response tofunctional activation, Neuroimage 13: 988–1001, 2001.

61. Marota JJ, Ayata C, Moskowitz MA, Weisskoff RM, et al., Investigationof the early response to rat forepaw stimulation, Magn Reson Med 41:247–252, 1999.



62. Silva AC, Kim SG, Pseudo-continuous arterial spin labeling techniquefor measuring CBF dynamics with high temporal resolution, MagnReson Med 42: 425–429, 1999.

63. Mayhew J, Johnston D, Martindale J, Jones M, et al., Increased oxygenconsumption following activation of brain: Theoretical footnotes usingspectroscopic data from barrel cortex, Neuroimage 13: 975–987, 2001.

64. Yacoub E, Le TH, Ugurbil K, Hu X, Further evaluation of the initialnegative response in functional magnetic resonance imaging, MagnReson Med 41: 436–441, 1999.

65. Logothetis N, Can current fMRI techniques reveal the micro-architecture of cortex? Nat Neurosci 3: 413–414, 2000.

66. Kim DS, Duong TQ, Kim SG, Reply to “Can current fMRI techniquesreveal the micro-architecture of cortex?” Nat Neurosci 3: 414, 2000.

67. CannestraAF, Pouratian N, Bookheimer SY, Martin NA, et al., Temporalspatial differences observed by functional MRI and human intraoper-ative optical imaging, Cereb Cortex 11: 2001.

68. Thompson JK, Peterson MR, Freeman RD, Single-neuron activity andtissue oxygenation in the cerebral cortex, Science 299: 1070–1072, 2003.

69. Duong TQ, Kim DS, Ugurbil K, Kim SG, Localized cerebral blood flowresponse at submillimeter columnar resolution, Proc Natl Acad Sci USA98: 10904–10909, 2001.

70. Luh WM, Wong EC, Bandettini PA, Ward BD, et al., Comparison ofsimultaneously measured perfusion and BOLD signal increases duringbrain activation with T(1)-based tissue identification, Magn Reson Med44: 137–143, 2000.

71. Haxby JV, Gobbini MI, Furey ML, Ishai A, et al., Distributed and over-lapping representations of faces and objects in ventral temporal cortex,Science 293: 2425–2430, 2001.

72. Ishai A, Ungerleider LG, Haxby JV, Distributed neural systems for thegeneration of visual images, Neuron 28: 979–990, 2000.

73. Kim DS, Kim M, Ronen I, Formisano E, et al., In vivo mapping of func-tional domains and axonal connectivity in cat visual cortex using mag-netic resonance imaging, Magn Reson Imaging 21: 1131–1140, 2003.

74. Kim M, Ducros M, Carlson T, Ronen I, et al., Anatomical correlatesof the functional organization in the human occipitotemporal cortex,Magn Reson Imaging 24: 583–590, 2006.

75. Singer, W, Time as coding space? Curr Opin Neurobiol 9: 189–194, 1999.


CHAPTER 12

Recent Advances in DiffusionMagnetic Resonance Imaging

Dae-Shik Kim and Itamar Ronen

Diffusion weighted magnetic resonance imaging (DWI) plays an increas-ingly important role in clinical and basic neurosciences. This is thanksto DWI’s exceptional capability in representing structural properties ofneural tissue as local water molecular displacements: changes in meandiffusivity reflect changes in macroscopic structural properties, whilegradient-direction encoded diffusion tensor imaging (DTI) can revealneuroanatomical connections in a noninvasive manner. Finally, recentadvances in compartmental-specific diffusion MRI suggest that micro-scopic cellular tissue properties might be measurable as well using diffu-sion MRI.

12.1 INTRODUCTION

Magnetic resonance imaging has paved the way for accurately map-ping the structural and functional properties of the brain in vivo.In particular, the intrinsic noninvasiveness of magnetic resonance(MR) methods and the sensitivity of the MR signal to subtle changesin the structural and physiological neuronal tissue fabric makeit an all but ideal research and diagnostic tool for characterizingintact neural tissue and studying processes that affect neural tis-sue properties such as cortical thinning, demyelination, and nervedegeneration/regeneration following injury. To this end, the tech-nique of diffusion weighted MRI (DWI) has become one of the pri-mary research and diagnostic tools in evaluating tissue structure

289


290 Dae-Shik Kim and Itamar Ronen

thanks to its ability to represent structural properties of neuraltissue as local water molecular displacements. For example, thesharp difference in structural characteristics between tissue prop-erties in the central nervous system has been extensively exploitedin countless DWI applications, ranging from the characterization ofischemia,1,2 demarcation of brain tumors3 and the extensive inves-tigation of connectivity through the use of diffusion tensor imaging(DTI).4,5 In addition, recent advances in diffusion tensor imaging(DTI) promises to label axonal connectivity pattern in a noninvasivemanner by utilizing directionally encoded local water diffusivity.Finally, recent advances in compartmental-specific diffusion MRIsuggest that diffusion MRI might be also able to provide semiquan-titative information about microscopic cellular tissue properties.

12.1.1 Brownian Motion and Molecular Diffusion

The essential nature of diffusion is that a group of molecules thatstart at the same location will spread out over time. Each moleculeexperiences a series of random displacements so that after a time T,the spread of position along a spatial axis x has a variance of:

σ2x = 2DT, (1)

where D is the diffusion coefficient, a constant characteristic of themedium. Diffusion of water molecules in most biological tissues isrecognized as being smaller than the value in pure water. In the braintissue, the diffusion coefficient is two to ten times lower than in purewater.6 It has been shown that in brain gray matter, the diffusionproperties are relatively independent of orientation (or isotropic).Conversely, in fibrous tissues such as brain white matter, the diffu-sion properties vary with orientation. A very important empiricalobservation is that the diffusion parallel to the fiber is much greaterthan the diffusion perpendicular to it.7 The variation with orienta-tion is termed diffusion anisotropy (Fig. 1). Isotropic diffusion mayindicate either a structurally isotropic medium, or the existence ofmultiple anisotropic structures that are randomly oriented in thesame sample volume.


Recent Advances in Diffusion Magnetic Resonance Imaging 291

Fig. 1. The upper panel shows a schematic representation of a typical white mat-ter voxel. The voxel is mostly occupied by closely packed myelinated axons. Watermolecule diffusion is restricted in the direction perpendicular to the axonal fibersleading to an anisotropic diffusion pattern. In the lower panel, a schematic repre-sentation of a gray matter voxel is shown. Although the presence of cell membranesstill poses restriction on diffusion, the well oriented structure of white matter fibertract no longer exists, and thus the diffusion pattern is more isotropic.

12.1.2 Anisotropic Diffusion

Whereas the factors that determine the lower diffusion coefficient inbrain tissue and the anisotropic water diffusion in white matter arenot completely understood, it is assumed that increased viscosity, tis-sue compartmentalization, as well as interaction with the structuralcomponents of the tissue such as macromolecules, membranes andintracellular organelles contribute to this phenomenon. One hypoth-esis of biological diffusion properties is related to the restriction of



diffusion by obstacles such as membranes.6,8 For very short diffu-sion times (i.e. if the diffusion path is short relative to the structuraldimensions), the molecular diffusion should resemble the free dif-fusion in a homogeneous medium. As the diffusion time increases,the water molecules diffuse far enough to encounter obstacles thatmay obstruct their movement. In certain media where the diffusionis restricted by impermeable barriers, it has been shown that as thediffusion time increases, the diffusion coefficient decreases whenthe diffusion distance is comparable with structure dimensions.6

Another hypothesis is that the behavior of water diffusion in tissuemay reflect rather hindered than restricted diffusion.6,9 The move-ment of water molecules may be hindered by much slower movingmacromolecules and by membranes, resulting in complicated, tortu-ous pathways. The anisotropic behavior of diffusion in white mattermay be also due to the intrinsic order of the axoplasmatic medium.6

The presence of microtubules and neurofilaments associated withaxonal transport and the lamellar structure of the myelin sheath mayinhibit motion perpendicular to axons, but does not restrict motionparallel to the fiber. When diffusion is hindered, the observed orapparent diffusion coefficient relates to the inherent diffusion coef-ficient, D0, through a tortuosity factor, λ9:

Dapp = D0

λ2 . (2)

12.1.3 Data Acquisition for DWI and DTI

As suggested by Stejskal and Tanner,10 the MR image is sensitizedto diffusion in a given direction using a couple of temporally sepa-rated pulsed B0 field gradients in the desired direction. The appli-cation of a magnetic field gradient pulse at e.g. one of the 3 spatialdimensions (x, y, z) dephases the protons (spin) along the respectivedimension (Fig. 2). A second pulse at the same direction, but oppo-site polarity (“refocusing pulse”), will rephase these spins. However,such rephasing cannot be perfect if the protons moved between the



δ

∆

90º 180º

TE/2TE/2

MR signal

gy

gz

gx

RF

TE

Fig. 2. MIR pulse sequence for diffusion tensor imaging (DTI). The direction ofthe magnetic field gradient is the one in which g(x) = g(y), and g(z) = 0, or g = (1,1, 0). See text for further details.

two gradient pulses. That is to say, the signal loss, which cannotbe recovered after the application of the second gradient pulse, is afunction of the local molecular motion. The amount of the moleculardiffusion is known to obey Eq. 3, assuming the sample is isotropic(no directionality in water diffusion):

SS0

= e−γ2G2δ2(�−δ/3)D, (3)

where S and S0 are signal intensities with and without the diffu-sion weighting, γ is a constant (gyromagnetic ratio), G and δ aregradient strength and duration, and � is the separation betweena pair of gradient pulses. Because these parameters are all known,from the amount of signal decrease (S/S0), diffusion constants ateach voxel can be derived. Such measurements have revealed thatdiffusion of brain water has strong directionality (anisotropy), which



is attributed to the existence of natural boundaries, such as axonsand/or myelination. The properties of such water diffusion can beexpressed as an ellipsoid — “diffusion ellipsoid.”11,12 This ellipsoidcan be characterized by six parameters; diffusion constants alongthe longest, middle, and shortest axes (λ1, λ2, and λ3, called princi-pal axes) and the direction of the three principal axes, perpendicularto each other. Once the diffusion ellipsoid is fully characterized ateach pixel of the brain images, local fiber structure can be derived.For example, if λ1 � λ2 ≥ λ3 (diffusion is anisotropic), it suggeststhe existence of dense and aligned fibers within each pixel, whereasisotropic diffusion (λ1 ≈ λ2 ≈ λ3) suggests sparse or unalignedfibers. When diffusion is anisotropic, the direction of λ1 indicates thedirection of the fibers.

12.1.4 Measures of Anisotropy Using Diffusion Tensors

One important application of the diffusion tensor is the quantita-tive characterization of the brain tissue structure and the degree ofanisotropy in brain white matter. Several scalar measures, whichemphasize different tensor features, have been derived from the dif-fusion tensor by different groups.7,13,14 To this end, diffusion tensorelements can be calculated by:

b = γ2δ2(� − δ/3)G2 (4)

S = S0 exp (−bD) (5)

D = 1b

lnS0

S. (6)

While the diffusion D is a scalar for conventional DWI, it is a ten-sor in case of DTI data. That is, instead of being characterized by asingle number, it is described by a 3×3 matrix of numbers. For exam-ple, if the diffusion-sensitizing gradient pulses are applied along thex-axis, u = (1, 0, 0), or if the measurement axis is at an angle θ to thex-axis and in the x + y plane, u = ( cos θ, sin θ, 0), then the measured



value of D along any axis u is given by:

D = (ux uy uz

) Dxx Dxy Dxz

Dxy Dyy Dyz

Dxz Dyz Dzz

ux

uy

uz

, (7)

D = u2xDxx + u2

yDyy + u2zDzz + 2uxuyDxy + 2uyuzDyz + 2uzuxDzx, (8)

∴ 1b

lnS0

S= u2

xDxx + u2yDyy + u2

zDzz + 2uxuyDxy

+ 2uyuzDyz + 2uzuxDzx. (9)

For example, for 12 directions,

1b

lnS0

S1····

1b

lnS0

S12

= U �D (10)

where,

U =

u2x1 u2

y1 u2z1 ux1uy1 uy1uz1 uz1ux1

· · · · · ·· · · · · ·· · · · · ·· · · · · ·

u2x12 u2

y12 u2z12 ux12uy12 uy12uz12 uz12ux12

and �D =

Dxx

Dyy

Dzz

2Dxy

2Dyz

2Dzx

.

Now, if we assume that the columns of U are linearly independent,then the matrix UTU is invertible and the least squares solution is

�D0 = (UTU)−1UT

1b

lnS0

S1····

1b

lnS0

S12

(11)



Since the 3×3 tensor matrix=D =

[Dxx Dxy DxzDxy Dyy DyzDxz Dyz Dzz

]is symmetric along

the diagonal, the eigenvalues and eigenvectors can be obtainedby diagonalizing the matrix using the Jacobi transformation. The

resulting eigenvalues=� =

[λ1 0 00 λ2 00 0 λ3

]and corresponding eigenvec-

tors=P = [−→p1

−→p2−→p3

]can then be used to describe the diffusivity

and directionality (or anisotropy) of water diffusion within a givenvoxel. An important measure associated with the diffusion tensor isits trace:

tr{D} = Dxx + Dyy + Dzz = 3 · 〈λ〉 = λ1 + λ2 + λ3. (12)

The trace has similar values in healthy white and gray matter(tr{D}∼2.1×10−3 mm2/s). However, the trace value drops consider-ably in brain tissue affected by acute stroke.15 This drop is attributedto an increase in tortuosity factor due to the shrinkage of the extra-cellular space.15 Consequently, the trace of the diffusion tensor canbe used as an early indicator of ischemic brain injury. Finally, theanisotropy of the diffusion tensor characterizes the amount of dif-fusion variation as a function of direction (e.g. the deviation fromisotropy). Several of these anisotropy measures are normalized toa range from 0 to 1. One of the most commonly used measures ofanisotropy is the fractional anisotropy (FA)7:

FA = 1√2

√(λ1 − λ2)2 + (λ2 − λ3)2 + (λ3 − λ1)2

λ21 + λ2

2 + λ23

, (13)

which is the ratio of the root-mean-square (RMS) of the eigenvaluesdeviation from their mean normalized by the eigenvalues Euclid-ian norm. FA has been shown to provide the best contrast betweendifferent classes of brain tissues.16 A useful way to display tract ori-entation is to use color to encode the direction of the tensor majoreigenvector.17,18 The 3D eigenvector space is associated with the3D RGB (Red-Green-Blue) color space by assigning a color to eachcomponent of the eigenvector (e.g. red to x, green to y, and blueto z). Consequently, the fibers that are oriented from left to right



Fig. 3. Color maps of several brain slices. (Left) axial, (middle) coronal, and (right)sagittal slices. See text for further details.

of the brain appear red, the fibers oriented anteriorly-posteriorly(front-back) appear green, and those oriented superiorly-inferiorly(top-bottom) appear blue (Fig. 3). All the other orientations are com-binations of these three colors. Color maps allow the identification ofdifferent white matter structures. Eigenvector color maps for threeorthogonal planes in a 3D brain volume are presented in Fig. 3.The color intensities are weighted by FA to emphasize white matteranatomy.

12.1.5 White Matter Tractography

White matter tractography (WMT) is based on the estimation ofwhite matter tract orientation using measured diffusion proper-ties of water as described in the previous sections. Some of themajor techniques for DTI based fiber tractography are discussedbelow:

12.1.6 Propagation Algorithms

In algorithms developed by many groups,11 a continuous represen-tation of the diffusion tensor and principal eigenvector ε1 are inter-polated from the discrete voxel data. The fiber track direction atany location along the tract is given by the continuous ε1. Typically,the tracking algorithm stops when the fiber radius of curvature or



Fig. 4. In vivo high-resolution diffusion tensor imaging (DTI) of the human corpuscallosum. The left panel depicts the user-defined seeding ROI for fiber reconstruc-tion, and the right panel shows the result of DTI based fiber tractography of humancorpus callosum.

the anisotropy factor falls below a threshold (Figs. 4 and 5). Withthis approach, the fiber is not represented by a succession of linesegments but by a relatively smooth curve that follows the localdiffusion direction and is more representative of the behavior ofreal fibers. These two approaches, often designated “streamline”approaches, are based on the assumption that diffusion is locallyuniform and can be accurately described by a single vector ε1. Unfor-tunately, this fails to describe voxels occupied by fibers with differentdiffusion tensors.19 Furthermore, the presence of noise in the diffu-sion MRI data induces a small uncertainty in the direction of thevectors ε1, that can lead to significant fiber tact propagation error.To try overcome these problems, tensorline approaches have beendeveloped such that the entire tensor information is used instead ofreducing it to a single eigenvector.20,21 Arecently proposed approachis a continuous approximation of the tensor field using B-splinesto derive fiber tracts. Tensorline algorithms seem to perform betterthan streamline algorithms for reconstructing low curvatures fibersand, in general, achieve better reproducibility. Poupon22,23 havedeveloped an algorithm based on a probabilistic approach aimedat minimizing fiber bending along the fiber tract. A regularizationstep based on the analogy between fiber pathways in white matterand so called “spaghetti plates” is used to improve robustness. A



Fig. 5. The explanatory power of DTI can further be increased by combining DTIfiber tractography with conventional functional imaging. Here, the areas of highfunctional MRI (fMRI) activity during visual stimulation along the human ventro-temporal cortex are used as seeding points for DTI based fiber reconstructions.

consequence of this approach is that it can represent fiber branchingand forks that are typically present in white matter fascicles, a clearadvantage over previously published methods.

12.1.6.1 Fiber assignment by continuous tracking

Mori et al.24 developed one of the earliest and most commonlyemployed algorithms: fiber assignment by continuous tracking(FACT). The FACT is based on extrapolation of continuous vectorlines from discrete DTI data. The reconstructed fiber direction withineach voxel is parallel to the diffusion tensor eigenvector (ε1) associ-ated with the greatest eigenvalue (λ1). Within each voxel, the fibertract is a line segment defined by the input position, the direction ofε1 and an output position at the boundary with the next voxel. Thetrack is propagated from voxel to voxel and terminated when a sharp



turn in the fiber orientation occurs. The FACT uses as propagationdirection the value corresponding to the current voxel: v∗

prop = vprop.The value of �p is chosen such as the current step crosses the entirevoxel and reaches its boundary. To this end, the trajectory will beformed by a series of segments of variable length. FACT integrationhas the advantage of a high computational efficiency.

12.1.6.2 Streamline tracking

The streamline tracking (STT) technique11,25,26 approximates vprop bythe major eigenvector of the tensor:

vprop = e1. (14)

This approach is analogous to simulated flow propagation in fluiddynamics including the study of blood flow phenomena from MRIflow measurements with 3D phase contrast.27

12.1.6.3 Tensor deflection

An alternative approach for determining tract direction is to usethe entire diffusion tensor to deflect the incoming vector (vin)direction14,28:

vout = D · vin. (15)

The incoming vector represents the propagation direction fromthe previous integration step. The tensor operator deflects the incom-ing vector towards the major eigenvector direction, but limits thecurvature of the deflection, which should result in smoother tractreconstructions. Tensor deflection (TEND) was proposed in order toimprove propagation in regions with low anisotropy, such as cross-ing fiber regions, where the direction of fastest diffusivity is not welldefined.29

12.1.6.4 Tensorline algorithms

The tensorline algorithm, described by Weinstein et al.,30 dynam-ically modulates the STT and TEND contributions to steer the



tract:

vout = fe1 + (1 − f )((1 − g)vin + gDvin), (16)

where f and g are user defined weighting factors that vary between0 and 1. The algorithm has 3D terms: (a) an STT term (e1) weightedby f , (b) a TEND term (D·vin) weighted by (1−f )g, and an undeviatedvin term weighted by (1 − f )(1 − g). The vectors and are normalizedto unify before being used in Eq. 16. Estimated trajectories with dif-ferent properties can be achieved by changing f and g. Tensorlinemay be considered as a family of tractography algorithms that canbe tuned to accentuate specific behavior. In the original implemen-tation of this algorithm, Weinstein et al. used a measure of prolatetensor shape, f = CL,14 to weight the STT term. Note that for f = 1,the tensorlines algorithm is equivalent to STT.

12.1.6.5 Probabilistic mapping algorithm

Diffusion tensor imaging is based on the assumption that the localorientation of nerve fibers is parallel to the first eigenvector of thediffusion tensor. However, due to issues such as imaging noises,limited spatial resolution and partial volume effect, the fiber orienta-tion cannot determined without uncertainty. Probabilistic methodsfor determining the connectivity between brain regions using infor-mation obtained from DTI have recently been introduced.31−33 Theseapproaches utilize probability density functions (PDFs) defined ateach point within the brain to describe the local uncertainty in fiberorientation. The probabilistic tractography algorithm reveals fiberconnectivity that progresses into the gray matter; while conventionalstreamlined algorithms failed to yield acceptable results. The goalof probabilistic tracking approaches is to determine the probabil-ity that fibers project from a starting point (or group of points) toregions of interest. In data analysis performed in this research, thelocal fiber orientation is given by the first eigenvector of the diffu-sion tensor that we call ε1. To perform probabilistic tacking, we needto introduce an uncertainty of ε1 orientation at every point along afiber created by a streamline tracking method. Then we repeat the



tracking a great number of times to generate a 3D probability map.The uncertainty in ε1 orientation can be described by the probabil-ity that it is deflected about its original position. The result of thisdeflection is a vector ε′

1. θ is the angle between ε1 and ε′1, and φ is

the rotation of ε′1 about ε1. The PDF for θ and φ are given by the 0-th

order model of uncertainty described in Ref. 32 φ is uniformly dis-tributed between 0 and 2π, and θ is normal about 0 with a standarddeviation Sigma linked to FA. Indeed, the smaller FA, the greaterthe fiber orientation uncertainty. We define Sigma = S(FA), S beinga sigmoid function. In our computation, we can modify the sigmoidfunction parameters: Sigma max, the standard deviation of θ as FAtends to 1, Sigma 0, the standard deviation of θ as FA tends to 1(i.e. a residual uncertainty), FA0, the value of FA for which Sigma =(Sigma0 + Sigma max)/2, and slope, the slope of the sigmoid. Tocreate a probabilistic map, a great number of fibers are generatedusing the streamline tracking algorithm. At each point along fiberpropagation, ε1 is modified into ε′

1, using a random number genera-tor and the PDF for φ and θ described above. The probability map isthe number of fibers reaching a voxel divided by the total number offibers that were generated. When probabilistic tracking is performedfrom multiple starting point (such as en entire ROI), the probabilityis multiplied by the number of starting points.

12.1.7 Limitations of DTI Techniques

Despite its great promise for visualizing and quantitatively char-acterizing white matter connections, DTI has some important lim-itations. It is not clear what is actually being measured with theanisotropy index. For example, the precise contribution of these twofactors, fiber density and myelination, on the anisotropy index hasnot been completely understood. Thus, it is not clear to what degreethe results of DTI correspond to the actual density and orientationof the local axonal fiber bundles. It is also important to understandhow white matter is, in general, organized. The most basic short-coming of DTI is that it can only determine a single fiber orienta-tion at any given location in the brain. This is clearly inadequate



in regions with complex white matter architecture, where differentaxonal pathways crisscross through each other. The crossing fiberscreate multiple fiber orientations within a single MRI voxel, wherea voxel refers to a 3D pixel, and constitutes the individual elementof the MR image. Since the diffusion tensor assumes only a sin-gle preferred direction of diffusion within each voxel, DTI cannotadequately describe regions of crossing fibers, or of converging ordiverging fibers. 3D DTI fiber tracking techniques are also foundin these regions of complex white matter architecture, since thereis no well defined single dominant fiber orientation for them tofollow.

In recent years, some of these problems have been addressedby measuring the full 3D dispersion of water diffusion in each MRIvoxel at high angular resolution. Thus, instead of obtaining diffusionmeasurements in only a few independent directions to determine asingle fiber orientation as in DTI, dozens or even hundreds of uni-formly distributed diffusion directions in 3D space are acquired toresolve multiple fiber orientations in high angular resolution diffu-sion imaging (e.g. HARDI). Each distinct fiber population can bevisualized on maps of the orientation distribution function (ODF),which are computed from the 3D high angular resolution diffusiondata through a projection reconstruction technique known as theFunk-Radon transform. This 3D projection reconstruction is verysimilar mathematically to the 2D method by which CT images arecalculated from X-ray attenuation data. Unlike DTI, HARDI hasthe advantage of being model-independent, and therefore does notassume any particular 3D distribution of water diffusion or any spe-cific number of fiber orientations within a voxel.

12.1.8 The Use of High b-value DWI for Tissue StructuralCharacterization

As a result of the structural heterogeneity of tissue in a spatialscale significantly smaller that the typical image voxel size, thediffusion-weighted signals display a multiexponential dependenceon diffusion weighting magnitude quantified with the parameter



b, where b = γ2δ2g2(� − δ/3) in a spin-echo diffusion experiment,where γ is the gyromagnetic ratio, g is the magnitude of the Stejskal-Tanner gradient pair each of which is of δ duration, and � is thetemporal separation of the gradient pair. The complexity of thismultiexponential behavior of the signal led to a more detailedinspection of diffusion properties in matter, as proposed.34,35 Themethod, known as q-space imaging, is based on the acquisition ofdata with multiple gradient strength values g. When a Fourier trans-formation is performed pixel by pixel with respect to the variableq = γg�/2π:

P(�R, �) = 12π

∫ ∞

−∞S(�q, �) · exp (−i2π�q · �R)d�q, (17)

the transformed data set P represents the displacement probabilityof the water molecules with respect to the axis which was sensi-tized to diffusion, at a given diffusion time �. This concept has beensuccessfully applied in various in vitro and in vivo applications,36−40

where the use of long diffusion times combined with gradation ofb-values and Fourier transformation has yielded displacement mapswith exquisite accuracy.

Although q-space imaging potentially yields detailed diffusiondata on heterogeneous tissue, the straightforward use of q-spacedata for imaging purposes has been mostly limited to displaying oneof the main parameters of the displacement distribution function, i.e.the zero displacement probability (amplitude at displacement = 0),and the displacement probability RMS (FWHM of the distributionfunction). This particular use clusters together the various diffusioncomponents, and thus it is particularly suitable for applications inwhich diffusion in a voxel is dominated by one component, eitherbecause of the nature of the tissue or by eliminating nonrestricteddiffusion components by means of a large � value.

The other approach of using diffusion data acquired with multi-ple b-values is to model the data according to a plausible model thatgoverns the diffusion pattern in each voxel. In this approach, the datais fitted to a multiparametric model function that best represents



the expected behavior of the signal with respect to b. The advan-tage of modeling the diffusion data is in the possibility to extractinformation about diffusion characteristics of water in various com-partments from the same data set, and thus simultaneously obtainvolumetric and structural information about those compartments.The most common and useful model for that matter is a biexpo-nential decay diffusion model, which partitions the diffusion datainto slow and fast diffusing components.41−46 It is now acceptedthat there is no soichiometric relation between the two componentsin the biexponential model and two distinct tissue compartments.However, it is widely accepted that the largest contribution to thenonmonoexponential behavior stems from restriction imposed ondiffusion, mostly on the intracellular and intra-axonal water pool.44

This view gains support from studies that measured diffusion ofintracellular metabolites such as N-acetyl aspartate (NAA), forwhich the diffusion attenuation curve as a function of b-value wasshown to be nonmonoexponential.47,48

12.2 SUMMARY AND CONCLUSIONS

Diffusion weighted magnetic resonance imaging (DWI) alreadyplans a crucial role in detecting neurostructural deviations at macro-scopic level. With recent advances in DTI, multimodal imagingand compartmental specific imaging, the importance of diffusionMRI for clinical and basic neurosciences plays are likely to increaseexponentially.


Drs Mina Kim and Susumu Mori provided crucial help in DWI/DTIdata acquisition and analyses. We also thank Mathieu Ducros, SahilJain and Keun-Ho Kim for their help with DTI postprocessing. Thiswork was supported by grants from NIH (RR08079, NS44825), TheMIND institute, Keck Foundation, and Human Frontiers ScienceProgram.



References

1. Moseley ME, Cohen Y, et al., Early detection of regional cerebralischemia in cats: Comparison of diffusion- and T2-weighted MRI andspectroscopy, Magn Reson Med 14(2): 330–346, 1990.

2. Moseley ME, Kucharczyk J, et al., Diffusion-weighted MR imag-ing of acute stroke: Correlation with T2-weighted and magneticsusceptibility-enhanced MR imaging in cats, AJNR Am J Neuroradiol11(3): 423–429, 1990.

3. Eis M, Els T, et al., Quantitative diffusion MR imaging of cerebral tumorand edema, Acta Neurochir Suppl (Wien) 60: 344–346, 1994.

4. Basser PJ, Mattiello J, et al., Estimation of the effective self-diffusiontensor from the NMR spin echo, J Magn Reson B 103(3): 247–254,1994.

5. Pierpaoli C, Jezzard P, et al., Diffusion tensor MR imaging of the humanbrain, Radiology 201(3): 637–648, 1996.

6. Le Bihan D, Turner R, et al., Imaging of diffusion and microcirculationwith gradient sensitization: Design, strategy, and significance, J MagnReson Imaging 1(1): 7–28, 1991.

7. Basser, PJ, Pierpaoli C, Microstructural and physiological features oftissues elucidated by quantitative-diffusion-tensor MRI, J Magn ResonB 111(3): 209–219, 1996.

8. Hajnal JV, Doran M, et al., MR imaging of anisotropically restricteddiffusion of water in the nervous system: Technical, anatomic, andpathologic considerations, J Comput Assist Tomogr 15(1): 1–18, 1991.

9. Norris DG, The effects of microscopic tissue parameters on the diffu-sion weighted magnetic resonance imaging experiment, NMR Biomed14(2): 77–93, 2001.

10. Stejskal EO, Tanner JE, Restricted self-diffusion of protons in colloidalsystems by the pulse-gradient, spin-echo method, J Chem Phys 49(4):1768–1777, 1968.

11. Conturo TE, Lori NF, et al., Tracking neuronal fiber pathways in theliving human brain, Proc Natl Acad Sci USA 96(18): 10422–10427, 1999.

12. Basser PJ, Mattiello J, et al., MR diffusion tensor spectroscopy and imag-ing, Biophys J 66(1): 259–267, 1994.

13. Conturo TE, McKinstry RC, et al., Encoding of anisotropic diffusionwith tetrahedral gradients: A general mathematical diffusion formal-ism and experimental results, Magn Reson Med 35(3): 399–412, 1996.

14. Westin CF, Maier SE, et al., Image Processing for Diffusion Tensor MagneticResonance Imaging, Springer, Cambridge, 1999.

15. Sotak CH, The role of diffusion tensor imaging in the evaluationof ischemic brain injury — A review, NMR Biomed 15(7–8): 561–569,2002.



16. Alexander AL, Hasan K, et al., A geometric analysis of diffusion tensormeasurements of the human brain, Magn Reson Med 44(2): 283–291,2000.

17. Makris N, Worth AJ, et al., Morphometry of in vivo human white mat-ter association pathways with diffusion-weighted magnetic resonanceimaging, Ann Neurol 42(6): 951–962, 1997.

18. Pajevic S, Pierpaoli C, Color schemes to represent the orientation ofanisotropic tissues from diffusion tensor data: Application to whitematter fiber tract mapping in the human brain, Magn Reson Med 43(6):921, 2000.

19. Alexander AL, Hasan KM, et al., Analysis of partial volume effects indiffusion-tensor MRI, Magn Reson Med 45(5): 770–780, 2001.

20. Weinstein D, Rabinowitz R, et al., Ovarian hemorrhage in women withVon Willebrand’s disease. A report of two cases, J Reprod Med 28(7):500–502, 1983.

21. Pajevic S, Basser P, A continuous tensor field approximation for DT-MRI data, 9th Annula Conference of the ISMRM, 2001.

22. Poupon C, Clark CA, et al., Regularization of diffusion-based directionmaps for the tracking of brain white matter fascicles, Neuroimage 12(2):184–195, 2000.

23. Poupon C, Mangin J, et al., Towards inference of human brain connec-tivity from MR diffusion tensor data, Med Image Anal 5(1): 1–15, 2001.

24. Mori S, Crain BJ, et al., Three-dimensional tracking of axonal projectionsin the brain by magnetic resonance imaging, Ann Neurol 45(2): 265–269,1999.

25. Basser PJ, Pajevic S, et al., In vivo fiber tractography using DT-MRI data,Magn Reson Med 44(4): 625–632, 2000.

26. Lori NF, Akbudak E, et al., Diffusion tensor fiber tracking of humanbrain connectivity: Aquisition methods, reliability analysis and bio-logical results, NMR Biomed 15(7–8): 494–515, 2002.

27. Napel S, Lee DH, et al., Visualizing three-dimensional flow with simu-lated streamlines and three-dimensional phase-contrast MR imaging,J Magn Reson Imaging 2(2): 143–153, 1992.

28. Lazar M, Weinstein DM, et al., White matter tractography using diffu-sion tensor deflection, Hum Brain Mapp 18(4): 306–321, 2003.

29. Westin CF, Maier SE, et al., Image Processing for Diffusion Tensor MagneticResonance Imaging, Springer, Cambridge, 1999.

30. Weinstein DM, Kindlmann GL, et al., Tensorlines: Advection-diffusionbased propagation through diffusion tensor fields, IEEE VisualizationProc, San Francisco, 1999.

31. Behrens TE, Johansen-Berg H, et al., Noninvasive mapping of connec-tions between human thalamus and cortex using diffusion imaging,Nat Neurosci 6(7): 750–757, 2003.



32. Parker GJ, Haroon HA, et al.,Aframework for a streamline-based prob-abilistic index of connectivity (PICo) using a structural interpretationof MRI diffusion measurements, J Magn Reson Imaging 18(2): 242–254,2003.

33. Jones DK, Pierpaoli C, Confidence mapping in diffusion tensor mag-netic resonance imaging tractography using a bootstrap approach,Magn Reson Med 53(5): 1143–1149, 2005.

34. Callaghan PT, Eccles CD, et al., NMR microscopy of dynamic displace-ments: k-space and q-space imaging, Journal of Physics — ScientificInstruments 21(8): 820–822, 1988.

35. Cory, DG, Garroway AN, Measurement of translational displacementprobabilities by NMR: An indicator of compartmentation, Magn ResonMed 14(3): 435–444, 1990.

36. King MD, Houseman J, et al., q-Space imaging of the brain, Magn ResonMed 32(6): 707–713, 1994.

37. King MD, Houseman J, et al., Localized q-space imaging of the mousebrain, Magn Reson Med 38(6): 930–937, 1997.

38. Assaf Y, Cohen Y, Structural information in neuronal tissue as revealedby q-space diffusion NMR spectroscopy of metabolites in bovine opticnerve, NMR Biomed 12(6): 335–344, 1999.

39. Assaf Y, Cohen Y, Assignment of the water slow-diffusing compo-nent in the central nervous system using q-space diffusion MRS:Implications for fiber tract imaging, Magn Reson Med 43(2): 191–199,2000.

40. Assaf Y, Ben-Bashat D, et al., High b-value q-space analyzed diffusion-weighted MRI:Application to multiple sclerosis, Magn Reson Med 47(1):115–126, 2002.

41. Niendorf T, Dijkhuizen RM, et al., Biexponential diffusion attenuationin various states of brain tissue: Implications for diffusion-weightedimaging, Magn Reson Med 36(6): 847–857, 1996.

42. Mulkern RV, Gudbjartsson H, et al., Multicomponent apparent diffu-sion coefficients in human brain, NMR Biomed 12(1): 51–62, 1999.

43. Clark, CA, Le Bihan D, Water diffusion compartmentation andanisotropy at high b values in the human brain, Magn Reson Med 44(6):852–859, 2000.

44. Inglis BA, Bossart EL, et al., Visualization of neural tissue water com-partments using biexponential diffusion tensor MRI, Magn Reson Med45(4): 580–587, 2001.

45. Mulkern RV, Vajapeyam S, et al., Biexponential apparent diffusion coef-ficient parametrization in adult vs newborn brain, Magn Reson Imaging19(5): 659–668, 2001.



46. Clark CA, Hedehus M, et al., In vivo mapping of the fast and slowdiffusion tensors in human brain, Magn Reson Med 47(4): 623–628, 2002.

47. AssafY, CohenY, In vivo and in vitro bi-exponential diffusion of N-acetylaspartate (NAA) in rat brain: A potential structural probe?, NMRBiomed 11(2): 67–74, 1998.

48. Assaf Y, Cohen Y, Non-mono-exponential attenuation of water andN-acetyl aspartate signals due to diffusion in brain tissue, J Magn Reson131(1): 69–85, 1998.


CHAPTER 13

Fluorescence Molecular Imaging:Microscopic to Macroscopic

Sachin V Patwardhan, Walter J Akers and Sharon Bloch

Medical imaging has revolutionized our understanding and ability tomonitor specific macroscopic physical, physiological, and metabolic func-tions at cellular and subcellular levels. In the years to come, it will enabledetection and characterization of disease even before anatomic changesbecome apparent. Fluorescence molecular imaging is revolutionarizingdrug discovery and development with real-time in vivo monitoring inintact tissues. Technological advancements have taken fluorescence basedimaging from microscopy to preclinical and clinical instruments for med-ical imaging. This chapter describes the current state of technology associ-ated with in vivo noninvasive or minimally invasive fluorescence imagingalong with the underlying principles. An overview of microscopic andmacroscopic fluorescence imaging techniques is presented and their rolein the development and applications of exogenous fluorescence contrastagents is discussed.

13.1 INTRODUCTION

Present medical imaging technologies rely on macroscopic physical,physiological, or metabolic changes that differentiate pathologicalfrom normal tissue rather than identifying specific molecular events(e.g. gene expression) responsible for disease.1 The human genomeproject is making molecular medicine an exciting reality. Develop-ments in quantum chemistry, molecular genetics and high speedcomputers have created unparallel capabilities for understanding

311


312 Sachin V Patwardhan, Walter J Akers and Sharon Bloch

complex biological systems. Current research has indicated thatmany diseases such as cancer occur as the result of the gradualbuildup of genetic changes in single cells.1−4 Molecular imagingexploits specific molecular probes as the source of image contrast forstudying such genetic changes at subcellular level. Molecular imag-ing is capable of yielding the critical information bridging molecularstructure and physiological function for understanding the integra-tive biology, which is the most important process in characterizationof disease, prevention, earlier detection, treatment, and evaluationof treatment.

The use of contrast agents for disease diagnostics and func-tionality is very common in established imaging modalities likepositron emission tomography (PET), magnetic resonance imaging(MRI), and X-ray tomography (CT). Contrast agents provide accu-rate difference images under nearly identical biological conditionsand yield superior diagnostic information. Fluorescence molecularimaging is a novel multidisciplinary field, in which fluorescencecontrast agents are used to produce images that reflect cellular andmolecular pathways and in vivo mechanisms of disease presentwithin the context of physiologically authentic environments. Thelimitation of fluorescence imaging is that the excitation light mustreach the fluorescent molecule which is governed by the absorptiondependent penetration depth of the light within the tissue. How-ever, fluorophores can be excited continuously and the signal is notgoverned by the inherent properties of the probe like the radioactivedecay. Further, a set of photophysical properties are accessible likefluorophore concentration, fluorescence quantum yield and fluores-cence lifetime. Some of these parameters are influenced by the localenvironment such as pH, ions, oxygen etc. and therefore, providemore relevant information about the physiological and molecularcondition. Most importantly, light is a nonionizing radiation, ren-dering it harmless and nontoxic.

Biophotonics can provide tools capable of identifying specificsubset of genes encoded within the human genome that can causethe development of cancer and other diseases. Photonic techniquesare being developed to image and identify the molecular alterations


Fluorescence Molecular Imaging: Microscopic to Macroscopic 313

that distinguish a diseased cell from a normal cell. Such technologieswill ultimately aid in characterizing and predicting the pathologi-cal behavior of the cell, as well as its responsiveness to drug treat-ment. The rapid development of laser and imaging technology hasyield powerful tools for the study of disease on all scales: singlemolecule to tissue materials and whole organs. Biochemical analysesof individual compounds characterize basic fluorescence propertiesof common fluorophores within the tissue. Additional informationassociated with complex systems such as cells and tissues structurecan be obtained from in vitro measurements. The study of in vivo ani-mal disease models provides information of about the intercellularinteractions and regulatory processes. Human clinical trials will thenlead to optical diagnostic, monitoring, and treatment procedures.

The purpose of this chapter is to provide an overview of micro-scopic and macroscopic fluorescence imaging techniques. Fluores-cence confocal microscopy, plan reflectance imaging, and diffuseoptical tomography techniques are discussed along with their rolein the development of exogenous fluorescence contrast agent forcellular level to in vitro and in vivo tissue imaging. For more spe-cific details on fluorescence contrast agents, measurement set upsand image reconstruction techniques and applications, the reader isencouraged to tap the extensive literature available on these subjects.

13.2 FLUORESCENCE CONTRAST AGENT:ENDOGENOUS AND EXOGENOUS

Light induced fluorescence is a powerful noninvasive method fortissue pathology recognition and monitoring.4−7 The attractivenessof fluorescence imaging is that fluorescent dyes can be detectedat low concentrations using non-ionizing harmless radiation thatcan be applied repeatedly to the patient. In fluorescence imag-ing, the energy from an external source of light is absorbed andalmost immediately re-emitted at a longer, lower energy wave-length that is related to the electronic transition from the excitedstate to the ground state of the fluorescent molecule. Fluorescence



that originates from chromophores naturally present in the tis-sue (endogenous) is known as autofluorescence. Synthesized chro-mophores (exogenous) may also be administered that target specifictissue type, or may be activated by functional changes in the tissue.

13.2.1 Endogenous Fluorophores

These fluorophores are generally associated with the structuralmatrix of tissue (e.g. collagen and elastin)8 or with the cellularmetabolic pathways (e.g. NAD and NADH).9 Cells in various dis-ease state often undergo different rates of metabolism or have differ-ent structures associated with a distinct fluorescent emission spectra.Fluorescence emission generally depends on the fluorophores con-centration, spatial distribution throughout the tissue, local microen-vironment, and light attenuation due to differences in the amountof nonfluorescing chromophores. Autofluorescence of proteins isassociated with amino acids such as tryptophan, tyrosin and pheny-lalanine with absorption maxima at 280 nm, 275 nm, and 257 nmrespectively, and emission maxima between 280 nm (phenylala-nine) and 350 nm (tryptophan). One of the main imaging applica-tions of fluorescent proteins is in monitoring tumor growth10,11 andmetastasis formation,12,13 as well as occasionally gene expression.4

Structural fluorophores like collagen or elastin have absorptionmaxima between 300 nm–400 nm and show broad emission bandsbetween 400 nm and 600 nm with maxima around 400 nm. Fluores-cence of collagen or elastin has been used to distinguish between var-ious tissue types e.g. epithelial and connective tissue.14−20 NADHis excited from 330 nm–370 nm wavelength range and is most con-centrated within the mitochondrial membrane where it is oxidizedwithin the respiratory chain. Its fluorescence is an appropriateparameter for detection of ischemic or neoplastic tissue. Fluores-cence of free and protein bounded NADH has been shown to besensitive to oxygen concentration.21 The main drawback of endoge-nous fluorophores is their low excitation and emission wavelength.In this spectral range, the tissue absorption is relatively high limitingthe light penetration.



13.2.2 Exogenous Fluorophores

Various fluorescing dyes can be use for probing cell anatomy andcell physiology. Exogenous fluorescence probes target specific cellu-lar and subcellular events, and this ability differentiates them fromnonspecific dyes, such as indocyanine green (ICG), which revealsgeneric functional characteristics such as vascular volume and per-meability. These fluorescence probes typically consist of the activecomponent, which interacts with the target (i.e. the affinity ligandor enzyme substrate); the reporting component (i.e. the fluorescentdye); and possibly a delivery vehicle (for example, a biocompati-ble polymer), which ensures optimal biodistribution. An importantcharacteristic in the design of active and activatable probes for in vivoapplications is the use of fluorochromes that operate in the NIR spec-trum of optical energy. This is due to the low light absorption thattissue exhibits in this spectral window, which makes light penetra-tion of several centimeters possible.

Exogenous targeted and activatable imaging probes yieldparticularly high tumor/background signal ratios because of theirnondetectability in the native state. In activatable probes, the fluo-rochromes are usually arranged in close proximity to each other sothat they self-quench, or they are placed next to a quencher usingenzyme-specific peptide sequences.22 These peptide sequencescan be cleaved in the presence of the enzyme, thus freeing thefluorochromes that can then emit light upon excitation. In contrastto active probes, activatable probes minimize background signalsbecause they are essentially dark at the absence of the target and canimprove contrast and the detection sensitivity. A variety of endoge-nous reporter probes have been used for enhanced detection of earlycancers, including somatostatin receptor targeted probes23−24; folatereceptor targeted agents25; tumor cell targeted agents26−29; agentsthat incorporate into areas of calcification; bone formation or both30;and agents being activated by tumor-associated proteases.31 Dyeslike fluorescein and indocyanine green are commonly used for flu-orescence angiography or blood volume determination in a clinicalsetup. Extensive research is also been carried out for development of



exogenous fluorophores with applications as activable probes thatcarry quenched fluorochromes24,33 and photosensitizer or tumorkilling agents for cancer treatment using photodynamic therapy.A photosensitizer is a drug that is preferentially taken up by malig-nant tissue and can be photoactivated. After an optimal time fromadministration, light is shown on the tissue area of interest andabsorbed by the sensitizer. The sensitizer then kills the surroundingtumor tissue, leaving the healthy tissue undamaged. Tissue local-ization, effectiveness in promoting cell death, and toxicity are someof the parameters that need to be characterized before human trials.

13.3 FLUORESCENCE IMAGING

Fluorescence imaging can provide information at different res-olutions and depth penetrations, ranging from micrometers(microscopy) to centimeters (fluorescence reflectance imaging andfluorescence molecular tomography).2−3 On microscopic level, fluo-rescent reporter dyes are typically used for monitoring the distribu-tion of important chemical species throughout the cell by obtainingfluorescence microscopy images of the cell after injecting it with thedye. Viability of the cell or permeability of its membrane can alsobe determined using fluorescence microscopy. Compared to micro-scopic cellular imaging, macroscopic in vitro tissue imaging allowsus to study interactions between cells and provide a platform muchcloser to true in vivo analysis in terms of structural architecture onmicroscopic and macroscopic scales. There is a significant differ-ence in tissue uptake and storage of various exogenous fluorophoresbetween in vitro and in vivo specimens. However, in vitro measure-ments can provide information associated with complex systemssuch as interaction of various biochemicals that are present in func-tional systems. Further, the effect of local environment on tissue opti-cal properties and properties such as reactivity to a specific chemicalcan be investigated prior to involving live subjects. For diagnos-tic purposes, the actual location and kinetics of tissue uptake areimportant. This information cannot be obtained using in vitro tissue



analysis. The pharmacokinetics, tissue discrimination capabilities,toxicity, and clearance pathways of fluorescence probes need to bestudied prior to use in human trials. Such studies are performedin vivo using animal models.

13.3.1 Fluorescence Microscopic Imaging

Fluorescence microscopy using endogenous fluorophores findsapplications in discriminating normal tissue from cancerous or evenprecancerous tissue in real-time clinical setting. Unique fluorescencespectral patterns associated with cell proliferation and betweenrapidly growing and slowly growing cells have been studied. Auto-fluorescence was used to identify terminal squamous differentia-tion of normal oral epithelial cells in culture and discriminationof proliferating and nonproliferating cell populations. Fluorescencemicroscopy using exogenous dyes is the most common techniqueused for monitoring the spatial distribution of a particular analytethroughout a cell. One or more exogenous dyes are introduced intothe cell and allowed to disperse. These dyes then interact with theanalyte of interest which in turn changes their fluorescence prop-erties. By obtaining a fluorescence image of the cell using excita-tion at specific wavelengths, relative concentrations of the analytecan be determined. Another important application of exogenousdyes is in elucidating the role of a particular chemical in cellularbiology.

In epifluorescence microscopy, the specimen is typically excitedusing a mercury or xenon lamp along with a set of monochroma-tor filters. The excitation light, after reflecting from a dichromaticmirror shines on to the sample through a microscope objective. Thedichromatic mirror reflects light shorter than a certain wavelength(excitation), and passes light longer than that wavelength (emission).Thus only the emitted fluorescence light passes onto the eye pieceor projected onto an electronic array detector positioned behind thedichroic mirror. While imaging thick specimens, the emitted fluo-rescent signal must pass through the volume of the specimen whichdecreases the resolution of objects in the focal plane. Additionally,



fluorescence emitted from excited objects that lie above and belowthe focal plane, obscures the emission from the in focus objects.

Laser-scanning confocal microscopy offers distinct advantagesover epifluorescence microscopy by using a pin hole aperture asshown in Fig. 1. The laser excitation light reflects off a dichromaticmirror and is focused on a single point within the tissue of inter-est rather than broadly illuminating the entire specimen using acomputer-controlled X-Y scanning mirror pair. With only a singlepoint illuminated, the illumination intensity rapidly falls off aboveand below the plane of focus as the beam converges and diverges,thus reducing excitation of fluorescence form interfering objects

Fig. 1. The principle of operation of a confocal microscope is shown on the left. Thepinhole aperture placed at the focal length of the lens blocks the light coming fromout-of focus planes (green and blue lines), while allowing the light coming fromthe plane-in-focus to reach the detector. Aschematic of point-scanning fluorescenceconfocal microscope is shown on the right. The dichromatic mirror reflects theemission light while allowing the excitation light to pass through. A motorized X-Yscanning mirror pair is used to collect the data from the selected sample area.



situated out of the focal plane being examined. The emitted fluo-rescence light from the sample gets descanned by the same mirrorsthat are used to scan the excitation light from the laser. The emittedlight passes through the dichromatic and is focused onto a pinholeaperture. The light that passes through the pinhole is measured by adetector, i.e. a photomultiplier tube. Any light emitting from regionsaway from the vicinity of the illuminated point will be blocked by thepinhole aperture, thus providing attenuation to out-of-focus inter-ference. Most confocal imaging systems provide adjustable pinholeblocking apertures. This enables a tradeoff to be made in verticalresolution and sensitivity. A small pinhole gives the highest resolu-tion and lowest signal and vice versa. With point-by-point scanning,there is never a complete image of the sample at any given instant.The detector is attached to a computer which builds up the image,one pixel at a time.

Point-scanning microscopes, when used with high numericalaperture lenses, have an inherent speed limitation in fluorescence.This arises because of a limitation in the amount of light thatcan be obtained from the small volume of fluorophore containedwithin the focus of the scanned beam (less than a cubic micron).At moderate levels of excitation, the amount of light emitted will beproportional to the intensity of the incident excitation. However, flu-orophore excited states have significant lifetimes (in the order if a fewnanosecond). Therefore, as the level of excitation is increased, thesituation eventually arises when most of the fluorophore moleculesare pumped up to their excited state and the ground state becomesdepleted. At this stage, the fluorophore is saturated and no more sig-nal may be obtained from it by increasing the flux of the excitationsource.

Despite their success, conventional microscopy methods suf-fer significant limitations when used in biological experimentation.They usually require chemical fixation of removed tissues, involvethe observation of biological samples under non-physiological con-ditions, can generally not resolve the dynamics of cellular processes,and most importantly, it is very difficult to generate quantitative datausing microscopy.



13.3.2 Fluorescence Macroscopic Imaging

Planar fluorescence imaging, transillumination and fluorescencemolecular tomography (FMT) are the most common imaging tech-niques used for obtaining fluorescence information at macroscopicresolution. Collapsing the volume of an animal or tissue into a sin-gle image, known as planar imaging, is generally fast, the data setsgenerated are small, and imaging can be done in high throughputfashion, at the expense of internal resolution. Tomographic imagingon the other hand allows a virtual slice of the subject to be obtainedand is more quantitative and capable of displaying internal anatomicstructures and/or functional information. However, FMT requireslonger acquisition times, generates a very large data set and is com-putationally expensive. Further, light becomes diffuse within a fewmillimeters of propagation within the tissues owing to elastic scatter-ing experienced by photons when they interact with various cellularcomponents, such as the membranes and different organelles. Diffu-sion results in the loss of imaging resolution. Therefore, macroscopicfluorescence imaging largely depends on spatially resolving andquantifying bulk signals from specific fluorescent entities reportingon cellular and molecular activity.

13.3.3 Planar Fluorescence Imaging

The most common technique to record fluorescence within a largetissues volume is associated with illuminating tissue with a planewave, i.e. an expanded light beam, and then collecting fluores-cence signals emitted towards a CCD camera.37 These methods canbe generally referred to as planar methods and can be applied inepi-illumination or transillumination mode. Figure 2 shows a typicalsetup of a planar reflectance imaging system. The imaging plane isuniformly illuminated using a particular wavelength light sourceand the light emitted by the fluorophore is captured using a CCDcamera. An illustrative image of a nude mouse with a subcutaneoushuman breast cancer xenograft obtained using a near-infrared fluo-rescent probe is also shown.



Fig. 2. Schematic diagram of a typical planner reflectance imaging system. Theimaging plane is uniformly illuminated using a particular wavelength light sourceand the light emitted by the fluorophore is captured using a CCD camera. Anillustrative image of a nude mouse with a subcutaneous human breast cancerxenograft MDA MD 361 obtained using a near-infrared fluorescent probe is alsoshown.

Planar imaging has the added advantage that same instru-mentation can be used to image fluorescence in solutions andexcised tissues. However, a significant drawback of this methodis that it cannot resolve depth and does not account for non-linear dependencies of the signal detected on propagation depthand the surrounding tissue. Superficial fluorescence activity mayreduce the contrast of underlying activity from being detectedowing to the simple projection viewing. Despite the drawbacks,planar imaging remains popular because setting up a reflectanceimaging system is comparatively easy and inexpensive. Planar flu-orescence imaging is a very useful technique when probing super-ficial structures (<5 mm deep), for example during endoscopy,41,42

dermatological imaging,43 intraoperative imaging,44 probing tissueautofluorescence45,46 or small animal imaging,47 with very highthroughputs.



13.3.3.1 Fluorescence molecular tomography

Recent technological evolutions have been the development of flu-orescence tomography for investigations at the whole-animal or tis-sue level. These technologies allow three-dimensional imaging offluorescence biodistribution in whole animals and account for tis-sue optical heterogeneity and the nonlinear dependence of fluo-rescence intensity on depth and optical properties. It can localizeand quantify fluorescent probes three-dimensionally in deep tis-sues at high sensitivities.48,49 The diffuse optical tomography (DOT)methods account for partial volume effects, reduce the influence ofsuperficial tissues and improve the contrast to noise ratio (CNR)of buried targets50−54 thereby overcoming the shortcomings of theplaner reflectance imaging.

Optical tomography is far more complex compared to X-ray CT.In X-ray CT, the radiation propagates through the medium in astraight line from the source to the detector. The forward problemthen becomes a set of integrals (Radon transform) and the inverseproblem is linear and well posed (back-projection methods). On theother hand, in optical imaging by the time the light reaches the detec-tor, it has lost all the information about the originating source due tomultiple scattering. Each measurement is therefore sensitive to thewhole tissue volume resulting into an ill posed, underdeterminedinverse problem. Mathematical models based on radiative transport(e.g. Monte Carlo techniques) or diffusion equation are required toreconstruct the most probable photon propagation path through tis-sue for a given source detector geometry (forward problem).55,56

Algorithms based on linear numerical inversion methods (inversesolution) start with the diffusion equation which is then transformedin to an integral equation via Green’s theorem. A linear version ofthe equation is then obtained using Born’s (or Rytov’s) approxi-mation and then discretized into a system of linear equations asfollows:

The fluorophore concentration is reconstructed by invertingratiometric data derived from the intensities of the excitation andfluorescence light measured on the detector plane for each source



position. The light intensity at the excitation wavelength is writtenas �(rs(i), rd(i), λexc), where rs(i), and rd(i) is the positions of the i-thsource and i-th detector locations respectively, and λexc is the excita-tion wavelength. Similarly, the fluence at the emission wavelengthλemi is written as �(rs(i), rd(i), λemi). Following the normalized Bornapproach, formulation of the ratiometric fluorescence/excitationmeasurements is written in discrete notation as y = Ax with thefollowing definitions21,22;

yi =[�(rs(i), rd(i), λemi) − θf�o(rs(i), rd(i), λexc)

�o(rs(i), rd(i), λexc)

](1)

Ai,j = −Sovh3

Do

G(rs(i), rj, λexc)G(rj, rd(i), λemi)G(rs(i), rd(i), λexc)

(2)

xj = ∂Nj (3)

Here, the two point Greens function, G, models light transport

for given boundary conditions and optical properties. Image voxel(xj) have concentration, ∂Nj and position rj. These equations arethen numerically solved using some type of regularization scheme.Singular value decomposition, algebraic reconstruction technique orconjugate gradient algorithms are used for example using Tikhonovregularization.57−60

The linear formulation works well when perturbations are smalland isolated, and when the background media is relatively uniform.However, the diffusion equation is inherently nonlinear becauseboth the photon fluence rate and the Green’s function are depen-dent upon the unknown quantities we are trying to solve. In algo-rithms based on nonlinear iterative methods, a global norm, suchas mean square error, is iteratively minimized. The unknown in-homogeneity is obtained that best predicts the measurement datasubject to some a priori knowledge. The unknown in-homogeneityis computed based on its current estimate and is compared to themeasurements every iteration.61−64

The fluorescence optical data can be obtained before and afteradministration of the absorbing fluorescence contrast agent and



the DOT images can be reconstructed and subtracted. However, amore robust approach is to use differential measurements due to theextrinsic perturbation. The two data sets (excitation and emission)are obtained within a short time of one another thereby minimiz-ing positional and movement errors and instrumental drift. Theuse of emission/excitation differential measurements eliminatessystematic errors associated with operational parameters and pro-vides a baseline measurement for independent reconstruction.3,65

Further, these ratio measurements reduce the influence of het-erogeneous optical properties and path lengths.65 FluorescenceDOT images have recently been demonstrated in vivo using bothfiber-coupled cylindrical geometries66−68 and lens-coupled planargeometries.69−71

The use of a lens to relay light from the tissue surface to a charge-coupled device (CCD) for detection69,71−73 permits dense spatialsampling and large imaging domains on the detected surface. Thefiber coupled illumination systems introduce an undesired asymme-try between the illumination plane (sparsely sampled by discretefibers) and the detection plane (densely sampled by a CCD arraydetector) and force tradeoffs between sampling density and fieldof view on the illumination plane. In addition, fiber optic switch-ing times (>0.1 seconds) limit data acquisition speeds. Rather thandirect lens coupling, other systems have used arrays of detector fiberto relay light from tissue to a CCD.66−68,74−76 While providing source-detector symmetry, this approach does not provide the dense sam-pling of the lens coupled detection. The source plane can be sampledusing fast acquisition, flexible, high-density, and large field-of-viewarrangements by raster scanning the source laser.

A schematic of the small animal continuous-wave fluorescenceDOT system is shown in Fig. 3.77 Here, the source illuminationis provided by a laser diode. The collimated output of the laserpasses through a beam splitter that deflects 5% of the beam toa photodiode for a reference measure of the laser intensity. Theremainder of the collimated beam (95%) passes through a lens, L,into a dual-axis XY galvanometer mirror system. The mirror pairsamples the source plane using a flexible, high-density and large



Fig. 3. Fluorescence tomography system. The mouse subject is suspended andheld in light compression between two movable windows (W1 and W2). Lightfrom a laser diode at 785 nm (LD) is collimated and passes through a 95/5 beamsplitter (BS). A reference photodiode (PD) collects 5% of the beam. The main 95%beam passes through lens (L1) into a XY galvo scanning system (XYGal). The mirrorpair scans the beam onto the illumination window (W1) of the imaging tank. Lightemitted from W2 is detected by an EMCCD via a filter (F1) and lens system (L2).77

field-of-view arrangements by raster scanning the focused illumi-nation (spot size = 100 µm) in two dimensions with a position Ato position B switch time of <0.5 ms. The 100 µm source spot sizeis similar to the multimode fibers sizes used in a wide variety ofDOT systems.66−69,71−76 The use of the galvanometer mirror pairpermits the system to scan an adjustable area of up to 8 cm × 8 cmwith flexible source positioning and source separations.After propa-gating through the sample volume, transmitted light passes througha selectable filter element and is detected on the opposite plane usinga lens coupled CCD camera. The typical scanning protocol consistsof two separate excitation and fluorescence scans. The excitationlight intensity profile is measured for each source position using aneutral density filter. The fluorescence emission light intensity pro-file is then measured by using a narrow band interference filter. Theexcitation and emission images obtained from the CCD camera arenormalized using the mean source intensity values obtained fromthe photodiode. This normalization compensates for the differencesin light levels between the excitation and emission scans.Afull frame4×4 binned image data (128×128) is collected for all the source posi-tions. The full detector images are cropped and binned to generate



set of detector measurement positions symmetrically arranged in thex-y plane such that for x-y source position, there is a matched x-ydetector position. With a total small animal whole body scan timeof ∼2.2 min, this fluorescence DOT system provides a 10× largerimaging domain (5 cm × 5 cm × 1.5 cm) compared to an equivalentfiber-switched system while maintaining the same resolution (smallobject FWHM ≤2.2 mm) and sensitivity (<0.1 pmole).69,77

Imaging the distribution of tumor-targeted molecular probessimultaneously in the liver, kidneys and tumors is demonstratedin Fig. 4 by imaging the uptake of a breast tumor-specific polypep-tide in nude mice bearing subcutaneously implanted human breast

Fig. 4. Representative slices from a 3D tomographic reconstruction of a nudemouse with a subcutaneous human breast cancer xenograft MDA MD 361. (A)a xy slice parallel to the detector plane at a depth of z = 2.5 mm and (B) a xz sliceextending from the source plane to detector plane at y = 12 mm.77



cancer carcinoma MDA-MB-361. The polypeptide was conjugatedwith a near infrared fluorescent probe cypate, which serves as thefluorescent contrast for optical imaging. For imaging the small ani-mal, anesthetized nude mice (ketamine/xylazine via intraperitonealinjection) were suspended between the source and detector win-dows. A matching fluid (µa = 0.3 cm−1, µ′

s = 10 cm−1) surroundsthe animal. With warmed matching fluid (T = 38◦C), the mice canbe imaged multiple times over the normal course of anesthetic dose(30 minutes–60 minutes). The full protocol for a combined fluores-cence/excitation scan (a 24 × 36 array with 2 mm square spacingbetween source positions, x = −24 mm to 24 mm, y = −36 mmto 36 mm) took 5 minutes–6 minutes including animal position-ing, emission and excitation scanning, retrieval of animal from thescanner and reconstructing the data. Figure 5 shows a 2D slice

Fig. 5. Retinal angiography images of a diabetic fundus (FA) showing loss ofnormal retinal capillaries and growth of abnormal ones that leak the fluorescein dye.(Source: Dr Levent Akduman, Saint Louis University Eye Center.)



parallel to the detector plane at a depth of z = 2.5 mm and a 2Dslice extending from the source plane to detector plane at y = 12 mm,obtained from the 3D tomographic reconstruction. The tumor (breastcancer) shows uptake of a fluorescing near infrared cypate derivativeprobe with a polypeptide that targets a protein receptor expressedin breast cancer. The kidneys also show contrast. The maximumvalues of probe concentration obtained from the tumor, liver andkidney volumes as a ratio of the background are 54.7, 32.4, and 58.3respectively.

Besides applications for disease diagnosis and monitoring,molecular imaging assays in intact living animals can also benefit inresolving biological questions raised by pharmaceutical scientists.Transgenic animals are useful in guiding early drug discovery by“validating” the target protein, evaluating test compounds, deter-mining whether the target is involved in any toxicological effects oftest compounds, and testing the efficacy of compounds to ensurethat the compounds will act as expected in man (Livingston, 1999).The implementation of molecular imaging approaches in this drugdiscovery process offers the strong advantage of being able to mean-ingfully study a potential drug labeled for imaging in an animalmodel, often before phenotypic changes become obvious, and thenquickly move into human studies. It is likely that preclinical trialscan be accelerated to rule out drugs with unfavorable biodistributionand/or pharmacokinetics prior to human studies. A further advan-tage over in vitro and cell culture experimentation may be achievedby repetitive study of the same animal model, using identical or alter-native biological imaging assays at different time points. This revealsa dynamic and more meaningful picture of the progressive changesin biological parameters under scrutiny, as well as possible temporalassessment of therapeutic responses, all in the same animal withoutrecourse to its death. This yields better quality results from far fewerexperimental animals. Another benefit of molecular imaging assaysis their quantitative nature. The images obtained are usually not justsubjective or qualitative, as is the case with standard use of sev-eral conventional medical imaging modalities, but instead, usually



provide meaningful numerical measures of biological phenomena(exemplified below). Such quantitative data could even be consid-ered more useful than similar data obtainable in vitro or ex vivo,on account of preserving the intactness and the physiology of theexperimental subject.

13.4 CONCLUSIONS

With the completion of several genome sequences, the next cru-cial step is to understand the function of gene products and theirrole in the development of disease. This knowledge will potentiallyfacilitate the discovery of informative biomarkers that can be usedfor the earliest detection of disease and for the creation of newclasses of drugs directed at new therapeutic targets. Thus, one ofthe capabilities most highly sought after is the noninvasive visu-alization of specific molecular targets, pathways and physiologicaleffects in vivo. Revolutionary advances in fluorescent probes, pho-toproteins and imaging technologies have allowed cell biologiststo carry out quantitative examination of cell structure and functionat high spatial and temporal resolution. Indeed, whole cell assayshave become an increasingly important tool in screening and drugdiscovery.

Fluorescence molecular imaging now creates the possibility ofachieving several important goals in biomedical research, namely,(1) to develop noninvasive in vivo imaging methods that reflect spe-cific cellular and molecular processes, for example, gene expres-sion, or more complex molecular interactions such as protein-proteininteractions; (2) to monitor multiple molecular events near simulta-neously; (3) to follow trafficking and targeting of cells; (4) to optimizedrug and gene therapy; (5) to image drug effects at a molecular andcellular level; (6) to assess disease progression at a molecular patho-logical level; and (7) to create the possibility of achieving all of theabove goals of imaging in a rapid, reproducible, and quantitativemanner, so as to be able to monitor time-dependent experimental,



developmental, environmental, and therapeutic influences on geneproducts in the same animal or patient.1

Fluorescein and ICG are FDA approved fluorescence dyes forhuman medical applications and are routinely used in clinical retinalangiography78 and liver function testing.79 Sample images of reti-nal angiography of a diabetic fundus (FA) showing loss of normalretinal capillaries and growth of abnormal ones that leak the fluores-cein dye are shown in Fig. 5. ICG exhibits favorable pharmacokineticproperties for assessment of hepatic function and cardiac output andhas been applied in clinical settings.80 ICG has also been reported asa NIR contrast agent for detection of tumors in animal research81,82

and at clinical level.83 The first fluorescence contrast-enhanced imag-ing in a clinical setting was reported by Ntziachristos et al.83 whodemonstrated uptake and localization of ICG in breast lesions usingDOT. Fluorescence imaging has shown very promising results as apotential imaging modality that will provide specific macroscopicphysical, physiological, or metabolic information at molecular level.With the current resources and research efforts, it won’t be longbefore a library of fluorescence biomarkers and photosynthesizersfor diagnosis, monitoring and treatment various diseases is formed.Technological advancements will soon take the fluorescence basedimaging devices from preclinical to clinical setups.

13.5 ACKNOWLEDGMENT

The authors acknowledge the help and support of Joseph P Culver,Samuel Achilefu and the entire team of the Optical Radiology Lab-oratory in the Department of Radiology at Washington UniversitySchool of Medicine, Saint Louis, Missouri. The authors are thankfulto Dr Levent Akduman, Saint Louis University Eye Center, SaintLouis, Missouri, for providing the retinal angiography images illus-trated in this chapter. Some of the work presented here was sup-ported in part by the following research grants: National Institutesof Health, K25-NS44339, BRG R01 CA109754, SmallAnimal ImagingResource Program (SAIRP) grant, R24 CA83060.



References

1. Massoud TF, Gambhir SS, Molecular imaging in living subjects: Seeingfundamental biological processes in a new light, Genes & Dev 17: 545–580, 2003.

2. Weissleder R, Ntziachristos V, Shedding light onto live moleculartargets, Nature Medicine 9(1): 123–128, 2003.

3. Ntziachristos V, Fluorescence molecular imaging, Annu Rev Biomed Eng8: 1–33, 2006.

4. Yang M, Baranov E, Moossa AR, et al., Visualizing gene expression bywhole body fluorescence imaging, Proc Natl Acad Sci USA 97: 12278–12282, 2000.

5. Tuchin VV, Handbook of Optical Biomedical Diagnostics, PM107, SPIEPress, Bellingham, WA, 2002.

6. Das BB, Lui F, Alfano RR, Time-resolved fluorescence and photonmigration studies in biomedical and random media, Rep Prog Phys 60:227, 1997.

7. Lakowicz JR, Principles of fluorescence spectroscopy, 2nd edn., KluwerAcademic, New York, 1999.

8. Denk W, Two-photon excitation in functional biological imaging,J Biomed Opt 1: 296, 1996.

9. Fujimoto D, Akiba KY, Nakamura N, Isolation and characterizationof a fluorescent material in bovine Achilles-tendon collagen, BiochemBiophys Res Commun 76: 1124, 1977.

10. Wagnieres GA, Star WM, Wilson BC, In vivo fluorescence spectroscopyand imaging for oncological applications, Photochem Photobiol 68: 603,1998.

11. Holfman RM, Visualization of GFP-expressing tumors and metastesisin vivo, Biotechniques 30: 1016–1022, 1024–1026, 2001.

12. Yang M, et al., Whole-body optical imaging of green fluorescent protin-expressing tumors and metastases, Proc Natl Acad Sci USA 97: 1206–1211, 2000.

13. Moore A, Sergeyev N, Bredow S, et al., Model system to quantitatetumor burden in locoregional lymph nodes during cancer spread, Inva-sion Metastasis 18: 192–197, 1998.

14. Wunderbaldinger P, Josephson L, Bremer C, et al., Detection of lymphnode metsastases by contrast — Enhanced MRI in an experimentalmodel, Magn Reson Med 47: 292–297, 2002.

15. Das BB, Lui F, Alfano RR, Time-resolved fluorescence and photonmigration studies in biomedical and random media, Rep Prog Phys 60:227, 1997.

16. Lakowicz JR, Principles of fluorescence spectroscopy, 2nd edn., KluwerAcademic, New York, 1999.



17. Schneckenburger H, Steiner R, Strauss W, et al., Fluorescence technolo-gies in biomedical diagnostics, in Tuchin VV (ed.), Optical BiomedicalDiagnostics, SPIE Press, Bellingham, WA, 2002.

18. Sinichkin Yu P, Kollias N, Zonios G, et al., Reflectance and fluorescencespectroscopy of human skin in vivo, in Tuchin VV (ed.), Handbook ofOptical Biomedical Diagnostics, SPIE Press, Bellingham, WA, 2002.

19. Sterenborg HJ, Motamedi M, Wagner RF, et al., In vivo fluorescencespectroscopy and imaging of human skin tumors, Lasers Med Sci 9:344, 1994.

20. Zeng H, MacAulay C, McLean DI, et al., Spectroscopic and microscopiccharacteristics of human skin autofluorescence emission, PhotochemPhotobiol 61: 645, 1995.

21. Schneckenburger H, Steiner R, Strauss W, et al., Fluorescence technolo-gies in biomedical diagnostics, in Tuchin VV (ed.), Optical BiomedicalDiagnostics, SPIE Press, Bellingham, WA, 2002.

22. Tung CH, Fluorescent peptide probes for in vivo diagnostic imaging,Biopolymers 76: 391–403, 2004.

23. Zaheer A, Lenkinski RE, Mahmood A, et al., In vivo near — infraredfluorescence imaging of osteobllastic activity, Nat Biotechnol 19: 1148–1154, 2001.

24. Weissleder R, Tung CH, Mahmood U, et al., In vivo imaging of tumorswith protease-activated near-infrared fluorescent probes, Nat Biotech-nol 17: 375–378, 1999.

25. Tung CH, Fluorescent peptide probes for in vivo diagnostic imaging,Biopolymers 76: 391–403, 2004.

26. Ballou B, et al., Tumor labeling in vivo using cyanine-conjugated mon-oclonal antibodies, Cancer Immunol Immunother 41: 257–263, 1995.

27. Neri D, et al., Targeting by affinity-matured recombinant antibody frag-ments on an angiogenesis-associated fibronectin isoform, Nat Biotech-nol 15: 1271–1275, 1997.

28. Muguruma N, et al., Antibodies labeled with Fluorescence-agentexcitable by infrared rays, J Gastroenterol 33: 467–471, 1998.

29. Folli S, et al.,Antibody — indocyanine conjugates for immunophotode-tection of human squamous cell carcinoma in nude mice, Cancer Res54: 2643–2649, 1994.

30. Zheer A, et al., In vivo near — infrared fluorescence imaging ofosteoblastic activity, Nat Biotechnol 19: 1148–1154, 2001.

31. Tung CH, Mahmood U, Bredow S, et al., In vivo imaging of proteolyticenzyme activity using a novel molecular reporter, Cancer Res 60: 4953–4958, 2000.

32. Bogdanov AA Jr, Lin CP, Simonova M, et al., Cellular activation of theself–quenched fluorescent reporter probe in tumor microenvironment,Neoplasia 4: 228–236, 2002.



33. Funovics M, Weissleder R, Tung CH, Protease sensors for bioimaging,Anal Bioanal Chem 377: 956–963, 2003.

34. Phair RD, Misteli T, Kinetic modeling approaches to in vivo imaging,Nat Rev Mol Cell Biol 2: 898–907, 2003.

35. Ke S, Wen XX, Gurfinkel M, et al., Near infrared optical imaging ofepidermal growth factor receptor in breast cancer xenografts, CancerRes 63: 7870–7875, 2003.

36. Zaheer A, Lenkinski RE, Mahmood A, et al., In vivo near — infraredfluorescence imaging of osteobllastic activity, Nat Biotechnol 19: 1148–1154, 2001.

37. Weissleder R, Tung CH, Mahmood U, et al., In vivo imaging of tumorswith protease-activated near-infrared fluorescent probes, Nat Biotech-nol 17: 375–378, 1999.

38. Wunder A, Tung CH, Muller-Lander U, et al., In vivo imaging of pro-tease activity in arthritis — A novel approach for monitering treatmentresponse, Arthritis Rheum 50: 2459–2465, 2004.

39. Mahmood U, Tung C, Bagdanov A, et al., Near infrared optical imagingsystem to detect tumor protease activity, Radiology 213: 866–870, 1999.

40. Yang M, Baranov E, Jinag P, et al., Whole–body optical imaging of greenfluorescent protin-expressing tumors and metastases, Proc Natl Acad SciUSA 97: 1206–1211, 2000.

41. Ito S, et al., Detection of human gastric cancer in resected speci-mens using a novel infrared fluorescent anti-human carcinoembry-onic antigen antibody with an infrared fluorescence endoscope in vitro,Endoscopy 33: 849–853, 2001.

42. Marten K, et al., Detection of dysplastic intestinal adenomas usingenzyme-sensing molecular beacons in mice, Gastroenterology 122: 406–414, 2002.

43. Zonios G, Bkowski J, Kollias N, Skin melanin, hemoglobin, and lightscattering properties can be quantitatively assessed in vivo using dif-fuse reflectance spectroscopy, J Invest Dermatol 117: 1452–1457, 2001.

44. Kuroiwa T, KajimotoY, Ohta T, Development and clinical application ofnear — infrared surgical microscope; preliminary report, Minim Inva-sive Neurosurg 44: 240–242, 2001.

45. Richards-Kortum R, Sevick-Muraca E, Quantitative optical spec-troscopy for tissue diagnosis, Annu Rev Physical Chem 47: 555–606, 1996.

46. Wang TD, et al., In vivo identification of colonic dysplasia using fluo-rescence endoscopic imaging, Gastrointest Endosc 49: 447–455, 1999.

47. Mahmood U, Tung C, Bagdanov A Jr, et al., Near-infrared opticalimaging of protease activity for tumor detection, Radiology 213: 866–870, 1999.

48. Ntziachistos V, Bremer C, Weissleder R, Fluorescence-mediated tomog-raphy resolves protease activity in vivo, Nat Med 8: 575–560, 2002.



49. Ntziachristos V, Weissleder R, Charge coupled device based scannerfor tomography of fluorescent near-infrared probes in turbid media,Med Phys 29: 803–809, 2002.

50. Hebden JC, Wong KS, Time-resolved optical tomography, App Optics32(4): 372–380, 1993.

51. Barbour RL, Graber HL, Chang JW, et al., MRI-guided optical tomog-raphy: Prospects and computation for a new imaging method, IEEECompu Sci & Eng 2(4): 63–77, 1995.

52. Pogue BW, Patterson MS, Jiang H, et al., Initial assessment of a simplesystem for frequency-domain diffuse optical tomography, Physics inMedicine and Biology 40(10): 1709–1729, 1995.

53. Oleary MA, Boas DA, Chance B, et al., Experimental images of hetero-geneous turbid media by frequency-domain diffusing-photon tomog-raphy, Optics Letters 20(5): 426–428, 1995.

54. Gonatas CP, Ishii M, Leigh JS, et al., Optical diffusion imaging using adirect inversion method, Phys Rev E 52(4): 4361–4365, 1995.

55. Gibson AP, Hebden JC, Arridge SR, Recent advances in diffuse opticalimaging, Phys Med Biol 50: R1–R43, 2005.

56. Arridge SR, Optical tomography in medical imaging, Inverse Problems15: R41–R93, 1999.

57. O’Leary MA, Boas DA, Chance B, et al., Experimental images of hetero-geneous turbid media by frequency-domain diffusing-photon tomog-raphy, Opt Lett 20: 426, 1995.

58. Yao Y, Wang Y, Pei Y, et al., Frequency-domain optical imaging ofabsorption and scattering distributions by a Born iterative method,J Opt Soc Am A14: 325, 1997.

59. Gaudette RJ, Brooks DH, DiMarzio CA, et al., A comparison study oflinear reconstruction techniques for diffuse optical tomography imag-ing of absorption coefficient, Phy Med Biol 45: 1051, 2000.

60. Pogue B, McBride T, Prewitt J, et al., Spatially variant regularizationimproves diffuse optical tomography, Appl Opt 38: 2950, 1999.

61. Ye JC, Webb KJ, Millane RP, et al., Modified distorted Born iterativemethod with an approximate Frechet derivative for optical diffusiontomography, J Opt Soc Am A16: 1814, 1999.

62. Hielsher AH, Klose AD, Hanson KM, Gradient-based iterative imagereconstruction scheme for time-resolved optical tomography, IEEETrans Med Imag 18: 262, 1999.

63. BluestoneAY,Abdoulaev G, Schmitz CH, et al., Three-dimensional opti-cal tomography of hemodynamics in the human head, Opt Express 9:272, 2001.

64. Roy R, Sevick-Muraca EM, A numerical study of gradient based non-linear optimization methods for contrast enhanced optical tomogra-phy, Opt Express 9: 49, 2001.



65. Soubret A, Ripoll J, Ntziachristos V, Accuracy of fluorescent tomogra-phy in the presence of heterogeneities: Study of the normalized Bornratio, IEEE Trans Med Imaging 24(10): 1377–1386, 2005.

66. Ntziachristos V, Weissleder R, Charge-coupled-device based scannerfor tomography of fluorescent near-infrared probes in turbid media,Med Phys 29(5): 803–809, 2002.

67. Ntziachristos V, Tung CH, Bremer C, Weissleder R, et al., Fluorescencemolecular tomography resolves protease activity in vivo, Nat Med 8(7):757–760, 2002.

68. Ntziachristos V, Bremer C, Tung C, et al., Imaging cathepsin B up-regulation in HT-1080 tumor models using fluorescence-mediatedmolecular tomography (FMT), Acad Radiol 9: S323–S325, 2002.

69. Graves EE, Ripoll J, Weissleder R et al., A submillimeter resolutionfluorescence molecular imaging system for small animal imaging, MedPhys 30(5): 901–911, 2003.

70. Graves EE, Weissleder R, Ntziachristos V, Fluorescence molecularimaging of small animal tumor models, Current Molecular Medicine 4(4):419–430, 2004.

71. Ntziachristos V, Schellenberger EA, Ripoll J, et al., Visualization of anti-tumor treatment by means of fluorescence molecular tomography withan annexin V-Cy5.5 conjugate, Proc Nat Acad Sci USA 101(33): 12294–12299, 2004.

72. Culver JP, Choe R, Holboke MJ, et al., Three-dimensional diffuse opticaltomography in the parallel plane transmission geometry: Evaluationof a hybrid frequency domain/continuous wave clinical system forbreast imaging, Med Phys 30(2): 235–247, 2003.

73. Schulz RB, Ripoll J, Ntziachristos V, Experimental fluorescence tomog-raphy of tissues with noncontact measurements, IEEE Transactions onMedical Imaging 23(4): 492–500, 2004.

74. Godavarty A, Eppstein MJ, Zhang CY, et al., Fluorescence-enhancedoptical imaging in large tissue volumes using a gain-modulated ICCDcamera, Physics in Medicine and Biology 48(12): 1701–1720, 2003.

75. Ntziachristos V, Weissleder R, Experimental three-dimensional fluo-rescence reconstruction of diffuse media by use of a normalized Bornapproximation, Optics Letters 26(12): 893–895, 2001.

76. Oleary MA, Boas DA, Li XD, et al., Fluorescence lifetime imaging inturbid media, Optics Letters 21(2): 158–160, 1996.

77. Patwardhan SV, et al., Time-dependent whole-body fluorescencetomography of probe bio-distributions in mice, Optics Express 13(7):2564–2577, 2005.

78. Richards G, Soubrane G, Yanuzzi L, Fluorescein and ICG Angiography,Thieme, Stuttgart, Germany, 1998.



79. Flanagan JH Jr, Khan S, Menchen S, et al., Functionalized tricarbocya-nine dyes as near-infrared fluorescent probes for biomolecules, Biocon-jug Chem 8: 751, 1997.

80. Caesar J, Shaldon S, Chiandussi L, et al., The use of indocyanine green inthe measurement of hepatic blood flow and as a test of hepatic function,Clin Sci 21: 43, 1961.

81. Gurfinkel M, Thompson AB, Ralston W, et al., Pharmacokinetics of ICGanf HPPH-car for the detection of normal and tumor tissue using flu-orescence, near-infrared reflectance imaging: A case study, PhotochemPhotobiol 72: 94, 2000.

82. Licha K, Riefke B, Ntziachristos V, et al., Hydrophilic cyaninedyes as contrast agents for near-infrared tumor imaging: Synthesis,photo-physical properties and spectroscopic in vivo characterization,Photochem PhotoBiol 72: 392, 2000.

83. Ntziachristos V, YodhAG, Schnall M, et al., Concurrent MRI and diffuseoptical tomography of breast after indocyanine green enhancement,Proc Natl Acad Sci USA 97: 2767, 2000.


CHAPTER 14

Tracking Endocardium Using OpticalFlow along Iso-Value Curve

Qi Duan, Elsa Angelini, Shunichi Hommaand Andrew Laine

In cardiac image analysis, optical flow techniques are widely used to trackventricular borders as well as estimate myocardial motion fields. The opti-cal flow computation is typically performed in Cartesian coordinates, andnot constrained from a priori knowledge of normal myocardium deforma-tion patterns. However, for cardiac motion analysis, displacements alongspecific directions and their derivatives are usually more interesting than2D or 3D displacement fields themselves. In this context, we propose twogeneral frameworks on optical flow estimation along iso-value curves.We applied the proposed frameworks in several specific applications: forendocardium tracking on cine cardiac MRI series and real-time 3D ultra-sound, and thickening computation in 2D ultrasound images. The endo-cardial surfaces tracked with the proposed algorithm were quantitativelycompared on manual tracing at each frame. The proposed method was alsocompared to the traditional Lucas-Kanade optical flow method directlyapplied to MRI image data in Cartesian coordinates and the standardcorrelation based optical flow estimation on real-time 3D echocardiogra-phy. Quantitative comparison showed a positive improvement in averagetracking errors or efficiency, through the whole cardiac cycle.

14.1 INTRODUCTION

Cardiac imaging techniques, including echocardiography, cardiacMRI, cardiac CT, and cardiac PET/SPECT, are widely used in clini-cal screening and diagnosis examinations as well as in research forin vivo studies. These imaging techniques provide structural and

337


338 Qi Duan et al.

functional information. In most clinical studies, quantitative evalu-ation of cardiac function requires endocardial border segmentationthroughout the whole cardiac cycle.

Recent advance in cardiac imaging technology have greatlyimproved the spatial and temporal resolution of acquired data, suchas with real-time three-dimensional echocardiography,1 and hightemporal resolution MRI.2 However, as information content is moredetailed, the amount of data needed to be analyzed for one cardiaccycle also increases dramatically, making manual analysis of thesedata sets prohibitively labor-intensive in clinical diagnosis centers.In this context, many computer-aided methods were developed toautomate or semi-automate endocardial segmentation or trackingtasks throughout the whole cardiac cycle. These computer-basedtechniques can be divided into two classes: segmentation methodsand motion tracking methods.

Today, cardiac image segmentation is a very active research area.Many techniques have been proposed, including active contour,3,4

level-set methods and deformable models,5–9 classification,10 activeappearance models,11 and other methods.12 Optical flow algorithmson tracking of the endocardial borders or other anatomical land-marks throughout the whole sequences were studied in severalrecent works.13–18 Optical-flow based tracking techniques offer thepossibility to compute myocardium motion field. Usually, thesemethods require initialization of the tracked points, either by manualtracing or with other segmentation techniques (as a preprocessingstep).

However, in cardiac motion analysis, displacements alongspecific directions are usually better indicators of wall motionabnormality. In this context, we propose a general framework foroptical flow estimation along iso-value curves. An additional con-straint related to specific motion direction was incorporated in theoriginal optical flow system of equations to properly constrainthe problem. A least-square fitting method was applied to smallneighborhoods for each point of interest to increase the robustnessof the method. The proposed method was then applied to endo-cardium tracking and results were quantitatively compared with


Tracking Endocardium Using Optical Flow along Iso-Value Curve 339

that obtained by manual tracing as well as tracking with the origi-nal Lucas-Kanade optical flow method.19

14.2 MATHEMATICAL ANALYSIS

14.2.1 Optical Flow Constraint Equation

Optical flow (OF) tracking refers to the computation of the displace-ment field of objects in an image, based on the assumption that theintensity of the object remains constant. This notion was first pro-posed by Horn20 and drove the active area of motion analysis inthe 1990s. Barron et al.21 wrote an extensive survey of the majoroptical-flow techniques at that time and drew the conclusion thatthe Lucas-Kanade and the Fleet-Jepson methods were the most reli-able among the nine techniques they implemented and tested onseveral image motion sequences.

Assuming the intensity at time frame t of the image point (x, y)is I(x, y, t), with u(x, y) and v(x, y) being the corresponding x and ycomponents of the optical flow vector at that point, it is assumed thatthe image intensity will remain constant at point (x + dx, y + dy) attime t+dt, where dx = udt and dy = vdt are the actual displacementof the point during time period dt, leading to the following equation:

I(x + dx, y + dy, t + dt) = I(x, y, t) (1)

If the image intensity is smooth with respect to x, y, and t, the left-hand side of Eq. (1) can be expanded into a Taylor series.20 Simplifica-tions, as detailed in Ref. 20, performed by ignoring the higher orderterms and taking limits as dt → 0, lead to the following equation:

∂I∂x

dxdt

+ ∂I∂y

dydt

+ ∂I∂t

= 0 (2)

Using the notations:

u = dxdt

, v = dydt

,

Ix = ∂I∂x

, Iy = ∂I∂y

, It = ∂I∂t

,(3)


340 Qi Duan et al.

Eq. (2) can be simplified as:

Ixu + Iyv + It = 0, (4)

Eq. (4) is called the optical flow constraint equation, as it expresses aconstraint on the components u and v of the optical flow. This sys-tem is under-constrained and with this equation alone, the opticalflow problem can not be uniquely solved. All gradient-based opticalflow methods try to add additional constraints to make the systemsufficiently constrained or even over-constrained. For example, theLucas-Kanade method19 tries to solve Eq. (2) through a weightedleast-squares fitting in each small spatial neighborhood � by mini-mizing the following equation, assuming a constant motion withinthe neighborhood: ∑

(x,y)∈�

W2(x, y)[Ixu + Iyv + It]2 (5)

where W (x, y) denotes a window function applied to the neighbor-hood. The solution to Eq. (5) is given by the following linear system:

ATW2A[uv

]= ATW2b (6)

where for n points in the neighborhood � at single time t,

A =[Ix1 . . . Ixn

Iy1 . . . Iyn

]T

,

W = diag[W (x1, y1), . . . , W (xn, yn)

],

b = −

It(x1, y1)...

It(xn, yn)

.

(7)

The system described in Eq. (6) can be solved by matrix inversionwhen the 2 by 2 matrix ATW2A is non-singular. The intrinsic least-square fitting property increases the robustness of the optical flowestimation for the Lucas-Kanade method.



14.2.2 Optical Flow along Iso-Value Curves

In cardiac motion analysis, motion along some iso-value curves isusually more interesting than the full 2D or 3D displacement itself.In both cardiac biomechanics,22 and cardiac imaging analysis, suchas Ref. 23, 2D or 3D displacement vectors are usually decomposedinto radial and circumferential displacement components. Thesecomponents and their derivatives (strains) are usually good indica-tors of ventricular abnormalities. For example, myocardium thick-ening, computed via radial derivatives of radial displacements, isthe best indicator for ischemia according to a recent biomechanicsstudy.24 With the correct use of a coordinate system, such as polarcoordinates25 in 2D and cylindrical coordinates23 in 3D, displace-ments along some directions (e.g. along radial directions) can bemathematically formulated as motion along some iso-value curves(e.g. θ = const). In this context, investigating optical flow along iso-value curves becomes important.

Given a time-varying N-dimensional time series I(−→X, t), where−→

X = [x1, . . . , xN]T is the spatial coordinates and t is the temporaldimension, the constant intensity constraint is

I(−→X, t) = I(

−→X + −→

dX, t + dt) (8)

where−→dX is the N-D displacement vector within time period dt for

the pixel located at−→X at time t. Using Taylor series expansion and

omitting higher order terms, we have

∇I(−→X, t) · −→

dX + ∂I(−→X, t)∂t

dt = 0 (9)

where ∇I(−→X, t) =

[∂I∂x1

, . . . , ∂I∂xN

]Tis the image spatial gradient vector

and the “·” represents the vector dot product.By defining the velocity vector (i.e. optical flow vector) as

−→v = d−→X

dt =[

dx1dt , . . . , dxN

dt

]T, the optical flow constraint equation for

N-dimensional time series can be derived as

∇I(−→X, t) · −→v + ∂I(

−→X, t)∂t

= 0 (10)

by taking limits as dt → 0.


342 Qi Duan et al.

Assume the optical flow estimation is performed along iso-valuecurves G(

−→X,

−→dX) = const. Note that in N-dimensional space, more

than one equations may be needed to represent iso-value curves orhyper-surfaces, so G could be a vector of functions and const couldbe a constant vector with same length as G. By letting F(

−→X,

−→dX) =

G(−→X,

−→dX)−const, the problem can be converted into an optical flow

estimation along the zero-value curve(s) F(−→X,

−→dX) = 0 (note F could

be a vector as the same reason as G). Thus, for a point−→X, two general

constraints are imposed on the optical flow vector −→v :

∇I(−→X, t) · −→v + ∂I(

−→X, t)∂t

= 0

F(−→X,

−→dX) = 0

(11)

There are many ways to solve the system described by Eq. (11).Here, we will propose a framework to solve this system via energyminimization since this framework can be easily extended to imagespaces with different dimensionalities, can easily incorporate neigh-borhood information, and can easily add additional constraints.

One straightforward way to solve the optical flow alongiso-value curves as in Eq. (11) is to follow the rationale of the Lucas-Kanade method. To increase the robustness of optical flow estima-tion, for each point

−→Xc, the final optical flow estimation is solved via

energy minimization of the energy defined in Eq. (12), in the leastsquare fitting sense, in an n-point neighborhood � centered at

−→Xc,

assuming a constant motion within the neighborhood:

−→v = arg min−→v

E1 = arg min−→v

(EOF + EISO)

= arg min−→v

(∥∥W(−→X ) · OF(

−→X )

∥∥2∣∣−→X ∈�

+ ∥∥F(−→Xc,

−→dXc)

∥∥2), (12)

where ‖·‖ represents the l2-norm, and the weighting vector W(−→X )

and optical flow constraint vector OF(−→X ) are defined as following



in the neighborhood �:

W(−→X ) = [

W (−→X1), . . . , W (

−→Xn)

]T

OF(−→X ) =

∇I(−→X1, t) · −→v + ∂I(

−→X1, t)∂t

.

.

.

∇I(−→Xn, t) · −→v + ∂I(

−→Xn, t)∂t

given (−→X1, . . . ,

−→Xn) ∈ �.

(13)

Generally solving the energy minimization problem in Eq. (12) isnot trivial depending upon the nonlinearity of the function F(

−→X,

−→dX).

One important feature of the proposed framework as in Eq. (12)is that everything is formulated in the original coordinate system ofthe input image series. There is no need to resample the image datato other coordinate system, e.g. polar coordinate, which is usuallydone in motion analysis or segmentation in one direction, such as inRef. 26. The main advantage of the proposed framework comparedwith these image resample frameworks are to avoid image resam-pling, which is a relative expensive step especially for 3D imagevolumes and may introduce some artifact depending upon the inter-polation scheme used. This will save a lot of computational powerwhen dealing with higher dimensional image series.

Another thing needed to be pointed out is that Eq. (12) is not theonly way to formulate the optical flow along iso-value curve. Actu-ally, another framework with identical optimum solution in idealcase will be proposed in the real-time 3D ultrasound application ina constrained energy minimization fashion.

In the following section, the proposed framework will be appliedto different applications. Specific zero-value curve(s) F(

−→X,

−→dX) will

be derived and the instants of Eq. (12) or other energy minimiza-tion schemes will be derived as well. The tracking results will


344 Qi Duan et al.

be quantitatively compared to the results derived from manualtracing through area-based index or finite-element model basedcontour/surface comparison.

14.3 METHODS AND RESULTS

14.3.1 Example I: Tracking Radial Displacements of theEndocardium in 2D Cardiac MRI Series

A direct application of the proposed framework was tested to trackthe endocardium motion along radial displacements. Previous workinvolving tracking endocardial borders using optical flow, such as,27

usually applied the optical flow algorithm directly on the Carte-sian image data without additional constraints on motion direction.Since radial displacements and its derivatives are the most inter-esting components of endocardial motion, we focused on OF radialdisplacement computation only.

14.3.1.1 Mathematical analysis

Usually in 2D cardiac images, a polar coordinate system is used todecompose the endocardium displacement field in radial and cir-cumferential directions. We followed the same coordinate systemconvention. The selection of the center of the polar coordinate sys-tem cannot simply be the centroid of the blood pool because of thewell known “floating centroid” problem in cardiac biomechanics.28

Following the proper ventricle axis selection protocol described inRef. 28, the long axis of left ventricle was first selected and thenthe center of the polar coordinate system was set as the intersec-tion of LV long axis and the imaging plane. In this coordinate sys-tem, radial displacements can be defined as displacements alongiso-value lines θ = const. The corresponding zero-value functionF(

−→X,

−→dX) = f (xc, yc, u, v) = 0, expressing the fact that the point (xc, yc)

and its motion vector (u, v) are along the line θ = const, is given by:{xc sin θ − yc cos θ = 0u sin θ − v cos θ = 0,

(14)



which can be simplified into

f (xc, yc, u, v) = ycu − xcv = 0. (15)

So the total energy associated with the optical flow along zero-value curve is:

E1 =∑

(x,y)∈�

W2(x, y)[Ixu + Iyv + It]2 + f 2(xc, yc, u, v). (16)

Similar to the original Lucas-Kanade method, the energy mini-mization problem described by Eq. (16) can be solved by least-squarefitting of the following equivalent over-constrained system:

∑

W2I2x

∑W2IxIy∑

W2IxIy

∑W2I2

y

yc − xc

[uv

]=

[ATW2b

0

], (17)

where W , A, and b are defined in Eq. (7).

14.3.1.2 Data and evaluation methods

The endocardial border tracking scheme developed in the previoussection was tested on two cardiac MRI protocols:

• A 2D cardiac MRI series with ECG gating acquired by a GE 1.5Tsystem using protocol FIESTA for 2D short axis stacks from anIRB approved experiment of LAD occlusion in sheep hearts. Thisprotocol, which is also called SSFP by other vendors, will generateclear anatomical image of the heart. For this reason, this protocolis widely used in cardiac MRI. This data set is selected to test theperformance of the optical flow on clear images with standardtemporal resolution in cardiac MRI.

• A 2D cardiac MRI series with ECG gating acquired by a Siemens1.5T system using a novel high-temporal resolution Phase TrainImaging (PTI) protocol proposed by Pai et al.2 for 2D short axisstacks from a volunteer heart. This novel high speed can pro-vide 2 ms temporal resolution on average and about four hundred


346 Qi Duan et al.

frames per cardiac cycle. The image quality is worse than FIESTAor SSFP protocol. This data is selected to test the performance ofthe optical flow on high-speed low quality image series and alsothe robustness on long-term tracking.

Endocardial border points for each time frame of the FIESTAdataand the last frame for the PTI data were traced by an experiencedexpert. The optical flow algorithm was initialized with manual trac-ing points on the first frame (end-diastole) and then automaticallyrun to track those points throughout the whole cardiac cycle (20frames in total for the FIESTA data and 412 frames for the PTI data).Two error measurements were used to evaluate the performanceof the optical flow: (1) the Tanimoto index TI = TP

1+FP = Seg1∩Seg2

Seg1∪Seg2,29

which is widely used in comparison of segmentation results; (2) rela-tive errors in radial coordinates. A 24 finite element model was usedto fit manually traced points or optical flow tracked points for eachframe of interest. The relative errors in radial coordinates of eachelement were then computed, with its mean serving as a perfor-mance indicator for each frame. The original Lucas-Kanade opticalflow method was also implemented and applied to the same data asa comparison method for endocardium tracking, without iso-valuecurve constraint.

14.3.1.3 Results

On the FIESTAdata, radial lengths for the endocardial border pointstracked by our method at end-diastole (ED) and end-systole (ES) areplotted in Fig. 1(a). When compared to endocardium obtained bymanual tracing, our proposed method has TI value 74.62% ± 8.54%,compared with that obtained by original Lucas-Kanade method as72.06% ± 9.13%. These results showed that our proposed methodhas more accurate and robust performance than the original Lucas-Kanade method. Example tracking results at frame 10 are shown inFig. 1(b), showing that our method is less likely to fail comparedwith the original method. Similar conclusion can be drawn from



(a)

(c) (d)

(b)

0 5 10 15 200

0.1

0.2

0.3

0.4

Frame Number

rel e

rr

Proposed MethodLK Method

-100 0 10012

14

16

18

20

22

24

(degree)

radi

al le

ngth

(m

m)

EDES

0 5 10 15 200

0.05

0.1

0.15

0.2

Frame Number

rel s

td

Proposed MethodLK Method

Fig. 1. On FIESTA sheep data: (a) Radial length of endocardium points at ED(solid line) and ES (dashed line); (b) Tracking result at frame 10 with proposedmethod (center of red circle) and Lucas-Kanade method (center of green cross);(c-d) Relative radial coordinates errors: (c) relative error and (d) standard deviation.(b-d) solid line: proposed method; dashed line: Lucas-Kanade method.

the comparison of the relative errors in radial coordinates plotted inFigs. 1(c) and 1(d), for which the proposed method has lower averageerrors as well as lower standard deviations in the relative errors inradial coordinates. The additional constraint of the OF motion alongiso-value curves improved the robustness and accuracy for trackingof the endocardium.

Error accumulation of consecutive frames in the OF estimationcan be noticed from the plots, which suggests that applying for-ward and backward tracking or adding more reference points mayimprove the performance of OF estimation.


348 Qi Duan et al.

Fig. 2. Tracking result at frame 412 with proposed method (red circle) and Lucas-Kanade method (green cross) on the high-speed PTI data.

On the PTI data, after tracking the endocardium through thewhole cardiac cycle, the TI values at the last frame are 85.40% forour method and 63.70% for the original Lucas-Kanade method.The relative errors are 7.10% ± 9.52% for our method and 96.49% ±126.91% for the original Lucas-Kanade method. Tracking results forthe last frame (the 412th frame) are shown in Fig. 2, which showsthat the additional constraint derived from the iso-value curveincreases the robustness of our method for high temporal resolutiontracking.

14.3.2 Example II: Tracking the Endocardium in Real-Time3D Ultrasound

Development of real-time 3D (RT3D) echocardiography startedin the late 1990s30 based on matrix phased arrays transducers.Recently, a new generation of RT3D transducers was introduced byPhilips Medical Systems (Best, The Netherlands) with the SONOS7500 transducer followed by the iE33 that can acquire a fullysampled cardiac volume in four cardiac cycles. This technicaldesign enabled a dramatic increase in spatial resolution and imagequality, which makes such 3D ultrasound techniques increasingly



attractive for daily cardiac clinical diagnosis. Since RT3D ultra-sound acquires volumetric ultrasound sequences with fairly hightemporal resolution and a stationary transducer, it can capturethe complex 3D cardiac motion very well. Advantages of usingthree-dimensional ultrasound in cardiology include the possibilityto display a three-dimensional dynamic view of the beating heart,and the ability for the cardiologist to explore the three-dimensionalanatomy at arbitrary angles, to localize abnormal structures andassess wall deformation. This technology has been shown, in thepast decade, to provide more accurate and reproducible screen-ing for quantification of cardiac function for two main reasons: theabsence of geometrical assumption for ventricular shapes and theaccuracy of the visualization planes for performing ventricular vol-ume measurements. It was validated through several clinical studiesfor quantification of LV function as reviewed in Ref. 31 and in Ref. 5.The development for computer aided tools for RT3D ultrasound isrelatively limited compared with the development of image pro-cessing techniques for other modalities. Early studies17 used simplesimulated phantoms while recent research32 used 3D ultrasounddata sequence for LV volume estimation. In Ref. 27, we proposeda framework based on correlation based optical flow estimation totrack the endocardium. The result is quantitatively validated againstmanual tracing result. In a recent study,33 3D speckle tracking tech-niques, which are similar to our method in Ref. 27, was tested mainlyon simulated data. All the tracking in previous studies was per-formed directly in 3D Cartesian coordinates. However, for trackingthe endocardium purpose, the problem can be reformulated as anoptical flow along iso-value curve problem, which will be muchmore efficient with comparable tracking results. Example frame ofRT3D ultrasound is shown in Fig. 3 in Phillips QLAB interface.

14.3.2.1 Mathematical analysis

As mentioned in previous section, Eq. (12) is not the only way toformulate the optical flow along iso-value curves. An equivalentframework to Eq. (12) can be formulated through constrained energy


350 Qi Duan et al.

Fig. 3. Example frame of RT3D ultrasound at ED for a patient with transplantedheart. (a) axial, (b) elevation and (c) azimuth views.

minimization:


EOF = ‖W(−→X ) · OF(

−→X )‖2|−→X ∈�

F(−→Xc,

−→dXc) = 0.

(18)

Equation (18) is an equivalent system of Eq. (12) in the sense thatboth systems have the same optimum solution under ideal case, i.e.the minimum value of EOF is zero.

In order to show that our framework is not limited to gradientbased optical flow framework, in this example, we will derive anenergy term that is equivalent to correlation based optical flow asused in Ref. 27. Since maximizing correlation coefficient is equiva-lent to minimizing the sum squared difference between two neigh-borhoods, the optical flow energy EOF can be simply defined withthis error energy. To properly define “radial displacement,” a prolate



spheroidal coordinate system (λ, µ, θ) with focus was established asdescribed in Refs. 27 and 34. So for each point

−→Xc with an n-point

neighborhood � centered at−→Xc, the tracking problem can be formu-

lated as:


EOF =∑−→X ∈�

(I(

−→X, t) − I(

−→X + −→v dt, t + dt)

)2

F(−→Xc,

−→dXc) =

[µt+dt

c − µtc

θt+dtc − θt

c

]= 0,

(19)

where

−→X =

x

yz

=

d sinh λ sin µ cos θ

d sinh λ sin µ sin θ

d cosh λ cos µ

. (20)

14.3.2.2 Data and evaluation method

The tracking approach was tested on one data set acquired with aSONOS 7500 3D ultrasound machine (Philips Medical Systems, Best,The Netherlands): One transthoracic clinical data set was aquiredfrom a heart transplant patient. Spatial resolution of the analyzeddata was (0.8 mm3) and 16 frames were acquired for one cardiaccycle. The endocardial surfaces were manually traced by one expe-rience expert for every frame between the end-diastole and the end-systole. The optical flow algorithms were initialized using manualtracing at ED and ES frames. The endocardial surfaces in betweenwere generated by averaging the results from forward and back-ward tracking by both methods. Manual tracing for each frame wasused as gold standard for surface comparison.

We evaluated OF tracking performance via visualization andquantification of dynamic ventricular geometry compared to seg-mented surfaces. Usually, comparison of segmentation results isperformed via global measurements like volume difference or mean-squared error. In order to provide local comparison, we proposeda novel comparison method in Ref. 35 based on a parameteriza-tion of the endocardial surface in prolate spheroidal coordinates36


352 Qi Duan et al.

and previously used for comparison of ventricular geometries fromtwo 3D ultrasound machines in Ref. 37. The endocardial surfaceswere registered using three manually selected anatomical land-marks: the center of the mitral orifice, the endocardial apex, and theequatorial mid-septum. The data as fitted in prolate spheroidal coor-dinates (λ, µ, θ), projecting the radial coordinate λ to a 64-elementsurface mesh with bicubic Hermite interpolation, yielding a realis-tic 3D endocardial surface. The fitting process was performed usingthe custom finite element package Continuity 5.5 developed at theUniversity of California San Diego (http://cmrg.ucsd.edu). The fit-ted nodal values and spatial derivatives of the radial coordinate,λ, were then used to map relative differences between two sur-faces, ε = (λseg − λOF)/λseg using custom software. A Hammermapping was used to flatten the endocardial surface via an areapreserving mapping,28 through which relative λ difference mapswere generated for end-systole (ES), providing a direct quantitativecomparison of ventricular geometry. These maps are visualized withiso-level lines, quantified in percentage values of radial difference.The area under 10% differences is used as the criteria for quantitativecomparison.

Average intra-observer variance and inter-observer variancewere also computed by the similar scheme using the two tracingsfrom a single user one month apart and two tracings from two dif-ference users at the same time.

14.3.2.3 Results

The area percentages under 10% difference are plotted in Fig. 4(a).The mean values are 69.66% ± 21.42% for proposed method and87.29%±10.38% for direct tracking scheme. Example Hammer mapsfrom both methods at one frame are shown in Figs. 4(b) and 4(c)respectively. The average intra- and inter-observer differences are79.38% and 55.33%, respectively, in terms of same surface compari-son criteria. Both methods are comparable to inter-observer varianceand the direct tracking has better performance than the proposedmethod.



(a) (b) (c)

1 2 3 4 5 6 70

20

40

60

80

100

Per

cent

age

Constrained

Direct

Intra

Inter

00

0

-0.1

0.1-0.2

-0.1

0.1

septumanterior lateral posterior

septum

apex

-0.2

0.2-0.1

0

-0.10

septumanterior lateral posterior

septum

apex

Fig. 4. Optical flow tracking results on RT3D ultrasound: (a) area percentage under10% difference from manual tracing for each frame generated by the proposedmethod (blue) and direct tracking method (green). Average intra-observer variance(red) and inter-observer variance (cyan) are also plotted for reference; (b) Hammermap of direct tracking result at frame 5; (c) Hammer map of constrained trackingresult at frame 5.

From computational cost point of view, the proposed methodused 5.3767 seconds on average to tracking the surface betweentwo frames, whereas the direct tracking method needed 112.78 sec-onds on average for the same task, which leads about 20 timessaving in computational power for our method compared withdirect tracking scheme. With performance comparable to inter-userdifference and much shorter time cost in computation than directtracking scheme, our method may be more suitable in clinical appli-cations, where the total analysis time is limited to 5–10 minutes foreach data set.

14.3.3 Example III: Thickening Computationon 2D Ultrasound Slices

In the previous two applications, optical flow along iso-value curveswas used mainly as a tracking tool. In this example, we will showthat the displacement estimated in this framework can also be usedin motion analysis and strain computation.

14.3.3.1 Data and method

One basal short-axis cross-section view was extracted from the RT3Dclinical data used in previous section. 2D versions of optical flow


354 Qi Duan et al.

(a) (b) (c)

spetum spetum

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Fig. 5. Results on thickening computation: (a) example 2D slice; (b) Segmentalaverage of thickening from direct tracking scheme; (c) Segmental average of thick-ening from our method.

methods detailed in previous section were also implemented. Bothoptical flow methods were initialized at end-diastole with man-ual tracing. Segmental average thickening from ED to ES werecomputed.

14.3.3.2 Results

Segmental average results of thickening computation from bothmethods are shown in Fig. 5. Both methods generated similar resultsand correctly indicated the reduced motion at the septum in theoriginal data set.

14.4 DISCUSSION

From the three examples we showed, we can conclude that, with theadditional energy term or constraint from the iso-value curve, theoptical flow algorithm can either perform better with roughly samecomputational cost or much efficiently without downgrading theaccuracy a lot, especially for the tracking tasks like tracking endo-cardium in cardiac imaging. Radial displacements and thickeningestimated derived from constrained scheme were similar to thoseobtained by direct tracking.



The frameworks proposed in Eqs. (12) and (18) are generic.They can be easily extended to higher dimensional space; and theenergy for optical flow estimation can be chosen different fromthe optical flow constrained equation. Actually with proper choice ofthe optical flow energy term, the well known intensity constancyassumption can be loosened, which could increase the robustnessof the estimation. Moreover, in addition to direct benefit from theenergy minimization framework, additional constraints, such assmoothness constraint, can be seamlessly incorporated by simplyadding weighted energy terms associated with these constraints.This framework could be also merged with variational optical flowapproaches, such as works in Refs. 38 and 39.

The proposed frameworks were formulated directly in the samecoordinate system as the input image, so there is no data resamplingrequired, which will reduce the overall computational cost andreduce the accuracy dependency on the interpolation methods. Thekey points in these frameworks are to properly define the zero-valuecurve function vector F and to properly minimize the energy. Thelatter one could be formulated as a non-linear problem for someapplications.

Although ideal systems described by Eqs. (12) and (18) havesame optimal solutions, the results on real image series from thesetwo frameworks may be different if the zero-minimum for the opti-cal flow energy can not be reached. In this case, framework definedby Eq. (18) will still give optical flow displacement vector along theiso-value curves whereas the framework defined by Eq. (12) mayloosen this constraint to get estimation with even lower energy,which yields the fact that the framework defined by Eq. (12) out-performs its constrained counterpart defined by Eq. (18). Consider-ing computational cost, framework defined by Eq. (12) will slightlyincrease the cost compared to direct tracking method with the addi-tional energy term; on the contrary, the framework defined byEq. (18) usually offers huge saving in computational power due todimensionality reduction. So for tracking purpose, if the accuracy ismore important than the efficiency, we would suggest the use of theunconstrained version as Eq. (12). If efficiency is more important or


356 Qi Duan et al.

displacement is required to strictly follow the iso-value curves, theconstrained version as Eq. (18) would be a good choice.

The last thing that needs to be pointed out is that the displace-ments estimated by the proposed frameworks cannot be used formotion analysis along directions other than the given iso-valuecurve. For example, if the displacement of the endocardium is esti-mated by optical flow along radial direction (θ = const) in 2D, thisestimation cannot be directly used to estimate the circumferentialdisplacement or cardiac twist. This is a limitation of the proposedframeworks since in some sense we are trading the universality infree motion estimation for much better accuracy or efficiency formotion estimation along specific iso-value curves. Fortunately, thislimitation would not limit the usefulness of the proposed frame-work a lot since in most of cardiac applications, landmark or surfacetracking and motion analysis along specific directions are more oftenthan free motion analysis.

14.5 CONCLUSION

Two generic frameworks for optical flow were proposed as an energyminimization problem with local constraints related to iso-valuecurves. Three applications of these frameworks were presented fortracking of the endocardium on 2D MRI data series (both FIESTAandPTI protocols) and real-time 3D ultrasound series. The endocardiumborders tracked by the proposed method as well as the Lucas-Kanade method were quantitatively compared to manual tracing oneach frame through the Tanimoto Index and relative errors in radialcoordinates after FEM fitting. The results showed superior perfor-mance for the proposed method in tracking the endocardium. Theconstrained version was applied on real-time 3D ultrasound data.Quantitative evaluation results yielded comparable performance tointer-observer variance with about 20-fold saving in computationalcost compared to direct tracking scheme. Thickening computationsfrom the proposed method and direct tracking method were com-pared with similar results. These frameworks are generic and can



be readily extended to n-dimensional spaces and seamlessly incor-porated additional constraints via a similar energy minimizationframework.

14.6 ACKNOWLEDGMENT

This work was funded by National Science Foundation grant BES-02-01617, American Heart Association #0151250T, Philips Medi-cal Systems, New York State NYSTAR/CAT Technology Program.Dr Andrew McCulloch at the University of California, San Diegoprovided the finite element software “Continuity” through theNational Biomedical Computation Resource (NIH P41RR08605).The authors also would like to thank Dr Todd Pulerwitz(Department of Medicine, Columbia University), Susan L. Herz,Christopher M. Ingrassia, Drs Jeffrey W. Holmes, Dr Kevin D. Costa(Department of Biomedical Engineering), and Dr Vinay M. Pai (Radi-ology, New York University).

References

1. Ramm OTV, Pavy JHG, Smith SW, Kisslo J, Real-time, three-dimensional echocardiography: The first human images, Circulation84: 685, 1991.

2. Pai V, Axel L, Kellman P, Phase train approach for very high temporalresolution cardiac imaging, J Cardiovasc Magn Reson 7: 98–99, 2005.

3. Drezek R, Stetten GD, Ota T, Fleishman C, et al., Active contour basedon the elliptical Fourier series, applied to matrix-array ultrasound ofthe heart, presented at 25th AIPR Workshop: Emerging Applicationsof Computer Vision, 1997.

4. Chalana V, Linker DT, Haynor DR, Kim Y, A multiple active con-tour model for cardiac boundary detection on echocardiographicsequences, IEEE Transactions on Medical Imaging 15: 290–298, 1996.

5. Angelini ED, Homma S, Pearson G, Holmes JW, et al., Segmentation ofreal-time three-dimensional ultrasound for quantification of ventricu-lar function: A clinical study on right and left ventricles, Ultrasound inMed & Biol 31: 1143–1158, 2005.

6. Paragios N, A level set approach for shape-driven segmentation andtracking of the left ventricle, IEEE Transactions on Medical Imaging 22:773–776, 2003.


358 Qi Duan et al.

7. Lin N, Duncan JS, Generalized robust point matching using anextended free-form deformation model: Application to cardiac images,presented at 2004 2nd IEEE International Symposium on BiomedicalImaging: Macro to Nano, 2004.

8. Rueckert D, Burger P, Geometrically Deformable Templates for Shape-basedSegmentation and Tracking in Cardiac MR Images, presented at EnergyMinimization Methods in Computer Vision and Pattern Recognition,Venice, Italy, 1997.

9. Montagnat J, Delingette H, Spatial and Temporal Shape ConstrainedDeformable Surfaces for 3D and 4D Medical Image Segmentation, INRIA,Sophia Antipolis RR-4078, 2000.

10. van Assen CH, Danibuchkine MG, Frangi AF, Ordas S, et al., SPASM:A 3D-ASM for segmentation of sparse and arbitrarily oriented cardiacMRI data, Medical Image Analysis 10: 286–303, 2006.

11. Mitchell SC, Lelieveldt BPF, van der Geest R, Schaap J, et al., Segmenta-tion of Cardiac MR Images: An Active Appearance Model Approach,presented at SPIE-The International Society for Optical Engineering,2000.

12. Setarehdan SK, Soraghan JJ, Automatic cardiac LV boundary detec-tion and tracking using hybrid fuzzy temporal and fuzzy multiscaleedge detection, IEEE Transaction on Biomedical Engineering 46: 1364–1378, 1999.

13. Veronesi F, Corsi C, Caiani EG, Sarti A, et al., Tracking of left ventricularlong axis from real-time three-dimensional echocardiography usingoptical flow techniques, IEEE Transactions on Information Technology inBiomedicine 10: 174–181, 2006.

14. Duan Q, Angelini E, Herz SL, Ingrassia CM, et al., Dynamic CardiacInformation From Optical Flow Using Four Dimensional Ultrasound, pre-sented at 27th Annual International Conference IEEE Engineering inMedicine and Biology Society (EMBS), Shanghai, China, 2005.

15. Loncaric S, Majcenic Z, Optical Flow Algorithm for Cardiac Motion Esti-mation, presented at 22nd Annual International Conference of the IEEEEngineering in Medicine and Biology Society, Jul 23–28 2000, Chicago,IL, 2000.

16. Gindi GR, Gmitro AF, Delorie DHJ, Velocity Flow-Field Analysis of Car-diac Dynamics, presented at Proceedings of the Thirteenth AnnualNortheast Bioengineering Conference, Philadelphia, PA, USA, 1987.

17. Gutierrez MA, Moura L, Melo CP, Alens N, Computing Optical Flowin Cardiac Images for 3D Motion Analysis, presented at Proceed-ings of the 1993 Conference on Computers in Cardiology, London,UK, 1993.



18. Suhling M, Arigovindan M, Jansen C, Hunziker P, et al., Myocardialmotion analysis from B-mode echocardiograms, IEEE Transactions onImage Processing 14: 525–536, 2005.

19. Lucas BD, Kanade T, An Iterative Image Registration Technique with anAppication to Stereo Vision, presented at International Joint Conferenceon Artificial Intelligence (IJCAI), 1981.

20. Horn BKP, Robot Vision, MIT Press, Cambridge, 1986.21. Barron JL, Fleet D, Beauchemin S, Performance of optical flow tech-

niques, Int Journal of Computer Vision 12: 43–77, 1994.22. Humphrey JD, Cardiovascular Solid Mechanics: Cells, Tissues, and Organs,

Springer, New York, USA, 2002.23. Papademetris X, SinusasAJ, Dione DP, Duncan JS, Estimation of 3D left

ventricular deformation from echocardiography, Medical Image Analy-sis 8: 285–294, 2004.

24. Azhari H, Sideman S, Weiss JL, Shapiro EP, et al., Three-dimensionalmapping of acute ischemic regions using MRI: Wall thichening versusmotion analysis, American Journal of Physiology 259: H1492–H1503, 1990.

25. Suhling M, Arigovindan M, Jansen C, Hunziker P, et al., Myocardialmotion analysis from B-mode echocardiograms, IEEE Transactions onImage Processing 14: 525–536, 2005.

26. Noble N, Hill D, Breeuwer M, Schnabel J, et al., Myocardial delineationvia registration in a polar coordinate system, Acad Radiol 10: 1349–1358,2003.

27. Duan Q, Angelini ED, Herz SL, Gerard O, et al., Tracking of LV Endo-cardial Surface on Real-Time Three-Dimensional Ultrasound with OpticalFlow, presented at Third International Conference on Functional Imag-ing and Modeling of the Heart 2005, Barcelona, Spain, 2005.

28. Herz S, Pulerwitz T, Hirata K, Laine A, et al., Novel Technique forQuantitative Wall Motion Analysis Using Real-Time Three-DimensionalEchocardiography, presented at Proceedings of the 15th Annual Scien-tific Sessions of the American Society of Echocardiography, 2004.

29. Theodoridis S, Koutroumbas K, Pattern Recognition, Academic Press,USA, 1999.

30. Ramm OTV, Smith SW, Real-time volumetric ultrasound imaging sys-tem, Journal of Digital Imaging 3: 261–266, 1990.

31. Krenning BJ, Voormolen MM, Roelandt JRTC, Assessment of left ven-tricular function by three-dimensional echocardiography, CardiovascUltrasound 1(1): 2003.

32. Shin I-S, Kelly PA, Lee KF, Tighe DA, Left Ventricular Volume EstimationFrom Three-Dimensional Echocardiography, presented at Proceedings of


360 Qi Duan et al.

SPIE, Medical Imaging 2004 — Ultrasonic Imaging and Signal Process-ing, San Diego, CA, United States, 2004.

33. Yu W, Yan P, Sinusas AJ, Thiele K, et al., Towards pointwise motiontracking in echocardiographic image sequences: Comparing the relia-bility of different features for speckle tracking, Medical Image Analysis10: 495–508, 2006.

34. Herz S, Ingrassia C, Homma S, Costa K, et al., Parameterization of leftventricular wall motion for detection of regional ischemia, Annals ofBiomedical Engineering 33: 912–919, 2005.

35. Duan Q,Angelini ED, Herz SL, Ingrassia CM, et al., Evaluation of OpticalFlow Algorithms for Tracking Endocardial Surfaces on Three-DimensionalUltrasound Data, presented at SPIE International Symposium, MedicalImaging 2005, San Diego, CA, USA, 2005.

36. Ingrassia CM, Herz SL, Costa KD, Holmes JW, Impact of IschemicRegion Size on Regional Wall Motion, presented at Proceedings of the2003 Annual Fall Meeting of the Biomedical Engineering Society,2003.

37. Angelini ED, Hamming D, Homma S, Holmes J, et al., Comparison of Seg-mentation Methods for Analysis of Endocardial Wall Motion with Real-TimeThree-Dimensional Ultrasound, presented at Computers in Cardiology,Memphis TN, USA, 2002.

38. Bruhn A, Weickert J, Feddern C, Kohlberger T, et al., Variational opticalflow computation in real-time, IEEE Transactions on Image Processing 14:608–615, 2005.

39. Ruhnau P, Kohlberger T, Schnorr C, Nobach H, Variational opticalflow estimation for particle image velocimetry, Experiments in Fluids 38:21–32, 2005.


CHAPTER 15

Some Recent Developments inReconstruction Algorithms for

Tomographic Imaging

Chien-Min Kao, Emil Y Sidky, Patrick La Rivièreand Xiaochuan Pan

Ionizing-radiation based imaging techniques play an extremely importantrole in non-invasively yielding information about the internal anatomicstructure and functional information within a subject under study. Com-puted tomography (CT), positron emission tomography (PET), and singlephoton emission computed tomography (SPECT) are the main imagingmodalities based upon ionizing radiation, and they have found applica-tions in virtually every discipline in science, engineering, biology, chem-istry, and, more notably, medicine. In these imaging techniques, one needsto develop algorithms for accurately reconstructing the underlying objectfunction from acquired projection data. In the last decade or so, in parallelto tremendous tomographic hardware advancement for data acquisition,there have also been important breakthroughs in the development of inno-vative algorithms for reconstructing the underlying object function. In thischapter, we briefly review some of the recent developments in reconstruc-tion algorithms for tomographic imaging in CT, PET, and SPECT.

15.1 IMAGE RECONSTRUCTION IN COMPUTEDTOMOGRAPHY

15.1.1 Introduction

X-ray projection imaging is the most common non-invasive scanemployed for probing the interior of a subject, and it found wideapplication very quickly after its initial discovery in 1895. For many

361


362 Chien-Min Kao et al.

purposes, the projection of the subject’s X-ray attenuation coeffi-cient yields important diagnostic information in medical imaging,or structural and compositional information in industrial imaging.There are, however, an increasing number of imaging applications,where it is desirable to have full 3D information of the X-ray atten-uation coefficient. Such information can be provided by combiningand processing X-ray projections of a subject taken from multipleview angles surrounding the subject. In the 1970s, as computer tech-nology began its rapid ascension, computed tomography (CT) wasdeveloped to address the need for internal 3D structural informa-tion. The early CT scanners obtained 3D images slice-by-slice by illu-minating the subject with a fan of X-rays, rotating the X-ray sourceand detector to obtain complete information to reconstruct the 2Dslice image. By translating the subject the subsequent slices couldbe obtained. The theory of image reconstruction lead to the filteredback projection (FBP) and fan-beam filtered back projection (FFBP)algorithms, corresponding respectively to parallel- and diverging-ray illumination. This step-and-shoot process was streamlined byintroducing the helical source trajectory. This trajectory is what isseen from the subject reference frame as it is translated at a con-stant rate through a rotating gantry that carries the X-ray sourceand detector on a circular trajectory. If the helical pitch, the distancecovered by the subject during a single turn of the gantry, is not toogreat then variations on the 2D FFBP algorithm can be utilized toobtain accurate reconstruction of the subject’s 3D X-ray attenuationcoefficient.

The trend in the technical development of CT scanners is toinclude more and more rows on the detector, extending its dimen-sion along the longitudinal axis of the helical scan (referred to assimply the longitudinal direction for short). Currently, commercialscanners employ up to 64 detector rows and this number will cer-tainly increase, because more detector rows allows for higher heli-cal pitches and more rapid coverage of 3D volumes. As the detectorsize increases longitudinally the X-ray source slit is opened up toilluminate the subject with an X-ray cone beam. This evolution ofCT scanners has increased the sense of urgency for the development


Some Recent Developments in Reconstruction Algorithms for Tomographic Imaging 363

of practical algorithms that can yield accurate image reconstructionfrom cone beam CT projection data.

A theory of 3D image reconstruction for cone beam CT has beenknown since work by Tuy,1 who derived a inversion formula thatyields a 3D distribution from its cone beam projection at views alonga general class of X-ray source trajectories. The helical trajectory fallsinto this class. Although the Tuy formula represents an importantadvance in the theory of cone beam CT image reconstruction, it hastwo major practical short-comings. First, direct implementation ofthis formula is numerically inefficient. Second, the projection datacannot be truncated; a complete projection of the subject is neededfrom all of the view angles. This is particularly impractical in helical,cone beam CT for human subjects; as the detector would have to havean extent larger than the body’s projection at all the sampled views.During the 1990s and early 2000s much effort was devoted to derivea practical image reconstruction algorithm, using a relation betweencone beam projection data and the 3D Radon transform of the objectfunction derived by Grangeat.2 These algorithms sought to solve theso called long object problem, where the cone beam projection dataare truncated only in the longitudinal direction.2 A breakthroughin image reconstruction theory occurred in 2001 when Katsevichpublished an exact formula for image reconstruction directly fromhelical, cone beam projection data.3 This algorithm, though related tothe Tuy formula,4 could support longitudinal truncation of the conebeam projection data, and requires only 1D filtering of the projectiondata thereby improving numerical efficiency.

The ideas of the Katsevich algorithm combined with the geomet-rical construct of the so called π-line in helical cone beam scanning,5

led Zou and Pan to develop a new class of cone beam CT imagereconstruction algorithms.6−10 These algorithms obtain the imagein a curvilinear coordinate system that is defined by the chords ofa general source trajectory. In helical cone beam CT, the π-lines canbe interpreted as a special set of chords. The new algorithms areefficient and create opportunities to design novel data acquisitionconfigurations that allow for dose reduction and increased scan-ning speed. The Zou-Pan image reconstruction formula involves



reversing the usual data processing steps of data filtration followedby back projection to the image array. These algorithms insteadperform the back projection step first, and are hence called backprojection filtration (BPF). The reversal of these operations improvesalgorithm efficiency, because the filtration in the image space is lesstime consuming than in the data space. More importantly, BPF canperform exact image reconstruction for projection data that is trun-cated both longitudinally and transversely. In the following sections,we introduce the data model for helical cone beam CT; we thenexplain the BPF image reconstruction algorithm; and finally we dis-cuss the implications for region-of-interest (ROI) imaging.

15.1.2 The Data Model of Helical Cone Beam CT

In helical cone beam CT, the X-ray source travels along a helical tra-jectory along with the 2D detector array. The detector shown in theimage is a flat-panel array, while current helical cone beam systemsgenerally use curved detector arrays. The image reconstruction the-ory, below, is presented in a detector independent formulation whichcan be easily adapted to either detector geometry. The data model forthe helical cone beam system assumes that the line integral of theX-ray attenuation coefficient for a ray originating from the sourceand terminating at a detector bin can be obtained from:

di = − ln

(Ii

(I0)i

), (1)

where i is a generic index for the rays specified by the combinationof all detector bin and source locations; (I0)i is the X-ray intensity innumber of photons that would be measured for the i-th ray if therewere no subject; Ii is the actual measured intensity; and di representsthe line integral of the X-ray attenuation for the i-th ray:

di =∫ ∞

0d�µ(�si + �θi). (2)

The vector �si is the X-ray source location and θi is the unit vector forthe i-th ray; µ is the spatially varying X-ray attenuation coefficient,ignoring energy dependence. The data model is idealized; X-ray



scatter, beam polychromatricity, partial volume averaging, etc.11 areall neglected here. The set of di is interpreted as the measurementsbecause they can be computed from the raw measurements throughEq. (1). The aim of the reconstruction algorithm is to find µ(�r ) givenmeasurements di.

As described by Eq. (2) the measurement set is a large but finiteset of line integrals. The theory of image reconstruction in helicalcone beam CT is formulated in terms of a continuous data function;thus we rewrite Eq. (2) to reflect this fact, and discuss discretiza-tion after the reconstruction formula is written down in Eq. (3). Todevelop the image reconstruction formula, we assume that we canobtain the continuous data function

g(λ, θ) =∫ ∞

0d�µ[�s(λ) + �θ], (3)

where λ is the continuously varying helical parameter indicatingthe source position. The source position is given in Cartesian coor-dinates by:

�s(λ) =(

R cos λ, R sin λ,h

2πλ

), (4)

where R is the helical radius, and h is the pitch length. The coordi-nate system is set up so that the axis of the helix is aligned along z.The detector bin locations are not specified. It is assumed that thedetector captures the attuation measurements along the necessaryrays originating at �s(λ) in the direction θ. The sufficient range of λ

and θ is discussed below.

15.1.3 The BPF Algorithm

The BPF algorithm for image reconstruction in helical cone-beam CTinvolves decomposing the imaging volume in chords of the sourcetrajectory. Mathematically, a single chord is described by:

�rc(λ1, λ2, t) = �s(λ1)(1 − t) + �s(λ2)t; t ∈ [0, 1]. (5)

The chord, specified by the helical parameters λ1 and λ2, is a line seg-ment that joins the source positions �s(λ1) and �s(λ2). The parameter



t locates a point on the chord. It has been previously observed thatall points internal to the convex hull of the helix can be uniquelyassigned to a point on a helical chord with the restriction that|λ1 − λ2| < 2π; chords that satisfy this restriction are called π-lines.5

The BPF algorithm obtains the volume image by reconstructing itchord by chord.

The main steps of the BPF algorithm involve taking a derivativeof the projection data, back projection of the data derivative to thechord to form an intermediate image function, and finally filtrationof the intermediate image to obtain the actual image function. Thefirst processing step for the data function follows this equation:

gD(λ, θ) = ∂

∂pg(p, θ)

∣∣∣p=λ

. (6)

The next step involves back projecting the data onto the chord:

fI(λ1, λ2, t) =∫ λ2

λ1

dλ1

|�s(λ) − �rc(λ1, λ2, t)|gD(λ, θc),

where θc = �rc(λ1, λ2, t) − �s(λ)|�rc(λ1, λ2, t) − �s(λ)| .

(7)

Before continuing on to the last step of the chord image reconstruc-tion, we note here that the above formula says something about theprojection data sufficient for reconstruction the image on the chord.The integration of λ for the back projection goes from λ1 to λ2, soprojection views for λ ∈ [λ1, λ2] are needed to form fI(λ1, λ2, t),and for each view the rays that intersect the chord need to bemeasured.

It turns out that the intermediate chord image fI(λ1, λ2, t) is sim-ply the Hilbert transform of the desired image function µc(λ1, λ2, t)along the chord:

fI(λ1, λ2, t) = 2∫ ∞

−∞dt′

µc(λ1, λ2, t′)t − t′

where µc(λ1, λ2, t) = µ[�rc(λ1, λ2, t)].(8)

The Hilbert transform involves an infinite range integration, but itis known that the object function µ has compact support. It turns



out that the solution µc(λ1, λ2, t) to the integral equation, Eq. (8), canbe expressed with a finite range integration because of the compactsupport property. Assuming that µc is compactly supported withinthe interval t ∈ [ta, tb], we have:

µc(λ1, λ2, t) =√

tb − tt − ta

×∫ tb

ta

dt′1

t − t′

√t′ − ta

tb − t′fI(λ1, λ2, t′). (9)

The fact that the t′ integration only runs from ta to tb has furtherimplications on the data sufficiency conditions. Only the projectionrays that intersect the chord for t ∈ [ta, tb] need to be measured, notthe complete π-line. This inverse for the finite Hilbert transform isactually only one of many possibilities.8,12 This completes the chainof operations needed to go from the projection data to the imagealong a π-line. The only issues that remain are how to obtain volumeimages, and what projection data are sufficient for reconstruction.

15.1.4 The Long Object Problem and ROI Reconstruction

The theory of π-line image reconstruction, above, tells how to obtainthe reconstructed image on the trajectory chords. The end goal, how-ever, is volume reconstruction. This section clarifies the connectionbetween the two cases, and along the way discusses scanning datarequirements for various scanning tasks.

For diagnostic helical cone-beam CT the most important taskthat the image reconstruction can fulfill is to provide numericallyexact images efficiently from projection data that are longitudi-nally truncated. The BPF algorithm does this. As the BPF algorithmitself provides the image on individual π-lines, the volume must beparameterized first in the curvilinear system specified by the inde-pendent variables λ1, λ2, and t. The variables λ1 and λ2 specify aπ-line, and t yields a specific point on that chord. We illustrate nowhow the volume coverage works in this coordinate system. First,one can fix one end of the chord, say λ1 = λA, then sweep λ2 inthe range [λA, λA + 2π]. Such a set of π-lines defines a π-surfacewhose geometry only depends on λA. To obtain the volume, λA isswept through an interval [λstart, λend]. The data sufficency condition



for such a volume scan is easy to derive from geometric considera-tions of the individual π-line is obviously λ ∈ [λstart, λend +2π]. Therequired projection data on the detector from each view, however, isless obvious but not difficult to derive. From Sec. 1.3, rays passingthrough the π-line defined by λ1 and λ2 must be detected. It turnsout that the area on the detector that should be measured is speci-fied by the so called Tam-Danielsson (TD) window.5,13 This windowrepresents the shadow of all π-lines to which the current view anglecontributes. Geometrically, the boundaries of the TD-window aredefined by the shadows of the helical scanning trajectory on thedetector within 2π of the current scanning angle λ. Note that thisgeometrical definition can be applied to any detector geometry aslong as the TD window fits within the detector. In practice, even theTD window is the upper limit on the detector area. If it is known thatthe subject support is confined well within the convex hull of the heli-cal scan, then the required detector area can be reduced further. Ineither case, the BPF reconstruction allows the utilization of projectiondata that are longitudinally truncated; thus solving the long objectproblem.

Because the BPF theory reconstructs a volume image chord-by-chord substantial reduction of scanning effort, even over long objectscanning, is possible when the image is desired only within a certainROI. Given the ROI, one needs only identify the π-lines that inter-sect with the ROI and then reconstruct them. The scanning range isfound by examining the volume parameterization in terms of λ1 andλ2. For example, spherical volume can be reparameterized in termsof λ1, λ2, and t, then subsequently projected down to the λ1, λ2-plane(by integrating over t). Each point within the area represents a sin-gle π-line that should be reconstructed. The actual volume that isreconstructed is the union of the support segments of all of theseπ-lines, which in general will be larger than the desired ROI. TrueROI reconstruction (known as the interior problem) is theoreticallynot possible. As with the long object scanning, the necessary projec-tion data are identified by the shadow of the support segments ofeach π-line on the detector. While long object scanning serves thebulk of diagnostic helical cone-beam CT, ROI scanning may prove



useful for specific protocols in image guided radiation therapy, CTbreast screening, or cardiac imaging.

15.2 IMAGE RECONSTRUCTION IN POSITRONEMISSION TOMOGRAPHY

15.2.1 Introduction

Positron emission tomography (PET) is a unique, functional imag-ing modality that is capable of producing quantitative in vivo assaysof a large variety of molecular pathways of biological systems. PEThas been routinely used in cancer diagnosis and evaluation.14 It isalso widely used in neurology15 and cardiology,16 and is promisingfor providing effective treatment outcome evaluations.17 Recently,there has been substantial interest in developing dedicated PET sys-tems for imaging small animals (such systems are referred to asmicroPET systems below).18 In combination with the use of animalmodels of human biology and diseases, microPET systems are pow-erful tools in preclinical research.19 MicroPET imaging of gene trans-fer, expression, and therapy have been successfully demonstrated20;and there are high expectations that microPET systems will playimportant roles in discovering new biology, as well as in drug andtreatment developments.21 In comparison with human PET imag-ing, microPET imaging demands much higher imaging performancecharacteristics,18 making microPET system development a usefultest bed for innovative PET designs and technologies. Because bothanimal and human PET systems are available, PET imaging is alsoa useful translational research tool. Finally, in recent years there hasalso been greatly renewed enthusiasm for time-of-flight (TOF) PETimaging due to its ability to produce improved image quality andthe availability of fast and dense scintillators adequate for imple-menting TOF-PET systems.22

The imaging performance of a PET system depends criticallyon both its instrumentation and reconstruction.23 Many discov-eries and innovations in PET instrumentation have taken placein recent years.18,23 These include new scintillators and detector



designs that enable substantial improvement of the spatial reso-lution and timing accuracy in cost effective manners. Currently,PET imaging can reach a spatial resolution of about or better than1 mm. On the other hand, there exist efforts in achieving excep-tional high sensitivity for microPET systems.24 Parallel to advancesin instrumentation, there are substantial advances in PET imagereconstruction as well.25 In practice, as we will discuss below, PETdata are significantly degraded, making the imaging model consid-erably different from the ideal. Major degradations in PET imag-ing include data noise, effects of finite detector size, and the pres-ence of unwanted radiation (scatter and random coincidences).18,23

These degradations need to be addressed in reconstruction in orderto produce high-quality PET images. As the application domain ofPET imaging enlarges, higher demands in all performance aspectsof PET imaging can be expected. These demands would requiremany more advances in PET instrumentation and reconstructionto be made.

Excellent review articles on PET instrumentation and reconstruc-tion can be found in Refs. 18, 23 and 25. Below, we will discussissues and challenges facing PET image reconstruction and describeapproaches for addressing them.

15.2.2 Imaging Model

PET imaging is based on the principle of annihilation coincidencedetection and tracer kinetic modeling.23 PET tracers are moleculesradioactively labeled with positron emitting isotopes, which includeF-18, C-11, N-13, and O-15. Positrons emitted by PET tracers willannihilate with electrons in their surroundings and give rise to apair of 511 keV photons traveling in opposite directions. Typically,rings of gamma ray detectors are placed around the subject beingimaged. A simultaneous detection of two 511 keV photons by thedetector rings, called a coincidence detection, registers an annihila-tion event. Generally, one can define the response function hi(�x ) torepresent the probability for a positron emission occurring at �x tobe detected by the i-th detector pair of a PET scanner. The response



function hi(�x ) include factors such as the (geometric) detection effi-ciency of the i-th detector pair to the position �x and the attenuationthat the annihilation photons are subject to before exiting the sub-ject. Because the annihilation photons travel in opposite directions,for small detectors we have, to a good approximation, hi(�x) = εiai

for �x ∈ Li and �hi(�x ) = 0 for �x �∈ Li, where Li denotes the line thatconnects the centers of the front faces of the two detectors of the i-thdetector pair; and εi and ai are the detection efficiency and subjectattenuation on Li respectively. In the literature, Li is called the line ofresponse (LOR). The number of the coincidence events collected atthe i-th detector pair, denoted by gi, is then related to the image func-tion f (�x ), i.e. the spatial density distribution of the positron decaystaking place during the imaging time, by

gi = εiai ×∫

Li

dl f (�x ), (10)

where∫

Lidl denotes the line integral along the LOR Li. Consequently,

under suitable conditions PET measurements are related to a collec-tion of line integrals of the image function, i.e. to certain samplingsof the Radon transform of the image function. Provided that theresulting samplings are adequate, according to the theory of Radontransform, the image function can be recovered from the acquiredPET measurements, up to the spatial resolution limit supported bythe samplings.

The above description provides the basic principle underlyingPET imaging. This description is greatly simplified; it omits manyphysical factors involved in the imaging process, including thepositron range, photon noncolinearity, the presence of scattered andrandom coincidences, and the effects of finite detector size. Positronrange is the finite distance between the location where a positronis emitted and where the annihilation takes place. Therefore, rigor-ously speaking, the image function f (�x ) refers to the density functionof the positron annihilation, rather than that of the positron emit-ter itself unless the positron range is negligibly small. Dependingon the positron emitting isotope employed, the positron range can



vary from 0.83 mm to 8.54 mm.18 Photon noncolinearity refers to thefact that the directions of the two annihilation photons can slightlydeviate from the ideal 180◦. The full-width-at-the-half-maximum ofthis angular deviation is small but finite (about 0.5◦). The departureof the annihilation position from the detected LOR due to photonnoncolinearity increases as the size of the detector ring increases.Scattered events are registered coincidence events of which at leastone annihilation event undergoes scattering before detection. Thereare also random events which chance coincidence detection of twophotons originating from two independent positron annihilations.These event types can significantly contaminate PET measurementsin 3D whole-body PET imaging and in applications that employ hightracer concentrations.23 A pair of detectors is sensitive to all annihi-lation events taking place inside the common volume seen by thedetectors. In the above simplified imaging model, we have assumedsufficiently small detectors such that the sensitive volume reduces tothe LOR. In practice, this is often a poor assumption. Furthermore,due to gamma-ray penetrations the sensitive volume can be muchlarger than that suggested by the dimension of the detector frontface, leading to the phenomenon of parallax errors.23 The sensitivityof a detector pair to points within the sensitive volume of the detectorpair can also vary considerably. In addition to the above describedphysical factors, radioactive decay and photon detection are ran-dom processes, giving rise to statistical variations in the number ofdetected events when given the same image function and imagingconditions. To include these physical and statistical factors in PETimaging, one can write:

gi = E{gi} =∫

d3�xhi(�x )f (�x ) + si + ri, i = 1, . . . , N, (11)

where si and ri are the expectations of the number of scattered andrandom events accumulated at the i-th detector pair during theimaging time, and E{gi} denotes the ensemble mean of gi. In PET, gi’sare independent Poisson variates. Therefore, the conditional prob-ability distribution of the measurement �g = [g1, . . . , gN]t given the



image function f (�x ), is equal to:

p(�g|�f ) =N∏

i=1

e−gi ggi

i /gi. (12)

It is well known that variance {gi} = gi. Therefore, generally PET datanoise is not stationary (i.e. variant with measurements), with the therelative standard deviation of the noise with respect to its meandecreasing with the number of detected events. From Eq. (12), thelog-likelihood function for the measurement is given by:

l(�f |�g) = log p(�g|�f ) = −N∑

i=1

gi +N∑

i=1

gilog gi + constant. (13)

15.2.3 Image Reconstruction

Under idealized conditions, PET measurements are related to cer-tain samplings of the Radon transform of the underlying imagefunction. After correcting for the detection sensitivity and subjectattenuation, analytic algorithms developed for inverting the Radontransform, such as the well celebrated filtered backprojection algo-rithm (FBP), can be employed for reconstructing the unknown imagefunction from PET measurements. Methods that can compensate forcertain deviations from the Radon transform, such as the positronrange and stationary detector response, have also been proposed.26

Analytic PET reconstruction methods, however, have two majorshortcomings. First, the tomographic reconstruction process is illconditioned such that small data noises can give rise to large errorsin the solution image. Unfortunately, PET data generated in typi-cal studies are quite noisy; therefore, achieving effective control ofthe negative effects of data noise is a concern of special importancein PET reconstruction. Noise reduction in analytic reconstructionmethods is typically achieved by employing ad hoc low-pass filters.By assuming stationary data noise (which is incorrect), Wiener fil-ters for reducing noise have also been developed and investigated.27

Generally speaking, analytic methods lack proper mechanisms forimplementing optimized handing of the nonstationary data noise



encountered in PET imaging. Second, analytic reconstruction algo-rithms are based on simplified imaging models that do not take intoaccount most physical factors present in PET imaging. Therefore, it isnecessary to apply prereconstruction corrections so that the imagingmodel for the corrected data can approximate the assumed models.Physical factors that are uncorrected for, or are only partially cor-rected, in the preprocessing can lead to image degradations, suchas image blur and spatial varying resolution, or even cause imageartifacts. Accurate prereconstruction data corrections are often diffi-cult to achieve. Furthermore, such prereconstruction corrections willdeteriorate the statistical nature of the acquired data and aggravatethe aforementioned concern regarding the inferior noise handlingcapability with the analytic reconstruction methods.

Model-based approaches that can fully account for the physicaland statistical models of the PET imaging process are necessary forachieving best image reconstructions. Iterative reconstruction meth-ods are such model-based techniques. For purpose of computation,the image function needs to be discretized:

f (�x ) =M∑

j=1

fjbj(�x), (14)

where bj(�x ), j = 1, . . . , M, is a expression set. The continuous imagemodel given by Eq. (11) then becomes:

E{�g} = H�f + �s + �r, (15)

where �f = [f1, . . . , fM]t, �s = [s1, . . . , sN]t, �r = [r1, . . . , rN]t, and His an N × M system response matrix having the elements Hij =∫

d3�xhi(�x )bj(�x). The probability model for �g is still given by Eq. (12).In the literature, the voxel representation, in which the image isassumed to consist of a lattice of cubic elements containing uni-form radioactivity within, is widely adopted for image discretiza-tion. Other discrete image representations have also been proposedfor PET image reconstruction. It is also common for researchers to



consider the following simplified PET imaging model:

E{�y} = H�f . (16)

In this case, one either removes scattered and random events inthe data by prereconstruction corrections, or simply ignores suchevents. In addition, the Poisson model is often assumed for �y eventhough the model is no longer valid after data corrections. Manyiterative methods for solving the discrete PET imaging model givenby Eqs. (15) and (16) having been developed. These methods differ inthe cost functions they employ for finding solutions to these mod-els. They also substantially differ in the quantitative performancecharacteristics (i.e. the trade-off behavior between the reconstruc-tion accuracy and noise sensitivity), the convergence behavior, andthe computational complexity.

Iterative PET reconstruction methods include the algebraicreconstruction techniques (ART),28 projection-onto-convex (POCS)techniques,29 penalized weighted least-square (PWLS) methods,30

maximum likelihood-based (ML) approaches,31 and the maximuma posteriori (MAP) approaches.32,33 In theART methods, one observesthat the solution to Eq. (16) can be interpreted as:

�f ∈N⋂

i=1

Ai, Ai = {�x : �hti�x = yi}, (17)

where �hi = [Hi1, . . . , HiM]t. The projection operator Pi that maps anarbitrary vector �x to the closest point on the hyperplane Ai is equal to:

Pi�x = �x + (�gi − �hti�x)�hi

/|�hi|2. (18)

The ART algorithm seeks to sequentially enforce the hyperplaneconstraints until convergence is reached, yielding the following algo-rithm: given an initial estimate �f (0), the nth estimate is equal to:

�f (n) = PN · · · P1�f (n−1). (19)

The order of the projection is arbitrary. The resulting algorithm is fastin terms of both convergence and the computation time needed eachiteration, but it lacks the ability to explicitly incorporate mechanisms



for handling data noise and often fails to converge when subject toinconsistent data. The POCS techniques are generalizations of theART methods in which the solution image is given by:

�f ∈N⋂

i=1

Ai, N ′ > N, (20)

where, in addition to the N hyperplane constraints given by themeasurements, Ai can include any convex set. Let Pi denote theprojection operator associated with the convex constraint sets Ai,the POCS update equation is then given by:

�f (n) = PN · · · P1 �f (n−1). (21)

Therefore, certain information regarding the data noise and solutionimages in the form of convex constraints can be specified for allevi-ating the negative effects of data noise. Convergence can still be anissue in POCS methods.

In contrast to ART and POCS methods, the ML, MAP, and PWLSmethods are statistical methods that explicitly employ the probabil-ity distributions of the data in reconstruction. In the MLmethods, thesolutions maximize the log likelihood functions given by Eq. (13),and they are often generated by using the expectation maximization(EM) algorithm given by:

f (n)j =

∑i

Hij∑i Hij

{gi∑

j′ Hij′ f(n−1)j′

}f (n−1)j , j = 1, . . . , M. (22)

When the measurements are strictly independent Poisson variates,given a positive initial estimate �f (0) the EM algorithm is guaranteedto converge to the ML solution.31 The EM algorithm has a relativelysimple update equation, offers favorable quantitative performancecharacteristics, and automatically enforces the voxel positivity con-dition. In the MAPapproach (also called the Bayesian approach), oneseeks to maximize the a posteriori distribution p(�f |�g) or, equivalently,



log p(�f |�g). According to the Bayes theorem, we have:

log p(�f |�g) = log(p(�g|�f )p(�f )/p(�g )

)= l(�f |�g) + log p(�f ) + constant.

(23)The a priori information p(�f ) imposes smoothness conditions onthe solution,32 or introduce structural information of the solutionderiving from associated anatomical images.33 In the literature,MAP methods are also called penalized maximum-likelihood meth-ods because the prior term penalizes the log-likelihood function inEq. (23). Many iterative algorithms for generating the MAP esti-mates have been proposed, including EM-like algorithms. BothML and MAP methods require exact knowledge of the probabil-ity distributions of the data. In many practical situations (such aswith pre-corrected data), such exact probability distributions arenot available. Approximate distributions for random-corrected datahave been proposed, including the shifted Poisson model and itsvariations, and the saddle-point approximation.34 Although exactdistributions are difficult to obtain, the second-order statistics of thecorrected data can be readily derived. It is therefore attractive toemploy PWLS methods that seek the minimize the cost function30:

�(�f ) = 12

(�y − H�f )tW (�y − H�f ) + β(�f )G(�f ), (24)

where the weighting matrix W is the inverse of an estimate of theconditional variances of �y, G(�f ) imposes penalties for image rough-ness, and β(�f ) provides a mechanism for preserving edge structuresin image.

The EM algorithm is quite attractive for PET image reconstruc-tion; the main drawback that limits its practical usefulness is itsslow convergence rate. An important variation of the EM algorithmis the ordered subsets EM (OSEM) algorithm, which has been widelyadopted as the de facto standard for practical applications.35 In thisalgorithm, the data are divided into a number of disjoint subsets andthe EM algorithm is sequentially applied to these subsets to consti-tute one iteration. This simple modification has been observed to



remarkably increases the algorithm’s convergence rate and, empir-ically, faster convergence rate is achieved with the use of more sub-sets. Unfortunately, under certain situations the OSEM algorithmmay not converge (similar to the situations with ART and POCS).Modifications to ensure the convergence of the OSEM algorithmhave been developed and investigated. Important examples are theraw-action maximum-likelihood (RAMLA) algorithm36 and the con-vergent OSEM (COSEM) algorithm.37

It is noted that, in practice, the properties of a statistical recon-struction method depend not only on the cost function it aims tooptimize but also on the specific updating equation it employs. Thisis because the algorithm is often terminated before reaching conver-gence and in some situations multiple solutions can exist. We alsonote that at convergence the EM algorithm is known to minimizethe Kullback-Leibler distance between the acquired data and thepredicted data based on the estimated solution. This interpretationof the EM algorithm is valid irrespective of the specific data noisemodel.

15.2.4 Three-Dimensional Imaging, Dynamic Imaging,and List Model Reconstruction

Most modern PET systems are fully 3D systems, with the so called3DRP algorithm of Kinahan38 being widely offered for perform-ing analytical 3D PET image reconstruction. Alternative rebinningapproaches that convert a fully 3D PET dataset to a collection of 2Ddatasets associated with individual transaxial image slices have alsobeen developed. The conversion process can be either approximate39

or mathematically exact.40 Hybrid iterative reconstruction meth-ods that first analytically rebin 3D PET datasets to 2D datasetsand employ 2D iterative reconstruction for achieving slice-by-slicereconstruction have also been developed and investigated. In suchhybrid approaches, system response matrices for, and the proba-bility distributions of, the rebinned data need to be determined.Generally, as expected, direct 3D iterative reconstruction producesthe best solutions. Hybrid approaches, nonetheless, greatly alleviate



the tremendous computation demands required by fully 3D itera-tive reconstructions and provide attractive tradeoffs between imageaccuracy and computation burden.

For derivations of certain local biochemical/physiologicalparameters within individual voxels, dynamic PET imaging is oftenperformed. Conventionally, dynamic PET data are stored as a tem-poral sequence of static PET data. The acquired data at each timepoint, called a frame, is separately reconstructed by using analytic oriterative reconstruction methods described above to generate a tem-poral sequence of PET images. Appropriate kinetic models are thenemployed to account for the observed temporal variations of the PETtracers within each voxel for deriving relevant parametric images.In this conventional approach, the spatial and temporal informa-tion available in the dynamic PET data are treated independently,although they are not uncorrelated. In Ref. 41, Kao et al. made theobservation that the temporal information available in the dynamicdata can be exploited for greatly reducing the data noise associatedwith each frame and hence significantly improving the resultingimage quality. By having obtained better frame images, more accu-rate kinetic parameters regarding the PET tracer are also obtained.Reconstruction approaches that generate parametric images directlyfrom the dynamic PET data have also been reported.42

So far, we have discussed the histogram data format in whichthe accumulated event counts at individual detector pairs (i.e. atindividual LORs) of a PET scanner are presented. In contrast, thelist-model data format presents a stream of individual event recordsthat are sequentially stored in the chronological order of event detec-tion. List-model data format is more versatile than the histogramformat. In principle, as much information as desired regarding thedetected events can be stored in the event records, therefore per-mitting maximal utilization of the detected event information forachieving optimized image reconstruction. Obviously, list-modeldatasets grow linearly in size with the imaging time while the his-togram datasets have a fixed size as determined the number of LORsof a PET scanner.As the number of LORs in a modern high resolutionPET system has drastically increased, the list-model data format



has gained popularity because its advantages are starting to out-weigh the storage disadvantage. Iterative algorithms basing on theML and MAP criteria for reconstructing list-model PET data havebeen developed.43,44 Methods for jointly estimating the image func-tions and the temporal basis functions underlying the tracer kineticsfrom list-model data have also been investigated.45 The combinationof list-model data and physiological gating information also pro-vides excellent mechanisms for performing cardiac and respiratorymotion corrections. Readers are referred to Ref. 25 for more detaileddiscussion on list-model PET image reconstruction.

15.3 IMAGE RECONSTRUCTION IN SINGLE PHOTONEMISSION COMPUTED TOMOGRAPHY

15.3.1 Introduction

In single-photon emission computed tomography (SPECT), a radio-pharmaceutical is injected into a patient with the expectation thatit will track some functional or physiological process of interest. Atany given time, one seeks to know the 3D distribution of the tracer.This can be achieved by employing one or more scintillation camerasplaced outside the patient, each of which records the 2D distributionof emitted photons incident on it.

In order to form a projection image that represents a knownmapping from the 3D distribution of activity to the 2D projection,the camera is generally equipped with a lead collimator that restrictsthe angular range of photons that reach the face of the camera. Inthis section, we focus on the case when a so-called parallel-hole col-limator is employed. Such collimators attempt to restrict attention tophotons that are travelling normal to the face of the camera, althoughin practice, they admit photons incident from a range of angles cen-tered around zero degrees. This acceptance cone leads to depth-dependent resolution and it is important to model and account forthis effect in order to obtain more accurate reconstructed images ofthe activity distribution.

Another physical effect that must be accounted for is attenua-tion of the photons as they travel through the patient from the point



at which they are emitted. SPECT is often performed with photonsaround 140 keV for which the attenuation coefficient in soft tissueis approximately 0.15 cm−1. In some areas of the body, such as theabdomen, it is reasonable to assume that attenuation is uniform.In other regions, such as the thorax, where the lungs, soft tissue,and bone all present significantly different attenuation coefficients, amore general model of nonuniform attenuation helps improve quan-titative accuracy.

To obtain sufficient data to invert the mapping from the 3D activ-ity distribution to 2D projections, the camera or cameras must berotated around the patient to a variety of angles. If we represent thecoordinates of the camera face by ξ and z and the angular positionof the camera by θ, then the mean of the set of measurements, whichwe denote p (ξ, z, θ), can be related to the 3D activity distributionby the following very general equation, adapted from Liang (PMB1997), and which includes the effect of nonuniform attenuation anddepth-dependent resolution:

p(ξ, z, θ) =∫ ∞

−∞dη

∫ ∞

−∞

∫ ∞

−∞dξ′dz′h (ξ − ξ′, z − z′, η) aθ (ξ′, z′, η)

×exp(

−∫

L0(ξ′,z′,η;ξ,z)µθ(ξ′′, z′′, η′′)dl

). (25)

Here, aθ(ξ, η, z) represents the activity distribution a(x, y, z) in a coor-dinate system rotated by θ about the z axis:

ξ = x cos θ + y sin θ(26)

η = x sin θ − y cos θ,

and µθ(ξ, η, z) represents the attenuation map µ(x, y, z) in the samerotated coordinate system. The detector response kernel is repre-sented by h(ξ, z, η) and it models blurring that is depth-dependent,but shift invariant at a specified depth. The attenuation term is writ-ten as a line integral through the attenuation map along the lineLθ(ξ′, z′, η; ξ, z) that connects the point (ξ′, z′, η) to the detector bin(ξ, z) at angle θ. This very general form accounts for the fact that



photons travelling from different portions of the field of view of agiven bin at a given depth could experience different amounts ofattenuation because of the different path they travel along towardthe detector.

Fortunately, it is very reasonable to simplify Eq. (25) by assumingthat the attenuation experienced by the photons traveling along anyof the lines contributing to a given projection bin is the same and canbe represented by the attenuation that takes place along the centralray of the bundle. We then obtain:

p(ξ, z, θ) =∫ ∞

−∞dη

∫ ∞

−∞

∫ ∞

−∞dξ′dz′h (ξ − ξ′, z − z′, η) aθ(ξ′, z′, η)

×exp(

−∫ η

−∞µθ(ξ′, z′, η′)dη′

). (27)

We will take Eq. (27) as our fundamental imaging equation and con-sider the approaches that have been developed to inverting it undera variety of special cases.

15.3.2 No Attenuation, No Depth-dependent Resolution

The simplest possible case arises when one ignores the effectsof attenuation and depth-dependent resolution effects. Then weobtain:

p (ξ, z, θ) =∫ ∞

−∞dηaθ(ξ, z, η), (28)

which is a slack of two-dimensional Radon transforms. This can beinverted by use of a number of standard reconstruction algorithms,including filtered backprojection and direct Fourier methods.

15.3.3 Uniform Attenuation Alone

The next simplest case arises when ignoring depth-dependent res-olution and assuming that the attenuation can be represented bya uniform attenuation coefficient within some closed boundary.



In this case, the imaging equation can be written:

p(ξ, z, θ) =∫ ∞

−∞dηaθ(ξ, z, η)e−µ[η+D(ξ,z,θ)], (29)

where D(ξ, z, θ) represents the distance from the point x = ξ cos θ, y =ξcos θ, z to the boundary in the direction of the projection. By defininga set of modified projections:

m(ξ, z, θ) ≡ eµ[D(ξ,z,θ)]p(ξ, z, θ), (30)

we obtain:

m(ξ, z, θ) =∫ ∞

−∞dηaθ(ξ, z, η)e−µη, (31)

an equation generally known as the exponential Radon transform(ERT).46

Tretiak and Metz developed an FBP-style reconstruction for-mula in which appropriately modified projections are subject toexponentially weighted backprojection.46 The reconstruction for-mula for the activity a(r, φ, z) given in cylindrical coordinates can beexpressed as:

a(r, φ, z) =∫ 2π

0eµη

∫|νm|≥νµ

|νm|2

ej2πνmξ

∫ ∞

−∞m(ξ′, z, θ)e−j2πνmξ′

dξ′dνmdθ,

(32)where vµ = µ/2π, ε = r cos (θ, −ϕ), and µ = r sin (θ−ϕ). Anumber ofdifferent analytic algorithms for inverting this imaging model wereproposed over the years. Bellini et al.47 and Inouye et al.48 developedmethods that worked in the spatial frequency domain to estimate the2D Fourier transform of the unattenuated sinogram, from which theexact image could be obtained by inverting the 2D Radon transform.Hawkins proposed a method based on the use of circularly harmonicBessel transforms.49 These algorithms are all exact in the face of per-fect data, but they propagate noise and inconsistencies differently.

In 1995, Metz and Pan50 analyzed the 2D Fourier transformof the 2D ERT and demonstrated that all these methods can be



interpreted as special cases of a broad class of methods. In particular,they showed that these methods represented different choices ofweighting coefficients implicitly being used to combine redundantdata that arise due to certain Fourier-domain symmetries. Metz andPan also showed that a new member of the class, given by a differ-ent choice of those weighting coefficients, had better noise propertiesthan those of the existing methods.50,51

Specifically, the method provides a means of estimating the coef-ficients of the angular Fourier series representation of the 2D Fouriertransform of a(r, φ, z) at a fixed z, which we denote Ak(νa), from the2D Fourier transform (actually a combination of a 1D Fourier trans-form and a 1D Fourier series expansion) of the modified data, whichwe denote Mk(νm), by use of:

Ak(νa) = ωγkMk(νm) + (1 − ω)(−1)kγ−kMk(− νm), (33)

where νm =√

ν2a + ν2

µ, γ =√

ν2m − ν2

µ/(νm + νµ), and 0 ≤ ω ≤ 1is a weight that allows the two independent estimates of Ak(νa)to be combined in a way that minimizes the variance of the finalimage. Metz and Pan showed that the existing algorithms can bederived by the selection of different ω and that new algorithms canbe derived that may have noise properties superior to the existingalgorithms.50,51

15.3.4 Distance-dependent Resolution Alone

If attenuation is ignored but distance-dependent resolution effectsmodeled, then Eq. (27) becomes:

p(ξ, z, θ) =∫ ∞

−∞dη

∫ ∞

−∞

∫ ∞

−∞dξ′dz′h(ξ − ξ′, z − z′, η)aθ(ξ′, z′, η). (34)

Appledorn52 presented an analytic solution to this equation for thecase when h(ξ, z, η) is a Cauchy function whose width parametergrows linearly with distance η.



15.3.5 Distance-dependent Resolution andUniform Attenuation

When both distance-dependent resolution and uniform attenuationare modeled, Eq. (27) becomes:

p(ξ, z, θ)

=∫ ∞

−∞dη

∫ ∞

−∞

∫ ∞

−∞dξ′dz′h(ξ − ξ′, z − z′, η)aθ(ξ′, z′, η)e−µ[η+D(ξ,z,θ)],

(35)

Soares derived the first analytic solution to this equation for thecase when h(ξ, z, η) is a Cauchy function.53 Van Elmbt and Walrand54

generalized Bellini’s method for inverting the ERT to invert Eq. (35)for the more practical case when h(ξ, z, η) is modeled as a Gaussianwhose standard deviation grows linearly with distance η. Pan andMetz extended the earlier work of Metz and Pan to this equationfor both the Cauchy form of h(ξ, z, η) considered by Soares and theGaussian form considered by van Elmbt and Walrand.55

15.3.6 Nonuniform Attenuation Alone

When the attenuation is nonuniform and distance-dependent reso-lution effects are ignored, Eq. (27) becomes:

p(ξ, z, θ) =∫ ∞

−∞dηaθ(ξ, z, η)exp

(−

∫ η

∞µθ(ξ, z, η′)dη′

). (36)

This equation is often referred to as the attenuated Radon transform.An approximate approach to inverting this equation was devel-

oped by Chang.56 The multiplicative Change method entails cal-culating the average fraction of photons that safely escape fromeach point in the reconstructed volume to the various detector loca-tions. The reconstructed image is then multiplied by the reciprocal ofthis average transmission factor map to obtain the corrected image.This correction is only approximate but it can be refined throughan iterative process in which the corrected image is reprojected andthe resulting data compared to the measured data. The difference



between the reprojections and the measured data are used to gener-ate an error image by reconstruction using FBP. The error images arecorrected for attenuation by the multiplicative trick and then addedto the original corrected image. This process could be continued fora desired number of iterations.

The explicit solution for the attenuated Radon transform wasfirst presented by Novikov.57 Natterer made significant contri-butions to the theory, including a different inversion formula aswell as an alternative, simpler derivation of Novikov’s formula.58

Kunyansky also provided a slightly modified version of Novikov’sformula, which allowed for simpler implementation and from whichFBP and the Tretiak-Metz algorithm could easily be obtained underthe cases when attenuation was zero or uniform, respectively.59

15.3.7 Short Scan and Region of Interest Imaging

While it is well known that inversion of the Radon transform onlyrequires data acquired over a 180◦ angular range, it was not knownuntil recently whether the exponential Radon transform and theattenuated Radon transform could also be inverted from so calledshort scan data.

In 2002, however, Noo andWagner showed that the ERT requiresdata on the angular interval θ ∈ [0, π].60 Pan et al.61 generalized thisresult to develop so called π-scheme short scan strategy in which thefull angular range of 2π is divided into a number of nonoverlappingangular intervals, and the data function is acquired only over dis-joint angular intervals whose summation without conjugate viewsis equal to π. This approach does not yield an explicit inversion for-mula, but it was demonstrated that an iterative algorithm is able togenerate high quality reconstructions from π-scheme data.

As for the attenuated Radon transform, Sidky et al., were able toshow, both heuristically and rigorously, that a short scan is sufficienthere as well by adopting the so called potato-peeler perspective toestablish that there is two-fold redundancy in a fullscan dataset.62

Recently, Noo et al. have developed approaches to reconstructingROI images from the ERT and attenuated RT with truncations.63




This work was supported in part by NIH grants EB00225 andEB02765. Dr Emil Sidky was supported NIH K01 EB003913. Theauthors are thankful to Mr Xiao Han and Mr Dan Xia for workingon the latex file of the manuscript.

References

1. Tuy HK, An inversion formula for cone-beam reconstruction, SIAMJ Appl Math 43: 546–552, 1983.

2. Grangeat P, Mathematical framework of cone-beam 3D reconstructionvia the first derivative of the Radon transform, in Herman GT, LouisAK, Natterer F (eds.), Mathematical Methods in Tomography, Lecture Notesin Mathematics, Springer-Verlag, Berlin, pp. 66–97, 1991.

3. Katsevich A, Analysis of an exact inversion algorithm for spiral cone-beam CT, Phys Med Biol 47: 2583–2597, 2002.

4. Chen G, An alternative derivation of Katsevich’s cone-beam recon-struction formula, Med Phys 30: 3217–3226, 2003.

5. Danielsson PE, Edholm P, Seger M, Towards exact 3D-reconstructionfor helical cone-beam scanning of long objects.Anew detector arrange-ment and a new completeness condition, in Townsend DW, Kinahan PE(eds.), Proceedings of the 1997 International Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine,Pittsburgh, pp. 141–144, 1997.

6. Zou Y, Pan X, Exact image reconstruction on PI-line from minimumdata in helical cone-beam CT, Phys Med Biol 49: 941–959, 2004.

7. Zou Y, Pan X, Image reconstruction on PI-lines by use of filtered back-projection in helical cone-beam CT, Phys Med Biol 49: 2717–2731, 2004.

8. Sidky EY, Zou Y, Pan X, Minimum data image reconstruction algo-rithms with shift-invariant filtering for helical, cone-beam CT, PhysMed Biol 50: 1643–1657, 2005.

9. Zou Y, Pan X, An extended data function and its backprojection ontoPI-lines in helical cone-beam CT, Phys Med Biol 49: N383–N387, 2004.

10. Zou Y, Pan X, Sidky EY, Theory and algorithms for image reconstruc-tion on chords and within region of interests, Journal of the Optical Soci-ety of America A 22: 2372–2384, 2005.

11. Jiang Hsieh, Computed Tomography — Principles, Designs, Artifacts, andRecent Advances, SPIE Press, Bellingham, WA, 2003.

12. Sidky EY, Pan X, Recovering compactly supported functions fromknowledge of its hilbert transform on a finite interval, IEEE SignalProcessing Lett 12: 97–100, 2005.



13. Tam KC, Samarasekera S, Sauer F, Exact cone-beam CT with a spiralscan, Phys Med Biol 43: 1015–1024, 1998.

14. Gambhir SS, Molecular imaging of cancer with positron emissiontomography, Nature Rev Cancer 2: 683–693, 2002.

15. Herholz K, Heiss WD, Positron emission tomography in clinical neu-rology, Molecular Imaging and Biology 6: 239–269, 2004.

16. Lance Gould K, PET perfusion imaging and nuclear cardiology, J NuclMed 32: 579–606, 1991.

17. Alexander GE et al., Longitudinal PET evaluation of cerebral metabolicdecline in dementia: A potential outcome measure in Alzheimer’s dis-ease treatment studies, Am J Psychiastry 159: 738–745, 2002.

18. Tai YC, Laforest R, Instrumentation aspects of animal PET, Annu RevBiomed Eng 7: 255–285, 2005.

19. Weissleder R, Scaling down imaging: Molecular mapping of cancer inmice, Nature Rev Cancer 2: 11–18, 2002.

20. Herschman HR et al., Seeing is believing: Non-invasive, quantita-tive and repetitive imaging of reporter gene expression in living ani-mals, using positron emission tomography, J Neurosci Res 59: 699–705,2000.

21. Kelloff GJ, Progress and promise of FDG-PET imaging for cancerpatient management and oncologic drug develpment, Clinical CancerResearch 11: 2785–2808, 2005.

22. Moses WW, Time-of-flight in PET revisited, IEEE Trans Nucl Sci 50:1325–1330, 2003.

23. Wernick MN, Aarsvold JN (eds.), Emisson Tomography: The Fundamen-tals of PET and SPECT, Elsevier Academic Press, San Diego, CA, 2004.

24. Kao CM, Chen CT, Development and evaluation of a dual-head PETsystem for high-throughput small-animal imaging, 2003 IEEE NuclearScience Symposium Conference Record, 2072–2076, 2003.

25. Qi J, Leahy RM, Iterative reconstruction techniques in emission com-puted tomography, Phys Med Biol 51: R541–R578, 2006.

26. Huesman R, Salmeron E, Baker J, Compensation for crystal penetrationin high resolution positron tomography, IEEE Trans Nucl Sci 36: 1100–1107, 1989.

27. Shao L, Karp JS, Countryman P, Practical considerations of the Wienerfiltering technique on projection data for PET, IEEE Trans Nucl Sci 41:1560–1565, 1994.

28. Herman GT, Meyer LB, Algebraic reconstruction techniques can bemade computationally efficient [positron emission tomography appli-cation], IEEE Trans Med Imag 12: 600–609, 1993.

29. Wernick MN, Chen CT, Superresolved tomography by convex projec-tions and detector motion, J Opt Soc Am A 9: 1547–1553, 1992.



30. Fessler JA, Penalized weighted least-squares image reconstruction forPET, IEEE Trans Med Imag 13: 290–300, 1994.

31. Shepp LA, Vardi Y, Maximum likelihood reconstruction for emissiontomography, IEEE Trans Signal Process 41: 534–548, 1982.

32. Hebert T, Leahy R, Ageneralized EM algorithm for 3D Bayesian recon-struction from Poisson data using Gibbs priors, IEEE Trans Med Imag8: 194–202, 1989.

33. Ouyang X, Wong WH, Johnson VE, Hu X, Chen CT, Incorporation ofcorrelated structural image in PET image reconstruction, IEEE TransMed Imag 13: 627–640, 1994.

34. Ahn S, Fessler JA, Emission image reconstruction for randoms-precorrected PET allowing negative singoram values, IEEE Trans MedImag 23: 591–601, 2004.

35. Hudson HM, Larkin RS, Accelerated image reconstruction usingordered subsets of projection data, IEEE Trans Med Imag 13: 601–609,1994.

36. Browne J, De Pierro AR, A row-action alternative to the EM algorithmfor maximizing likelihoods in emission tomography, IEEE Trans MedImag 15: 687–699, 1996.

37. Hsiao IT, Rangarajan A, Khurd P, Gindi G, An accelerated convergentordered subsets algorithm for emission tomography, Phys Med Biol 49:2145–2156, 2004.

38. Kinahan PE, Rogers WL, Analytic 3D image reconstruction using alldetected events, IEEE Trans Nucl Sci 36: 864–968, 1989.

39. Duabe-Witherspoon ME, Muehllehner G, Treatment of axial data inthree-dimensional PET, J Nucl Med 28: 1717–1724, 1987.

40. Defrise M et al., Exact and approximate rebinning algorithm for 3D PETdata, IEEE Trans Med Imag 16: 167–186, 1997.

41. Kao CM, Yap JT, Mukherjee J, Wernick MN, Image reconstruction fordynamic pet based on low-order approximation and restoration of thesinograms, IEEE Trans Med Imag 16: 738–749, 1997.

42. Kamasak ME, Bouman CA, Morris ED, Sauer K, Direct reconstructionof kinetic parameter images from dynamic PET data, IEEE Trans MedImag 25: 636–650, 2005.

43. Barrett HH, White T, Parra L, List-model likelihood, J Opt Soc Am A 14:2914–2923, 1997.

44. Rahmin A, Blinder S, Cheng JC, Sossi V, Statistical list model recon-sturction in quantitative dynamic imaging using high resolutionresearch tomograph, 8th Fully 3D Meeting 117–120, 2005.

45. Reader AJ, Sureau FC, Comtat C, Trebossen R et al., Joint estimationof dynamic PET images and temporal basis functions using fully 4DML-EM, Phys Med Biol 51: 5455–5474, 2006.



46. Tretiak O, Metz CE, The exponential radon transform, SIAM J ApplMath 39: 341–354, 1980.

47. Bellini S, Piacenti M, Caffario C, Rocca F, Compensation of tissueabsorption in emission tomography, IEEE Trans Acoust Speech, Sig Pro-cessing 27: 213–218, 1979.

48. Inouye T, Kose K, Hasegawa A, Image reconstruction algorithm forsingle-photon-emission computed tomography with uniform attenu-ation, Phys Med Biol 34: 299–304, 1989.

49. Hawkins WG, Leichner PK, Yang NC, The circular harmonic trans-form for spect reconstruction and boundary conditions on the fouriertransform of the sinogram, IEEE Trans Med Imaging 7: 135–148, 1988.

50. Metz CE, Pan X, A unified analysis of exact methods of inverting the2-D exponential radon transform, with implications for noise controlin spect, IEEE Trans Med Imaging 14: 643–658, 1995.

51. Pan X, Metz CE, Analysis of noise properties of a class of exact meth-ods of inverting the 2-D exponential radon transform, IEEE Trans MedImaging 14: 659–668, 1995.

52. Appledorn CP, An analytical solution to the nonstationary reconstruc-tion problem in single photon emission computed tomography, inOrtenhdahl DA, Llacer J (eds.), Information Processing in Medical Imag-ing, Wiley-Liss, New York, pp. 69–79, 1990.

53. Soares EJ, Byrne CL, Glick SJ, Appledorn CR et al., Implementationand evaluation of an analytical solution to the photon attenuation andnon-stationary resolution reconstruction problem in SPECT, IEEE TransNucl Sci 40: 1231–1237, 1993.

54. van Elmbt L, Walrand S, Simultaneous correction of attenuation anddistance-dependent resolution in SPECT: An analytical approach, PhysMed Biol 38: 1207–1217, 1993.

55. Pan X, Metz CE, Analytical approaches for image reconstruction in 3dspect, in Grangeat P,Amans J (eds.), 3D Image Reconstruction in Radiologyand Nuclear Medicine, KluwerAcademic Publishers, NewYork, 103–116,1996.

56. Chang LT, A method for attenuation correction in radionuclide com-puted tomography, IEEE Trans Nucl Sci 25: 638–643, 1978.

57. Novikov RG, An inversion formula for the attenuated X-ray transfor-mations, Dèpartment deMathèmatique, Universitè de Nantes, Nantes,France (preprint), 2000.

58. Natterer F, Inversion of the attenuated Radon transform, Inv Probs 17:113–119, 2001.

59. Kunyansky LA, A new SPECT reconstruction algorithm based uponthe Novikov’s explicit inversion formula, Inv Prob 17: 293–306,2001.



60. Noo F, Wagner JM, Image reconstruction in 2D spect with 180-degreeacquisition, Inv Probs 17: 1357–1372, 2001.

61. Pan X, Kao CM, Metz C, A family of pi-scheme exponential radontransform and the uniqueness of their inverses, Inv Prob 18: 825–836,2002.

62. Sidky E, Pan X, Variable sinogram and redundant information in spectwith non-uniform attenuation and the uniqueness of their inverses, InvProbs 18: 1483–1497, 2002.

63. Noo F, Defrise M, Pack JD, Clackdoyle R, Image reconstruction fromtruncated data in single-photon emission computed tomography withuniform attenuation, Inv Prob 23: 645–667, 2007.


CHAPTER 16

Shape-Based Reconstruction fromNevoscope Optical Images of

Skin Lesions

Song Wang and Atam P Dhawan

Optical imaging of skin lesions has been of significant interest for earlydiagnosis and characterization of skin cancers. The work presented inthis chapter is in continuation of the development of an optical imagingbased portable system with computerized analysis to detect skin cancers,particularly melanomas in early curable stage. The method developedin this paper can provide reconstructions of melanin and blood contentsassociated with a skin lesion from its multispectral transillumination basedoptical images. The results of simulation of a skin lesion for reconstructionof melanin and blood (hemoglobin) information from multispectral opticalimages are presented. Changes in melanin and hemoglobin contents in askin lesion detected over time using the proposed method would allowearly detection of malignant transformation and the development of acancerous lesion.

16.1 INTRODUCTION

In recent years, optical medical modalities have drawn significantattention from researchers. Visible and near-infrared light wave-lengths have been used of surface reflectance, transillumination andtransmission based methods.1 Also, optical modalities can providea portable imaging system for routine screening and monitoring ofskin lesions. Optical modalities usually make use of light within thelower part of magnetic electric spectra, which is believed that thiskind of light is not going to poison the interrogated tissue or the

393


394 Song Wang and Atam P Dhawan

side effect would be greatly reduced. In the so called “therapeuticwindow,” including part of visible light and infrared light, physio-logically meaningful chromophores like melanin, oxyhemoglobinand deoxyhemoglobin have relative low absorption coefficients.Meanwhile, the scattering coefficients of human tissues are rela-tively high in this range, resulting in a penetrating depth favor-able for investigating brain, breast and skin etc. More importantly,the optical coefficients of these chromophores are strongly wave-length dependent. The discrepancy of chromophores therefore givesus a chance to reveal their distributions using multispectral light.1

Though the work presented in this chapter is motivated by theneed of developing a computer-aided optical imaging system fordiagnosis and characterization of skin cancers, the methods pre-sented in the paper are generally applicable for optical imagereconstruction.

Skin cancer is one of fastest growing cancer among all cancers.2

The majority of skin cancers are nonmelanoma skin cancers. Thecancer is derived from keratinocytes, the main type of cell of epider-mis. Malignant melanoma results from uncontrolled growth of mela-somes originally existed in epidermis. Though nonmelanoma cancerprevails among all kinds of skin cancers, malignant melanoma isthe most fatal form which accounts for 90% death.2 It is fatal ifnot detected in early stages. It can be cured with nearly 100% sur-vival rate, if removed at its early stage. Malignant melanoma iscurrently diagnosed by dermatologists according to its color andmorphology.2 However, such diagnosing process to large extentis subjective and diagnostic accuracy rests on the dermatologist’sindividual experience. There is an urgent need for developing anoninvasive modality to reveal physiological features of malignantmelanoma quantitatively and objectively so that even an unskilleddermatologist is able to make right decision.

As an effective utility to diagnosing malignant melanoma, thelight-based device should be able to provide both morphologicaland physiologic information. Morphological information may beutilized to determine the depth of invasion of malignant melanoma


Shape-Based Reconstruction from Nevoscope Optical Images of Skin Lesions 395

while physiologic characteristics like distribution of melanin dustand blood vessels are essential to differentiate it from a benignone. Within visible and infrared spectra, major chromophores ofmalignant melanoma include melanin, oxyhemoglobin and deoxy-hemoglobin. The absorption of water is negligible compared to thesemajor ones. Among these chromophores, melanin presents a higherabsorption than oxyhemoglobin and deoxyhemoglobin. Also, theirabsorption spectra are not linear dependent. Hence, it is possibleto uncover distributions of these chromophores by multispectraloptical measurement. Like mentioned before, malignant melanomahave a nonblood, melanin-rich core and a hemoglobin-rich periph-eral blood net. Once distributions of chromophores are rendered, itsstructure is available simultaneously. Having investigated the phys-ical properties of malignant melanoma under visible and infraredlight, it is obvious that an optical device would be a reasonable choicefor imaging malignant melanoma.

An optical transillumination imaging device, Nevoscope isused for imaging skin lesions. It was introduced by Dhawan3 fornoninvasive diagnosis of malignant melanoma and other skin can-cers. In its transillumination mode, light is directed by a channel 45◦with respect to the normal of skin and enters skin though a ring lightsource. The reemerged light gets captured by the CCD camera andforms the transillumination image. This image contains the informa-tion of underlying optical properties. Within the optical tomographyframework, it is possible to retrieve two key signatures of malignantmelanoma, the spatial distribution of melanin and blood from opti-cal reflectance measurements.

16.2 OPTICAL IMAGING METHODS

During the last several decades, various optical imaging modalitieshave been developed.10 These methods can be divided into five cat-egories: surface imaging, fluorescence imaging, optical coherenceimaging (OCT), optical spectroscope and optical tomography (OT).



16.2.1 Surface Imaging

Surface imaging methods provide a specific light source to illumi-nate the surface of the skin and skin lesion. Surface reflectance basedmeasurements are then stored as a high-resolution image using ahigh-resolution CCD digital camera via a magnification optical lens.For example, “Dermoscope” has been used for surface-reflectancebased imaging of skin lesion.2,3 A better accuracy for detectingmelanoma can be obtained through the use of the EpiluminescenceLight Microscopy (ELM) imaging method, where the reflection ofthe surface light is reduced by either an oil-glass interface on theskin or cross-polarization of the surface and reflected light to cancelthe surface reflection.2 Dermoscopy utilizes surface reflectance dom-inant illumination methods that are to render the skin translucentand thereby allowing for the visualization of subsurface structuresand colors. These subsurface structures and colors in combinationwith their location and distribution (pattern) have been shown toimprove a clinician’s ability to detect early melanoma and basalcell carcinoma. Dermoscopy can be performed utilizing polarizedor nonpolarized light. Cross-polarization method for epilumines-cence uses linear polarizers in the incident light and a viewing lensto cancel the light that is reflected from the skin. Since most of thereflected light from the skin surface has, for the most part, the samepolarization angle as the incident light, cross-polarization blocksmost of the surface reflected light and only the light that is diffusedbelow the skin surface is visualized.

A novel optical imaging system, the Nevoscope that usestransillumination as well as a combination of surface illuminationand transillumination, has been developed by Dhawan to provideimages with significant information about skin-lesion subsurfacepigmentation architecture.3,4 Nevoscope consists of a digital CCDcamera hooked up to a zoom lens and a customized optical assemblyto obtain surface and/or transillumination-based images of the skinand skin lesion. In the Nevoscope transillumination method, lightis transmitted into the skin area surrounding the lesion at 45◦ angle.A virtual light source is thus created a few millimeters below theskin surface for uniform transillumination of a small area of the skin



containing the skin lesion. In such a side-transillumination method,no surface light is used. An annular transillumination ring providesfiber optics directed light to illuminate the region of interest uni-formly. The skin lesion is positioned inside the transillumination ringthrough the opening providing a direct field of view to the digitalcamera through a zoom lens assembly. The light from the illuminatorring that is not reflected back due to a mismatch in refractive indices,enters into the skin and goes through multiple internal reflectionsand scattering. This light eventually gets diffused across the layers ofthe skin and back-scattered diffused light photons emerge from theskin to form a transilluminated image of the skin and skin lesion. Forsurface illumination, additional fiber optics directed point sourcesdistributed around the internal wall of the Nevoscope are providedto reflect light through the surface of the skin lesion. The surfacelight intensity can be adjusted and is polarized. Another polarizinglens (cross-polarized by 90◦) is used with cross-polarization methodfor the imaging of skin lesion. The Nevoscope by virtue of its designprovides three different ways of imaging a skin lesions.

Besides using the pigment and color information from surfacereflectance information, optical models to relate the reflectance mea-surements to underlying optical properties have been developed.For instance, Claridge etc.15 use a Kubelka-Munk model to simu-late the formation of color images of melanoma. They eventuallyare able to recover blood and melanin distribution in various skinlayers. Kubelka-Munk model is basically a one-dimensional theory.Using this model and multispectral imaging, the usually ill-posed,underdetermined inverse problem occurred in optical tomographyis dealt for image reconstruction.

16.2.2 Fluorescence Imaging

Fluorescence imaging uses ultraviolet light to excite fluorophoresand collects emitted light at a higher wavelength. Fluorophoresinclude endogenous and exogenous fluorophores. The former refersto natural fluorophores intrinsic inside the skin such amino acidand structural protein. These fluorophores are randomly distributed



in skin. So direct reconstructing their distributions is meaninglessif not possible. The latter usually refers to some smart polymernanoparticles targeting at specific molecules like hemoglobin.

Due to the disparity of the metabolism states, some kinds ofexogenous fluorophores have unique distributions in malignantmelanoma compared to those in normal tissue, which may suggestthe presence of a cancer. Fluoresence imaging has a similar mech-anism as single particle emission computed tomography (SPECT).The purpose is to recover source distribution given a boundary mea-surement. Aspiring results are obtained by several groups.16,17

16.2.3 Optical Coherence Tomography

The relatively new modality OCT makes use of coherent proper-ties of light.18 In OCT system, light with a low coherence lengthis divided into two parts. One serves as reference while the otheris directed into tissue. When light travels in the tissue, it encoun-ters interface with different refractive index and part of the light isreflected. This reflectance is mixed with the reference subsequently.Once the difference of optical path length between reference lightand reflected light is less than the coherence length, coherenceoccurs. By observing the coherence pattern and changing the opticalpath length of reference light with a mirror, a cross section of skincan be rendered.

With a sufficient low coherence length, the resolution of OCTmay reach a magnitude of micrometer hence can disclose subtlechanges in cancer tissue at a cellular level. OCT recovers the struc-ture of interrogated tissue in a mechanism analogous to ultrasonicimaging. The latter modality sends sound wave into tissue and thesound wave reflects when encountering impedance varied inter-face. However, the resolution of OCT is much higher than ultrasonicimaging.

16.2.4 Optical Spectroscope

Another optical imaging modality is an optical spectroscope. Itsapplication dates back several decades when spectroscope was



first used to evaluate blood oxygenation. The spectroscope samplesinvestigated tissue using reasonable distanced detector and source.It measures re-emerging light at multiple wavelengths usually rang-ing from visible to infrared spectra. The measured absorption spec-trum is a direct reflectance of what happens within the samplingvolume.

In skin, the absorption spectrum is an overlap effect of severalchromophores. However, to recognize the fraction of these chro-mophores is difficult. The common assumption is that these chro-mophores are homogenously distributed in the sampling volume.In fact, it is not true for skin, a complex and heterogeneous layeredtissue. Even a further assumption that chromophores in each skinlayer are homogenous is against reality. Having said that, the absorp-tion spectrum is indeed related to the underlying composition of tis-sue. It may therefore provide significant signatures to some diseasesby itself.

Malignant melanoma contains more melanin and blood thannormal tissue hence more light is absorbed and the absorption variesin terms of characteristics of melanin and hemoglobin. Study showsthat the absorption spectrum of malignant melanoma differs signif-icantly from that of normal tissue. Features extracted from the spec-trum can be subsequently used to identify malignant melanoma.Tomatis etc.19 used artificial neural network as the classifier. Theyreported a sensitivity of 80.4% and a specificity of 75.6% in 1391 caseswhere 184 are melanoma. A study based on multivariate discrimi-nant analysis20 also shows promising results.

16.2.5 Optical Tomography

When discussing about optical tomography, we refer to the opticalimaging system aimed to reconstruct inside spatial-resolved opticalproperties by multiple source detector channels. Though some opti-cal tomography systems borrow similar ideas from other well estab-lished tomography systems like CT, the fundamental design conceptof optical devices may deviate greatly from these well establishedones. The characteristic of the system is related to the configuration



of source detector channels. It is also related to the tissue since theoptical properties of the tissue affects light transportation and con-sequently affects the spatial sensitivity of the system. The reason isthat, unlike the straight line trajectory of CT, light becomes diffusein tissue. Both system configuration and tissue optical propertieschange the trajectories of light photons.

The reconstruction of optical properties differs from most wellestablished modalities in that it usually does not have access to thestandard algorithms such as filtered back-propagation algorithm.With the under-determined and ill-posed nature of the inverse prob-lem, an optical reconstruction algorithm is to seek an optimal solu-tion from numerous possible ones meeting some priori. The opticalreconstruction algorithms are far away from perfect and it is stilla hot research topic in the optical communities. As about Nevo-scope, because of its reflectance geometry, the measurements ofsource detector channels are highly dependent. That is, effectivemeasurements are greatly reduced. In other words, there are fewerconstraints on parameter space under this scenario, which makesthe inverse problem even harder to solve. The shape based multi-constrants algorithm presented in this chapter has several advan-tages over the conventional voxel by voxel approaches. It has fewerparameters and more constraints. And it has a global method tosearch the parameter space. The algorithm will be illustrated anddiscussed in rest of the chapter.

16.3 METHODOLOGY: SHAPE-BASED OPTICALRECONSTRUCTION

Figure 1 shows a flow chart of the proposed reconstruction methodin terms of Nevoscope transillumination images. Firstly, the overallgoal is to minimize difference between the real measurement andthe predicted measurement. This minimization problem is solvedby genetic algorithms to offer a global searching. Secondly, a lin-earized forward model is adopted and evaluated by Monte Carlosimulation in terms of typical optical properties of normal skin.Thirdly, the malignant melanoma is represented by shapes of its



Normal Skin Image

Skin Lesion Image

realM∆

Jacobian Matrix

Normal Skin Optical Properties

calM∆Genetic

Algorithm

Predicted Shape

Sampling Function

Monte Carlo Simulation

Fig. 1. Flow chart of proposed method.

melanin part and blood part. These parameters are lumped intogenetic algorithms.

16.3.1 Forward Modeling

To develop an optical tomographic system, a forward model isrequired to relate the measurement to the optical properties of theinvestigated tissue. Regardless of what kind of imaging geometrywe are using, an optical system may be described as:

M = F(x), (1)

where M is the measurement and F is a forward model. x is a distri-bution of unknown optical properties.

Given a reasonable initial guess x0 of the background opticalproperties, we may expand Eq. (1) into:

M = F(x0) + F′(x0)(x − x0) + 12

F′′(x0)(x − x0) + · · · , (2)

where F′ and F′′ are first order and second order Frechet derivativesrespectively.



Let �Mcal = M−F(x0) and �x = x−x0, Eq. (2) may be rearrangedas:

�Mcal = F′�x + 12

F′′�x + · · · . (3)

The discrete form of Eq. (3) turns to be:

� �Mcal = J��x + 12

H��x + · · · . (4)

Here, J is the Jacobian matrix and H is the Hessian Matrix. � �Mcal

is the measurement vector and ��x is the vector that gives the varia-tions from the background �x0.

Neglecting higher order terms in Eq. (4), we get a simplifiedlinear system:

� �Mcal = J��x. (5)

The formulation in terms of Eq. (5) leads to linear optical tomog-raphy which is also known as “difference imaging.” That is, twomeasurements are taken. One is for background tissue (that is, x0)and one is for abnormal tissue (that is, unknown x). Their differenceis then fed to the reconstruction algorithm to obtain the optical prop-erties. In this study, the linear approach is adopted for Nevoscopeand the Jacobian matrix is extracted by Monte Carlo simulation interms of a seven layered optical skin model.

16.3.2 Shape Representation of Skin-Lesions

There are a variety of shape representation methods adopted byauthors working on optical tomography. In their 2D shape-basedreconstruction, Kilmer et al.5 used a B-spline curve to describethe 2D shape. Babaeizadeh6 used tensor-product B-spline to cre-ate 3D heart shape when studying electrical impedance tomogra-phy. The B-spline curve can sufficiently describe complex shapegiven a few control points. Kilmer7 later used an ellipsoid in their3D study. To fully determine the ellipsoid, they need three param-eters to represent the centroid, three parameters to represent thelengths of the semiaxes and three parameters to represent the direc-tion of the ellipsoid. The advantage of their approach is that only



nine parameters are required, which is a quite small number ofparameters in 3D geometry. However, this simplification makes itimpossible to describe more complex 3D shapes. Zacharopoulos8

uses the spherical harmonic representation in their 3D study. Intheir study, Zacharopoulos shows that the eleven degree sphericalharmonic representation can describe the shape of neonatal headfairly well.

A melanoma basically is a 3D object. Therefore, the 3D descrip-tion like spherical harmonic is appropriate. In addition, we canobserve malignant melanoma one step further. Malignant melanomais a result of uncontrolled replication of melasome cells sitting inthe basal layer of epidermis. The shape of the melanoma is hencebounded by the epidermis layer. All we need to describe the lesionis therefore reduced to 2D surfaces in the 3D domain.

In order to represent malignant melanoma with 2D surfaces,we break it into two parts: the melanin part and the blood part.The melanin part is a 3D region bounded by a single surface andthe epidermis layer. Within the region, the optical properties areconstant and the only absorber is melanin. Furthermore, a secondsurface which sits below the first surface is used to represent theblood part. The region bounded by the first surface and the secondsurface is blood only. This model mimics the deteriorated lesion andits x-z intersection is shown in Fig. 2.

Let the first surface be represented as f1(x, y) which correspondsto the depth of lesion from the epidermis layer at the position (x, y).The idea is to represent the continuous surface with limited parame-ters. Firstly, we put a N×N rectangular grid to lie over the epidermallayer. Secondly, the function f1(x, y) is sampled to N ×N discrete val-ues fd1(X, Y). Here, (x, y) is continuous and (X, Y) is N × N numbersof discrete sampling positions. Third, the discrete values are inter-polated by the cubic tensor-product B-spline which satisfies the fol-lowing condition:

fd1(X, Y) =N∑

i=1

N∑j=1

c1(i, j)β3(X − i, Y − j). (6)



Epidermis

Dermis

Fat

Melanin

Blood

f2(x,y)f1(x,y)

Stratum Corneum

Fig. 2. Shape representation of malignant melanoma.

The original function f1(x, y) can then be approximated by:

fB1(x, y) =N∑

i=1

N∑j=1

c1(i, j)β3(x − i, y − j), (7)

where

β3(x − i, y − j) = β3(x − i)gβ3(y − j), (8)

is the tensor product of one-dimensional cubic B-spline basis β3(x−i)and β3(y − j). And c1(i, j) is B-spline coefficient.

Similarly, the second surface can be defined by N × N discretevalues fd1(X, Y) or, equivalently, zd(X, Y) = fd2(X, Y)−fd1(X, Y) whichis the thickness of blood region between the first surface and thesecond surface.

16.3.3 Reconstruction Algorithm

To reconstruct the surfaces and piecewise constant optical proper-ties, the continuous surface representation should be incorporatedinto the forward photon transportation model. In our study, the lin-earized forward model (Eq. 5) is kept intact and the continuous rep-resentation is sampled into the discrete vector of unknowns ��x.



That is:

� �M = J · ��x = J · S(fd1(X, Y), mp1, zd(X, Y), bp2), (9)

where S( · ) is the sampling function which converts the continuousshape representation to the voxel-based optical properties in for-ward model, mp1 is the fraction of melanin and bp2 is fraction ofblood.

The inverse problem can therefore be formulated as minimizingthe objective function:

Fobj = 12‖� �Mreal − J · ��x‖2

= 12‖� �Mreal − J · S(fd1(X, Y), mp1, zd(X, Y), bp2)‖2. (10)

Now, the unknowns of this inverse problem are reduced tofd1(x, y), zd(X, Y), mp1 and bp2.

The multispectral shape reconstruction using N wavelengthscan be formulated as a multiobjective optimization problem andits objective function is given as:

Fobj = α1 · Fλ1

obj + α2 · Fλ2

obj + · · · + αN · FλN

obj, (11)

where, {α1, α2, . . . , αN} is a set of coefficients to balance the contribu-tions from different single-wavelength objective functions.

In our study, we use a genetic algorithm to solve the optimiza-tion problem8 for the following reasons. First, in genetic algorithm,the gradient need not be evaluated which simplifies the computationand provides reliability. Second, genetic algorithm is one of the mostpopular methods used to seek global minimal. Third, among theglobal optimization techniques, genetic algorithm provides a rea-sonable convergence rate due to its implicit parallel computation.Its elements include the fitness function, coding the chromosome,reproduction and crossover (breeding) and mutation.

As to the optimization problem occurred in the shape-basedreconstruction, the objective function (Eq. 11) is selected as the fitness



function. Its parameters is coded into chromosome like:{

fd1(X1, Y1) − fd1(X2, Y2) − · · · − fd1(XN×N , YN×N) − mpl− zd(X1, Y1) − zd(X2, Y2) − · · · − zd(XN×N , YN×N) − bp2

}. (12)

Reproduction is governed by the “Roulette wheel” rule andcrossover and mutation events occur according to some predefinedprobabilities.

Before working on the optimization algorithm, we may addsome reasonable constraints to the parameters. Firstly, the regionof support of the lesion in x-y plane is readily available in terms ofits surface shape. This further reduces the number of parameters torepresent a surface from N × N to a smaller set. As a consequence,the optimization algorithm has a faster convergence rate. Secondly,the blood region is typically thin layers within a few hundreds ofmicrometers which put a constraint on zd(X, Y). Thirdly, the frac-tions of melanin and blood are not free parameters. They can alsobe bounded according to the appearance of melanoma and the clin-ical experience. Lastly, multispectral imaging provides implicit con-straints. Given the distinct absorption spectra of blood and melanin,a reasonable solution must satisfy the measurements of all involvedwavelengths.

16.3.4 Phantom and Error Evaluation

To validate the shape-based multispectral algorithm, a double-surface phantom is created to represent malignant melanoma. Thefirst and second surfaces are described by a mixed Gaussian functionwhich are given as:

f1(x, y) = MAX(peak1 · G(x, y, µ1a, µ2a, σa), peak2 · G(x, y, µ1b, µ2b, σb))(13)

f2(x, y) = MAX(peak3 · G(x, y, µ1a, µ2a, σa), peak4 · G(x, y, µ1b, µ2b, σb))

where the Gaussian function is:

G(x, y, µ1, µ2, σ) = 12πσ2 exp

(− (x − µ1)2 + (y − µ2)2

2σ2

). (14)



And the parameters used in Eq. (13) are:

µ1a = 0.0375 × 2 cm µ2a = −0.0375 × 3 cmµ1b = −0.0375 × 2 cm µ2b = 0.0375 × 3 cmσa = 0.0375 × 2.5 cm σb = 0.0375 × 2 cmpeak1 = 100 × 6 µm peak2 = 100 × 4 µmpeak3 = 100 × 8 µm peak4 = 100 × 6 µm

(15)

The fraction of melanin is set to 5% between the epidermal layerand the first surface f1(x, y). And the fraction of blood is set to 20%between the first surface f1(x, y) and the second surface f2(x, y). Thismodel has sufficient variation in order to verify the reliability of thereconstruction algorithm. Figure 3(A) displays the 3D view of thismodel.

To further evaluate the reconstruction result, we introduce thevolume deviations Volerr1 and Volerr2. They are defined as:

Volerr1 = |vol1c − vol1m|vol1m

. (16)

Here, vol1c is the calculated volume bounded by the first surface andvol1m is the corresponding volume from the model:

Volerr2 = |vol2c − vol2m|vol2m

. (17)

Here, vol2c is the calculated volume bounded by the first and secondsurfaces and vol2m is the corresponding volume from the model.

16.4 RESULTS AND DISCUSSIONS

We select 580 nm and 800 nm to validate the reconstruction algo-rithm since at these two wavelengths the absorption of oxy- anddeoxyhemoglobin is equivalent. In addition, absorption of melaninand blood at the two wavelengths has considerable difference whichprovides excellent constraints to the solution. First of all, the dou-ble surface continuous model is sampled and two “real” measure-ments �M580 and �M800 are calculated by Monte Carlo simulationat 580 nm and 800 nm respectively. Next, a nine by nine rectangu-lar grid is overlapped on epidermal layer. The region of support of



Fig. 3. Reconstruction results: (A) Double-surface model (B–E) Reconstructedsurfaces with different constraints.



the lesion is counted as 10 discrete control points. As a result, thechromosome contains 22 genes and is coded as:{

fd1(X1, Y1) − fd1(X2, Y2) − · · · − fd1(X10, Y10) − mpl− z(X1, Y1) − z(X2, Y2) − · · · − z(X10, Y10) − bp2

}. (18)

The fitness function of the genetic algorithm is given as:

Fobj = α1F580obj + α2F800

obj , (19)

where in our simulation, α1 is 0.3 and α2 is 1.Four simulations with different constraints are implemented in

real number represented genetic algorithm and results are summa-rized in Table 1. The recovered surfaces are displayed in Figs. 3(B–E).In each case, the left is the first surface and the right is the secondsurface. There is no constraint for the first surface while the thicknessbetween the first surface and the second surface is set to be 300 µmto represent a thin layer of blood net. In addition, the deformationprocess of the surfaces during optimization is shown in Fig. 4.

In terms of Table 1, the constraints have significant impacts onthe reconstructed surfaces. Except the loosest constrained case (E), allcases present reasonable reconstructions which are consistent withthe model. Moreover, the reconstructed first surface has a smallervolume error than the second surface. There are several reasons toexplain the larger volume error of the second surface. Firstly, theabsorption coefficient of blood is smaller than that of melanin. As aresult, the change in blood region has less contribution to the fitnessfunction. Secondly, since a reflectance geometry is adopted in Nevo-scope, the sensitivity decreases at deeper layers. This also influences

Table 1. Summary of Reconstruction Results

Melanin Blood Recovered RecoveredBounds Bounds Melanin Blood Volerr 1 Volerr 2

Case (%) (%) (%) (%) (%) (%)

(b) 5–5 20–20 5 20 2.68 16.58(c) 4.5–5.5 10–30 5.10 18.33 3.81 20.89(d) 4–6 10–30 4.64 18.97 5.18 24.71(e) 3–7 10–30 3.37 10.08 44.09 63.50



Fig. 4. Deformation process during optimization (From left to right and from topto bottom): (A) the first surface (B) the second surface.

the accurate reconstruction of the second surface. Thirdly, becausethe two surfaces are attached together, the error resulting from thefirst surface would inevitably propagate to the second surface. Inthe worst case (E), a large error has been observed. The fraction ofmelanin and blood is underestimated, which associates with over-estimated volumes. It is therefore still a reasonable result for theoptimization problem.



16.5 CONCLUSION

A shape based reconstruction method using a genetic algorithm hasbeen presented in this chapter. Though the reconstruction algorithmhas been described for optical images of skin lesions for the detectionof malignant melanoma, the framework of the shape based imagereconstruction can be applied to other optical imaging applications.


This research was partially funded by grants from George A Ohl JrTrust Foundation and Gustarus and Louise Pfeiffer Research Foun-dation. The work presented in this report is also a part of the doctoraldissertation work of Song Wang with Dr Atam Dhawan as his Ph.D.advisor.

References

1. Abramovits W, Stevenson LC, Changing paradigms in dermatology:New ways to examine the skin using noninvasive imaging methods,Clinics in Dermatol 21: 353–358, 2003.

2. Bashkatov AN, Genina EA, et al., Optical properties of human skin,subcutaneous and mucous tissues in the wavelength range from 400to 2 000 nm, Journal of Physics D: Applied Physics 38: 2543–2555, 2005.

3. Dhawan AP, Gordon R, Rangayyan RM, Nevoscopy: Three-dimensional computed tomography for nevi and melanoma by trans-illumination, IEEE Trans on Medical Imaging MI-3(2): 54–61, 1984.


5. Misha E Kilmer, Eric L Miller, David Boas, et al., A shape-based recon-struction technique for DPDW data, Optics Express 7(13): 481–491, 2000.

6. Saeed Babaeizadeh, Dana H Brooks, David Isaacson, A deformable-radius B-spline method for shape-based inverse problems, as appliedto electrical impedance tomography, acoustics, speech, and signal pro-cessing 2005 (ICASSP ’05).

7. Misha E Kilmer, Eric L Miller, et al., Three-dimensional shape-basedimaging of absorption perturbation for diffuse optical tomography,Applied Optics 42(16): 3129–3144, 2003.



8. Zacharopoulos A, Arridge S, Dorn O, et al., 3D shape reconstruction inoptical tomography using spherical harmonics and BEM, Progress inElectromagnetics Research Symposium 2006.

9. Chris Houck, Jeff Joines, Mike Kay, A Genetic Algorithm for FunctionOptimization:AMatlab Implementation, NCSU-IE TR, pp. 95–09, 1995.

10. Balch CM, et al., Prognostic factors analysis of 17,600 melanomapatients: Validation of the American joint committee on cancermelanoma staging system, Journal of Clinical Oncology 19(16): 3622–3634, 2001.

11. Marchesini R, et al., Optical imaging and automated melanoma detec-tion: Questions and answers, Melanoma Research 12: 279–286, 2002.

12. Ganster H, Pinz A, Kittler H, et al., Computer aided recognition ofpigmented skin lesions, Melanoma Research 7: 1997.

13. Seidenari S, et al., Digital video-microscopy and image analysis withautomatic classification for detection of thin melanomas, MelanomaResearch 9(2): 163–171, 1999.

14. Menzies S, Crook B, McCarthy W, et al., Automated instrumentationand diagnosis of invasive melanoma, Melanoma Research 7: 1997.

15. Claridge E, Cotton S, et al., From color to tissue histology: Physics-based interpretation of images of pigmented skin lesion, Medical ImageAnalysis, 489–502, 2003.

16. Claridge E, Preece SJ, An inverse method for recovery of tissue param-eters from colour images, Information Processing in Medical Imaging,Springer, Berlin, LNCS2732, pp. 306–317, 2003.

17. Churmakov DY, et al., Analysis of skin tissues spatial fluorescence dis-tribution by the Monte Carlo simulation, J Phys D: Applied Phys 36:1722–1728, 2003.

18. Chang J, Graber HL, Barbour RL, Imaging of fluorescence in highlyscattering media, IEEE Trans on Biomedical Engineering 44(9): 810–822,1997.

19. Fercher AF, et al., Optical coherence tomography — Principles andapplications, Rep Prog Phys 66: 239–303, 2003.

20. Tomatis S, et al., Automated melanoma detection: Multispectral imag-ing and neural network approach for classification, Med Phys 30(2):212–221, 2003.

21. Tomatis S, Bartoli C, et al., Spectrophotometric imaging of subcuta-neous pigmented lesion: Discriminant analysis, optical properties andhistological characteristics, J Photochem Photobiol 42: 32–39, 1998.

22. Young AR, Chromophores in human skin, Physics in Medicine andBiology 42: 789–802, 1997.


CHAPTER 17

Multimodality Image Registrationand Fusion

Pat Zanzonico

Imaging has long been a vital component of clinical medicine and, morerecently, of biomedical research in small animals. In addition, image reg-istration and fusion have become increasingly important components ofboth clinical and laboratory (i.e. small-animal) imaging and have leadto the development of a variety of pertinent software and hardware tools,including multimodality, e.g. PET-CT, devices which “automatically” pro-vide registered and fused three-dimensional (3D) image sets. This chapteris a brief, largely non-mathematical review of the basics of image regis-tration and fusion and of software and hardware approaches to 3D imagealignment, including mutual information algorithms and multimodalitydevices.

17.1 INTRODUCTION

Since the discovery of X-rays, imaging has been a vital compo-nent of clinical medicine. Increasingly, in vivo imaging of smalllaboratory animals, i.e. mice and rats, has emerged as an impor-tant component of basic biomedical research. Historically, clinicaland laboratory imaging modalities have often been divided intotwo general categories, structural (or anatomical) and functional (orphysiological). Anatomical modalities, i.e. depicting primarily mor-phology, include X-rays (plain radiography), CT (computed tomog-raphy), MRI (magnetic resonance imaging), and US (ultrasound).Functional modalities, i.e. depicting primarily information related

413


414 Pat Zanzonico

to underlying metabolism, include (planar) scintigraphy, SPECT(single-photon emission computed tomography), PET (positronemission tomography), MRS (magnetic resonance spectroscopy),and fMRI (functional magnetic resonance imaging). The functionalmodalities form the basis of the rapidly advancing field of “molecu-lar imaging,” defined as the direct or indirect noninvasive monitor-ing and recording of the spatial and temporal distribution of in vivomolecular, genetic, and/or cellular processes for biochemical, bio-logical, diagnostic, or therapeutic applications.1

Since information derived from multiple images is often comple-mentary, e.g. localizing the site of an apparently abnormal metabolicprocess to a pathologic structure such as a tumor, integration ofthis information may be helpful and even critical. In addition toanatomic localization of “signal” foci, image registration and fusionprovide: intra- as well as intermodality corroboration of diverseimages; more accurate and more certain diagnostic and treatment-monitoring information; image guidance of external-beam radia-tion therapy; and potentially, more reliable internal radionuclidedosimetry, e.g. in the form of radionuclide image-derived “isodose”contours superimposed on images of the pertinent anatomy. Theproblem, however, is that differences in image size and dynamicrange, voxel dimensions and depth, image orientation, subject posi-tion and posture, and information quality and quantity make it dif-ficult to unambiguously co-locate areas of interest in multiple imagesets. The objective of image registration and fusion, therefore, is (a) toappropriately modify the format, size, position, and even shape ofone or both image sets to provide a point-to-point correspondencebetween images and (b) to provide a practical integrated displayof the images thus aligned. This process entails spatial registrationof the respective images in a common coordinate system based onoptimization of some “goodness-of-alignment,” or “similarity,” cri-terion (or metric). This chapter is a brief, largely nonmathematicalreview of the basics of image registration and fusion and of soft-ware and hardware approaches to 3D image alignment and presentsillustrative examples of registered and fused multimodality imagesin both clinical and laboratory settings.


Multimodality Image Registration and Fusion 415

17.2 BACKGROUND

The image registration and fusion process2−5 is illustrated diagram-matically and in general terms in Fig. 1. The first step is reformattingof one image set (the “floating,” or secondary, image) to match thatof the other image set (the reference, or primary, image). Alter-natively, both image sets may be transformed to a new, commonimage format. Three-dimensional (3D), or tomographic, image setsare characterized by: the dimensions (e.g. in mm), i.e. the length

Fig. 1. The image registration and fusion process. See text for details.


416 Pat Zanzonico

(�X), width (�Y), and depth (�Z), of each voxel; the image matrix,X ×Y× Z = number of rows, X × number of columns, Y× numberof tomographic images (or “slices”), Z; and the image depth (e.g. inbytes), which defines the dynamic range of signal display-able ineach voxel (e.g. a word-mode, i.e. one word- or two byte-“deep,”PET image can display up to 216 = 65 536 signal levels for 16-bitwords). The foregoing image parameters are provided in the image“header,” a block of data which may either be in a stand-alone textfile associated with the image file or incorporated into the image fileitself. Among the image sets to be registered, either the finer matrixis reformatted to the coarser matrix by combining of voxels or thecoarser matrix is reformatted to the finer matrix by interpolationof voxels. One of the resulting 3D image sets is then magnified orminified to yield primary and secondary images with equal voxeldimensions. Finally, the “deeper” image is rescaled to match thedepth of the “shallower” matrix. Usually, the higher spatial resolu-tion and finer matrix structural (e.g. CT) image is the primary imageand the functional (e.g. PET) image the secondary image.

The second step in image registration is the actual transformation[translation, rotation, and/or deformation (warping)] of the refor-matted secondary image set to spatially align it, in three dimensions,with the primary image set.

The third and fourth steps are, respectively, the evaluation ofthe accuracy of the registration of the primary and transformed sec-ondary images and adjustment, iteratively, of the secondary imagetransformation until the registration (i.e. the goodness-of-alignmentmetric) is optimized.

The fifth and final step is image fusion, the integrated display ofthe registered images.

17.3 PROCEDURES AND METHODS

17.3.1 “Software” versus “Hardware” Approachesto Image Registration

In both clinical and laboratory settings, there are two practi-cal approaches to image registration and fusion, “software” and



“hardware” approaches. In the software approach, images areacquired on separate devices, imported into a common image-processing computer platform, and registered and fused usingthe appropriate software. In the hardware approach, images areacquired on a single, multimodality device and transparently regis-tered and fused with the manufacturer’s integrated software. Bothapproaches are dependent on software sufficiently robust to recog-nize and import diverse image formats. The availability of industry-wide standard formats, such as the ACR-NEMA DICOM standardi.e. theAmerican College of Radiology (ACR) and National ElectricalManufacturers Association (NEMA) for Digital Imaging and Com-munications in Medicine (DICOM) standard,6−9 is therefore critical.

17.3.2 Software Approaches

17.3.2.1 Rigid versus non-rigid transformations

Software-based transformations of the secondary image set to spa-tially align it with the primary image set are commonly character-ized as either “rigid” or “nonrigid”.2−5 In a rigid transformation,the secondary image is only translated and/or rotated with respectto the primary image. However, the Euclidean distance betweenany two points (i.e. voxels) within an individual image set remainsconstant. In nonrigid, or deformable, transformations (commonlyknown as “warping”), selected subvolumes within the image setmay be expanded or contracted and/or their shapes altered. Trans-lations and/or rotations may be performed as well. Such warping istherefore distinct from any magnification or minification performedin the reformatting step, where distances between points all changeby the same relative amount. Unlike rigid transformations, whichmay be either manual or automated, non-rigid transformations aregenerally automated.

17.3.2.2 Feature- and intensity-based approaches

Registration transformations are often based on alignment of specificlandmarks visible in the image sets; this is sometimes characterized


418 Pat Zanzonico

as the “feature-based” approach.2−5 Such landmarks may be eitherintrinsic, i.e. one or more well-defined anatomic structure(s) or thebody contour (i.e. surface outline), or extrinsic, i.e. one or more fidu-cial markers placed in or around the subject. Feature-based registra-tion generally requires some sort of preprocessing “segmentation” ofthe image sets being aligned, that is, identification of the correspond-ing features (e.g. fiduciary markers) of the image sets. Feature-basedimage registration algorithms may be automated by minimizationof the difference(s) in position of the pertinent feature(s) betweenthe image sets being aligned.

Other registration algorithms are based on analysis of voxelintensities (e.g. counts in a PET or SPECT image) and are character-ized as “intensity-based” approaches.2−5 These include: alignmentof the respective “centers of mass” (e.g. counts) and orientation (i.e.principal axes) calculated for each image set; minimization of abso-lute or sum-of-square voxel intensity differences between the imagesets; cross-correlation (i.e. maximizing the voxel intensity correla-tion between the image sets); minimization of variance (i.e. match-ing of identifiable homogeneous regions in the respective imagessets); and matching of voxel intensity histograms (discussed fur-ther in the Results and Findings section).2 Such intensity-basedapproaches implicitly assume that the voxel intensities in the imagesbeing aligned represent the same, positively correlated parameters(e.g. counts) and thus are directly applicable only to intramodalityimage registration.

17.3.2.3 Mutual information

A relatively new but already widely used automated registra-tion algorithm is based on the statistical concept of mutualinformation,3,10 also known as transinformation or relative entropy.The mutual information of two random variables A and B is aquantity that measures the statistical dependence of the two vari-ables, that is, the amount of information that one variable containsabout the other. Mutual information measures the information aboutA that is shared by B. If A and B are independent, then A contains noinformation about B and vice versa and their mutual information is



therefore zero. Conversely, if A and B are identical, then all informa-tion conveyed by A is shared with B and their mutual informationis maximized. Accurate spatial registration of two such image setsthus results in the maximization of their mutual information andvice versa.

The concepts of entropy and mutual information are developedmore formally in the following. Given “events” (e.g. gray-scale val-ues) e1, e2, . . . , en with probabilities (i.e. frequencies of occurrence)p1, p2, . . . , pn in an image set, the entropy (specifically, the so-called“Shannon entropy”) H is defined as follows3:

H ≡n∑

1

pi log1pi

(1a)

= −n∑

1

pi log pi. (1b)

The term, log 1pi

, indicates that the amount of information providedby an event is inversely related to the probability (i.e. frequency)of that event: the less frequent an event, the more significant is itsoccurrence. The information per event is thus weighted by the fre-quency of its occurrence. The uniform “background” (eBG) occu-pies a large portion of a CT image (i.e. pBG is large), for example,and therefore contributes relatively little information (i.e. log 1

PBGis

small) — and would not contribute substantially to accurate align-ment with an MR image. The Shannon entropy is also a measure ofthe uncertainty of an event. When all events (e.g. all gray scale valuesin an image) are equally likely to occur (as in an highly heteroge-neous image), the entropy is maximal.a When an event or a range ofevents is more likely to occur (as in a uniform image), the entropy isminimal. Additionally, the entropy is a measure of dispersion of animage’s probability distribution (i.e. the probability of a grey scalevalue versus the grey scale values): a highly heterogeneous imagehas a broad dispersion and a high entropy while a uniform imagehas no dispersion and minimal entropy. Entropy thus has several

aThe analogy between signal entropy, used in the context of mutual information, andthermodynamic entropy thus becomes clear.


420 Pat Zanzonico

interpretations: the information content per event (e.g. grey-scalevalue), the uncertainty per event, and the statistical dispersion ofevents in an image.

For two images A and B, the mutual information MI(A,B) maybe defined as followsb,3:

MI(A, B) ≡ H(B) − H(B|A). (2)

H(B) is the Shannon entropy of image B (derived from the proba-bility distribution of its grey-scale value) and H(B|A) is the condi-tional entropy of image B with respect to image A [derived from theconditional probabilities p(b|a), the probability of grey scale value boccurring in image B given that grey scale value a occurs in the cor-responding voxel in image A]. When interpreting entropy in termsof uncertainty, MI(A,B) thus corresponds to the uncertainty in imageB minus the uncertainty in image B when image A is known. Intu-itively, therefore, MI(A,B) — the image-B information in image A —is the amount by which the uncertainty in image B decreases whenimage A is given. Because images A and B can be interchanged,MI(A,B) is also the information image B contains about image A andit is therefore mutual information. Registration thus corresponds tomaximizing mutual information: the amount of information imageshave about each other is maximized when, and only when, theyare aligned. If a subject is imaged by two different modalities, thereis presumably considerable mutual information between the spatialdistribution of the respective signals in the two images sets no matterhow diverse (i.e. unrelated) they may appear to be. For example, thedistribution of fluorine-18-labeled fluorodeoxyglucose (FDG) visu-alized in a PET scan is, at some level, dictated by (i.e. dependent on)the distribution of different tissue types imaged by CT.

17.3.2.4 Goodness-of-alignment metrics

Regardless of the algorithm employed, the evaluation and adjust-ment of the registration requires some metric of its accuracy. It may

bIn information theory, there are actually a number of different definitions of mutualinformation.



be as simple as visual (i.e. qualitative) inspection of the alignedimages and a judgment by the operator that the registration isor is not “acceptable.” A more objective, and ideally quantitative,evaluation of the accuracy of the registration is, of course, pre-ferred. One goodness-of-alignment metric, for example, is the sumof the Euclidean distances between corresponding fiduciary mark-ers (or anatomic landmarks) in the two image sets; the optimumalignment corresponds to the transformation yielding the mini-mum sum of distances. Another similarity metric, as discussedabove, is the mutual information: when the mutual informationbetween the two image sets is maximized, they are optimallyaligned.

17.3.3 Hardware Approaches

The major manufacturers of PET and CT scanners now also marketmultimodality scanners,11−13 combining high performance state-of-the-art PET and CT scanners in a single device. These instrumentsprovide near-perfect registration of images of in vivo function (PET)and anatomy (CT) using a measured, and presumably fixed, rigidtransformation between the image sets. These devices have alreadyhad a major impact on clinical practice, particularly in oncology,and PET-CT devices are currently outselling “PET-only” systems bya two-to-one ratio.14 Although generally encased in a single seam-less housing, the PET and CT gantries in such multimodality devicesare separate; the respective fields of view are separated by a distanceof the order of 1 m and the PET and CT scans are performed sequen-tially (Figs. 2 and 3). In one such device (Gemini, Philips Medical),the PET and CT gantries are actually in separate housings with anadjustable separation (up to ∼1 m) between them; this not only pro-vides access to patients but also may minimize anxiety among claus-trophobic subjects (Fig. 4).

In addition to PET-CT scanners, SPECT-CT scanners are nowcommercially available. The design of SPECT-CT scanners is similarto that of PET-CT scanners in that the SPECT and CT gantries areseparate and the SPECT and CT scans are acquired sequentially, not


422 Pat Zanzonico

Fig. 2. Schematic diagram (side view) of a commercially available clinical PET-CTscanner. From reference11 by permission of the authors. Inset: Photo of the PET-CTscanner in the diagram, the Biograph™ (Siemens-CTI).

simultaneously. In such devices, the separation of the SPECT andCT scanners is more apparent (Fig. 5) because the rotational andother motions of the SPECT detectors effectively precludes encas-ing them in a housing with the CT scanner. Multimodality imagingdevices for small animals (i.e. rodents) — PET-CT, SPECT-CT, andeven SPECT-PET-CT devices — are now commercially available aswell (Fig. 6).

Multimodality devices simplify image registration and fusion —conceptually as well as logistically — by taking advantage of thefixed geometric arrangement between the PET and CT scanners orthe SPECT and CT scanners in such devices. Further, because thetime interval between the sequential scans is short (i.e. a matter ofminutes) and the subject remains in place, it is unlikely that sub-ject geometry will change significantly between the PET or SPECT



Fig. 3. Atypical imaging protocol for a combined PET-CT study: (A) the topogram,or scout CT scan, for positioning; (B) the CT scan; (C) generation of CT-based atten-uation correction factors; (D) the PET scan over the same longitudinal range of thepatient as the CT scan; (E) reconstruction of the attenuation-corrected PET emissiondata; (F) the attenuation-corrected PET images; and (G) display of the final fusedPET-CT images. From Ref. 13 by permission of the authors.

scan and the CT scan. Accordingly, a rigid transformation matrix(i.e. translations and rotations in three dimensions) can be used toalign the PET or SPECT and the CT image sets. This matrix can bemeasured using a “phantom,” i.e. an inanimate object with PET- orSPECT- and CT-visible landmarks arranged in a well-defined geom-etry. The transformation matrix required to align these landmarkscan then be stored and used to automatically register all subsequentmultimodality studies, since the device’s geometry and thereforethis matrix should be fixed.

17.3.4 Image Fusion

Image fusion may be as simple as simultaneous display of imagesin a juxtaposed format. A more common, and more useful, format isan overlay of the registered images, where one image is displayedin one color table and the second image in a different color table.


424 Pat Zanzonico

Fig. 4. Photo of a commercially available clinical PET-CT scanner, theGemini™ (Philips Medical), which allows variable separation of the PET and theCT subsystems.

Typically, the intensities of the respective color tables as well as the“mixture” of the two overlaid images can be adjusted. Adjustment(e.g. with a slider) of the mixture allows the operator to interactivelyvary the overlay so that the designated screen area displays only thefirst image, only the second image, or some weighted combinationof the two images, each in its respective color table.

17.4 RESULTS AND FINDINGS

17.4.1 Software Approaches to Image Registration

17.4.1.1 Feature-based approach: Extrinsic fiduciary markers

Comparative imaging of multiple radiotracers in the same subjectcan be invaluable in elucidating and validating their respectivemechanisms of localization. Comparative imaging of PET trac-ers, particularly in small animals, is problematic, however: such



Fig. 5. Photo of a commercially available clinical SPECT-CT scanner, thePrecedence™ (Philips Medical).

tracers must be administered and imaged separately because simul-taneously imaged positron emitters cannot be separated based onenergy discrimination. In one such study (Fig. 7),15 the intratumoraldistributions of sequentially administered F18-FDG and the hypoxiatracer F18-fluoromisonidazole (FMiso) were compared in rats byregistered R4 microPET™ imaging with positioning of each animalin a custom-fabricated whole-body mold. Custom-manufacturedgermanium-68 rods were reproducibly positioned in the moldas external fiduciary markers. The registered microPET™ imagesunambiguously demonstrate grossly similar though not identicaldistributions of FDG and FMiso in the tumors — a high-activity rimsurrounding a lower-activity core. However, there were subtle butpossibly significant differences in the intratumoral distributions ofFDG and FMiso, and these may not have been discerned withoutcareful image registration.


426 Pat Zanzonico

Fig. 6. Photos of two commercially available laboratory (i.e. rodent) SPECT-CTscanners: (A) the X-SPECT™ (Gamma Medica); (B) the Inveon™ (Siemens Preclin-ical Solutions), which allows the detachment and separate use of the CT and thePET subsystems.

17.4.1.2 Intensity-based approach: Minimizationof intensity differences

As illustrated in Fig. 8 showing sequential PET brain images of thesame patient,2 misalignment of the image sets produces visualiz-able structure in the difference images (the bottom row of Fig. 8(A)),i.e. the voxel-by-voxel intensity differences are not zero. In con-trast, accurate registration yields differences images whose voxel-by-voxel intensity differences are equal to zero within statisticaluncertainty (i.e. “noise”) and therefore an absence of visualizablestructure (bottom row of Fig. 8(B)).

17.4.1.3 Intensity-based approach: Matching of voxelintensity histograms

For two image sets A and B, a 2D joint histogram (also known asthe “feature space”) (Fig. 9)2 can be constructed by plotting, foreach combination of intensity a in image A and intensity b in imageB, the point (a, b) whose darkness or lightness reflects the numberof occurrences of the combination of intensities a and b. Thus, adarker point in the joint histogram indicates a larger number and



Fig. 7. (A) Left panel: An anesthetized tumor-bearing rat in its custom-fabricatedmold (Rapid-Foam™, Soule Medical) for immobilization and reproducible posi-tioning for repeat and/or intermodality imaging studies. Right panel: Three custom-manufactured 68Ge fiduciary markers (10 µCi each, 1 × 10 mm) (Sanders MedicalProducts) reproducibly inserted into the mold and used as extrinsic fiduciary mark-ers for software registration of serial microPET™ images. (B) The appearance ofthe 68Ge markers on overlaid F18-FDG and -FMiso transverse-section microPET™images before and after registration based on the rigid transform consisting oftranslations �x, �y, and �z and rotations �θx, �θy, and �θz. (C) Registered andfused 18F-FDG (gray scale) and FMiso (hot iron) transverse, coronal, and sagit-tal microPET™ images; the sagittal views are through a R3327-AT rat prostatetumor xenograft in the animal’s right hindlimb. Discordant areas of FDG and FMisouptakes are indicated by the white arrows for the R3327-AT tumor and by the yel-low arrows for a FaDu human squamous cell carcinoma tumor xenograft. Bothtumors, 20 mm × 20 mm × 30 mm in size, were significantly hypoxic. From Ref. 15by permission of the authors.

a lighter point a smaller number of occurrences of the combina-tion (a,b). When two identical image sets are aligned (matched), allvoxels coincide and the plot in the voxel intensity histogram is theline of identity (i.e. a = b for all voxels). As one of the image sets


428 Pat Zanzonico

Fig. 8. Intramodality image registration based on minimization of voxel intensitydifferences. (A) Selected brain images of sequential misaligned (i.e. nonregistered)PET studies of the same patient, with the section-by-section difference images inthe bottom row. (B) The same image sets as in (A), now aligned by minimization ofthe voxel-by-voxel intensity differences. From Ref. 2 by permission of the authors.

Fig. 9. Intramodality image registration based on matching of voxel intensity his-tograms. The joint intensity histograms of a transverse-section brain MR imagewith itself when the two image sets are originally matched (i.e. aligned) and whenmisaligned by counterclockwise rotations of 10◦ and 20◦, respectively. See text fordetails. Adapted from Ref. 2 by permission of the authors.



Fig. 10. An intermodality (CT and MR) joint intensity histogram. The feature-less (i.e. uniform) area corresponding to brain tissue in the transverse-section headCT image (left panel), in contrast to the anatomic detail in the corresponding areaof the MR image (middle panel), yields a distinct vertical cluster (arrow) in theCT-MR joint histogram (right panel). Adapted from Ref. 3 by permission of theauthors.

is rotated relative to the other (by 10◦ and then by 20◦), for exam-ple, the joint histogram becomes increasingly blurred (i.e. dispersed)(Fig. 9). Alignment of the images can therefore be achieved by min-imizing the dispersion in the joint intensity histogram. Like otherintensity-based approaches, this approach is most readily adapt-able to similar (i.e. intramodality) images sets but in principle canbe applied to dissimilar (i.e. intermodality) images by appropri-ate mapping of one image intensity scale to the other intensityscale (Fig. 10).3

17.4.1.4 Mutual information

As illustrated in Fig. 113 for registration of a brain MR image withitself, the joint histogram of two images changes as the alignmentof the images changes. When the images are registered, correspond-ing signal foci overlap and the joint histogram will show certainclusters of grey scale values. As images become increasingly mis-aligned (illustrated in Fig. 11 with rotations of 2◦, 5◦, and then10◦ of the brain MRI relative to the original image), signal fociwill increasingly overlap that are not their respective counterpartson the original image. Consequently, the cluster intensities for


430 Pat Zanzonico

Fig. 11. Effect of misregistration on joint intensity histograms and mutual informa-tion (MI) between a transverse-section brain MR image (top row) and itself. Shownare the joint intensity histograms and mutual information (MI) (middle row) whenthe two image sets are originally matched (i.e. aligned) and when misaligned byclockwise rotations of 2◦, 5◦, and 10◦, respectively (bottom row). See text for details.Adapted from Ref. 3 by permission of the authors.

corresponding signal foci (e.g. skull and skull, brain and brainetc.) will decrease and new noncorresponding combinations of greyscale values (e.g. of skull and brain) will appear. The joint his-togram will thus become more dispersed; as described above, min-imization of this dispersion is the basis of certain intensity-basedregistration algorithms. At the same time, the mutual information(MI) (see Eqs. 1 and 2), which is minimized when the two imagesare aligned, will increase. However, unlike other intensity-basedapproaches, no assumptions are made in the MI approach regard-ing the nature of the relationship between image intensities (e.g.a positive or a negative correlation). MI is thus a completely gen-eral goodness-of-alignment metric and can be applied to inter- aswell as intramodality registration and automatically without priorsegmentation.



17.4.2 Hardware Approaches to Image Registration

Multimodality devices simplify image registration and fusion —conceptually as well as logistically — by taking advantage of thefixed geometric arrangement between the PET and CT scanners orthe SPECT and CT scanners in such devices. Further, because thetime interval between the sequential scans is short (i.e. a matter ofminutes), it is unlikely that a subject’s geometry will change signif-icantly between the PET or SPECT scan and the CT scan. Accord-ingly, a rigid transformation matrix (i.e. translations and rotationsin three dimensions) can be used to align the PET or SPECT andthe CT image sets. This matrix can be measured using a “phan-tom,” i.e. an inanimate object with PET- or SPECT- and CT-visiblelandmarks arranged in a well defined geometry. The transforma-tion matrix required to align these landmarks can then be stored andused to automatically register all subsequent multimodality studies,since the devices mechanics and therefore this matrix are presum-ably fixed.

To illustrate the utility of registered and fused multimodalityimaging studies in both clinical and laboratory settings, examplesare presented in Figs. 1216 and 13.

17.5 DISCUSSION AND CONCLUDING REMARKS

In practice, two basic approaches to image registration and fusion,“software” and “hardware” approaches, have been developed.

In the software approach, images are acquired on separatedevices and registered and fused using the appropriate software.Rather robust and user-friendly software for image registration andfusion is now widely available. Software approaches to registra-tion of images acquired on separate devices have been particularlysuccessful in the brain because of the ability to reliably immobilizeand position the head, the pronounced contrast between the bonyskull (an intrinsic landmark) and the brain, and the lack of motionor deformation of internal structures. Outside the brain, however,software registration is more difficult because of the many degrees


432 Pat Zanzonico

Fig. 12. Registered and fused FDG PET and CT scans of a patient with lung cancerand an adrenal gland metastasis. (A) Coronal PET images show typically increasedFDG uptake in the primary lung tumor (single arrow in left panel) and in the metas-tasis in the left adrenal gland (double arrow in left panel) but also in the left sideof the neck (arrow in right panel). (B) Transaxial PET and CT images through thisneck lesion. Reading these images separately or in the juxtaposed format shown,it is difficult to definitively identify the anatomic site (i.e. tumor versus normalstructure) of the focus of activity in the neck. (C) The registered and fused PET-CTimages, using the fused, or overlay, display, unambiguously demonstrate that theFDG activity is located within muscle, a physiological normal variant. Because it isbest visualized using the original color display, an arrow is used to identify the loca-tion of this unusual, but nonpathologic, focus of FDG activity on the fused images.Therefore, the FDG activity in the neck was not previously undetected disease, afinding which would significantly impact the subsequent clinical management ofthe patient. Adapted from Ref. 16 with permission of the authors.



Fig. 13. Registered and fused SPECT-CT images (coronal views) of a mouse witha LAN1 neuroblastoma tumor xenograft in its hindlimb (arrow). The radiotracerwas iodine-125-labeled 3F8, an antibody directed against the ganglioside 2 (GD2)antigen, which is overexpressed on neuroblastomas (including LAN1). The imageswere acquired at two days postinjection with the X-SPECT™ [Fig. 6(A)]. The CTimage shows the tumor as a space-occupying structure along the contour of the ani-mal (left panel). The specific targeting of the radiolabeled 3F8 to the GD2-expressingtumor xenograft is demonstrated by the high-contrast SPECT image (middle panel).The registered and fused PET-CT images, again using the fused, or overlay, display,unambiguously demonstrate that the 3F8 activity is located in the tumor, confirm-ing that the focus of activity represents specific tumor-targeting by this antibodyand not, for example, excreted activity in the urinary bladder or radioactive con-tamination. The images are provided courtesy of Drs Shakeel Modak and Nai-KongCheung, Memorial Sloan-Kettering Cancer Center.

of freedom of the torso and its internal structures when imaged atdifferent times by different devices and with the subject in differ-ent positions. For example, depending on the variable degree offilling of the bladder with urine or the intestines with gas, pelvicand abdominal structures may be significantly displaced from oneimaging study to the next. The registration process may therefore berather time-consuming and labor-intensive.

In the hardware approach, images are acquired on a single,multimodality device and transparently registered and fused. Todate, such multimodality devices have been restricted almost exclu-sively to PET-CT and SPECT-CT scanners. While MRI-CT scan-ners might have little practical advantage, since both MRI andCT are both anatomic imaging modalities, PET-MRI and SPECT-MRI devices would be highly attractive. Combining PET or SPECTand MRI remains problematic, however, because the magnetic


434 Pat Zanzonico

fields proximal to an MRI scanner interfere with the scintillationdetection process in all current generation PET and SPECT scan-ners. Nonetheless, practical PET-MRI scanners are currently underdevelopment.17

Both intra- and intermodality image registration and fusion willno doubt become even more widely used and increasingly importantin both clinical and laboratory settings.

References

1. SNA News, RSNA, SNM Urge Interdisciplinary Cooperation to AdvanceMolecular Imaging, 2005.

2. Hutton BF, Braun M, Thurfjell L, et al., Image registration: An essentialtool for nuclear medicine, Eur J Nucl Med 29: 559–577, 2002.

3. Maintz JBA, Viergever MA, A survey of medical image registration,Med Image Anal 2: 1–36, 1998.

4. Hajnal JV, Hill DLG, Hawkes DJ (eds.) Medical Image Registration, BocaRaton, FL, CRC Press, 2001.

5. Hill DLG, Batchelor PG, Holden M, et al., Medical image registration,Phys Med Biol 46: R1–R45, 2001.

6. American College of Radiology, National Electrical ManufacturersAssociation, “ACR-NEMA Digital Imaging and CommunicationsStandard,” NEMA Standards Publication No. 300–1985, Washington, DC,1985.

7. American College of Radiology, National Electrical ManufacturersAssociation, “ACR-NEMA Digital Imaging and CommunicationsStandard: Version 2.0,” NEMA Standards Publication No. 300–1988,Washington, DC, 1988.

8. American College of Radiology, National Electrical ManufacturersAssociation, “Digital Imaging and Communications in Medicine(DICOM): Version 3.0,” Draft Standard,ACR-NEMACommittee, Work-ing Group VI, Washington, DC, 1993.

9. Mildenberger P, Eichelberg M, Martin E, Introduction to the DICOMstandard, Eur Radiol 12: 920–927, 2002.

10. Viola P, Wells III WM, Alignment by maximization of mutual informa-tion, Inter J Computer Vision 22: 137–154, 1997.

11. Beyer T, Townsend DW, Brun T, et al., A combined PET/CT scanner forclinical oncology, J Nucl Med 41: 1369–1379, 2000.

12. Townsend DW, Carney JPJ, Yap JT, et al., PET/CT today and tomorrow,J Nucl Med 445: 4S–14S, 2004.



13. Yap JT, Carney JPJ, Hall NC, et al., Image-guided cancer therapy usingPET/CT, Cancer J 10: 221–223, 2004.

14. J Nucl Med (Newsline), PET on Display: Notes from the 59th SNM AnnualMeeting, pp. 24N–26N, 2003.

15. Zanzonico P, Campa J, Polycarpe-Holman D, et al., Animal-specificpositioning molds for registration of repeat imaging studies: Com-parative microPET™ imaging of F18-labeled fluoro-deoxyglucose andfluoro-misonidazole in rodent tumors, Nucl Med Biol 33: 65–70, 2006.

16. Schoder H, Erdi Y, Larson S, et al., PET/CT: A new imaging technologyin nuclear medicine, Eur J Nucl Med Mol Imaging 30: 1419–1437, 2003.

17. Catana C, Wu Y, Judenhofer MS, et al., Simultaneous Acquisition ofmultislice PET and MR images: Initial results with a MR-compatiblePET scanner, J Nucl Med 47: 1968–1976, 2006.


CHAPTER 18

Wavelet Transform and Its Applicationsin Medical Image Analysis

Atam P Dhawan

Recently, wavelet transform has been found to be a very productive andefficient tool for image processing, analysis and compression for medicalapplications. Wavelet transform provides complete spatiofrequency local-ization for medical images that may be used to remove noise, undesiredfeatures and artifact, or to extract useful features for image characteriza-tion and classification. This chapter provides an introduction of wavelettransform with decomposition and reconstruction methods for medicalimage analysis.

18.1 INTRODUCTION

Wavelet transform has recently emerged as an efficient signal pro-cessing tool for the localization of frequency or spectral componentsin the data. As a historical perspective of signal analysis, the Fouriertransform has proved to be an extremely useful tool for decompos-ing a signal into constituent sinusoids of different frequency com-ponents. However, Fourier analysis suffers from a drawback of theloss of localization or time information when transforming infor-mation from the time domain to the frequency domain. When thefrequency representation of a signal is looked into, it is impossibleto tell when a particular event took place. If the signal properties donot change much over time, this drawback may be ignored. How-ever, signals change with interesting properties over time or space.

437


438 Atam P Dhawan

An electrocardiogram signal changes over the time marker withrespect to heart beat events. Similarly, in the context of two-dimensional and three-dimensional images, a signal or a propertyrepresented by the image changes over the sampled data pointsin space. Fourier analysis, in general, does not provide a specificevent (frequency) localized information with respect to time (in timeseries signals) or space (in images). This drawback of Fourier transfercan be somewhat addressed by using short-time fourier transform(STFT).1−4 This technique adapted the Fourier transform to analyzeonly a small section of the signal at a time. As a matter of fact, STFTmaps a signal into separate functions of time and frequency. TheSTFT provides some information about frequency localization withrespect to a selected window. However, this information is obtainedwith limited precision determined by the size of the window.Amajorshortcoming with STFT is that the window size is fixed for all fre-quencies once a particular size for the time window is chosen. In realapplications, signals may require a variable window size in order toaccurately determine event localization with respect to frequencyand time or space.

Wavelet transform may use long sampling intervals where lowfrequency information is needed, and shorter sampling intervalswhere high frequency information is available. The major advan-tage of wavelet transform is its ability to perform multiresolu-tion analysis for event localization with respect to all frequencycomponents in data over time or space. Thus, wavelet analysis iscapable of revealing aspects of data that other signal analysis tech-niques miss, such as breakdown points, and discontinuities in higherderivatives.1−4

Wavelet transform theory uses two major concepts: scaling andshifting. Scaling, through dilation or compression, provides a capa-bility of analyzing a signal over different windows or samplingperiods in the data while shifting, through delay or advancement,provides translation of the wavelet kernel over the entire signal.Daubechies wavelets1 are compactly orthonormal wavelets whichmake discrete wavelet analysis practicable. Wavelet analysis hasseen numerous applications in statistics,1−4 time series analysis1−2


Wavelet Transform and Its Applications in Medical Image Analysis 439

and image processing.5−8 Generalized wavelet basis functions havebeen studied for image processing applications.6−8 Furthermore,wavelet transform has also been used in data mining field and otherdata intensive applications because of its many favorable proper-ties, such as vanishing moments, hierarchical and multiresolutiondecomposition structure, linear time and space complexity of thetransformations, decorrelated coefficients and a wide variety of basisfunctions.

18.2 WAVELET TRANSFORM

Wavelet transform is the decomposition of a signal, f (t), with afamily of real orthonormal bases ψj,k(t) obtained through trans-lation and scaling of a kernel function ψ(t) in the Hilbertspace L2(R) of square integrable functions, known as the motherwavelet, i.e.

ψj,k(t) = 2j/2ψ(2jt − k); j, k ∈ Z, (1)

where j and k are integers representing, respectively, scaling andshifting indices. Using the orthonormal property, the waveletcoefficients of a signal f (t) can be computed as:

cj,k =∫ +∞

−∞f (t)ψj,k(t)dt. (2)

The signal f (t) can be fully recovered or reconstructed from thewavelet coefficients as:

f (t) =∑j,k

cj,kψj,k(t). (3)

To obtain wavelet coefficients from Eq. (2), ψj,k(t), the translated andscaled versions of the mother wavelet ψ(t), are obtained using ascaling function. Using a scale resolution of multiples of two, thescaling function φ(t) can be obtained as:

φ(t) = √2∑

n

h0(n)φ(2t − n). (4)


440 Atam P Dhawan

Then, the wavelet kernel ψ(t) is related to the scaling function as:

ψ(t) = √2∑

n

h1(n)φ(2t − n), (5)

where

h1(n) = ( − 1)nh0(1 − n). (6)

The coefficients h(n) in Eq. (4) must satisfy several conditions forthe set of basis wavelet functions defined in Eq. (1) to be unique,orthonormal, and with a certain degree of regularity.1−4

18.3 SERIES EXPANSION AND DISCRETE WAVELETTRANSFORM

Let x[n] be an arbitrary square summable sequence representing asignal in the time domain such that:

x[n] ∈ l2(Z). (7)

The series expansion of a discrete signal x[n] using a set of ortho-normal basis functions ϕk[n] is given by:

x[n] =∑k∈Z

〈ϕk[l], x[l]〉ϕk[n] =∑k∈Z

X[k]ϕk[n]

where X[k] = 〈ϕk[l], x[l]〉 =∑

l

ϕ∗k [l]x[l]. (8)

where X[k] is the transform of x[n]. All basis functions must satisfythe orthonormality condition, i.e.

〈ϕk[n], ϕl[n]〉 = δ[k − l]with

‖x‖2 = ‖X‖2 (9)

where 〈〉 represents the inner product.



The series expansion is considered to be complete if every signalfrom l2(Z) can be expressed as shown in Eq. (8). Similarly, using aset of biorthogonal basis functions, the series expansion of the signalx[n] can be expressed as:

x[n] =∑k∈Z


X[k]ϕk[n]

=∑k∈Z


X[k]ϕk[n]

where X[k] = 〈ϕk[l], x[l]〉 and X[k] = 〈ϕk[l], x[l]〉and 〈ϕk[n], ϕl[n]〉 = δ[k − l]. (10)

Using a quadrature mirror filter theory, the orthonormal bases ϕk[n]can be expressed as low pass and high pass filters for the decompos-tion and reconstruction of a signal. It can be shown that a discretesignal x[n] can be decomposed into X[k] as:

x[n] =∑k∈Z

〈ϕk[l]x[l]〉ϕk[n] =∑k∈Z

X[k]ϕk[n]

where

ϕ2k[n] = h0[2k − n] = g0[n − 2k]ϕ2k+1[n] = h1[2k − n] = g1[n − 2k]and

X[2k] = 〈h0[2k − l], x[l]〉X[2k + 1] = 〈h1[2k − l], x[l]〉. (11)

In Eq. (11), h0 and h1 are respectively, the low pass and high passfilters for signal decomposition or analysis, and g0 and g1 are respec-tively, the low pass and high pass filters for signal reconstruction orsynthesis. A perfect reconstruction of the signal can be obtained ifthe orthonormal bases are used in decomposition and reconstructionstages as:

x[n] =∑k∈Z

X[2k]ϕ2k[n]+∑k∈Z

X[2k + 1]ϕ2k+1[n]

=∑k∈Z

X[2k]g0[n − 2k]+∑k∈Z

X[2k + 1]g1[n − 2k]. (12)


442 Atam P Dhawan

As described above, the scaling function provides low pass fil-ter coefficients and the wavelet function provides the high passfilter coefficients.Amultiresolution signal representation can be con-structed based on the differences of information available at twosuccessive resolutions 2j and 2j−1. Such a representation can be com-puted by decomposing a signal using the wavelet transform. First,the signal is filtered using the scaling function, a low pass filter. Thefiltered signal is then subsampled by keeping one out of every twosamples. The result of low pass filtering and subsampling is calledthe scale information. If the signal has the resolution 2j, the scaleinformation provides the reduced resolution 2j−1. The difference ofinformation between resolutions 2j and 2j−1 is called the “detail”signal at resolution 2j. The detail signal is obtained by filtering thesignal with the wavelet, a high pass filter, and subsampling by afactor of two.

In order to decompose an image, the above method for 1D sig-nals is applied first along the rows of the image, and then along thecolumns. The image, at resolution 2j+1, represented by Aj+1, is firstlow pass and high pass filtered along the rows. The result of eachfiltering process is subsampled. Next, the subsampled results arelow pass and high pass filtered along each column. The results ofthese filtering processes are again subsampled. The combination offiltering and subsampling processes essentially provides the bandpass information. The frequency band denoted by Aj in Fig. 1 is

Fig. 1. A three-level wavelet decomposition tree, where A means approximationand D means detail.



2 H 1

2 H 0

2 H 1

2 H 0

2 H 1

2 H 0

Horizontal Subsampling Vertical Subsampling

2 H 1

2 H 0

2 H 1

2 H 0

2 2 1

2 2 H 0

2 H 1

2 H 0

2 H 1

2 H 0

2 2 H 1

2 2 H 0

2 H 1

2 H 0

2 H 1

2 H 0

2 2 H 1

2 2 0

Horizontal Subsampling Vertical Subsampling

Low-Low Aj

High-High Dj3

High-Low Dj2

Low-High Dj1

Fig. 2. Multiresolution decomposition of an image using the wavelet transform.

referred to as the low-low frequency band. It contains the scaled lowfrequency information. The frequency bands labeled Dj

1, Dj2, and

Dj3 denote the detail information. They are referred to as low-high,

high-low, and high-high frequency bands, respectively (Fig. 2). Thisscheme can be iteratively applied to an image to further decomposethe signal into narrower frequency bands, i.e. each frequency bandcan be further decomposed into four narrower bands. Since eachlevel of decomposition reduces the resolution by a factor of two,the length of the filter limits the number of levels of decomposition(Fig. 3).

The signal decomposition at the j-th stage can thus be general-ized as:

x[n] = ∑Jj=1

∑k∈z X(j)[2k + 1]g(j)

1 [n − 2jk]+∑k∈z X(j)[2k]g(j)

0 [n − 2jk]X(j)[2k] = 〈h(j)

0 [2jk − l], x[l]〉X(j)[2k + 1] = 〈h(j)

1 [2jk − l], x[l]〉. (13)

Wavelet based decomposition of a signal x[n] using a low-pass filterh0[k] (obtained from the scaling function) and a high-pass filter h1[k]is shown in Fig. 4(A) while the reconstruction of the signal fromwavelet coefficients is shown in Fig. 4(B).


444 Atam P Dhawan

D13 High-High

Component D1

2 High-Low Component

D11 Low-High

Component

A1 Low-Low Component

A2 D21

D23D2

2

Fig. 3. Wavelet transform based image decomposition: the original resolutionimage (NxN) is decomposed into four low low A1, low-high D1

1, high-low D12, and

high-high D13 images each of which is subsampled to resolution

(N2 X N

2

). The low-

low image is further decomposed into four images of(

N4 X N

4

)resolution each in the

second level of decomposition. For a full decomposition, each of the “detail” com-

ponent can also be decomposed into four subimages with(

N4 X N

4

)resolution each.

The “least asymmetric” wavelets were computed and reportedby Daubechies.1 Different least asymmetric wavelets were com-puted for different support widths as larger support widths pro-vide more regular wavelets, a desired property in signal and imageprocessing. A least asymmetric wavelet is shown in Fig. 5 with thecoefficients of the corresponding low pass and high pass filters givenin Table 1.

18.4 IMAGE PROCESSING USING WAVELET TRANSFORM

The wavelet transform provides a set of coefficients representingthe localized information in a number of frequency bands. A pop-ular method for denoising and smoothing is to threshold these



2H1

2H0 2H1

2H0 2H1

2H0

22H1

22H0 2H1

2H0

22H1

2H0

H1

22H0 2H1

2H0

22H1

2H0

H1

22H0

x[n] X(1)[2k+1]

2 G1 +

G022 G1 +

G022 G1 +

G02

2 G122 G1 +

G0222 G1 +

G02

22 G1 +

G0222 G1 +

G02

22 G1 +

G022

][nx

(A)

(B)

X(1)[2k] X(2)[2k+1]

X(2)[2k] X(3)[2k+1]

X(3)[2k]

X(3)[2k+1]

X(3)[2k]

X(2)[2k+1]

X(1)[2k+1]

Fig. 4. (A) A multiresolution signal decomposition using wavelet transform and(B) the reconstruction of the signal from wavelet transform coefficients.

coefficients in those bands that have high probability of noise andthen reconstruct the image using the reconstruction filters. Thereconstruction filters, as described in Eq. (12), can be derived fromthe decomposition filters using the quadrature mirror theory.1−4

The reconstruction process integrates information from specificbands with successive upscaling of resolution to provide the finalreconstructed image at the same resolution as of the input image. Ifcertain coefficients related to the noise or noise like information arenot included in the reconstruction process, the reconstructed imageshows a reduction of noise and smoothing effects. As can be seenin Fig. 6.22, the coefficients available in the low-high, high-low andhigh-high frequency bands in the decomposition process, provideedge related information that can be emphasized in the reconstruc-tion process for image sharpening.5−8 Figure 6 shows an X-ray mam-mogram original image that is smoothed using the wavelet shown inFig. 5. To obtain the smoothed image shown in Fig. 7, a hard thresh-olding method was used in which high-high frequency waveletcoefficients was equated to zero and not used in the reconstructionprocess. The loss of high-frequency information can be seen in the


446 Atam P Dhawan

Fig. 5. The least asymmetric wavelet with eight coefficients.

Table 1. The Coefficients for the Corresponding Low Passand High Pass Filters for the Least AsymmetricWavelet

N High Pass Low Pass

0 −0.107148901418 0.0455703458961 −0.041910965125 0.0178247014422 0.703739068656 −0.1403176241793 1.136658243408 −0.4212345342044 0.421234534204 1.1366582434085 −0.140317624179 −0.7037390686566 −0.017824701442 −0.0419109651257 0.045570345896 0.107148901418



Fig. 6. An original digital mammogram image.

smoothed image. Figure 8 shows the reconstructed image of fromthe high-high wavelet coefficients only.

18.5 FEATURE EXTRACTION USING WAVELETTRANSFORM FOR IMAGE ANALYSIS

Two-dimensional wavelet transform is widely used in image pro-cessing applications. Its ability to repeatedly decompose an imagein the low frequency channels makes it ideal for image analysissince the lower frequencies dominate the real images. The smoothimage has strong components only in the low frequencies whereasthe textured image has substantial components in the wide fre-quency/scale spectrum. Features related to spatiofrequency rep-resentation of the image can be efficiently extracted and analyzedusing wavelet transform method. Wavelet transform provides one of


448 Atam P Dhawan

Fig. 7. Asmoothed version of the image shown in Fig. 6 obtained through wavelettransform based smoothing method.

the best representation methods for analysis of texture informationin the image. Texture has been widely used in image analysis forbiomedical applications and satellite image analysis. It is an impor-tant characteristic of an image and is useful for image interpretationand recognition. The application of wavelet orthogonal representa-tion to texture discrimination and fractal analysis has been discussedby Mallat.2 Feature extraction for texture analysis and segmentationusing wavelet transforms has been applied by Chang and Kuo,9

Laine and Fan,10 Unser,11 and others.12−15

Each level of decomposition provides band pass filtered spa-tiofrequency information that can be used for feature extraction,representation and analysis. For example, energy ratios in spe-cific subbands from the wavelet transform based multiresolutiondecomposition have been used in characterization of skin lesionimages for detection of skin cancer, malignant melanoma.16−18 The



Fig. 8. An image reconstructed using high-high wavelet coefficients from thewavelet transform of the image shown in Fig. 6.

epiluminesence images of skin lesion were obtained using Nevo-scope and used for classification using texture based featuresextracted through the wavelet transform based decompositionmethod.19−22 The method is briefly described here.21−22 Figure 9

Fig. 9. Sample images (A) dysplastic nevus and (B) malignant melanoma.


450 Atam P Dhawan

shows sample images of a dysplastic nevus (nonmalignant lesion)and a malignant melanoma.

18.5.1 Feature Extraction Through Wavelet Transform

Three-level wavelet transform was applied to the epiluminesenceimages using the Daubechies 3 wavelet to obtain the 10 main waveletsubbands. Figure 10 shows three-level wavelet decomposition cod-ing of the image.

These channels (subbands) were further grouped into low-(channels 1–4), middle- (channels 5–7) and high-frequency (chan-nels 8–10). The ratio of the mean energy in the four low-frequencychannels (1–4) to the mean energy in the three middle-frequencychannels (5–7) is proposed as a criterion for optimal feature selec-tion by R Porter and N Canagarajah.12 Similarly, a set of ratios ofthe wavelet coefficients are studied for the textural analysis and theoptimal set of features is obtained by statistical analysis.

The set of ratios studied are:

r1 = m(c1)m(c12)

; r2 = m(c12)m(c11)

; r3 = m(c2) + m(c3) + m(c4)m(c5) + m(c6) + m(c7)

;

r4 = m(c5) + m(c6) + m(c7)m(c8) + m(c9) + m(c10)

;

r5 = m(c1)m(c12)

∗ m(c2) + m(c3) + m(c4)m(c5) + m(c6) + m(c7)

;

Fig. 10. Three-level wavelet decomposition of an image.



r6 = m(c12)m(c11)

∗ m(c5) + m(c6) + m(c7)m(c8) + m(c9) + m(c10)

;

r7 = m(c1)m(c2) + m(c3) + m(c4)

÷ m(c12)m(c5) + m(c6) + m(c7)

;

r8 = m(c11)m(c5) + m(c6) + m(c7)

÷ m(c12)m(c8) + m(c9) + m(c10)

, (14)

where ci stands for the different wavelet channels i = 1..10, of decom-position and m stand for the mean value of the wavelet coefficientsfor different channels given by:

m =∑

i∑

j xij

length ∗ breadth, (15)

where xij is the computed coefficient of wavelet transform; thelength and breadth are the dimensions of the respective channelsdecomposed.

The variance of the wavelet coefficient is given by:

ε =∑

i∑

j (xij − mean)2

length ∗ breadth, (16)

where mean represents the mean of the wavelet coefficients.The entropy measure for texture analysis can be defined as:

H =∑

i∑

j x2ij ∗ log (x2

ij)

length ∗ breadth. (17)

The energy of the wavelet coefficients defined as follows:

E =∑

i∑

j x2ij

length ∗ breadth. (18)

The set of ratios mentioned earlier is calculated for mean, vari-ance, energy and entropy of wavelet coefficients giving in all 32ratios, which henceforth are referred to as features. Also the graylevel features such as the mean and standard deviation of the imageintensity were included in the feature set. Thus, 34 features wereconsidered in this texture analysis.


452 Atam P Dhawan

Statistical correlation analysis was performed on extractedfeatures to select statistically significant and correlated features.The statistical correlation analysis provided a reduced set of featuresof the following five features with highest statistical significance:

f1 = m(c1)m(c12)

; f2 = m(c12)m(c11)

; f3 = e(c12)e(c11)

;

f4 = et(c1)et(c2) + et(c3) + et(c4)

÷ et(c12)et(c5) + et(c6) + et(c7)

;

f5 = ln (std + 1), (19)

where m, e, et and std stands for the mean, energy and entropy of thewavelet coefficients and standard deviation of the image intensityrespectively.

The selected features were then used in training a nearest-neighborhood classifier (described in Chapter 10) using a training setof pathologically validated labeled set of images. The trained clas-sifer was then used to classify those images that were not includedin the training set. Results of the nearest-neighborhood classifierwere compared to the pathology to obtain true positive and false-positive rates of melanoma detection. A true positive rate of 93% formelanoma detection was obtained with a false positive rate of 0%through this analysis.20


Wavelet transform has been effectively used for one- and multi-dimensional data analysis with a number of applications includingmedical image analysis. Wavelet transform provides a simple seriesexpansion based signal decomposition and reconstruction methodsfor localization of characteristic events associated with frequencyand time/space information. Utilizing the property orthonormalbasis functions with scaling and shifting operations, multiresolu-tion wavelet packet analysis provides localized responses equiva-lent to multiband filters but in a computationally efficient manner.Wavelet transform can be implemented through a simple modular



algorithm suitable for fast or real-time applications of any kind ofdata analysis.

Wavelet transform has been used for image enhancement,restoration and reconstruction for medical images. The localizedspatiofrequency information available through wavelet transformcan be effectively used for defining specific features for image rep-resentation, characterization and classification. Multidimensionalexpansion of wavelets transform and adaptive design of waveletfor specific image processing tasks have become areas of significantresearch interest in the recent years and will continue to be a pro-ductive research area in the near future.

References

1. Ingrid Daubechies, Ten Lectures on Wavelets, Society for Applied Math-ematics, Philadelphia, PA, 1992.

2. Mallat S, A theory for multiresolution signal decomposition: Thewavelet representation, IEEE Transactions on Pattern Analysis andMachine Intelligence 11: 674–693, 1989.

3. Stephane Mallat, Wavelets for a Vision, Proceedings of the IEEE 84:604–614, 1996.

4. Cohen A, Kovacevic J, Wavelets: The mathematical background, Pro-ceedings of the IEEE 84: 514–522, 1996.

5. BovikA, Clark M, Geisler W, Multichannel texture analysis using local-ized spatial filters, IEEE Transactions on Pattern Analysis and MachineIntelligence 12: 55–73, 1990.

6. Weaver JB, Yansun X, Healy Jr DM, Cromwell LD, Filtering noise fromimages with wavelet transforms, Magnetic Resonance in Medicine 21:288–295, 1991.

7. Alex P Pentland, Interpolation using wavelet bases, IEEE Trans-actions on Pattern Analysis and Machine Intelligence 16: 410–414,1994.

8. Ming-Haw Yaou, Wen-Thong Chang, Fast surface interpolation usingmultiresolution wavelet transform, IEEE Transactions on Pattern Anal-ysis and Machine Intelligence 16: 673–688, 1994.

9. Chang T, Kuo CCJ, Texture analysis and classification with tree-structure wavelet transform, IEEE Trans Image Process 2(4): 429–447,1993.

10. LaineA, Fan J, Texture classification by wavelet packet signatures, IEEETrans Pattern Anal Mach Intell 15(11): 1186–1191, 1993.


454 Atam P Dhawan

11. Unser M, Texture classification and segmentation using waveletframes, IEEE Trans Image Process 4(11): 1549–1560, 1993.

12. Porter R, Canagarajah N, A robust automatic clustering scheme forimage segmentation using wavelets, IEEE Trans Image Process 5(4):662–665, 1996.

13. Wang JW, Chen CH, Chien WM, Tsai CM, Texture classification usingnon-separable two-dimensional wavelets, Pattern Recognition Letters19: 1225–1234, 1998.

14. Chitre Y, Dhawan A, M-band wavelet discrimination of natural tex-tures, Pattern Recognition Letters, 773–789, 1999.

15. vanErkel AR, ThPattynama PM, Receiver operating characteris-tic (ROC) analysis: Basic principles and applications in radiology,European Journal of Radiology 27: 88–94, 1998.

16. KopfA, Saloopek T, Slade J, MarghoodA, et al., Techniques of cutaneousexamination for the detection of skin cancer, Cancer Supplement 75(2):684–690, 1994.

17. Koh H, Lew R, Prout M, Screening for melanoma/skin cancer: Theo-retical and practical considerations, J Am Acad Dermatol 20: 159–172,1989.

18. Stoecker W, Moss R, Skin Cancer Recognition by Computer Vision:Progress Report, National Science Foundation Grant ISI 8521284,August 29, 1988.

19. Dhawan AP, Early detection of cutaneous malignant melanoma bythree dimensional Nevoscopy, Computer Methods and Programs inBiomedicine 21: 59–68, 1985.

20. Nimukar A, Dhawan A, Relue P, Patwardhan S, Wavelet and StatisticalAnalysis for Melanoma Classification, SPIE International Conference onMedical Imaging, MI 4684, 1346–1353, Feb 24–28, 2002.

21. Patwardhan S, Dhawan AP, Relue P, Classification of melanoma usingtree-structured wavelet transform, Computer Methods and Programs inBiomedicine 72(3): 223–239, 2003.



CHAPTER 19

Multiclass Classification for TissueCharacterization

Atam P Dhawan

Computer aided diagnostic applications such as cancer detection mayrequire a binary classification into benign and malignant classes. How-ever, there are many medical imaging applications requiring multiclassclassifications to categorize image data into more than two classes for tis-sue or pathology characterization. This chapter provides an introductionof some of the approaches such as Bayesian classification, support vec-tor machine, and neuro-fuzzy systems that can be applied in multiclassclassification.

19.1 INTRODUCTION

Conventional methods for computer-aided medical image analy-sis for the detection of an outcome or pathology such as cancerusually require a binary classification of acquired image data. How-ever, other medical image analysis applications such as segmenta-tion and tissue characterization from multiparameter images mayrequire multiclass classification. For example, brain images acquiredthrough multiparameter multidimensional imaging protocols maybe analyzed for multiclass segmentation for tissue characteriza-tion for the evaluation and detection of critical neurological func-tions and disorders. Several chapters in this book describe currentand merging trends in multiparameter brain imaging and radia-tion therapy that can be benefited using multiclass classification

455


456 Atam P Dhawan

approaches. Fusion of anatomical, metabolic and functional infor-mation usually leads to multidimensional data sets leading to anal-ysis of local regions that can be obtained from segmentation anddetection approaches based on multiclass classification. In this chap-ter, we present some of the multiclass classification methods suitablefor multiparameter medical image analysis.

19.2 MULTICLASS CLASSIFICATION USING MAXIMUMLIKELIHOOD DISCRIMINANT FUNCTIONS

Medical images preprocessing and feature extraction analysis leadsto a set of spatially distributed multidimensional data vectors ofraw measurements and computed features. The total number ofmeasurements and computed features allocated to each pixel in theimage sets up the dimension d of the feature space. Let us assumethat we have an image of m rows and n columns with mn numberof pixels to be classified into k number of classes. Thus, we have mndata vectors X = {xj; j = 1, 2, . . . , mn} distributed in a d-dimensionalfeature space. Thus, each element of the data vector (i.e. pixel in theimage) is associated with d-dimensional feature vector. The pur-pose of multiclass classification is to find a mapping f (X) to map theinput data vectors into k classes denoted by C = {ci; i = 1, 2, . . . , k}.In order to learn such a mapping, we can use a training set S ofcardinality l with labeled input vectors such that:

S = {(x1, cl), . . . , (xl, cl)}, (1)

xi ∈ χ are provided in the inner-product space of and χ ⊆ Rd andCi ∈ γ = {1, . . . , k} the corresponding class or category label.

As shown in Eq. (1), there is a pair relationship of the assignmentof each input pixel X to a class C. Let us assume that each class ci

model obtained from the training set has a mean vector µi and acovariance represented by

∑i such that:

µi = 1n

∑j

xj, (2)

where i = 1, 2, . . . , k; and j = 1, . . . , n; n is the number of pixel vectorsin the i-th class, and xj is the j-th of n multidimensional vectors that


Multiclass Classification for Tissue Characterization 457

comprise the class. The dimension of xj corresponds to the numberof image modalities used in the analysis. The covariance matrix ofclass i,

∑i, is:

∑i= 1

n − 1

∑j

(xj − µi)(xj − µi) . (3)

For developing an estimation model,1–3 let us assume that the imageto classify is a realization of a pair of random variables {Cmn, Xmn};where Cmn is the class of the pixel mn. Cmn represents the spatialvariability of the class in the image and can take the values in adiscrete set {1, 2, . . . , k}. Xmn is a d-dimensional random variable ofpixel mn describing the variability of measurements for that pixel.Xmn describes the variability of the observed values x in a particularclass. Given that Cmn = i, (i = 1, 2, . . . , k), the distribution of Xmn

is estimated to obey the general multivariate normal distributiondescribed by the density function:

p(x) = 1

(2π)d/2

∣∣∣∣∑i

∣∣∣∣1/2 exp

−(x − µi)

2∑

i(x − µi)

, (4)

where x is a d-element column vector, µi is a d-element estimatedmean vector for the class i calculated from the training set,

∑i is the

estimated d × d covariance matrix for class i also calculated fromthe training set, and d is the dimension of multiparameter or featurevector.

For maximum likelihood based discriminant analysis to assigna class to a given pixel in the image.1–4 For each pixel, four tran-sition matrices Pr(m, n) = [pijr(m, n)] can be estimated, where ris a direction index (following four spatial connectedness direc-tions in the image) and pijr(m, n) are the transition probabilitiesdefined by:

pij1(m, n) = P{Cmn = j|Cm,n−1 = i}, (5)

pij2(m, n) = P{Cmn = j|Cm+1,n = i}, (6)


458 Atam P Dhawan

pij3(m, n) = P{Cmn = j|Cm,n+1 = i}, (7)

pij4(m, n) = P{Cmn = j|Cm−1,n = i}. (8)

A generalized estimation of transition probabilities for classescan be obtained using b images in the training set and aver-aged over small neighborhood of h pixels around the pixel mnas:

pij1(m, n) =∑

b

∑n{pix|Cmn = j, Cm,n−1 = i}∑

b

∑n{pix|Cm,n−1 = i}

pij2(m, n) =∑

b

∑n{pix|Cmn = j, Cm+1,n = i}∑

b

∑n{pix|Cm+1,n = i}

pij3(m, n) =∑

b

∑n{pix|Cmn = j, Cm,n+1 = i}∑

b

∑n{pix|Cm,n+1 = i}

pij4(m, n) =∑

b

∑n{pix|Cmn = j, Cm−1,n = i}∑

b

∑n{pix|Cm−1,n = i}

,

(9)

where∑

b{pix|CP} denotes the number of pixels with theproperty CP in the images used in the training set usedto generate the model and

∑n represents the number of

pixels with the given property in the predefined neighbor-hood.

The equilibrium transition probabilities can then be estimatedusing a similar procedure as:

πi(mn) =∑

b

∑n{pix|Cmn = i}∑

b

∑n{pix}

. (10)

19.2.1 Maximum Likelihood Discriminant Analysis

The class random variable Cmn is assumed to constitute a k-stateMarkov random field. Rows and columns of Cmn constitute segments



of k-state Markov chains. Chains are specified by the k × k transitionmatrix P = [pij] where:

pij = P{Cmn = j|Cm,n−1 = i}, (11)

which leads to the equilibrium probabilities (π1, π2, . . . , πK).Using the above model,1 the probabilities of each pixel belonging

to a specific class i is:

P{Cmn = i|xkl, (k, l) ∈ N(m, n)}, (12)

where N(m, n) is a predefined neighborhood of the pixel (m, n). Forexample, a four-connected neighborhood around a pixel mn can bedefined as:

N(m, n) = {(m, n), (m − 1, n), (m, n − 1), (m + 1, n), (m, n + 1)}. (13)

It follows that:

P{Cmn = i|xkl, (k, l) ∈ N(m, n)} = P{Cmn = i, Xmn|Xm±1,n, Xm,n±1

}P

{Xmn|Xm±1,n, Xm,n±1

}(14)

and

P{Cmn = i|xkl, (k, l) ∈ N(m, n)}

= P{Xmn |Cmn = i, |Xm±1,n, Xm,n±1

}P

{Cmn = i|Xm±1,n, Xm,n±1

}P

{Xmn|Xm±1,n, Xm,n±1

} ,

(15)

where

P{◦|Xm±1,n, Xm,n±1} ≡ P{◦|Xm−1,n}P{◦|Xm,n−1}× P{◦|Xm+1,n}P{◦|Xm,n+1}. (16)

Taking into account the class conditional independence Eq. (15) canbe stated as:

P{Cmn = i|xkl, (k, l) ∈ N(m, n)}= P{Xmn|Cmn = i}P{Cmn = i|Xm±1,n, Xm,n±1}

P{Xmn|Xm±1,n, Xm,n±1} . (17)


460 Atam P Dhawan

With the Bayes estimation method, the above expression leads to:

P{Cmn = i|xkl, (k, l) ∈ N(m, n)}= P{Xmn|Cmn = i}P{Xm±1,n, Xm,n±1|Cmn = i}P{Cmn = i}

P{Xmn|Xm±1,n, Xm,n±1}P{Xm±1,n, Xm,n±1}(18)

where

P{Xm±1,n, Xm,n±1|◦} ≡ P{Xm−1,n|◦}P{Xm,n−1|◦}× P{Xm+1,n|◦}P{Xm,n+1|◦}. (19)

The terms in Eq. (19) can be further expressed as:

P{Xm−1,n|Cmn = i} =N∑

j=1

P{Xm−1,n|Cmn = j}P{Cm−1,n = j|Cmn = i}

≡ Hm−1,n(i). (20)

Finally, substituting Eqs. (19) and (20) into Eq. (18), the probabilityof the current pixel, mn belonging to class i given the characteristicsof the pixels in the neighborhood of mn can now be defined as:

P{Cmn = i|xkl, (k, l) ∈ N(m, n)}= P{Cmn = i|Xmn}P{Xmn}Hm−1,n(i)Hm,n−1(i)Hm+1,n(i)Hm,n+1(i)

P{Xmn|Xm±1,n, Xm,n±1}P{Xm±1,n, Xm,n±1} .

(21)

Equation (21) shows the conventional expression for the class proba-bilities, denoted by P{Cmn = i|Xmn}P{Xmn}, is modified by the factorsHij according to the evidence found in the immediate neighborhood.Pixels are classified based on the class that maximizes.

19.3 NEURO-FUZZY CLASSIFIERS FOR MULTICLASSCLASSIFICATION

Thepattern recognitionsystemssuchasbackpropagationneuralnet-work, radial basis function (RBF) network or of k-nearest-neighbor(KNN) can provide multiclass classification using crisp decisionsurfaces that often suffer from low immunity to noise in the training



patterns. Neural networks and clustering methods for classificationare described in Chapter 10 of this book. To overcome problems ofcrisp function based classifier, fuzzy functions have been used forclassification applications.

Several approaches using fuzzy set theory for pattern recogni-tion can be found in a number of publications.5–19 A novel patternrecognition method using fuzzy functions with a winner-take-allstrategy is presented here that can be used for multiclass classifica-tion. In this approach, the feature space is first partitioned into allcategories using the training data. The data is thus transformed intoconvex sets in the feature space. It is achieved by dividing them intohomogeneous (containing only points from one category), nonover-lapping, closed convex subsets, and then placing separating hyper-planes between neighboring subsets from different categories. Thehyperplane separation of the obtained subsets with homogenousconvex regions provides the consecutive network layer to deter-mine what region a given input pattern belongs to. In our approach,a fuzzy membership Mf function is devised for each created con-vex subset (f = 1, 2, . . . , k). The classification decision is made by theoutput layer based on the “winner-take-all” principle. The resultingcategory C is the convex set category with the highest value of mem-bership function for the input pattern. A schematic diagram of sucha neuron-fuzzy classification system is shown in Fig. 1.5

19.3.1 Convex Set Creation

There are two requirements for the convex sets: they have to behomogeneous and nonoverlapping. To satisfy the first condition,one needs to devise a method of finding one category points withinanother category’s hull. Thus, two problems can be defined: (1) howto find whether the point P lies inside of a convex hull (CH) of points;(2) how to find out if two convex hulls of points are overlapping. Thesecond problem is more difficult to examine because hulls can beoverlapping over a common (empty) space that contains no pointsfrom either category. This problem can be defined as a generalizationof the first one,20 and the first condition can be seen as a special case of


462 Atam P Dhawan

M1

winner-take-alloutput layer

ϕL

ϕ1

fuzzy membershipfunction layer

x1

xi

xd

hyperplanelayer

inputlayer

max

M2

MK

C

Fig. 1. Architecture of a neuro-fuzzy pattern classifier.

the second requirement, when one of the convex sets is a single point.An interesting discussion on the first problem and its complexity canbe found in Refs. 22 and 23. The complexity of the second problemis far greater. For more detailed analysis of the problem, see Ref. 20.

In real world situations, when training samples are not com-pletely noise free, it would not be advisable to insist on high accu-racy of solutions to problems 1 and 2. In such a case, compromisebetween computational efficiency and accuracy should be reached.With this in mind, a new algorithm for solving problem 1 is proposedbelow. It is based on another property of convex sets, described bythe separation theorem,24 which states that for two closed nonover-lapping convex sets S1 and S2 there always exists a hyperplane thatseparates the two sets — separating hyperplane.

19.3.1.1 Algorithm A1: Checking point B to be withinconvex hull (CH)

An algorithm to check a point P to be within a convex hull (CH) mayhave the following steps:

(1) Put P in origin;(2) normalize points of CH (the vectors V = (v1, v2, . . . , vn) from

the origin);



(3) find min and max vector coordinates in each dimension;(4) find set E of all vectors V that have at least one extreme coor-

dinate;(5) take their mean and use it as projection vector φ:

φ = (vi|∀ vi ∈ E);

(6) set a maximum number of allowed iterations (usually = 2n);(7) find a set U = (u1, u2, . . . , um) (where m ≤ n) of all points in CH

that have negative projection on φ;(8) if U is empty (P is outside of CH) exit, else proceed to 9;(9) compute coefficient ψ:

ψ = φTU

U = 1m

m∑i=1

ui;

(10) calculate correction vector δφ by computing all of itsk-dimensional components δφ :

Uk �= 0 ⇒ δφk = ψ

Ukd

Uk �= 0 ⇒ δφk = ψ

d

, k = 1, 2, . . . , d

where d is the data’s dimension;(11) update φ · φ = φ − η · δφ, where η > 1 is a training parameter;(12) if iteration limit exceeded exit (assume P inside of CH), other-

wise go to 7.

The value of the training parameter η should be close to 1, soeven the points lying outside but close to the hull can be found.Heuristically, it has been found that the values of α should fall inthe range 1.0001 < η < 1.01. They are, however, dependent on theprecision of the training data and should be adjusted accordingly.

The principle idea is to find the hyperplane (defined by itsorthogonal vector φ) separating P and CH. If such a hyperplaneis found within a certain amount of iterations, the point is definitelyoutside of CH. If the hyperplane has not been found, it is assumedthat P is inside.


464 Atam P Dhawan

19.3.1.2 Algorithm A2: Creation of convex subsets

An algorithm to create convex sets may have the following steps.

(1) Select one category. Consider the set of all its training points.This is a positive set of samples. The training points from all theremaining categories constitute a negative set. Both sets are ind-dimensional linear space L. Mark all positive points as “not yettaken” and order them in a specific way. For example, choose anarbitrary starting point in the input space and order all positivepoints according to their Euclidean distance from that point. Usean index array to store the order.

(2) Build the convex subsets. Initialize current subset S by assigningto it the first point in . Loop over ordered positive categorypoints (in ) until there are no more points remaining. Consideronly points that have not yet been “taken”:

(a) Add the current point P to the subset S.(b) Loop over points from negative category. Consider only neg-

ative points that are closer than P to the middle of the currentsubset. Using A1, look for at least one negative point insideof conv S. If there is one, disregard the latest addition to S.Otherwise mark the current point P as “taken”.

(c) Update . Reorder the “not yet taken” positive categorypoints according to their distance from the mean of points inthe current subset.

(3) If all points in the category have been assigned to a subset, pro-ceed to step 4, otherwise go back to step 2 and create the nextconvex subset. The starting point is the first “not yet taken” pointin the list.

(4) Check if all categories have been divided into convex subsets. Ifnot, go back to step 1 and create subsets of the next category.

In the step 2(b), it is not always necessary to use algorithm A1 forchecking the presence of every single negative point within the cur-rent convex subset. Once a separating hyperplane is found for onenegative point, it should be used to eliminate all other negative



points that lie on the opposite side of the hyperplane than the convexsubset, from the checklist. Thus, both presented algorithms shouldbe used together in order to save computations. Using proceduresA1 and A2 does not guarantee that the constructed convex subsetsare not overlapping, since problem 2 is essentially not addressed. Itis of no significance when the subsets are from the same category.However, when they are not, this could result in linear nonsepara-bility of the neighboring subsets. This might seem as a drawback ofthe proposed solution since the overall accuracy seems to have beencompromised for the benefit of computational efficiency. However,the results of performed test show that this compromise is accept-able, since the performance of the NFPC was equal, or better thanthat of the backpropagation network classifier. Not directly address-ing problem 2 does not mean that the constructed subsets are alwaysoverlapping. Contrarily, the more representative the training set (i.e.greater number of training samples is), the smaller probability of theoverlap becomes, as the likelihood of finding a common empty spacedecreases. As shown in Ref. 5, this approximation yields acceptableresults that are comparable to and often better than that of othermethods.

In Ref. 23, the authors proposed a different method for solvingproblem 1 — separating hyperplane detection (SHD) algorithm. Asopposed to the approximate procedure A1, SHD always providesa definite answer. However, as the results in section III show, itscomputational complexity is always higher. This is because the sep-arating hyperplane is not found, only detected, so no negative pointscan be eliminated from the checklist in step 2(b) of A2.

19.3.1.3 Initial subset point selection

The presented algorithm requires initialization in the form of startingpoints for convex subsets from each category (step 1 of A2). Theremay be many possible ways of finding these starting points. In thesimplest case, they may be chosen randomly or by taking the mean of


466 Atam P Dhawan

all category points’ coordinates. The starting points for each categorycan be obtained by resolution coarsing, which may be performed toplace the starting point in an area with the greatest concentration ofthat particular category’s points.5,20

19.3.1.4 Placing hyperplanes — Hyperplane layer creation

Once the convex subsets have been found, it is assumed that they arenot overlapping, so that only one hyperplane is needed to separatetwo neighboring subsets. The program loops over subsets from allcategories and places a hyperplane between two sets from differentcategories that have not yet been separated by existing hyperplanes.Thus, a number of hyperplanes can vary depending on the trainingset. Several algorithms can be used to place a separating hyperplane,however it has been proven20 that backpropagation with batch train-ing performs better than other methods when the two classes arelinearly separable. Since we are primarily dealing with linearly sepa-rable convex subsets, backpropagation with batch training was usedin our implementation. A hyperplane was represented by a singleneuron trained to output a positive value (+1) for one category and anegative value (−1) for the other. The NPFC hyperplane layer com-prises a set of all hyperplanes needed to fully separate all convexsubsets from different categories.

19.3.2 Fuzzy Membership Function Construction

The hyperplanes define the convex regions trained from the pre-sented samples. These regions are the bases for constructing fuzzymembership functions, which represent the point’s relative mem-bership in a given convex subset, rather than in a category. It meansthat for a single point, the sum of its membership values for dif-ferent convex clusters is bound from below — it can never be neg-ative — and from above by a total number of convex subsets forall categories. The utilized fuzzy membership function Mf has to beflexible to reflect the true shape of the convex subset with the great-est precision possible. In our case, it was defined for each subset



f (f = 1, 2, . . . , k) as follows:

Mf (x) = Lf

√√√√ Lf∏i=1

θi, θi = 1(1 + eλif ϕix)

, (22)

where Lf — number of separating hyperplanes for the subset f ,ϕi — i-th separating hyperplane function for the subset, in

the vector form,x — network’s input vector in the augmented form,λif — steepness (scaling) coefficient for the i-th hyperplane

in the subset f .

The value of λif depends on the depth of convex subset f , as projectedonto the separating hyperplane Hi (defined by ϕi):

λif =−log

(1−χχ

)µif

, µif = 1n

n∑j=1

ϕixj, (23)

where n is the number of training points in the covex subset f ,ϕi — i-th hyperplane equation in the vector form,µif — depth of the convex subset f , as projected onto i-th

hyperplane,xj — augmented coordinate vector of the j-th point in the

subset,χ — center value of the membership function.

Since the sigmoidal function in Eq. (22) is continuous and onlyreaches the value of 1 in infinity, the resulting maximum value ofMf is less than 1. In practice, the maximum possible value is con-trolled by the center value χ, which is the goal membership valuefor a point with the mean projection value onto Hi for the entiresubset. In the performed tests, χ was set to 0.99. Other versions offuzzy membership functions are possible. An alternative approachis represented by two examples shown in Eqs. (24) and (25) below:

M∗f (x) = Lf

√√√√ Lf∏i=1

θi, θi = 1(1 + eλif ϕx)(1 + e−λif (ϕix+δif ))

, (24)


468 Atam P Dhawan

where Lf — number of separating hyperplanes for the subset f ,ϕi — i-th separating hyperplane function for the subset, in

the vector form,x — network’s input vector in the augmented form,λif — steepness (scaling) coefficient for the i-th hyperplane

in the subset f , defined by Eq. (23),δif — width of the subset f as projected on the i-th hyperplane.

Mf (x) = Lf

√√√√ Lf∏i=1

θi, θi = 1√2πσif

e−(ϕix− δif

2 )2

2σ2if , (25)

where Lf — number of separating hyperplanes for the subset f ,ϕi — i-th separating hyperplane function for the subset,

in the vector form,x — network’s input vector in the augmented form,λif — fuzziness coefficient for the i-th hyperplane in the

subset f ,δif — width of the subset f as projected on the i-th hyperplane.

The structure of the designed fuzzy membership function neuron isshown in Fig. 2. Scaling and multiplication stages are represented byEqs. (23) and (22), respectively. The input to the neuron is the hyper-plane layer, created as described in previous section. The neuron’soutput is the fuzzy membership function Mf for convex subset f . Theneuron structure for fuzzy membership functions from Eqs. (24) and(25) is analogous.

19.3.3 Winner-Take-All Output for Classification

The output Out of the classifier is the category C of the convex setfuzzy membership function Mi that attains the highest value for thespecified input pattern x, i.e.:(

Out = C∣∣∣f �=i

1 ≤∀f ≤ KMf (x) < Mi(x), Mi ∈ C

),



ϕL

Input fromhyperplane

layer

ϕ2

ϕ1

Scaling

λ1f

λ2f

λLf

Πf

Multiplication

Mf

Outputfuzzy

function

Fig. 2. Fuzzy membership function neuron.

where Out — output of the classifier,x — input pattern,K — number of convex sets obtained during training (num-ber of fuzzy function neurons in the fuzzy membership func-tion layer),Mi — the highest fuzzy membership function value for theinput x,C — category of convex subset used to construct membershipfunction Mi.

In other words, the output is based on the winner-take-all principle,with the convex set category corresponding to Mi determining theoutput. A decision surface for each category can be determined bythe fuzzy union of all of the fuzzy membership functions for the con-vex subsets belonging to this category. Thus, if the decision surfacefor a particular category can be defined as:(

Mcategory(x) = max(Mi(x))∣∣∣ ∀i,Mi∈category

)

where Mcategory(x) — decision surface for the category,Mi — the fuzzy membership functions for convex cluster i.


470 Atam P Dhawan

19.4 SUPPORT VECTOR MACHINE (SVM) FORMULTICLASS CLASSIFICATION

As described above, winner-take-all strategy can be effectively usedfor multiclassification.Aset of prototypes such as fuzzy membershipfunctions in the above described approach can be defined. A scoringfunction φ : χ × M ⇒ � is also defined measuring the similarity ofan input feature vector, an element in χ with Mi prototypes definedin space M. Thus a most similar prototype is selected to assign therespective class C, from a set γ , to the feature vector for classification.A multiprototype approach for multiclass classification using thewinner-take-all method can thus be expressed as F. Aiolli et al.25:

H(x) = C

(argmax

i∈�

φ(x, Mi)

), (26)

where x is an input feature vector, � is the set of prototypes indexes,Mi (i = 1, 2, . . . , k) are prototypes, and C : � ⇒ γ is the function toassign the class associated to a given prototype.

The use of large margin kernels for search of a linear disc-rimanant model in high-dimensional feature space for pattern clas-sification has been investigated by several investigators.25–30 Forexample, a radial basis function (RBF) can be used as a kernel func-tion. The RBF kernel function (see Chapter 10 for RBF network archi-tecture) can be defined as:

k(x, y) = exp (−λ‖x − y‖2), λ ≥ 0. (27)

A generalized kernel function can be expressed as:

k(x, y) = (〈x, y〉 + u)d, u ≥ 0, d ∈ N, (28)

where d is the dimensionality of classification space.The relevance vector machine (RVM)26 uses a model prototype

for regression and classification exploiting a probabilistic Bayesianprinciple similar to the approach presented in the first section ofthis chapter. There are several other models investigated for patternclassification using theoretic approaches from kernel-based classifierto linear programming perturbation based methods.25–30



A single prototype per class (as described above) can be used formulticlass classification using a Bayesian probabilistic model. Fora correct classification using a multiclass classifier, the prototype ofthe correct class should have a larger score than the maximum scoresrelated to all other incorrect classes. The multiclass margin for inputvector xi is then defined as F. Aiolli et al.25:

p(xi, ci|M) = 〈Myi , xi〉 − maxr �=yi

〈Mr, xi〉, (29)

where yi chosen such that C(yi) = ci., is the index of the prototypeassociated to the correct label for the training example xi. In thesingle prototype case, with no loss of generality, the associated classindices to be coincident, that is yi = ci.

It follows that for a correct classification of xi with a margin ofgreater or equal to 1, the following condition has to be satisfied asdescribed in F. Aiolli et al.25:

〈Myi , xi〉 ≥ θi + 1 where θi = maxr �=yi

〈Mr, xi〉. (30)

Recently, the above single-prototype-based approach has beenextended to multiprototype based SVM for multiclass classificationby Aiolli and Sperduti.25

19.5 MULTICLASS CLASSIFICATION OFMULTIPARAMETER MR BRAIN IMAGES

MR brain image segmentation into several tissue classes is of sig-nificant interest to visualize and quantify individual anatomicalstructures. The model developed in Ref. 1 employed 15 brain tis-sue classes instead of the commonly used set of four classes, whichwere of clinical interest to neuroradiologists for following-up withpatients suffering from cerebrovascular deficiency (CVD) and/orstroke. The model approximates the spatial distribution of tissueclasses by a Gaussian Markov random field and uses maximumlikelihood method to estimate the class probabilities and transitional


472 Atam P Dhawan

probabilities for each pixel of the image. Multiparameter MR brainimages with T1, T2, proton density, Gd + T1, and perfusion imagingwere used in segmentation and classification. In the developmentof the segmentation model, true class-membership of measuredparameters was determined from manual segmentation of a set ofnormal and pathologic brain images by a team of neuroradiologists.

An initial set of 15 tissue classes, as shown in Table 1 wasidentified by the neuroradiologist. Gray matter was divided intosuperficial and deep gray matter structures because pathologic pro-cesses often discriminate between involvement of the superficialcortical or deep basal ganglia. The deep gray matter was furtherdivided into four classes, caudate head, putamen, globus pallidus,and thalamus. White matter was divided into three classes: super-ficial white matter and two deeper white matter tracts, the corpuscallosum and the internal capsule. The superficial white matter con-sisted primarily of white matter within the cortical pathways of thecentrum semiovale. The CSF spaces were divided into two classesbased on the ventricular system. The first class was that of the CSFcontained within the ventricular system and the second class was for

Table 1. List of Classes used in Proposed Classification Scheme. TheClasses are Color Coded to Facilitate Manual Classification

Class Number Color Code Class Name

C1 White White MatterC2 Yellow Corpus CallosumC3 Gray Superficial GrayC4 Blue Green CaudateC5 Blue ThalamusC6 Light Blue PutamenC7 Dark Blue Globus PallidusC8 Light Cream Internal CapsuleC9 Light Violet Blood VesselC10 Dark Violet VentricleC11 Dark Green Choroid PlexusC12 Green Septum PellucidiumC13 Pale Green FornicesC14 Orange Extraaxial FluidC15 Pale Violet Zona Granularis



CSF outside the ventricular system (within the extra-axial spaces).This selection is based on the understanding that CSF within the ven-tricular system may have some signal variation due to the influenceof pulsatile blood flow.

Using the maximum likelihood discriminant function methoddescribed above, MR brain images were analyzed and seg-mented into 15 classes using multiclass classification approach. Fivecomplete sets of MR T1-weighted, T2-weighted, proton density,Gd + T1-weighted and perfusion brain images were used to deter-mine the class signatures. These images were obtained at 1 mm inter-slice and 5 mm intraslice resolution. The images with no observedpathology were used in manual classification by two expert neurora-diologists for a 15-class classification. Figure 3 shows a T2-weightedslice image used for manual classification to create a 5-parameter sig-nature database used in adaptive classification as described above.

The results of manual segmentation are presented in Fig. 4.Segmentation by one expert covering the whole area is shown inFig. 4(A) while the segmentation by another classifying only those

Fig. 3. T2-weighted MR slice image of the brain used for manual classification forcomputing multiparameter signatures for classification.


474 Atam P Dhawan

(A) (B)

Fig. 4. Manual segmentations of the same slice shown in Fig. 3 into 15 classes.In (A), segmentation by Expert-1 for classification of the whole area. In (B), seg-mentation by Expert-2 classifying only those pixels for which there was completecertainty of the class.

pixels for which the neuroradiologist was completely certain of theclass is shown in Fig. 4(B). Experts were required to repeat the man-ual classification with the both criteria to study the interobservervariability matrix. The interobserver variability is very high.1

The automated classifications were obtained using two differentcriteria over different probability cell size and averaging operations.The first criterion was based on the classification of a pixel with theclass that provided maximum probability for the given pixel overall classes. In addition, no pixel is assigned to any class if the maxi-mum class probability is less than 0.5. This minimum threshold wasselected as the acceptance criterion so that new classes whose sig-natures are not included into the candidate classes could be labeledas unclassified pixels and then learned as new classes later. Thoughthis criterion provided classification of all pixels, it caused some dis-agreement among experts on the pixels belonging to the boundaryof regions. The interobserver variability was observed largely dueto the disagreement for the boundary pixels because the brain tissueclasses do not exhibit sharp isolating boundaries. On the other hand,



if a volumetric quantification needs to be obtained, the regions haveto be closed with all pixels classified. The second criterion was estab-lished to reduce the classification error by only classifying those pix-els that has class probability better than 0.9. This criterion does notclassify all pixels leaving out pixels on the boundary of two regionsbut provides better confidence in the tissue classification.

The automated classifications using the probability cell size of2×2 with a one pixel wide averaging are shown in Fig. 5. Theclassification obtained using the maximum class probability forselecting the classification label is shown on the left while the clas-sification as obtained using the class probability with p > 0.9 forselecting the classification label is shown on the right. The pixelsthat could not be classified because the maximum probability forany class was less than 0.9 remained unclassified and are displayedas black in the image on the right in Fig. 5. To demonstrate the effectof probability cell size and averaging length, the same slice wasclassified using 4 × 4 pixels probability cell size and 4 pixel wide

(A) (B)

Fig. 5. Results of automatic classification with 2 × 2 pixel probability cell size andone pixel averaging. For (A) classification is obtained on the basis of maximumprobability. For (B) classification is obtained for pixels with p > 0.9.


476 Atam P Dhawan

(A) (B)

Fig. 6. Results of automatic classification with 4 × 4 pixel probability cell size and4 pixel wide averaging. Left (A): classification is obtained on the basis of maximumprobability. Right (B): Classification is obtained for pixels with p > 0.9.

averaging. The results of this segmentation for two classificationcriteria: maximum probability and p > 0.9 are shown in Figs. 6(A)and (B) respectively.

The signatures or prototypes were developed for all classes.After learning, new images of a patient were analyzed. The MRbrain images of the patient were taken about 48 hours after thepatent suffered from a stroke. An arbitrarily selected slice passingthrough the stroke area is shown in Fig. 7(A). The patient brain wasscanned with the same protocol to obtain T1-weighted, proton den-sity, T2-weighted, Gd + T1 and perfusion MR images. The perfu-sion image of the corresponding slice is shown in Fig. 7(B). Figure 8shows the results of automatic segmentation with maximum classprobability classification criterion at the left and “p > 0.9” classi-fication criterion at the right. The manual segmentation is shownin Fig. 9. It can be seen from the classification results that thoughno signatures for the pathological regions such as primary hem-orrhage and edema were provided to the computer classification



(A) (B)

Fig. 7. Proton density MR and (B) perfusion image of a patient 48 hours afterstroke.

(A) (B)

Fig. 8. Results of automatic classification with 4 × 4 pixel probability cell size and4 pixel wide averaging. (A) classification as obtained on the basis of maximumprobability, (B) as obtained with p > 0.9.


478 Atam P Dhawan

Fig. 9. The manual classification as obtained through the commonality matrix oftwo experts.

method, the adaptive segmentation method as described abovecreated three new classes corresponding to the primary hemorrhage,edema and effected white matter (shown in rose color). Six newclasses were learned from using the above method as groups ofunclassified pixels with clustered signatures in the 5-dimensionalparameter space. With the feedback from experts, these new classeswere added in the signature database.


There are very exciting and important applications of multicall clas-sification in medical image analysis and tissue characterization. Themaximum likelihood discriminant function based multiclass clas-sification method described above with results on multiparameterMR brain images demonstrates the usefulness and capability of anadaptive multiclass classification method for segmentation and tis-sue characterization. Recent approaches in neuron-fuzzy and multi-prototype support vector machine pattern classification have shown



a great potential in providing accurate and robust multiclass classi-fication for medical imaging applications.

References

1. Zavaljevski A, Dhawan AP, Holland S, Ball W, et al., Multispectral MRbrain image classification, Computerized Medical Imaging, Graphics andImage Processing 24: 87–98, 2000.

2. Atam P Dhawan, Medical ImageAnalysis, John Wiley Publications andIEEE Press, June 2003, Reprint, 2004.

3. Vannier M, Pilgram T, Speidel C, Neumann, et al., Validation of mag-netic resonance imaging (MRI) multispectral tissue classification, Com-puterized Medical Imaging and Graphics 15: 217–223, 1991.

4. Clarke L, Velthuizen R, Phuphanich S, Schellenberg J, et al., MRI: Sta-bility of three supervised segmentation techniques, Magnetic ResonanceImaging 11: 95–106, 1993.

5. Grohman W, Dhawan AP, Fuzzy convex set based pattern classifica-tion of mammographic microcalcifications, Pattern Recognition 34(7):119–132, 2001.

6. Pal SK, Mitra S, Multilayer perceptron, fuzzy sets and classification,IEEE Trans Neural Networks 3: 683–697, 1992.

7. Mitra S, Pal SK, Fuzzy multi-layer perceptron, inferencing and rulegeneration, IEEE Trans Neural Networks 6: 51–63, 1995.

8. Liu JNK, Sin KY, Fuzzy neural networks for machine maintenance inmass transit railway system, IEEE Trans Neural Networks 8: 932–941,1997.

9. Gader P, Mohamed M, Chiang JH, Comparison of crisp and fuzzy char-acter neural networks in handwritten word recognition, IEEE TransFuzzy Systems 3: 357–364, 1995.

10. Chang J, Han G, Valverde JM, Griswold NC, et al., Cork quality clas-sification system using a unified image processing and fuzzy-neuralnetwork methodology, IEEE Trans Neural Networks 8: 964–974, 1997.

11. Zhang YQ, Kandel A, Compensatory neurofuzzy systems with fastlearning algorithms, IEEE Trans Neural Networks 9: 83–105, 1998.

12. Simpson PK, Fuzzy Min-Max Neural Networks — Part 1: Classifica-tion, IEEE Trans Neural Networks 3: 776–787, 1992.

13. Pedrycz W, Computational Intelligence: An Introduction, CRC Press,New York, 1998.

14. Zhang J, Morris AJ, Recurrent neuro-fuzzy networks for nonlinear pro-cess modeling, IEEE Trans Neural Networks 10: 313–326, 1999.


480 Atam P Dhawan

15. Suh IH, Kim TW, Fuzzy Membership function based neural networkswith applications to the visual servoing of robot manipulators, IEEETrans Fuzzy Systems 2: 203–220, 1994.

16. Kwan HK, Cai Y, A fuzzy neural network and its application to patternrecognition, IEEE Trans Fuzzy Systems 2: 185–193, 1994.

17. Petridis V, Kaburlasos VG, Fuzzy lattice neural network (FLNN):A hybrid model for learning, IEEE Trans Neural Networks 9: 877–890,1998.

18. Chiang JH, Gader PD, Hybrid fuzzy-neural systems in handwrittenword recognition, IEEE Trans Fuzzy Systems 5: 497–510, 1997.

19. Purushothaman G, Karayiannis NB, Quantum Neural networks(QNN’s): Inherently fuzzy feedforward neural networks, IEEE TransNeural Networks 8: 679–693, 1997.

20. Grohman WM, Neuro-fuzzy pattern classifier with convex sets, PhDdissertation, Dept of Bioengineering, University of Toledo, 1999.

21. Nilsson NJ, The Mathematical Foundations of Learning Machines, SanMateo, CA, Morgan Kaufmann, 1990.

22. Luenberger DG, Optimization by Vector Space Methods, John Wiley &Sons Inc, New York, 1969.

23. Suh IH, Kim JH, Rhee FChH, Convex-set-based fuzzy clustering, IEEETrans Fuzzy Systems 7: 271–285, 1999.

24. Hiriart-Urruty JB, Lemaréchal C, Convex Analysis and MinimizationAlgorithms, Springer Verlag, Berlin, 1993.

25. Aiolli F, Sperduti A, Multiclass classification with multiprototype sup-port vector machines, Journal of Machine Learning Research 6: 817–850,2005.

26. Tipping ME, Sparse bayesian learning and the relevance vectormachine, Journal of Machine Learning Research 1: 211–244, 2001.

27. Downs T, Gates KE, Masters A, Exact simplification of support vectorsolutions, Journal of Machine Learning Research 2: 293–297, 2001.

28. Fung GM, Mangasarian OL, Smola AJ, Minimal kernel classifiers,Journal of Machine Learning Research 3: 303–321, 2002.

29. Aiolli F, Sperduti A, A reweighting strategy for improving margins,Artificial Intelligence Journal 137: 197–216, 2002.

30. Aiolli F, Sperduti A, Multiprototype support vector machine, in Pro-ceedings of International Joint Conference of Artificial Intelligence (IJCAI),2003.

31. Crammer K, Singer Y, On the algorithmic implementation of multiclasskernel-based machines, Journal of Machine Learning Research 2: 265–292,2001.


CHAPTER 20

From Pairwise Medical ImageRegistration to Populational

Computational Atlases

M De Craene and AF Frangi

Medical image registration is a widely used strategy for intrasubject andintersubject matching. Over the last twenty years, numerous strategieshave been designed to address the challenges raised by medical appli-cations including the capability to handle images acquired by differentimaging sensors and the estimation of flexible but realistic transforma-tions for modeling intersubject and pathology-induced deformations. Thischapter presents a survey of pairwise intensity-based automatic regis-tration algorithms by classifying them according to the similarity costfunction and extracted features used for quantifying the matching, therepresentation of the space of allowed transformations and the regular-ization strategy used to ensure continuous and smooth transformations.Joint alignment of a population of subjects is an interesting problem thatcan be seen as a natural extension of pairwise registration. It brings for-ward the open question of generalizing pairwise similarity cost functionsand the definition of a common reference space where the populationunder study can be projected. An overview of existing techniques forthe construction of computational atlases based on a collection of sub-jects is presented. The two main representations of such atlases — prob-abilistic and statistical — are described. Finally, we present a review ofstate of the art techniques for the joint alignment of a population ofimages.

20.1 INTRODUCTION

Information gained from two or several image modalities acquiredin the track of anatomical features is usually of a complementarynature. Therefore, a proper integration of useful data obtained from

481


482 M De Craene and AF Frangi

the separated images is often desired. The preliminary step in thisintegration process is to bring the acquired images into spatial align-ment. The term registration refers to this alignment procedure.

The first registrations techniques were performed by manualadjustment of rotations and translations. In this scenario, the prac-titioner proceeds by translating the contours of one image onto asecond image. The main drawback of manual registration is the lackof reproducibility and therefore the intra and inter observers errorsthat result from experience or external conditions (tiredness, timepressure, . . .).

Over time, automated rigid registration algorithms have beendeveloped, by minimization of the mean square error betweenmonomodal images and by matching corresponding boundary sur-faces extracted from different modalities in multimodal images. Asmany applications require to estimate more complex transforma-tions, automated nonrigid registration algorithms have been investi-gated by constraining the transformation with landmarks in eachmodality and then projecting the transformations onto basis func-tions such as thin plate splines.1

Registration algorithms have first been introduced in otherdomains than medical imaging, mainly in video image processing,for the estimation of movement between two consecutive frames.One of the most classical technique for movement estimation invideo sequences, called optical flow,2 was rediscovered years laterfor brain matching.3

In the medical imaging community, image registration has founda large spectrum of applications. Below is a short list — far to beexhaustive — of applications where registration plays a key role.

20.1.1 Fusion of Multimodal Images

Complementary information is provided by different imageacquisition sensors in a large number of medical applications.In radiotherapy planning, computed tomography (CT) offer a ref-erence image for anatomy while positron emission tomography(PET) images reveal the presence of tumor tissues using injection


From Pairwise Medical Image Registration to Populational Computational Atlases 483

of a contrast agent like 18F-fluorodeoxyglucose (FDG). The fusion ofthese two modalities enables the delineation of the tumor and struc-tures that need to be preserved as much as possible from irradiationlike saliva glands. Such registration problem is both challenging interms of image resolution4 (PET has a fairly poor image resolutioncompared to CT) and by the difference of information provided byeach modality.5

20.1.2 Atlas-Based Segmentation

Atlas-based segmentation is a standard paradigm for the automaticsegmentation of medical images. It relies on the existence of a ref-erence subject (atlas) in which structures of interest have been care-fully segmented, usually by hand. To segment a new image volume,a transformation that registers (i.e. puts in point-to-point spatial cor-respondence) the atlas to this volume is first computed. This trans-formation is then used to project labels assigned to structures fromthe atlas onto the image volume to be segmented. Thus, the segmen-tation problem is reduced to a registration problem which tried tocapture and compensate the normal anatomical variability. Figure 1illustrates this concept for brain images.

Fig. 1. Atlas-based segmentation is a technique that consists of mapping anatomi-cal structures of a reference individual to another subject for performing automaticsegmentation. The quality of the segmentation depends both of the performanceof the registration algorithm and the representativity of the reference subject.Population-based atlases are an extension of classical atlases that encodes anatom-ical variability of a population in the atlas representation.



20.1.3 Quantifying Temporal Deformations

Image registration has been applied to quantify the evolution ofanatomical structures or pathologies like tumors pre- and post-treatment (e.g. liver tumors6 or lymph nodes in the neck).7 Shenand Davatzikos8 proposed a group-wise (4D) framework to esti-mate such changes. Movement compensation between consecutiveframes of an image sequence is an important topic in cardiac imag-ing for estimating deformation or strain maps over the cardiaccycle. Involved modalities in this application are cine MR images9

and tagged MRI images.10 When the time step between consecu-tive images is small, more temporal coherency is expected in thesequence of transformations. This coherence can be enforced by theuse of dedicated regularization techniques.

20.1.4 Surgery and Preoperative Roadmap

Registration between a preoperative image and the physical scene inthe operating room has brought forward the possibility to visualizea preoperative surgical roadmap during surgery. Since such appli-cations require almost real time computations, the matching is oftenperformed using implantable markers11 both visible in the imageand being optically tracked during surgery or fiducial points thatare identified on the image and using a mapping catheter record-ing its physical location during surgery.12 Non-rigid deformationmodels have been proposed to characterize complex deformationsoccurring over surgery like intracranial brain deformations occur-ring after skull opening13–15 or liver deformations.16

20.1.5 Voxel-Based Morphometry

Voxel-based morphometry which consists of, in a first step, align-ing two groups of patients or a temporal sequence to the samereference space, and, is a second step, to perform a statistical analysisto identify significant changes over time17 or between groups. Voxel-based morphometry has been widely used for analyzing functional



MRI images18 and analyzing structural changes between sane andpathological groups (e.g. schizophrenia).19

20.2 IMAGE FEATURES AND SIMILARITY METRICS

In this section, we will concentrate on metrics that are mainly basedon image intensity functions to find the optimal matching. Such met-rics operate directly on the image intensity values, without reducingfirst the data to a set of segmented structures to be matched20 or with-out any identification by the user of key points in the modalities tomatch.21

A wide set of metrics have been proposed22: sum of squareddifferences23–25 (only valid for the same modality with properlynormalized intensities), normalized cross-correlation26–28 (whichallows for linear relationship between the intensities of the twoimages), mutual information,29,30 minimization of variance of inten-sity ratios.31,32

Due to its capability to deal with multimodal images, mutualinformation has been associated with various transformation mod-els and optimization strategies for solving challenging registrationproblems.33 Mutual information was first used to align MR and CTimages of the brain. The two first implementations by Viola andWells,30 Maes and Collignon,29 and Collignon et al.,34 proposedalmost simultaneously, differ by the probability estimation tech-nique and the optimization strategy. Viola has proposed Parzen win-dowing for probability density estimation and a gradient descentfor optimization while Maes used joint histograms and a Powelloptimization scheme. Extension of mutual information for deal-ing with nonrigid registration problems has been first proposedby Rueckert et al.,35 who introduced the use of a B-Spline nonrigidtransformation model for capturing breast nonrigid deformations.Further extensions of this method have used more complex opti-mization schemes like Broyden-Fletcher-Goldfarb-Shanno (BFGS)36

or to enforce positiveness of the deformation jacobian and selectactive control points where the deformation needs to be refined.37



Normalized versions of mutual information have been reported tobe more robust to changes of overlap between the two images38

or to enable the selection of optimal features over the registrationprocess.39

Other statistical divergence estimators have been proposedas similarity measure for registration. Pluim et al.40 performeda comparison of five different divergence measures between thejoint intensity distribution obtained at alignment and the prod-uct of marginal intensity distributions. The Kullback divergenceassumes to have a priori knowledge on the joint intensity distributionexpected to be obtain at alignment41 and measures the divergencebetween the actual joint distribution and the target one.

Several authors have also been looking at alternative featuresthan image intensities to generate metrics than are more robust

Table 1. Short Survey of Most Classical Similarity Metrics for Intensity-Based Image Registration Listing the Different Metrics and theFeatures from Which They are Computed

Features Metric Multimodal References

Intensity Normalizedcross-correlation

no [26–28]

Intensity Mutual information yes [29, 30, 33,35–37, 51–53](review)

Intensity Normalized entropy yes [38, 39]Intensity Correlation ratio yes [53]Intensity F-information

measuresyes [54]

Edge and ridge Cross-correlation yes [26]Edge type, intensity

and geometricmoment

Vector similarity no [42, 43]

Intensity and tissueclasses

Conditional entropyover tissue classes

yes [54]

Waveletcoefficients44,45

and sliceaccumulationpyramid44

Alpha mutualinformation,45

normalizedmutualinformation44

yes [44, 55]



to image artifacts or that can enable multiscale analysis23 of theimage set to match for faster and safer convergence. Edge andRidge measures based on first and second order image derivativeshave been proposed by Maintz et al.26 The HAMMER algorithm,42,43

dedicated to brain image matching, uses a local similarity met-ric based on a vector of features that incorporates edge infor-mation, image intensity and geometric moments. The edge typerequires a prior segmentation and encodes all the possible tran-sition between voxel classes (Cerebrospinal fluid, white and graymatter in the case of the brain). Geometric moments are computedin a spherical neighborhood around each voxel and for each tis-sue class. The algorithm searches for each driving voxel (the setof these driving voxels is iteratively increasing over iterations) inthe fixed image the voxel with the most similar attribute vector inthe other image. Wavelet transformation has also been proposedas a feature generator for image registration. The difficulty withsuch feature set is the definition of a similarity metric able of deal-ing with high dimensional similarity vectors. Xu and Yao44 usedstandard normalized mutual information for matching images ata given scale of the wavelet decomposition. A comparison is per-formed with a simple pyramidal representation of the image gener-ated by the accumulation of slices. Oubel et al.45 used an estimatorof alpha mutual information based on entropic graphs. The pro-posed method estimates probability densities from K-nearest neigh-bor (KNN) graphs.46 This probability estimation technique enablesto work with vector features of arbitrary dimension. Histogram-based method become impractical when the number of dimen-sions of the feature space increases beyond ten due to the curseof dimensionality: for fixed resolution per dimension the numberof histogram bins increases geometrically in dimension.47 KNNgraphs, in turn, have a storage and computation complexity thanincreases linearly in feature dimension.48 Similarity metrics dealingwith high-dimensional feature spaces are also required when theimages are intrinsically multichannel like diffusion tensor images(DTI).49,50



20.3 TRANSFORMATION REPRESENTATION

20.3.1 Dense Deformation Field Versus PriorTransformation Model

This section classifies intensity-based automatic registration algo-rithms depending on the representation of the transformation beingoptimized. A clear distinction in the huge variety of proposedmethods22,33 can be made following the representation of the trans-formation: some algorithms (dense field representation) considerthe displacement at each voxel independently, some others (priortransformation models) use a simplified representation of the trans-formation using a global model or local basis functions.

— Dense field representation registration algorithms address theestimation of a dense deformation field, considering indepen-dently the three components of the displacement at each voxel.Variational methods, incrementally estimating a dense deforma-tion field bringing the two images to the closest local minimum ofthe metric, are the only option for efficiently dealing with the highdimension of the optimization space. The theory beyond thesealgorithms is related to functional analysis and Euler-Lagrangeequations. The reader is invited to refer to Hermosillo et al.55 formore details about the mathematical concepts behind variationalanalysis applied to medical image registration. The general iter-ative scheme of registration algorithms using a dense field rep-resentation is given in the top part of Fig. 2.

— Prior transformation models registration algorithms model-ing the transformation using a simplified model. The first pro-posed models were rigid or affine. More sophisticated modelshave been proposed for non-rigid registration (B-Splines byRueckert et al.).56 The use of a prior transformation model sim-plifies the estimation of a function at each point of the imagedomain by estimating a limited set of parameters. The itera-tive estimation of a dense deformation field at each iterationis simplified to the estimation of the metric derivative regard-ing the set of transformation parameters. This derivative canbe estimated analytically or using finite difference techniques



Fixed Image

Moving Image

fieldCurrent disp.

Update disp.field

Fixed Image

Moving Image

Interpolator

Metric

Optimizer

Transform

Fig. 2. Iterative schemes of intensity-based registration methods according to therepresentation of the transformation. A dense deformation field approach offersless modularity since it is typically based on a variational analysis where an updatedisplacement field is estimated at each iteration and composed with the currentestimation of the displacement field. By variational analysis, we mean methodsthat address the estimation of a non-parametric displacement i.e. methods thatoptimize a function instead of a discrete set of parameters. Prior deformation modelsare naturally regularized and based on a small number of parameters. They can becombined with optimization method that do not require the analytical computationof the metric derivative. The regularization of the transformation over the iterationsis discussed in Sec. 20.4.

and be used for gradient descent or quasi-Newton methods.57,58

Some numerical optimization schemes do not even require thecomputation of a derivative like the Brent-Powell method59 orstochastic methods.60,61 The main algorithmic blocks making upregistration algorithms based on a prior transformation modeland the interactions between blocks are presented in the bottompart of Fig. 2.

In the case of a dense field representation of the transforma-tion, an incremental dense displacement field is estimated at eachiteration. The similarity metric has to be differentiable for computing



at each point the infinitesimal variation of the metric due to a vari-ation in the three directions of space of the current displacement atthis point.An extremely popular registration algorithm belonging tothis class is the optical flow algorithm. Initially proposed for move-ment detection in video sequences,2 it has been applied with suc-cess to medical image registration, firstly reintroduced by Thirion.3

Optical flow algorithms typically minimize the sum of squared dif-ferences between fixed and moving images by iteratively solvingordinary differential equations that provide an update to the cur-rent displacement field. The various implementations of optical flowmostly differ from the regularization strategy used to constrain theregularity of the displacement field. The regularization techniqueproposed by Horn and Schunk2 incorporate in the functional a globalsmoothness term minimizing the absolute gradient of the displace-ment field. This lead to a system of recursive equations that must besolved iteratively. A simpler implementation choice is to perform adirect smoothing of the displacement field at each iteration using aGaussian kernel. The smoothing can operate on the update or thecumulated displacement field. Another critical parameter for ensur-ing global continuity of the recovered displacement field is the timestep used in the evolution equations. Vermuri et al.62 have proposedto control the time step using the Courant-Friedrichs-Levy (CFL)condition. This leads to a maximal time step lead by the currentforce and the moving image gradient. Recently, a new class of regis-tration algorithms, that makes use of a diffeomorphic representationof the deformation, has gained growing interest. A diffeomorphicrepresentation63,64 of the displacement field adds a time dependencyto the displacement field, which in practice represents the time ofsuccessive iterations. The final displacement field is obtained by inte-gration over time of the velocity field which is a function of space andtime. The key idea is to model the matching process as a dynamicprocess where the smoothness of the velocity field at each iterationis kept under control.65

The major part of dense field estimation algorithms optimizethe sum of squared differences between fixed and moving imagesor a correlation measure. Extension to global similarity metrics like



Fig. 3. Main nonrigid transformation representations used in medical-imagingregistration. The dense deformation field represent the displacement of each voxelindependently and offers the most local representation of the deformation. A sim-pler representation of the deformation can be obtained by basis functions likeB-Splines. B-Splines are defined on a regular lattice of control points but canaccurately represent high-order nonlinear functions. Finite element models rep-resent the transformation by interpolating displacements on a nonregular grid.The most common interpolation method uses linear shape functions on tetrahedralelements.

entropy-based similarity measures has been investigated by sev-eral authors.54,55 The more complicated is the similarity cost func-tion, the more expensive is the computation of the velocity field.However, dense deformation field estimation algorithms based onglobal metric like mutual information generate a local deformationtake into account global image features like intensity histogram. Inbrain matching, if a mask is used to measure mutual informationonly inside the intracranial cavity, a mutual information based algo-rithm can be more sensitive to white matter/gray matter contrastdifferences than an optical flow algorithm. Figure 4 illustrates fora specific example (atlas to patient with tumor matching) the dis-placement fields generated by the Demons algorithm and a mutualinformation flow algorithm.66



Fig. 4. This figure illustrates the use of two algorithms using a dense field repre-sentation for mapping an atlas on a patient with tumor. Such registration problemsare challenging because a pathology-induced deformation is added to the inter-subject variations of anatomy. Both methods uses the same regularization scheme(Gaussian smoothing of the accumulated displacement) but differ by the similaritycost function: the demons optimize a quadratic error between fixed and movingimage intensities while mutual information optimizes mutual information. In thisspecific case, the mutual information flow show higher deformations around theventricles and the corpus callosum than optical flow. However, mutual informationis only measured inside the brain to give a better sensitivity to the gray/white mat-ter contrast. For this reason, demons are more accurate around brain contours. Thissimple example show that each application might require a different algorithm:an analysis of brain functional maps might be more focused on brain surface andsulci matching while other applications will require more accuracy around deepstructures.



The second class of algorithms (prior transformations models)uses numerical optimization schemes instead of a variational anal-ysis to optimize the metric. Some optimization algorithms do notrequire any computation of analytical derivatives (if the number ofparameters is reasonably small, they can be estimated using finitedifferences35 or a stochastic estimate of the gradient67) which makethe implementation simpler. In this class, the different techniquescan be classified following three main components:

— A transformation model which takes as input a set of parame-ters characterizing the transformation. It transforms coordinatesin the fixed image domain to coordinates in the moving imagedomain.

— The registration metric which provides a score (cost function)for a set of transformation parameters.

— The parametric optimizer, searching for the set of transformationparameters yielding the best metric score.

In this category of algorithms, one can distinguish betweenglobal and local transformation models. In a global transformationmodel, the modification of one parameter affects all points of theimage. Most frequently used global transformation include rigid,rigid and scale (also called similarity transformation), affine, andperspective transformations.

In a local transformation model, each parameter acts on a limitedsupport in the image domain. Such transformation models enable toestimate non-rigid transformations for instance to map two differentsubjects. Two local transformation models are particularly popularin the medical image registration literature:

B-Spline transformation as proposed by Rueckert et al.35 In this model,the displacement is represented as a sum of B-Spline basis func-tions. Each three dimensional B-Spline is obtained by a tensor prod-uct of one dimensional B-Splines. Such basic functions offers theadvantages of having a limited support (this property can fastenthe computation of metric derivatives since one parameter actson a restricted region) and a reasonably low computational cost.



B-Splines also provide a natural regularization of the displacementfield since, at each voxel, the displacement is a weighted sum of thedisplacement at neighbor control points. A drawback of B-Splines itthat it uses a regular grid (even if Rohde et al.37 selects active nodesin a regular grid) which does not necessarily fit the geometry of theobjects in the scene.

Finite Elements (FE) models divide the volume of interest in polygo-nal elements (hexahedral or tetrahedral elements). The main advan-tage of this representation is its ability to sample irregularly thevolume of interest. Faces of the mesh elements can track anatomicalborders of structures of interest in the image. Finite element mod-els have been applied to model cardiac deformations (see Refs. 68,69 and 70). Montagnat and Delingette71 introduced time dependentconstrains in deformable models of the hearth. Finite element mod-els are an efficient strategy to propagate sparse displacements to theentire image. Ferrant et al.72 estimates boundary displacements ofthe brain surface and ventricles and propagates them to the entirevolume to estimate the brain shift occurring during surgery. Cashet al.16 applied a similar strategy for liver surgery where the vol-umetric displacement is computed by applying displacements onexposed liver surface to the entire model. Surface displacements aretracked by a laser range scanner. FE models also offer the possibilityof including various elasticity properties to elements belonging todifferent anatomical structures. This feature enables adaptive reg-ularization strategy if it is a priori known that some structures areexpected to deform more than others.

Aside from this two main types of deformation models, Arsignyet al.73 proposed an infinitesimal approach to compose locally affinetransformations into an invertible transformation. The resultingclass of transformations is called polyrigid or poly-affine transfor-mations. Other transformation models aim at modelling the effectof a specific pathology. An example is the atlas-based segmenta-tion of patients with tumors. Since the atlas does not contain anyrepresentation of the lesion, a specific transformation model has tobe designed for propagating the radial transformation of a seed to



the entire brain to simulate repulsion effects induced by the tumor.74

Some strategies used an optical flow algorithm after introducing aseed by assigning to a small set of voxels an intensity similar to theone of the tumor. Bach et al.75 proposed the joint use of two deforma-tion models, a typical optical flow force to drive the deformation out-side the tumor and a radial growth inside the tumor. Kyriacou andDavatzikos76 proposed a biomechanical model of the brain usinga finite-element method. They first model soft tissue deformationsinduced by the tumor growth and, then, they registered the anatom-ical atlas with a transformed patient image from which the tumorwas removed.

20.4 REGULARIZATION AND PRIOR KNOWLEDGE

The space of nonrigid transformations explored by an automaticregistration algorithm might generate discontinuous transforma-tions (in the case of a dense displacement field for instance)or noninvertible transformations. Various strategies have beendesigned to constrain the space of allowed transformations.

A first category of methods controls the magnitude of the trans-formation jacobian, defined for each voxel x as ∂T(x)/∂x. Rohdeet al.37 proposed a set of conditions on the jacobian of the updatedisplacement field (computed at each iteration) that ensures posi-tiveness of the jacobian of the overall cumulated displacement field.A more common strategy is to add a penalty term to the cost func-tion based on the transformation jacobian unity.77,78 This constraintypically enforces the determinant jacobian to get globally close to1 which is equivalent to impose that the involved tissues behaveas incompressible materials.79 The assumption of incompressibil-ity holds for small deformations or short time periods79 but mightbecome unrealistic in some applications like cardiac displacementstracking. Haber and Modersitzki80 proposed to relax the unity con-strain by admitting a range of values close to 1. In this work, alogbarrier method optimization scheme is used to progressivelyreduce the weight of the jacobian constrain in the global objective



function over the iterations. It is expected that such constrained opti-mization strategies will gain growing interest in the image registra-tion community even if they are more expensive than conventionalapproaches.

A second category of regularization term is based on defor-mation energies derived from mechanics like, for instance, elasticenergy.81 These regularization terms are mathematically more com-plex that the transformation jacobian but they offer the possibil-ity to model heterogeneous materials. A typical example in brainmatching is the attribution of different mechanical properties to theventricles (like different Young Modulus72). Davatzikos82 assigns anonzero initial strain to the ventricles to cause a uniform ventricu-lar expansion. The value of this initial strain is determined by theventricle volumes in the two images. Elastic regularization can alsomodel anisotropic materials like ventricle fibers in the heart68 forproducing realistic strain maps of the left ventricle. Image deforma-tion has also been modeled as a viscous fluid whose motion is gov-erned by Navier-Stokes equation of conservation of momentum.83,84

Christensen83 solves Navier-Stokes equation by successive overrelaxation. A simpler approach84 to model fluid regularization con-sists of convoluting the dense force field by a Gaussian kernel toproduce a velocity field. Velocities are then iteratively accumulatedto form the global displacement field. Mechanical regularizationhas also been coupled with statistical knowledge over the space ofpossible transformations. Wang et al.85 has introduced a Bayesianformulation to couple image matching, mechanical and boundarystatistical shape model forces. Mohamed et al.86 proposed a frame-work for generating, for each patient, a statistical model of tumormass-effect deformations. The global transformation has two com-ponents, one representing interindividual differences in brain shape,and the other representing tumor induced deformation.

A third category of algorithm introduces a symmetric similar-ity cost function.87 A symmetric cost function depends on forwardand reverse transformations between the two images. Optimizingsuch metric provides a bidirectional mapping but gives no guaran-tee that the composition of forward and backward transformations



Fig. 5. Projection anatomical structures (ventricles, thalamus and central nuclei)in a reference subject (atlas) on a patient with tumor after mutual information flowmatching. Contours of the tumor are obtained using an independent segmenta-tion method. Atlas-segmentation is typically less accurate but topologically morecoherent than intensity-based classification techniques segmentation methods thatdo not use any prior information about a reference anatomy. Therefore, severalmethods attempt to unify these two strategies by using an atlas in an expectation-minimization framework124 or for adding localization information channels to theintensity channel.125

outputs an identity transformation. Christensen and Johnson87 com-putes the inverse of backward and forward transformations andintroduces a penalty term in the cost function. Rogelj and Kovavic88

uses an action-reaction principle to compose image matching forcesobtained by permuting fixed and moving images so that the sum ofthese forces is summed to zero.

These three categories do not cover the whole extent of pub-lished regularization strategies. One could mention regularization



based on the squared norm of the transformation Laplacian35 alsoreferred as curvature-base regularization.89 An interesting problemis the regularization of the global displacement field when one of theimages contains partial data. Periaswamy and Farid90 proposes anEM approach to account for partial data where each pixel is charac-terized by a probability to be an outlier. The similarity cost functionuses these probabilities to weight the contribution of the differentvoxels. Elastic regularization (already mentioned above) is also aclassical strategy to propagate sparse boundary displacements tothe entire image volume.14,16

20.5 ATLAS CONSTRUCTION

20.5.1 Individual Atlases

Over the last century, the construction of anatomical atlases has beenessential to understanding anatomy. The first atlases have been builtfor the brain from histological slices and were two dimensional only.With the emergence of threedimensional imaging modalities such asmagnetic resonance imaging, research has been aimed at developing3D digital atlases.

The first digital atlases have been focusing primarily on thehuman brain91,92 and were built from a single subject.

The work of Talairach and Tournoux92 has produced a3-dimensional, reference system for comparing brains of differentsizes in which the brain is organized into areas delimited by anatom-ical planes. Talairach and others have further defined proportion-ality rules that map coordinates extrapolated from the subject toatlas coordinates. The surgical planning laboratory atlas93 defines amore complex subdivision of the brain volume obtained by a man-ual labeling. 3D models are then computed from the label maps andused for visualization and educational purposes.

Brain atlases have been used to project functional informationpresent in the atlas on a subject. For instance, Rasser et al.94 devel-oped a nonlinear registration technique to project the Broadmannareas onto 3D coregistered functional MRI datasets. Their technique



Fig. 6. Comparison between global and local deformation models. The top rowshow rigid, affine, and prospective transformations. For such transformations, eachparameter of the transformation affects the entire image domain. In the case oflocal deformations (like B-Splines), each parameter has a limited support and actson a limited part of the image domain. This property can be used for an efficientcomputation of the metric gradient, since the modification of one transformationparameter only requires to recompute the contribution of a few voxels to the metric.

uses an elastic surface matching for matching landmarks basedon sulci and hemisphere boundary or margins. Nowinski andThirunavuukarasuu95 have used a brain atlas to assist a method forlocalization analysis of functional images. They use an enhanced andextended electronic Talairach-Tournoux brain atlas. Dinov et al.96

used a probabilistic atlas for incorporating a priori anatomical infor-mations in a subvolume thresholding method for the analysis ofPET (Positron Emission Tomography) and SPECT (single photonemission computed tomography) images. The atlas is also used todetermine the statistical significance of the effects of motor stimuluson brain perfusion.

Although atlases based on a single subject provide a standard ref-erence, they fail to illustrate the variability of anatomical structures



Fig. 7. Rigid registration algorithms often need to compensate for large offsetsbetween images acquired using different sensors. The top part of this figure showthe matching of a preoperative T1 image (left) on a intraoperative T2 MR image(middle). This matching enables to compare the image acquired during surgeryat a lower resolution with a high resolution preoperative image. It also enablesto project a preoperative planning in the physical space of the operating room.Even if this matching requires high-deformable nonrigid registration tools, rigidregistration is an initial calibration phase which affects the quality of the nonrigiddeformation model used for tumor resection.

existing among individuals since a single subject cannot representthe human variability. However, it is expected that these atlases willbe very useful for characterizing anatomical structures without largeintersubject variability.

Accurate depiction of variability, on the other hand, will helpin the automatic detection of abnormalities in a pathological image.Atlases computed from a collection of subjects may also enable adifferentiation of different populations if significant differences existin their respective anatomy. As an example of population specificatlas, Hill et al.97 proposed an approach to build customized atlases



by giving to the user the possibility to look in a database for subjectsmeeting particular criteria such as age and sex.

20.5.2 Probabilistic and Statistical Atlases

Characterizing the variability of a population under a given alignmentcan be performed using two representations. First, a probabilistic atlas,can be obtained by averaging aligned subjects to compute a meanrepresentation of image intensities or of spatial densities of the dif-ferent classes. Second, the variability stored in the transformationunder which the probabilistic atlas has been computed can be repre-sented using statistical methods like principal components analysis.Such representation enables the generation of statistical atlases i.e.atlases that encode the shape variability in a population.

20.5.2.1 Probabilistic atlases

A probabilistic atlas gives an estimate of the mean anatomy amonga population by averaging all subjects in a standard space. Collinset al.28 proposed a methodology for mapping MR images from 305subjects into the same stereostatic space. In this approach, intensitiesare normalized and averaged on a voxel-by-voxel basis. Mazziottaet al.98 presented a probabilistic brain atlas which includes bothmacroscopic and microscopic information on the function and struc-ture of the human brain. The goal is to collect information from 7 000volunteers of different ages and from different countries.

20.5.2.2 Statistical atlases

Awidely used technique to model the statistical variation of a popu-lation around a mean subject is the active shape models (ASM) repre-sentation. ASM have been introduced by Cootes et al.99 for modelingthe shape of anatomical structures by gathering statistical informa-tion from a large set of images.Active appearance models (AAM) arean extension ofASM that not only model the shape of the anatomicalfeatures but also their appearance.100



These tools have been applied to cardiac anatomy for buildingstatistical shape models,99,101–103 and statistical appearance models(e.g. active appearance models).104,105 Usually, ASM techniquesrequire a large number of landmarks to be identified in all the imagesused for the construction of the model. Automatic landmarking foreasier ASM generation has been approached using various strate-gies: polygonal correspondence,106,107 image registration techniquesfor landmarks propagation.102

Recently, statistical deformation models have been developed tomodel the anatomy of the brain.56,108 The main idea is to carry thestatistical analysis directly on the deformation fields which describea dense correspondence between the atlas and each subject. In thesemethods, the deformation fields can be obtained by nonrigid regis-tration eliminating the need for image segmentation. A strategy forbuilding cardiac motion atlas has been proposed by Rao et al.109 Inthis framework, the motion fields from different subjects are mappedinto the same coordinate system using a vector field transformationtechnique. Rougon and Petijean developed a similar approach forbuilding a statistical 3D motion MRI atlas for modeling and analyz-ing myocardial contraction.110

20.5.3 Alignment of an Image Population

The alignment of a population in a common space of coordinates isan interesting extension of the classical pairwise registration prob-lem. It requires either the generalization of pairwise similarity costfunctions to group wise similarity measures or the definition of a ref-erence in which all subjects can be projected. Two main streams canbe identified in the various strategies proposed for joint alignmentof a population.

The first stream (reference-free) captures the best group wise align-ment by optimizing a global cost function requiring the joint opti-mization of transformation parameters over the entire population.Studholme111 simultaneously aligns a group of images to a commonspace by building a joint histogram whose dimension is equal tothe number of subjects in the database. Bhatia et al.112 has proposedthe selection of one arbitrary database image to act as an intensity



reference. A joint histogram is built with pairs of intensities, eachpair comprises the voxel intensity in the reference and the intensityat the same location in each subject. The transformation parame-ters among all subjects are optimized at the same time to maximizethe normalized entropy computed from the joint histogram. Evenif these authors demonstrate that such algorithms are suitable forgroup wise alignment, the curse of dimensionality (in the joint his-togram or optimization space dimensions) may induce convergenceproblems for large databases. Zollei et al.113,114 have proposed a con-gealing approach for dealing with the reference-free alignment oflarge databases. In their scheme, they minimize the sum of per-voxel entropy by considering separately the contribution of eachsubject to this global cost function. Finally, Van Essen115 has pro-posed to extract the cortical surface of each subject and to inflatethese surfaces to a standard spherical configuration. A reference iscomputed in the spherical system of coordinates by averaging con-tours of selected sulci. Each subject is then mapped to this targetusing landmark-constrained registration.

The second stream (reference-based) of group wise alignmentalgorithms maps a single reference subject to the population beingstudied (see for instance Refs. 116–118). A statistical analysis of thetransformations can then be used to characterize different popula-tions as performed in Refs. 19, 119 and 120.

Within these two categories of algorithms, reference-free andreference-based, most of the authors have focused on the challengeof defining a common space of coordinates that approximates thatof an average subject. Indeed, the choice of a non-representative ref-erence would introduce a bias in the atlas. By bias, we mean the pro-jection of the population into a common space of coordinates whichdoes not reflect a central tendency in the database, creating this wayunnecessary complicated registration problems. If the constructedatlas is biased and depicts peculiar anatomical features for the popu-lation being studied, there is a risk that when normalizing a sampleof individual brains to this space, systematic deformations couldarise, which, in turn, could affect statistical or functional studies.121

Both in the reference-free and reference-based categories, variousapproaches have been proposed that would project all population



subjects to a common space of coordinates that represents best thepopulation average subject.

First, it is possible to generate a mean transformation by sum-ming all atlas-to-subjects transformations. The magnitude of thismean transformation could then be used as an initial measure of thebias introduced by the current reference. In the reference-free cate-gory, both Studholme111 and Bhatia112 have used this measure of thebias. Studholme111 incorporates into the cost function a term penaliz-ing large atlas to subjects transformations. Bhatia et al.112 explicitlyenforces at each iteration the sum of atlas to subjects transforma-tions to be zero. In the reference-based category, Guimond et al.122

and Rueckert et al.56 have used a similar approach. In their work,they picked one of the subjects in the population subject as a ref-erence and then nonrigidly align all other subjects to this subject.After alignment, all subjects are intensity averaged. Lastly, the meantransformation of all atlas-to-subjects transformations is applied tothe sum of aligned subjects, thus removing the bias introduced bythe choice of the initial reference.

Another approach, specific to the reference-based category, is toconstruct at each iteration a reference reflecting the mean anatomy ofthe population under the current alignment. In this scheme, the groupwise alignment algorithm iterates between the computation of thereference and the alignment of all subjects on this reference. Mars-land et al.123 iteratively updates the chosen reference by selecting asreference the subject minimizing the sum of distances between itselfand the other ones. Joshi et al.120 has proposed the arithmetic meanof all subjects as choice of reference. All subjects are successivelyaligned to the reference by an optical flow technique. After havinggone through all subjects, the mean is recomputed and becomes thenew reference for the next iteration.

20.6 CONCLUSION

In this chapter, we have put in perspective the classical prob-lem of pairwise registration and the joint alignment of an image



population. The joint alignment of an image population of subjectsnaturally extends the challenges already encountered in pairwiseregistration. Common challenges to these two registration prob-lems encompass the efficient representation of smooth large non-rigid deformations, the ability to deal with multimodal or noisyimages, the inclusion of a priori physical or statistical knowledgeabout the transformation, the modeling of pathology-induced defor-mations that can create or destroy some structures (like lesions ortumors) etc.

Challenges specific to the alignment of a population of subjectsare the definition of a central and unbiased reference that is mostrepresentative of the population average subject, the extension ofpairwise similarity cost functions to group wise metrics, and thestatistical representation of the variability encoded in the alignedpopulation and the set of transformations. Still, a lot of open ques-tions remain unsolved and require additional research efforts in thisfield: How identify subgroups automatically when more than onereference is required to depict the variability in the population? Howto quantify confidence intervals of normal anatomical variabilityto automatically detect pathologies? How to integrate informationacquired at different scales in a common atlas? Such new challengesare expected to motivate the development of new registration tech-niques and foster this promising and exciting field of research.


This work was partially performed in the framework of the Inte-grated Project @neurIST (IST-2005-027703), which is cofinanced bythe European Commission. This work was also partially supportedby grants MEC TEC2006-03617, ISCIII FIS2004/40676, and CDTICENIT-CDTEAM. The work of AFF is supported by the Span-ish Ministry of Education and Science under a Ramon y CajalResearch Fellowship. The CILab is part of the ISCIII CIBER-BBN(CB06/01/0061) and also acknowledges financial support fromPhilips Medical Systems BV.



References

1. Bookstein FL, Principal warps: Thin-plates splines and the decom-position of deformations, IEEE Transactions on Pattern Analysis andMatching Intelligence 11(6): 567–585, 1989.

2. Horn BKP, Schunck BG, Determining optical flow Artificial Intelligence17(1–3): 185–203, 1981.

3. Thirion JP, Image matching as a diffusion process: An analogy withmaxwell’s demons, Medical Image Analysis 2(3): 243–260, 1998.

4. Lavely WC, Scarfone C, Cevikalp H, et al., Phantom validation ofcoregistration of PET and CT for image guided radiotherapy, MedicalPhysics 31: 1083–1092, 2004.

5. Rizzo G, Cattaneo GM, Castellone P, Castiglioni I, et al., Multimodalmedical image integration to optimize radiotherapy planning in lungcancer treatment, Annals of Biomedical Engineering 32(10): 1399–1408,2004.

6. Charnoz A, Agnus V, Malandain G, Forest C, et al., Liver registrationfor the follow-up of hepatic tumors, In Medical Image Computing andComputer-Assisted Intervention 8: 155–162, IRCAD R&D, Strasbourg,France, 2005.

7. Yan J, Zhao B, Curran S, Zelenetz A, et al., Automated matching andsegmentation of lymphoma on serial CT examinations, Medical Physics34(1): 55–62, 2007.

8. Shen D, 4D image warping for measurement of longitudinal brainchanges, in International Symposium on Biomedical Imaging, pp. 904–907, 2004.

9. Perperidis D, Mohiaddin RH, Rueckert D, Spatiotemporal free-formregistration of cardiac MR image sequences, Medical Image Analysis9(5): 441–456, 2005.

10. Chandrashekara R, Mohiaddin RH, Rueckert D, Cardiac motiontracking in tagged MR images using a 4D b-spline motion model andnonrigid image registration, in International Symposium on BiomedicalImaging 1: 468–471, 2004.

11. Maurer CR Jr, Fitzpatrick JM, Wang MY, Galloway Jr RL, et al., Reg-istration of head volume images using implantable fiducial markers,IEEE Transactions on Medical Imaging 16(4): 447, 1997.

12. Sra J, Registration of three dimensional left atrial images with inter-ventional systems, Heart 91(8): 1098–1104, 2005.

13. Skrinjar O, Spenser D, Duncan J, Brain shift modeling for use in neu-rosurgery, in Medical Image Computing and Computer-Assisted Interven-tion, pp. 641–649, 1998.



14. Ferrant M, Nabavi A, Macq B, Black PmcL et al., Serial registrationof intraoperative MR images of the brain, Medical Image Analysis 6(4):337–359, 2002.

15. Clatz O, Delingette H, Talos IF, Golby AJ, et al., Robust nonrigid reg-istration to capture brain shift from intraoperative MR, IEEE Transac-tions on Medical Imaging 24(11): 1417–1427, 2005.

16. Cash DM, Miga MI, Sinha TK, Galloway RL, et al., Compensatingfor intraoperative soft-tissue deformations using incomplete surfacedata and finite elements, IEEE Transactions on Medical Imaging 24(11):1479–1491, 2005.

17. Bookstein FL, Voxel-based morphometry should not be used withimperfectly registered images, Neuroimage 14(6): 1454–1462, 2001.

18. Liao R, Krolik JL, Mckeown MJ, An information-theoretic criterion forintrasubject alignment of FMRI time series: Motion corrected indepen-dent component analysis, IEEE Transactions on Medical Imaging 24(1):29–44, 2005.

19. Csernansky JC, Joshi S, Wang L, Gado M, et al., Hippocampal mor-phometry in schizophrenia by high dimensional brain mapping, Pro-ceedings of the National Academy of Science, 11406–11411, 1998.

20. Marai GE, Laidlaw DH, Crisco JJ, Super-resolution registration usingtissue-classified distance fields, IEEE Transactions on Medical Imaging25(2): 177–187, 2006.

21. Rohr K, Stiehl HS, Sprengel R, Buzug TM, et al., Landmark-based elas-tic registration using approximating thin-plate splines, IEEE Transac-tions on Medical Imaging 20(6): 526–534, 2001.

22. Maintz JBA, Viergever MA, A survey of medical image registration,Medical Image Analysis 2(1): 1–36, 1998.

23. Unser M, Thevenaz P, Lee C, Ruttimann UE, Registration and statis-tical analysis of PET images using the wavelet transform, IEEE Engi-neering in Medicine and Biology 14(5): 603–611, 1995.

24. Christensen GE, Miller MI, Marsh JL, Vannier MW, Automatic analy-sis of medical images using a deformable textbook, Computer AssistedRadiology, 146–151, 1998.

25. Haller JW, Hippocampal MR imaging morphometry by means of gen-eral pattern matching, Radiology 199(3): 787–791, 1996.

26. Maintz JBA, van den Elsen PA, Viergever MA, Comparison of edge-based and ridge-based registration of CT and MR brain images, Med-ical Image Analysis 1(2): 151–161, 1996.

27. Studholme C, Hill DLG, Hawkes DJ, Automated 3D MR and PETbrain image registration, Computer Assisted Radiology, pp. 248–253,1995.



28. Collins DL, Neelin P, Peters TM, EvansAC,Automatic 3D intersubjectregistration of MR volumetric data in standardized talairach space,Journal of Computer Assisted Tomography 18(2): 192–205, 1994.

29. Maes F, Collignon A, Vandermeulen D, Marchal G, et al., Multimodal-ity image registration by maximization of mutual information, IEEETransactions on Medical Imaging 16(2): 187–198, 1997.

30. Viola P, Wells WM, Alignment by maximization of mutual informa-tion, in Fifth International Conference on Computer Vision, pp. 16–23,1995.

31. Zuo CS, Jiang A, Buff BL, Mahon TG, et al., Automatic motion correc-tion for breast MR imaging, Radiology 198(3): 903–906, 1996.

32. Hill DLG, Hawkes DJ, Harrison NA, Ruff CF, A strategy for auto-mated multimodality image registration incorporating anatomicalknowledge and imager characteristics, in Barrett HH, GmitroAF, eds.,Information Processing in Medical Imaging, Volume 687, pp. 182–196,Springer-Verlag, Germany, 1993.

33. Pluim JPW, Maintz JBA, Viergever MA, Mutual-information-basedregistration of medical images: A survey, IEEE Transactions on MedicalImaging 22(8): 986–1004, 2003.

34. Collignon A, Vandermeulen D, Suetens P, Marchal G, 3D multimodal-ity medical image registration using feature space clustering, in NAyache ed., Computer Vision, Virtual Reality, and Robotics in Medicine,Volume 905 of Lecture Notes in Computer Science, pp. 195–204. Springer-Verlag, Germany, 1995.

35. Ruekert L, Sonoda I, Hayes C, et al., Nonrigid registration using free-form deformations: Applications to breast MR images, IEEE Transac-tions on Medical Imaging 18(8): 712–721, 1999.

36. Mattes D, Haynor DR, Vesselle H, Lewellen TK, et al., PET-CT imageregistration in the chest using free-form deformations, IEEE Transac-tions on Medical Imaging 22(1): 120–128, 2003.

37. Rohde GK, Aldroubi A, Dawant BM, The adaptive bases algorithmfor intensity-based nonrigid image registration, IEEE Transactions onMedical Imaging 22(11): 1470–1479, 2003.

38. Studholme C, Hill DLG, Hawkes DJ, Overlap invariant entropy mea-sure of 3D medical image alignment, Pattern Recognition 32(1): 71–86,1999.

39. Butz T, Thiran JP, From error probability to information theoretic (mul-timodal) signal processing, Signal Processing 85(5): 875–902, 2005.

40. Pluim JP, Maintz JB, Viergever MA, F-information measures in med-ical image registration, IEEE Transactions on Medical Imaging 23(12):1508–1516, 2004.



41. Chung ACS, Wells III WM, Norbash A, Grimson WEL, Multimodalimage registration by minimising Kullback-Leibler distance, in Med-ical Image Computing and Computer-Assisted Intervention 2: 525–532,Springer, 2002.

42. Shen D, Davatzikos C, Hammer: Hierarchical attribute matchingmechanism for elastic registration, IEEE Transactions on Medical Imag-ing 21(11): 1421–1439, 2002.

43. Shen D, Davatzikos C, Very high-resolution morphometry usingmasspreserving deformations and hammer elastic registration, Neu-roimage 18(1): 28–41, 2003.

44. Xu P, Yao D, A study on medical image registration by mutual infor-mation with pyramid data structure, Computers in Biology and Medicine37(3): 320–327, 2007.

45. Oubel E, Frangi AF, Hero AO, Complex wavelets for registration oftagged MRI sequences, International Symposium on Biomedical Imaging,pp. 622–625, 2006.

46. Neemuchwala HF, Hero AO, Multisensor Image Fusion and its Applica-tions, Chapter Entropic Graphs for Registration, Marcel-Dekker Inc,2004.

47. Neemuchwala H, Hero AO, Carson PL, Image matching usingalphaentropy measures and entropic graphs, Signal Processing 85: 277–296, 2005.

48. Neemuchwala HF, Entropic Graphs for Image Registration, PhD Thesis,The University of Michigan, 2005.

49. Cao Y, Miller MI, Winslow RL, Younes L, Large deformation diffeo-morphic metric mapping of vector fields, IEEE Transactions on MedicalImaging 24(9): 1216–1230, 2005.

50. Ruiz-Alzola J, Westin CF, Warfield SK, Alberola C, et al., Nonrigidregistration of 3D tensor medical data, Medical Image Analysis 6(2):143–161, 2002.

51. Pluim JP, Maintz JB, Viergever MA, Mutual-information-based reg-istration of medical images: A survey, IEEE Transactions on MedicalImaging 22(8): 986–1004, 2003.

52. Wells WM, Viola P, Atsumi H, Nakajima S, et al., Multi-modal volumeregistration by maximization of mutual information, Medical ImageAnalysis 1(1): 35–51, 1996.

53. Collignon A, Maes F, Delaere D, et al., Automated Multimodality ImageRegistration Using Information Theory, pp. 263–274, 1995.

54. D’Agostino E, Maes F, Vandermeulen D, Suetens P, An informationtheoretic approach for non-rigid image registration using voxel classprobabilities, Medical Image Analysis 10(3): 413–431, 2006.



55. Hermosillo G, Chefd’Hotel C, Faugeras O, Variational methods formultimodel image matching, International Journal of Computer Vision50(3): 329–343, 2002

56. Rueckert D, FrangiAF, Schnabel JA,Automatic construction of 3D sta-tistical deformation models of the brain using nonrigid registration,IEEE Transactions on Medical Imaging 22(8): 1014–25, 2003.

57. Nocedal J, Wright SJ, Numerical Optimization, Springer-Verlag, 2000.58. Beasley D, Bull DR, Martin RR, An overview of genetic algorithms:

Part 1, fundamentals, University Computing 15(2): 58–69, 1993.59. Brent RP, Algorithms for minimization without derivatives, 3–4,

Prentice-Hall, 1973.60. Goldberg DE, Genetic Algorithms in Search, Optimization, and Machine

Learning, Addison-Wesley Publishing Company, Reading, MA, 1989.61. Spall JC, Overview of the simultaneous perturbation method for

effcient optimization, http://techdigest.jhuapl.edu/td/td1904/spall.pdf,Hopkins APL Technical Digest 19: 482–492, 1998.

62. Vemuri BC, Ye J, Chen Y, Leonard CM, Image registration via lev-elset motion: Applications to atlas-based segmentation, Medical ImageAnalysis 7(1): 1–20, 2003.

63. Joshi SC, Miller MI, Landmark matching via large deformation diffeo-morphisms, IEEE Transactions on Imaging Processing 9(8): 1357–1370,2000.

64. Joshi S, Lorenzen P, Gerig G, Bullitt E, Structural and radiometricasymmetry in brain images, Medical Image Analysis 7(2): 155–170, 2003.

65. Avants BB, Schoenemann PT, Gee JC, Lagrangian frame diffeomor-phic image registration: Morphometric comparison of human andchimpanzee cortex, Medical Image Analysis 10(3): 397–412, 2006.

66. De Craene M, du Bois d’AischeA, Macq B, Warfield SK, Incorporatingmetric flows and sparse jacobian transformations in ink, The InsightJournal, January–June 2006.

67. Spall JC, Adaptive stochastic approximation by the simultaneous per-turbation method, IEEE Transactions on Automatic Control 45: 1839–1853, 2000.

68. Papademetris X, Sinusas A, Dione D, Constableand RT, et al., Estima-tion of 3D left ventricular deformation from medical images usingbiomechanical models, IEEE Transactions on Medical Imaging 21(7):786–799, 2002.

69. Sermesant M, Forest C, Pennec X, Delingette H, et al., Deformablebiomechanical models: Application to 4D cardiac image analysis,Medical Image Analysis 7(4): 475–488, 2003.



70. Kaus MR, Berg JV, Weese J, Niessen W, et al.,Automated segmentationof the left ventricle in cardiac MRI, Medical Image Analysis 8(3): 245–254, 2003.

71. Montagnat J, Delingette H, 4D deformable models with temporal con-straints: Application to 4D cardiac image segmentation, Medical ImageAnalysis 9(1): 87–100, 2005.

72. Ferrant M, Nabavi A, Macq B, et al., Registration of 3D intraoperativeMR images of the brain using a finite element biomechanical model,IEEE Transactions on Medical Imaging 20: 1384–1397, 2001.

73. Arsigny V, Pennec X, Ayache N, Polyrigid polyaffine transforma-tions: Anovel geometrical tool to deal with non-rigid deformations —Application to the registration of histological slice, Medical ImageAnalysis 9(6): 507–523, 2005.

74. Dawant BM, Hartmann SL, Gadamsetty S, Brain atlas deformationin the presence of large space-occupying tumors, in Medical ImageComputing and Computer-Assisted Intervention, pp. 589–596, 1999.

75. Bach Cuadra M, Pollo C, Bardera A, et al., Atlas-based segmentationof pathological MR brain images using a model of lesion growth, IEEETransactions on Medical Imaging 23(10): 1301–1314, 2004.

76. Kyriacou SK, Davatzikos C, Nonlinear registration of brain imageswith tumor pathology using a biomechanical model, IEEE Transactionson Medical Imaging 18: 580–592, 1999.

77. Christensen GE, Consistent linear-elastic transformations for imagematching, in Information Processing in Medical Imaging, pp. 224–237,Springer, 1999.

78. Droske M, Rumpf M,Avariational approach to non-rigid morpholog-ical registration, SIAM Journal on Applied Mathematics 64(2): 668–687,2004.

79. Rohlfing T, Maurer Jr CR, Bluemke DA, Jacobs MA, Volume-preserving nonrigid registration of MR breast images using free-formdeformation with an incompressibility constraint, Medical Imaging,IEEE Transactions 22(6): 730–741, 2003.

80. Haber E, Modersitzki J, Image registration with guaranteed displace-ment regularity, International Journal of Computer Vision 71(3): 361–372,2007.

81. Bajcsy R, Kovacic S, Multiresolution elastic matching, Computer Vision,Graphics and Image Processing 46: 1–21, 1989.

82. Davatzikos C, Spatial transformation and registration of brain imagesusing elastically deformable models, Computer Vision and Image Under-standing 66(2): 207–222, 1997.



83. Christensen GE, Rabbitt RD, Miller MI, Deformable templates usinglarge deformation kinematics, IEEE Transactions on Medical Imaging5(10): 1435–1447, 1996.

84. D’Agostino E, Maes F, Vandermeulen D, Suetens P, A viscous fluidmodel for multimodal non-rigid image registration using mutualinformation, Medical Image Analysis 7(4): 565–575, 2003.

85. Wang Y, Staib LH, Physical model-based non-rigid registration incor-porating statistical shape information, Medical Image Analysis 4(1):7–20, 2000.

86. Mohamed A, Zacharaki EI, Shen D, Davatzikos C, Deformable regis-tration of brain tumor images via a statistical model of tumor-induceddeformation, Medical Image Analysis 10(5): 752–763, 2006.

87. Christensen GE, Johnson HJ, Consistent image registration, IEEETransactions on Medical Imaging 20(7): 568–582, 2001.

88. Rogelj P, Kovacic S, Symmetric image registration, Medical Image Anal-ysis 10(3): 484–493, 2006.

89. Fischer B, Modersitzki J, Curvature based image registration, Journalof Mathematical Imaging and Vision 18(1): 81–85, 2003.

90. Periaswamy S, Farid H, Medical image registration with partial data,Medical Image Analysis 10(3): 452–464, 2006.

91. Schaltenbrand G, Bailey P, Introduction to Stereotaxis with an Atlas ofthe Human Brain, Georg Thieme Verlag, 1959.

92. Talairach J, Tournoux P, Coplanar stereotaxic atlas of the human brain:3-dimensional proportional system — An approach to cerebral imag-ing, Thieme Medical Publishers, New York, 1998.

93. Kikinis R,Adigital brain atlas for surgical planning, model driven seg-mentation and teaching, IEEE Transactions on Visualization and Com-puter Graphics 2(3): 232–241, 1996.

94. Rasser PE, Johnston PJ, Ward PB, Thompson PM, A deformable brod-mann area atlas, in International Symposium on Biomedical Imaging 1:400–403, 2004.

95. Nowinski WL, Thirunavuukarasuu A, Atlas-assisted localizationanalysis of functional images, Medical Image Analysis 5(3): 207–220,2001.

96. Dinov ID, Mega MS, Thompson PM, Lee L, et al., Analyzing func-tional brain images in a probabilistic atlas: A validation of subvolumethresholding, Journal of Computer Assisted Tomography 24(1): 128–138,2000.

97. Hill DLG, Hajnal JV, Rueckert D, Smith SM, et al., A dynamicbrain atlas, in Dohi T, Kikinis R (eds.), Medical Image Computing andComputer-Assisted Intervention, Volume 2489 in Lecture Notes in Com-puter Science, pp. 532–539 (part I), Tokyo (Japan), 2002.



98. Mazziotta J, Toga A, Evans A, Fox P, et al., A four-dimensional proba-bilistic atlas of the human brain, Journal of the American Medical Infor-matics Association 8(5): 401–430, 2001.

99. Cootes TF, Taylor CJ, Cooper DH, Graham J, Active shape models:Their training and application, Computer Vision and Image Understing61(1): 38–59, 1995.

100. Cootes TF, Taylor CJ, Statistical models of appearance for medicalimage analysis and computer vision, in Sonka M, Hanson KM (eds.),Proc. SPIE 4322: 236–248, Medical Imaging 2001: Image Processing, 236–248, 2001.

101. Cootes TF, Hill A, Taylor CJ, Haslam J, The use of active shape modelsfor locating structures in medical images, Image and Vision Computing12(6): 355–366, 1994.

102. Frangi AF, Rueckert D, Schnabel JA, Niessen WJ, Automatic construc-tion of multiple-object three-dimensional statistical shape models:Application to cardiac modeling. IEEE Transactions on Medical Imaging21(9): 1151–1166, 2002.

103. Lötjonen J, Kivistö S, Koikkalainen J, Smutek D, et al., Statistical shapemodel of atria, ventricles and epicardium from short- and long-axisMR images, Medical Image Analysis 8(3): 371–386, 2004.

104. Mitchell SC, Bosch JG, Lelieveldt BPF, et al., 3D active appearancemodels: Segmentation of cardiac MRI and ultrasound images, IEEETransactions on Medical Imaging 21(9): 1167–1178, 2002.

105. Uzumcu M, Frangi AF, Sonka M, Reiber JHC, et al., ICA vs PCAactive appearance models: Application to cardiac MR segmenta-tion, in Medical Image Computing and Computer-Assisted Intervention,Volume 2878 in Lecture Notes in Computer Science, pp. 451–458,2003.

106. Hill A, Taylor CJ, Automatic landmark generation for point distribu-tion models, in Proceedings of the Conference on British Machine VisionVolume 2, pp. 429–438, Surrey, UK, 1994.

107. Brett AD, Taylor CJ, Method of automated landmark generation forautomated 3D PDM construction, Image and Vision Computing 18(9):739–748, 2000.

108. Gee JC, Bajcsy RKS, Brain Warping, Chapter Elastic Matching: Contin-uum Mechanical and Probabilistic Analysis, pp. 183–197. AcademicPress, 1999.

109. RaoA, Sanchez-Ortiz GI, Chandrashekara R, Lorenzo-Valdes M, et al.,Comparison of cardiac motion across subjects using non-rigid regis-tration, in Medical Image Computing and Computer-Assisted Interven-tion, Volume 2488, Lecture Notes in Computer Science, pp. 722–729,Springer, 2002.



110. Rougon N, Petitjean C, Ph. Cluzel, Prteux F, et al., A non rigid reg-istration approach for quantifying myocardial contraction in taggedMRI using generalized information measures, Medical Image Analysis9(4): 353–375, 2005.

111. Studholme C, Simultaneous population based image alignment fortemplate free spatial normalisation of brain anatomy, in Workshop onBiomedical Image Registration, pp. 81–90, 2003.

112. Bhatia KK, Hajnal JV, Puri BK, Edwards AD, et al., Consistent group-wise non-rigid registration for atlas construction, in International Sym-posium on Biomedical Imaging, pp. 908–911, 2004.

113. Zollei L, Learned-Miller E, Grimson E, Wells III WM, Efficient pop-ulation registration of 3D data, in Workshop on Computer Vision forBiomedical Image Applications, Lecture Notes in Computer Science,pp. 291–301, ICCV, 2005.

114. Zollei L, A Unified Information Theoretic Framework for Pair- and Group-wise Registration of Medical Images, PhD thesis, Massachusetts Instituteof Technology Computer Science and Artificial Intelligence Labora-tory, 2006.

115. Van Essen DC, A population-average, landmark- and surface-based(PALS) atlas of human cerebral cortex, Neuroimage 28(3): 635–662,2005.

116. Miller MI, Joshi SC, Christensen GE, Large deformation fluid dif-femorphisms for landmark and image matching, Brain Warping, Aca-demic Press, 1999.

117. Toga AW, Brain Warping, Academic Press, 1999.118. Dinggang S, Davatzikos C, Hammer, hierarchical attribute matching

mechanism for elastic registration, IEEE Transactions on Medical Imag-ing 21(11): 1421–1439, 2002.

119. Thompson PM, Moussai J, Zohoori S, Goldkorn A, et al., Corticalvariability and asymmetry in normal aging and Alzheimer’s disease,Cerebral Cortex 8(6): 492–509, 1998.

120. Joshi S, Davis B, Jomier M, Gerig G, Unbiased diffeomorphic atlasconstruction for computational anatomy, Neuroimage 23(Suppl. 1),S151–S160, 2004.

121. Diedrichsen J, A spatially unbiased atlas template of the human cere-bellum, Neuroimage 33(1): 127–138, 2006.

122. Guimond A, Meunier J, Thirion JP, Average brain models: A conver-gence study, Computer Vision and Image Understanding 77(2): 192–210,1999.

123. Marsland S, Twining C, Taylor C, Groupwise non-rigid registrationusing polyharmonic clamped-plate splines, in Medical Image Analysis,Volume 2879, 771–779, Lecture Notes in Computer Science, 2003.



124. Pohl KM, Fisher J, Grimson W, Kikinis R, et al., A Bayesian model forjoint segmentation and registration, Neuroimage 31(1): 228–239, 2006.

125. Warfield SK, Kaus M, Jolesz FA, Kikinis R, Adaptive, template moder-ated, spatially varying statistical classification, Medical Image Analysis4(1): 43–55, 2000.

126. Gerardo Hermosillo, Christophe Chefd’Hotel, Olivier Faugeras, AVariational Approach to Multimodal Image Matching, Technicalreport, INRIA, 2001.

127. Hermosillo G, Chefd’Hotel C, Faugeras O, Variational methods formultimodal image matching, International Journal of Computer Vision50(3): 329–343, 2002.


CHAPTER 21

Grid Methods for Large Scale MedicalImage Archiving and Analysis

HK Huang, Zheng Zhou and Brent Liu

Grid computing is the integrated use of geographically distributed com-puters, networks, and storage systems to create a virtual computing andcommunication system environment for solving large scale, data-intensiveproblems, for example, in various medical image applications. This chap-ter uses medical imaging picture archiving and communication system(PACS), described in Sec. 1, as a means to present how grid comput-ing can be used to solve several difficult problems facing PACS clin-ical operations. In Sec. 2, we introduce grid computing fundamentalsincluding topics in the Globus 4.0 five-layered toolkit and the integra-tion of PACS DCIOM technology with the Globus. Section 3 describesthe concept of data grid and its applications in PACS. We first define theneeds of fault-tolerant PACS archive and backup, and three main tasks ofdata grid during disaster recovery. Two important new developments indata grid are the dynamic metadata database and the management sys-tem which are essential to guarantee the fault-tolerance of an enterprisePACS. Section 4 introduces grid computing which extends the data gridinfrastructure with computational services to cover certain application ori-ented computing requirements when computer-aided diagnosis (CAD) isintegrated with PACS for daily clinical operation. The CAD of multiplesclerosis of the brain on MRI is used as an example to illustrate stepsinvolved.

517


518 HK Huang, Zheng Zhou and Brent Liu

21.1 INTRODUCTION

21.1.1 Background

Grid computing is the integrated use of geographically distributedcomputers, networks, and storage systems to create a virtual com-puting system environment for solving large scale, data-intensiveproblems in science, engineering, commerce, and healthcare.1−7

A grid is a high-performance hardware and software infrastructureproviding scalable, dependable and secure access to the distributedresources. Unlike distributed computing and cluster computing,the individual resources in grid computing maintain administra-tive autonomy and are allowed system heterogeneity; this aspectof grid computing guarantees scalability and vigor. Therefore, thegrid’s resources must adhere to agreed upon standards to remainopen and scalable. A formal taxonomy, composed of five layers hasbeen defined by the Globus Toolkit 4.0 of grid computing to assurethis standardization (Fig. 1) which will be described in greater detailin Sec. 2.2.8

InternetTransport

Application

LinkOS

I in

tern

et p

roto

col a

rch

itec

ture Application

Fabric

Connectivity

Resource

Collective

Grid

Computers, networks, storage system, instruments, etc.

Communicates between resources

Controls access & monitors data grid resources

Finds resources within grid

Interface of inputs to data grid

Layered Grid Architecture

Fig. 1. The five layer grid architecture defined by the globus toolkit 4.0: fabric, con-nectivity, resource, collective, and application. The left-hand side depicts its corre-spondence to the open system interconnection (OSI) seven-layer internet protocol.


Grid Methods for Large Scale Medical Image Archiving and Analysis 519

21.1.2 Large-Scale Medical Imaging Systems — PACS

PACS (Picture Archiving and Communication system) is a largescale integral imaging system responsible for 24/7 clinical diagnosticoperation in a healthcare delivery system.9 We use it to illustrate theusefulness of grid computing in medical image application. Two cur-rent research topics are discussed, the fault-tolerance image archiveand recovery after disaster,10 and the integration of computer-aideddiagnosis (CAD) with PACS.11 These two applications explain theconcepts of data grid and grid computing in large scale medicalimaging systems.

A PACS is a system integration of computers, servers, work-stations, communication networks, and software to form a systemfor medical image information archive, distribution, and display. Itconsists of the following components:

• A data acquisition gateway connected to the radiology informa-tion system (RIS) and the hospital information system (HIS) foracquiring patient and examination related data.

• An array of image acquisitions gateways connected to variousmedical imaging modalities including light image sensors, filmdigitizer, computed radiography (CR), digital radiology (DR), dig-ital mammography (DM), ultrasonic (US), computed tomography(CT), magnetic resonance image (MRI), Single Photon Emissiontomography (SPECT), and Positron Emission Tomography (PET).

• A PACS controller and archive server including storage devices.• Image display workstations (WS).• An image management system software for image archival, dis-

tribution, and manipulation.

These components are interconnected by digital networks, com-munication protocols and application software using the digitalimaging and communications in medicine (DICOM)12 standard fordata communication protocol and image data format, and Health7 (HL7) standard13 for data format shown in Fig. 2. The DICOMstandard contains many parts each of which was designed for a



RIS and HIS Database

Database Gateway

Imaging Modalities

Acquisition Gateway

PACS Controller & Archive Server

Application Servers

Web Server

Workstations

Generic PACS Components & Data Flow

Reports

Fig. 2. Generic PACS components and its data flow. DICOM and HL7 are twostandards used for image and textual information, respectively.

given type of connection, and as a whole, it is commonly referred toas DICOM technology, resources, or services. When multiple PACsystems are connected together to share some resources and opera-tions such as storages and image distribution, it is referred to as theEnterprise PACS.

21.2 GRID COMPUTING FUNDAMENTALS

21.2.1 Grid Computing

Grid computing is based on an open set of standards and protocols inits core infrastructure. In this chapter, we use the open grid servicesarchitecture (OGSA)5,6 as a guide to discuss the computational ser-vices and the data services of the Globus Toolkit 4.0 codeveloped byArgon National Laboratory (ANL), University of Chicago and Infor-mation Sciences Institute (ISI), University of Southern California2−4

for medical image applications.8

(a) Computational Services support specific applications on dis-tributed computational resources, such as supercomputers or



a cluster of computers. A grid for this purpose is called acomputational grid.

(b) Data services allow the sharing and management of distributeddatasets. A grid for this purpose is called a data grid.

21.2.2 The Globus Five-Layer Toolkit

Figure 1 shows the five layers of the grid computing technologydefined by the Globus toolkit 4.0 and the layers correspondencewith the OSI (Open System Interconnection) architecture. Figure 3describes the tools available in each of the five layers.

21.2.3 Integration of DICOM with Globus

Grid Computing technology can be used for specific PACS oper-ations by integrating a selected subset of DICOM resources withthe Globus toolkit. We present two examples, the PACS Data Gridand the CAD/PACS (computer-aided diagnosis) computing grid.The former is for PACS backup archive and disaster recovery oper-ation by using the DICOM image store, query and retrieve (Q/R)services. The CAD/PACS Computing Grid is for integrating CADwith PACS application, which requires additional DICOM resourcesincluding the Screen Captured (SC) and Structure Report (SR).12 Inorder to assure the fault-tolerance of the integration, a new databaseand service dedicated for CAD/PACS computing grid defined asthe DICOM data model metadata database and metadata catalogservice are also necessary. Figure 3 depicts the positions of theseDICOM services and the Metadata in the Globus Grid five layerinfrastructure. Sections 3 and 4 detail the characteristics, functions,and workflow of the data grid and the CAD/PACS computing gridof these applications, respectively.

21.3 DATA GRID: LARGE-SCALE MEDICAL IMAGEMANAGEMENT SYSTEMS FOR CLINICAL SERVICES

Three topics will be presented: use data grid in large scale enterprisePACS operation, methods of integrating multiple PAC systems with



IPI data grid layered infrastructure based on globus

NetworkI2, RNP2

PACSSimulator

IPI SAN

HCCIISAN

Resources (Fabric)

Replica Location Service

ReliableFile

Transfer

Database Access

IntegrationSecurity

GridFTP Service

Core Middleware/Globus Toolkit 4.0.2 (Connectivity and Resource)

MetaDataCatalogService

User-Level Middleware (Collective)

Data Grid (Application)

DICOMStorage Service

DICOMQuery Service

DICOMRetrieve Service

Replica Database

DICOM DataModel

MetaDataDatabase

SJHCSAN

Info Services

ExecutionMgmt

DataReplication

Resource Mgmt Data Management

Developed by IPI

Integration of DICOM to Data Grid

Fig. 3. The five layer grid architecture for medical image PACS data grid andCAD/PACS computing grid (see shadow boxes) applications. Fabric Layer: Theleftmost five clear boxes are existing resources from PAC systems, SAN: storagearea network14; replica database is a globus tool; the rightmost Metadata Databaseis for fault tolerant data grid and computing grid (shadow) application. Core mid-dleware (connectivity and resource layers): The four boxes at the leftmost are globustools used for data management in PACS data grid, the rest are other globus tools.Replica (shadow) and resource mgmt (shadow) are also used for computing grid.User level middleware (collective) layer includes the info service globus tool andthe metadata catalog service for fault tolerance. Both resources are also used forcomputing grid applications (shadow boxes). The data grid application layer con-sists of the DICOM storage, query, and retrieve services. Light shaded boxes withbold external rectangles are DICOM resources, and the Metadata database for faulttolerance developed at the Image Processing and Informatics (IPI) Laboratory, USC.

the data grid, and three tasks of the data grid during a PACS disasterrecovery.

21.3.1 Data Grid for PACS Archive and Q/R

Figure 3 illustrates the integration of DICOM image store andDICOM image Q/R in the application layer of Globus toolkit to formthe data grid for storage backup of multiple PAC systems. We usethree PACS sites shown in Fig. 4 to demonstrate the fault tolerancefeatures of the data grid.



Modalities

PACS Server

SAN

Site 1

Backup

Data Grid

PACSGAPPACS

GAP

PACSGAP

Storage Node

SAN

Metadata DB

ReplicaDB

Grid Services

Web clientCommon View

PACS Simulator

SAN

IPIDICOM

GAP

WS

Modalities

PACS Server

SAN

Site 2

Backup

WS

Modalities

PACS Server

SAN

Site 3

Backup

WS

Fig. 4. Three PACS sites operate total independently as three separate PAC sys-tems, each supports its own clinical site. Each site has a standalone PACS with itsown server, workstations (WS), storage area network (SAN) archive and storagebackup. An enterprise PACS is when these three PAC systems (or more) are con-nected together. In an enterprise PACS, a WS at each site can Q/R images from itsown SAN for image display. A WS of any three PAC systems can also Q/R imagesfrom other sites using a web client common view mechanism. The weaknesses ofthis three PACS system interconnection are the two single-points-of-failure. Wheneither each PACS Server or SAN fails, the interconnectivity of three PAC systemsbreaks down. Data grid architecture can take away the backup and the connectionto the web client common view by each PACS. It maintains interconnectivity ofthese three systems in real-time without human intervention. There are two typesof PACS GAP in this architecture, DICOM GAP (bottom) and PACS GAP (middleleft). The former is for PACS WS which uses DICOM standard for image Q/R, thelatter is for none DICOM file transfer used by some PAC systems.

The operation environment is as follows: three PACS sites oper-ate total independently as three separate PAC systems, each sup-ports its own clinical site. Each site has a standalone PACS withits own Server, workstations (WS), SAN (Storage Area Network14)archive and storage backup. A WS at each site can Q/R images from



its own SAN to display image data. A WS of any of these three PACsystems can also Q/R images from other sites using a Web clientCommon View mechanism.

There are several weaknesses of using this method of integratingmultiple PACS operations:

(1) The two single-points-of failure (SPOF) in each PACS are theserver and the SAN archive.

(2) If the server of a PACS goes down, its WS would not be ableto retrieve images from the SAN of its own PACS or reviewimages of other PAC systems because the workflow (see work-flow arrows in Fig. 4) relies on the availability of the PACS server.

(3) If the SAN of a PACS goes down, two possibilities could hap-pen. First, its WS would not be able to view its own images fromSAN. Even though the backup archive may work, it will taketime for it to be online and supply images for its own WS. It isbecause most of the backup storage nowadays is low cost and itspriority is to preserve a second copy of the archive data insteadof immediately failover for continuing operation. The backup isusually without an automatic switch function for primary oper-ation. Second, the PACS would not be able to support Q/R of itsimages by a WS from other PAC systems.

Therefore, two problems under consideration for the data grid areto minimize the impact due to the failure of the Server or the SANof each PACS.

The PACS data grid can be designed as a means of linking thesethree sites together such that the data grid can be used: (1) to supportthe backup archive and disaster recovery for the three sites, and(2) to allow a WS of any site to retrieve and review image data fromany other sites. The former is a data grid with functions for PACSbackup and disaster recovery, and the latter involves functions ofimage distribution and review.

In the PACS data grid, there are three primary components tothe DICOM imbedded data grid. Figure 4 illustrates the overallarchitecture of the data grid which is located at the IPI, USC; other



SANs are located at three clinical PAC systems as shared storageresources.

(1) Storage Node: Computer(s) and storage devices, for examples,multiple copies of SAN, provide storage resources for the datagrid. In this case, each image has three copies, one is in itsown PACS SAN, and two are in two SANs within the data grid(see Fig. 4).

(2) Database: A service that keeps track of metadata as well asfile locations of different storage nodes within the data grid.Dynamic and robust access to data is provided by the dataaccess interface (DAI) in the Globus toolkit integrated with thedatabase.15

(3) PACS or DICOM Grid Access Point (GAP): A service providesDICOM compliant storage and Query/Retrieve capabilities forWS of any PAC system to access data within the data grid. Thereare multiple GAPs in the data grid (see Fig. 4) and can be usedas the backup for each other.

21.3.2 Data Back Up and Disaster Recovery

Let us consider the workflows of data backup and disaster recoveryof the Data Grid.

21.3.2.1 The GAP

Under normal operation condition (Fig. 5(A), solid lines), the imageis sent from the DICOM 1 to the data grid through its designatedGAP1 (Fig. 5(A)). GAP1 is then sent two copies of the image to twogrid storage resource SAN1, and SAN2 respectively. Suppose GAP1fails (cross lines), GAP2 (dotted lines) would take over and completethe task.

21.3.2.2 DICOM Q/R

Figure 5(B) solid lines show the normal operation of DICOM Q/Rat DICOM 1. Under normal operation condition (solid lines), the



GAP2GAP1

SAN1SAN2

DCIOM 1 DICOM 2

Copy 1 Copy 2

Data Grid

PrimaryConne

(A)

ction

SecondaryConnection

GAP2GAP1

SAN1SAN2

DCIOM 1WS

DICOM 2 WS

Data Grid

Q/R

(B)

PrimaryQuery

PrimaryRetrieve

FailoverQuery Failover

Retrieve

Fig. 5. Workflows of data grid during image data store and query/retrieve.5(A) Image backup: Solid lines show the normal operation, the image is sent fromthe DICOM 1 to the data grid through its designated GAP1 for backup storage.Dotted lines show GAP1 fails (cross lines), and GAP 2 takes over automatically.5(B) Query/retrieve: Solid lines show the normal operation, DICOM 1 queriesimages then retrieves from SAN1 through GAP1. Dotted lines show SAN1 fails(cross-lines), GAP1 finds SAN2 automatically and completes the task.



WS queries GAP1 for the location of required image data at themetadata database which identifies SAN1 as the location. It returnsquery results to the GAP, then the WS. The WS retrieves the imagedata from SAN1. Suppose SAN1 fails during Q/R (cross lines), GAP1would find SAN2 and pass Q/R information to SAN2 which wouldcomplete the task (dotted lines).

21.3.2.3 The metadata database

Metadata includes all DICOM image header and data model infor-mation extract from each image when it is acquired from the imagingmodality. It is organized and stored in the metadata database (Fig. 3)which provides all necessary information of the image including thepointer to where the image is located in the data grid SANs. Prop-erly query the database, any image data can be retrieved by the WSthrough the GAP. The metadata database without backup databasesbecomes a single-point-of-failure in the data grid. For this reason,a middle layer called the Data Access Interface (DAI)8 servers isadded in between GAPs and metadata storage nodes. Therefore,there are two layers in the metadata database, the multiple DAIservers and multiple metadata storage nodes (or SANs) shown inFig. 6 which allows multiple GAPs access multiple DAI servers andmultiple metadata storage nodes. The three main roles of the DAIserver are: centralization of metadata access, replication of metadatainto multiple storage nodes, and handling metadata for differentPACS archive.15

21.3.3 Three Tasks of the Data Grid During the PACS Serveror Archive Failure

Following Fig. 4 in which each PACS relies on its own backup archiveand the connection of the server through a Web client common viewmechanism for Q/R images of other PAC systems. The failure ofthe server or the SAN in a PACS would shut down its connectiv-ity with the enterprise PACS. With the data grid connected to the



Fig. 6. General architecture of the fault-tolerance metadata system for the datagrid. Fault-tolerance is from top: Multiple GAPs; middle: DAI servers; and lower:Multiple metadata storage nodes (SANs).

multiple PAC systems architecture shown in Fig. 7, the enterprisePACS could achieve the fault-tolerance status. Note that in this archi-tecture, which differs from that shown in Fig. 4, all backup storagesand connections to the web client common view for image Q/R fromthree PACS servers are discarded. Also, the connection of PACS tothe data grid is at WSs instead of servers.

In this data grid architecture, two single-points-of-failure (SPOF)of any PACS are still the PACS server and the SAN storage device.When these two SPOF fail, the data grid has to overcome three majortasks in order to be fault tolerant. First, it has to maintain continuingclinical operation allowing WSs of this PACS Q/R images from thedata grid. Second and third, after the sever and the SAN have beenrepaired, it has to rebuild the PACS own archive, and the backuparchive for other PAC systems. Figure 7 describes, as an example,these three tasks during the PACS failure (dotted lines) at site 2 in



Modalities

PACS Server

SAN1

Site 1

Data Grid

PACSGAP

PACSGAP

PACSGAP

Storage Node

SAN

Metadata DB

ReplicaDB

Grid Services

PACS Simulator

SAN

IPIDICOM

GAP

WS

Modalities

PACS Server

SAN2P1 P2

Site 2

WS

Modalities

PACS Server

SAN3

Site 3

WS

Task 1

Task 2 Task 3

Fig. 7. Three Tasks (dotted lines) of the Data Grid during PACS Server or archivefailure. The PACS at Site 2 is used as an example in which either the server orthe SAN or both fails (cross lines). SAN2 is partitioned into P1 and P2. P1 is itsown PACS archive, and P2 is Site 2’s storage resource committed to the data grid.Task 1. Allowing its WS to Q/R its own images from the Data Grid for continuingclinical operation. Task 2. After its server and SAN have been repaired, the datagrid rebuilds P1 of SAN2 installing the PACS own images. Task3. After its serverand SAN have been repaired, the data grid rebuilds P2 of SAN2 which have thebackup images of other PAC systems connected to the data grid. All three tasks areperformed automatically without human intervention.

Note that all backup storages and connections to the Web client common viewfor image Q/R from the three PACS servers are not necessary and thus discarded(Compare with Fig. 4).

which either the server or the SAN or both fails (cross lines). SAN2is partitioned into P1 and P2. P1 is for its own PACS archive, and P2is Site 2’s storage resource committed to the data grid. Task 1 has thehighest priority among three tasks shown in dotted lines. All threetasks are performed automatically without human intervention.



21.4 GRID COMPUTING — COMBINING IMAGEMANAGEMENT AND ANALYSIS

A computational service infrastructure in the data grid providesdependable, consistent, pervasive, and inexpensive access to com-putational capabilities. Globus toolkit described in Secs. 2 and 3including fabric, core and user-level middleware services (refer toFig. 3 shadow boxes) enable the expansion of Data Grid to GridComputing applications. The Grid execution environment includescomputing and storage services with diverse capabilities.16 Fivecomputational services using the Globus toolkit in the data gridinfrastructure have been developed. We first describe the basicinfrastructure and then in Secs. 4.2 and 4.3 use this infrastructureto present CAD of multiple sclerosis (MS) on MRI, and CAD/PACSintegration, respectively.17

21.4.1 Computational Services Architecture in the Data Grid

In grid environment, an application component can be implementedin different source files; each compiled to run in a different type oftarget architecture. Exact replicas of the executable file can be storedin many locations, which helps reduce execution time. Data files canalso be replicated in various locations. Each file has a descriptionof its contents in terms of application-specific metadata. The meta-data service including catalog service (see user-level shadow box ofFig. 3) responds to queries based on application-specific metadataand returns the logical names of files containing the required data.Given a logical file name that uniquely identifies a file without spec-ifying a location, the replica location service (RLS, Shadow in CoreMiddleware, Fig. 3) can be used to find physical location for the fileon the grid.

In grid computing, a specific application may require a certaintype of resources for execution. Figure 8 shows the operation archi-tecture of the computational services as follows:

(1) The client requests resources from GRID Monitoring and Dis-covery System (MDS) server, which manages the resources anddistributes the jobs to computational services.



Submit (Client)

Application

Globus Client APIs

IndexService

GRID MDS Server

Execute(Computational Services )

Executable Code

Scheduler

GRAM Service

Remote Procedure Call :Client Side

Remote Procedure Call :Server Side

1

2, 4

2

3

Fig. 8. Operation architecture of the grid computing. Left: a client, right: compu-tational services. MDS: monitoring and discovery system, GRAM (grid resourceallocation and management). Numerals represent the workflow, see text.

(2) The index service finds resources appropriate to the require-ments of application components and notifies the client to sendthe application to the grid resource allocation and management(GRAM) service.

(3) The GRAM service acknowledges MDS after it receives theapplication, jobs that completely specified for execution are sentto the scheduler that manage the resources and monitor execu-tion progress. Execute acknowledges MDS server the completionof the job.

(4) MDS notifies the client that the job is completed.

21.4.2 An Example of the Computing Grid — CADof Multiple Sclerosis (MS) on MRI

21.4.2.1 Multiple sclerosis

Multiple sclerosis (MS) is a progressive neurological disease affect-ing myelin pathways. Multiple lesions in the white matter (myelin



pathways) can cause paralysis and severe motor disabilities. Symp-toms are changes in sensation, visual problems, muscle weakness,depression. MRI has become the medical imaging study of choiceboth for the diagnosis and for the follow-up and monitoring of multi-ple sclerosis. The progression of the disease is variable, and requiresroutine follow-up to document disease exacerbation, improvement,or stability of the characteristic MS lesions. Current status is thatit is easy to make the diagnosis from MRI, but time consuming toquantify the number and size of lesions, and its poor reproducibility.Imaging Informatics using CAD is considered an ideal quantitativetool to monitor progression of MS.18 Two-Dimensional CAD usingMRI T1 and FLAIR sequences with 5 mm slice are the most com-mon imaging techniques to make MS diagnosis, with the former forbrain anatomy and the latter for MS locations. Several commerciallyavailable CAD techniques for MS detection are being used in clini-cal environment, however, these methods have several weaknesses.First, they are mostly 2D and often require human intervention toidentify the MS lesions. Second, these methods lack the informat-ics component to organize and accurately measure the many lesionsdetected for disease progression comparison with time and/or treat-ment. Third, these methods are mostly stand alone CAD and cannot be readily integrated with PACS for routine use in daily clin-ical operation. Data grid and CAD/PACS computing grid can beused to alleviate these shortcomings of current methods. Based onanatomical and pathological knowledge, we have developed a 3DCAD to quantify MS lesions on MRI with thinner slice for compar-ative quantitative studies. Here, we will not discuss the details of3D CAD methodology except the image processing steps, insteadwe describe the data grid and computing aspect of the 3D MS CAD.Section 4.3 will present the CAD-PACS integration with the datagrid and computing grid.

21.4.2.2 Integration of MS CAD with data grid and grid computing

The steps required in the computational procedure of quantitativediagnosis of MS using CAD are shown in Fig. 9. Assuming that such



PREPROCESSING: 3D head mask detection,3D bias field correction

T1, FLAIR, newstudy from PACS

Studies comparison

3D side by side view

MS lesionsegmentation

MS lesionsquantification

3D brain mask

3D CSF spaces mask

Corrected FLAIR images

T1, FLAIR images

Series standardization

3D Brain mask (BM)

T1, FLAIR images

Quantification of MS lesions progression using 3D CAD

3D H

ead

mas

k (H

M)

T1, FLAIR, priorstudy from PACS

Prior study: lesion number

and volume

New study: lesion number

and volume

Courtesy of Dr. A. Gertych

Fig. 9. The image analysis work flow of 3D MS CAD. T1 and FLAIR sequences arefirst standardized and preprocessed followed by detection, segmentation, quantifi-cation, and visualization of 3D MS lesions. Quantitative and visualization compar-ison between original and follow-up 3D MRI studies are also available.

a 3D CAD software package is available, Fig. 10 depicts the workflowsteps 1–4 of the MS CAD modified for the computational resourcesbased on the data grid and the computing grid architectures dis-cussed in Fig. 8 and CAD workflow in Fig. 9. Figures 11(A)–(D)show MS CAD results and the corresponding DICOM structuredreport file of a patient with 26 lesions.

(1) MRI T1 and FLAIR image sequences are sent by the data gridclient to the data grid Monitoring and Discovering System(MDS) server.

(2) MDS distributes sequence images to the preprocessing resourcewhich performs series standardization, 3D head mask detection,and 3D bias field correction.



MDS

Pre-Processing

3-D StudiesCompariosn

3-D Brain & CSFMasks

1 MRI T1FLAIR

3-D MS lesion Quantification

3-D MSQuantitative

Results

MSComputationalServices in the

Data Grid

2

3

4

Client

4 CADResults

Fig. 10. Operation workflow of MS CAD in computational services of the data gridand computing grid. The MDS allocates the computation requirements accordingto available services in the data grid. Numerals, see text, represent the workflowwithin the MS computational services. MDS: monitoring and discovering systemserver.

(3) MDS distributes preprocessed sequence images to 3D brain andcranial spinal fluid (CSF) masks, MS lesion detection and quan-tification, and studies comparison computational services forprocessing; and receives processed results.

(4) MDS sends processed results to 3D quantification resultsresource for organization and receives the compiled single orcomparison study results. MDS returns compiled study resultsto the client with visualization and display.

The advantages of developing the computational services in the datagrid for MS CAD versus using the conventional standalone CADmethod are:

(1) It utilizes the existing data grid technology which saves the jobdistribution and computation time.

(2) It does not require to significant rewrite the image process-ing codes for the computational services in the data grid, this



Quantification and tracking of multiple sclerosis (MS)

Truth CAD Result

(A)

Fig. 11. CAD identified, organized with provided quantitative measurements oftwenty-six Multiple Sclerosis (MS) lesions of a patient shown in the DICOM Struc-tured Report. Structured Report information could be extracted by the Metadatadatabase in the Data Grid for storage, distribution and future analysis.(A) Identifying and Quantification of MS in an MRI FLAIR image, comparing

radiologist reading (true, left) with CAD results (right).(B) Two consecutive images in which MS lesions were identified and aligned.(C) Twenty-six MS lesions were identified and organized in colors.(D) Twenty-six (A-Z) MS lesions were detected and organized, the quantita-

tive results of each in number of pixels shown in the DICOM structuredreport.

(E) A screen capture of the top view of 3D rendering of the brain with MS lesions.The brain ventricular system was used as a reference of positions of thelesions.

approach of distributing computations to available resourceswould result in substantial acceleration of data analysis speed.

(3) With the progressively increasing use of MS CAD in clinical cen-ters, utilizing data grid architecture can assure easier distributionof computational services throughout the data grid.



CAD multiple sclerosis (MS), two images

Next Axial imageoverlaid

(B)

DICOM structured report of MS in a PACS WS

Image areas mappedin colors maybe separatedby distance.

Color mapof all MS areas

(C)

Fig. 11. (Continued)



(D)

DICOM structured report of multiple sclerosis in a PACS WS

Fig. 11. (Continued)



21.4.3 Integration of CAD/PACS with the ComputationalServices in the Data Grid

Most CAD method is a developed for standalone operation as asecond reader, it does not integrate with PACS clinical workflow.Recently, a CAD-PACS toolkit was introduced which allows CADto be integrated with PACS workflow enhancing the usefulness ofCAD.17 Integration of the CAD-PACS toolkit with data grid andgrid computing would allow streamlining of CAD results and PACSoperation in grid environment. The PACS-CAD toolkit consists oftwo components, a DICOM secondary capture (DICOM-SCTM) anda DICOM-IHETM component to accommodate various PACS oper-ations (Fig. 12).

The DICOM-SCTM component installed on a CAD workstationconverts the screen shot of video display of CAD results to a DICOMimage file for storing in a PACS server and displaying on PACSworkstations. The workflow is as follows (Fig. 13):

(1) The PACS WS sends DICOM image files to CAD WS for process.The CAD WS receives DICOM files and performs the CAD.

Fig. 12. PACS-CAD Model Toolkit. The PACS-CAD toolkit consists of two com-ponents, a DICOM secondary capture (DICOM-SCTM) and a DICOM-IHETM com-ponent to accommodate various PACS operations. DICOM-SC is easy to implementbut the data is in screen capture format which does not allow for data manipulationand analysis. DICOM-IHE involves system integration which requires cooperationof PACS manufacturers to implement. Once implemented CAD results are in thePACS workflow which allows for data analysis and mining.



Fig. 13. DICOM-SC (Secondary Captured). The DICOM-SC toolkit installed ona CAD workstation converts the screen shot of CAD results to a DICOM image filefor storing in a PACS server and displaying on PACS workstations. See text for itsworkflow.

(2) The DICOM-SC with i-CAD-SC package installed in CAD WSconverts the screen shot of CAD results to DICOM files and sendsit to PACS server. If the CAD does not have the capability toprovide screenshot, the i-CAD screen capture is used to capturethe CAD windows creating the screen shot. This output image iscreated as a DICOM secondary capture image having the samepatient information of the original DICOM image with a newgenerated series information called screen capture in DICOMheader. Therefore, it will be stored as additional series under thesame study of patient data model in PACS server.

(3) When the PACS WS queries the PACS Server, the new seriescontaining the CAD results will appear as a series under thestudy of patient. Radiologists can retrieve the CAD results withthe study to their PACS WS. The CAD results are shown as aDICOM image.

The DICOM-IHETM component follows DICOM standard andIntegrating the Healthcare Enterprise (IHE)19 workflow profilesusing DICOM structured report and post-processing workflow pro-files. Thus, results from various CAD software can be integrated intodiagnosis workflow of a PACS having DICOM and IHE-complianceand, most importantly, these quantified CAD results in structured



Fig. 14. DICOM-IHE Workflow. The DICOM-IHE toolkit follows DICOM stan-dard and IHE workflow profiles using DICOM structured report and Post-processing workflow profiles. Results from CAD can be integrated into diagnosisworkflow of a PACS having DICOM and IHE-compliance. Quantified CAD resultsin Structured Report format can be archived in the metadata services of the datagrid (Fig. 3).

report format can be archived in the metadata services of the datagrid (Fig. 3). Computational services can be developed in the gridcomputing to directly query and retrieve CAD results within PACSfor future data analysis and mining applications. Figure 14 showsthe workflow.

(1) PACS server pushes a DICOM worklist to i-PPM (post-processing manager) to requesting a CAD process for the studies.If PACS server cannot push the worklist, the i-PPM can querythe PACS server for DICOM worklist automatically.

(2) The CAD WS queries the CAD worklist from i-PPM. The CADclaims work items to be performed.

(3) The CAD WS queries/retrieves DICOM images from the PACSserver for CAD process.



(4) The CAD WS sends “work item in progress” message to thei-PPM.

(5) The CAD WS performs CAD process and stores the CAD resultsin the Receive-SR installed in the PACS server. The CAD WS usescomputational services described in Fig. 10 for CAD process.

(6) The CAD WS reports work item PPS and work item completedmessage to the i-PPM.

(7) The PACS WS retrieves the DICOM SR CAD results for physi-cians’ review. The web-based Display-SR can be used in thePACS WS to view the CAD results and perform future analysis.

21.5 SUMMARY

Grid computing is a powerful tool for large scale computation andstorage requirements. In this chapter, we present a novel conceptof data grid for medical image application, in particular, for dailyclinical PACS on site archive, off site backup, and disaster recovery.PACS data grid is based on the Globus 4.0 toolkit, with SAN storagetechnology and some DICOM resources to form an integrated faulttolerant archive system.

Grid computing utilizes the data grid infrastructure for specificmedical imaging applications by adding necessary computationalservices. We use computer-aided diagnosis (CAD) of multiple scle-rosis (MS) of brain on MRI as an example. The computational ser-vices required in CAD MS include image preprocessing followed bydetection, segmentation, quantification, and visualization of 3D MSlesions. Grid computing for large scale medical image archiving andanalysis is still in its infancy, we anticipate many fruitful results willmaterialize in the near future.


This research has been partially supported by NIH R01 EB 00298and NIH R01 LM07606 grants, and a contract from MI2.



References

1. What is grid computing, http://www-1.ibm.com/grid/about_grid/what_ is.shtml.

2. Baker M, et al., Grids and Grid technologies for wide area distributedcomputing, Proceedings of SPIE Medical Imaging, February, 2002.

3. The Grid: A New Infrastructure for 21st Century Science, http://www.aip.org/pt/vol-55/iss-2/p42.html.

4. Morgan K, Computational Grids, The Grid: Blueprint for a New Comput-ing Infrastructure, Chapter 2, 1999.

5. Foster I, Kesselman C, Nick J, Tuecke S, The Physiology of the Grid: AnOpen Grid Services Architecture for Distributed Systems Integration.In Open Grid Service Infrastructure WG, Global Grid Forum, June 22,2002.

6. Foster I, Kesselman A, Tuecke S, The Anatomy of the Grid: EnablingScalable Virtual Organizations, International J Supercomp Applications15(3): 2001.

7. Foster I, Kesselman C, Nick J, Tuecke S, Grid Services for DistributedSystem Integration, Computer 35(6): 2002.

8. Globus Toolkit 4, http://www.globus.org/toolkit/docs/4.0/.9. Huang HK, PACS and imaging informatics, John Wiley & Sons,

Hoboken, NJ 2004.10. Liu BJ, Zhou MZ, Documet J, Utilizing data grid architecture for the

backup and recovery of clinical image data, J Comput Med Imag Graph29: 95–102, 2005.

11. Doi K, Huang HK, Special issue editorial: Computer-aided diagnosisand image-guided decision support, J Comput Med Imag Graph 31(3–4):195–197, 2007.

12. DICOM, http://medical.nema.org/, accessed on 18 November 2006.13. HL7, http://www.hl7.org/, accessed on 20 April, 2007.14. SAN Technology: http://www.storage.ibm.com/ibmsan/whitepaper.

html.15. Lee J, Zhou Z, Talini E, Documet J, et al., Design and implementation

of a fault-tolerant and dynamic metadata database for clinical trials,Proceedings of SPIE Medical Imaging 6516, February 2007.

16. Blythe J, Deelman E, Transparent grid computing: A knowledge-basedapproach, Fifteenth Innovative Applications of Artificial Intelligence Con-ference (IAAI-03), Acapulco, 12–14 August 2003.

17. Zhou Z, Liu BJ, LeAH, CAD-PACS integration toolkit based on DICOMsecondary capture, structured report and IHE workflow profiles,J Comput Med Imag Graph 31(3–4): 346–352, 2007.



18. Wong A, Gertych A, Zee CS, Guo B, et al., A CAD system for assess-ment of MRI findings to track the progression of multiple sclerosis,Proceedings of SPIE Medical Imaging 651614(1–7): February 2007.

19. IHE, http://www.ihe.net/, accessed on 18 November 2006.


CHAPTER 22

Image-Assisted Knowledge Discoveryand Decision Support in Radiation

Therapy Planning

Brent J Liu

This chapter introduces an application of a medical imaging informat-ics approach to develop a unified patient oriented information systemto handle complex radiation therapy (RT) imaging and informatics dataduring the course of the patient treatment. Currently, this data is scat-tered throughout each of the different treatment and information systemsin the oncology department. Additionally in this chapter, as an exam-ple, the methodology will be used to develop quantified knowledge anddecision support tools for a brain tumor patient case treated by inten-sity modulated radiation therapy (IMRT). The use of the “inverse treat-ment planning” nature of IMRT allows for the extraction of quantifiedknowledge and the development of decision-support tools to assist inthe decision-making process. This a priori knowledge, whether the dosedistribution to the target tumor is acceptable while limiting dose to crit-ical structures, resides within the expertise of oncologists and physicists.It is currently used in evaluating the acceptance of a treatment planand this research will quantify this knowledge to derive decision sup-port tools. As a result, the development of quantified knowledge canaugment the conventional “inverse treatment planning” approach intoan improved “knowledge-based treatment planning” process with betterworkflow efficiency and more precise dose predictions. The imaging andinformatics methodology and approach can be extended to various clini-cal decision making scenarios during the course of the patient’s treatmentfor not only a specific RT treatment type but also a specific lesion type inany body region.

545


546 Brent J Liu

22.1 INTRODUCTION

22.1.1 Need for Imaging Informatics in RadiationTherapy Planning

The essence of medical imaging informatics is to use informat-ics methods to extract and synthesize the information-rich picturearchiving and communication systems (PACS) and other patient-related databases to further advance medical research, education,and clinical services. Along the way, we need to develop method-ology, tools, infrastructure, and applications. In this chapter, wefirst present the basic fundamental concepts of building a patient-oriented information system to integrate standardized imaging andinformatics data based on DICOM and then introduce method-ologies for developing quantified knowledge and decision-supporttools during the radiation therapy (RT) planning process. This appli-cation not only relies on PACS as a decision support tool but alsomust combine PACS image data with other medical specialty’s datalike in radiation therapy to form a new medical informatics serverwith decision support.

The need for comprehensive clinical imaging informatics inimage-intensive radiation therapy (RT) is steadily recognizablebecause of ever increasing demands for better diagnostic and treat-ment equipment and more accurate information. Traditionally, mul-tiple information systems acquire the necessary data during the RTtreatment course of a patient, however, most of the data is scatteredthroughout each of the varying treatment and information systems.In addition, RT utilizes some of the most technological advance-ments in diagnostic imaging, therapeutic radiation, and computer-ized treatment planning systems, which adds to the complexity ofthe collection and navigation of pertinent RT data. Therefore, imag-ing informatics tools are needed for knowledge discovery to extracttreatment planning data in support of oncologists and physicists’decision making process during daily practice. Digital Imaging andCommunication in Medicine (DICOM) is the de facto imaging stan-dard for imaging departments like radiology, along with clinicalworkflow profiles (Integrating the Healthcare Enterprise). This inturn led to their successful development and utilization of PACS


Image-Assisted Knowledge Discovery and Decision Support in Radiation Therapy Planning 547

which has become an indispensable integrated imaging system indiagnostic radiology. Furthermore, recently accepted concepts ofcomputer-aided diagnosis (CAD) integrated with PACS advancesradiology to the next level of excellence in clinical care.1−6 The read-ily available HL7 (Health Level 7), DICOM, and IHE are basic toolsin the realm of medical imaging informatics which can be appliedto RT. In addition, other more advanced and powerful imaginginformatics methods such as data mining for knowledge discov-ery, CAD, and outcomes analysis can also be adopted and inventedfor the benefit of more accurate and efficient patient treatmentplanning.

22.1.2 Current State of Imaging Informatics in RT

Currently in RT, the practical use of imaging informatics tools islimited. DICOM is mostly used for transmitting PACS images toan RT system; and imaging-guided treatment planning systems arelimited to dose computations and graphical data displays. PertinentRT data results do not have a standardized protocol.

To address these shortcomings, the DICOM Standard Committeeextended DICOM for the RT application by ratifying seven DICOM-RT objects.7 Although some of these objects are utilized within thedaily clinical operation in piece-meal fashion, they are not inte-grated. There are still data crucial to the decision making processthat has not utilized these standards. The need for a system integra-tion infrastructure based on standards is crucial for the establish-ment of patient outcomes related medical informatics research. Themethodology of developing quantified knowledge with decision-support that would augment the current robust and complex thera-peutic clinical processes to ultimately improve the quality of patientcare is needed. One such system integration infrastructure is theimaging-based electronic patient record (ePR), which is a patient-based digital virtual folder of clinical information obtained from var-ious information sources.8 The inclusion of imaging data and built-indecision support makes the ePR stand out amongst general clinicalinformation systems, thus opening new doors to the possibility ofimprovement in clinical decision outcomes of the future in RT.


548 Brent J Liu

This chapter will discuss a methodology for research and devel-opment of a DICOM-Based ePR system with quantified knowledge-base decision-support tools to facilitate RT in therapeutic treatmentplanning. As an example, this chapter will discuss the develop-ment of necessary imaging informatics research tools on a spe-cific clinical scenario for RT of brain tumor patients treated withintensity-modulated radiation therapy (IMRT), and specifically, the“inverse treatment planning” nature of IMRT using a treatmentplanning system (TPS). Quantified knowledge and decision-supporttools based on the expertise of oncologists and physicists to assistin their decision-making process are designed and developed,thus augmenting the conventional treatment planning approachinto a “knowledge-based treatment planning.” This methodologycan be extended for future clinical decision-making scenarios dur-ing the course of the patient’s treatment for not only a specificRT treatment type but also a specific lesion type in any bodyregion. Let us review briefly the concept of ePR in the nextsection.

22.1.3 Review of Electronic Patient Record (EPR)

Electronic patient record is an emerging concept to replace or supple-ment the hospital- or clinic-based healthcare information systems.The concept of the ePR is a patient-based digital virtual folder ofclinical information obtained from various information sources. Thecomponents of an ePR include an information model, a clinical datarepository, a web-based application for the users, along with a secu-rity model and built in decision support. The inclusion of imag-ing data and built in decision support makes the ePR stand outamongst general clinical information systems such as HIS and RIS(Hospital and Radiology Information Systems). The imaging datawithin the ePR data model has opened new doors to the possibility ofimprovement in clinical decision outcomes of the future. However,the difficulties involved in system integration across the healthcareenterprise have slowed the developmental progress. Currently, theUnited States Department of Veterans Affairs Healthcare Enterprise



(VAHE) information system, VistA,8 is probably the most advancedenterprise-level ePR integrated with images compared with othersin the field.

The major functions of an ePR system are:

• Accept direct digital input of patient data.• Analyze across patients and providers.• Provide clinical decision support and suggest courses of

treatment.• Perform outcome analysis, and patient and physician profiling.• Distribute information across different platforms and health

information systems.

The concept of the DICOM-RT based ePR System uses the ePR archi-tecture with an individual patient as the focus.


22.2.1 Introduction to the Medical Imaging InformaticsApproach for Developing Quantified Knowledgeand Decision-Support Tools

Figure 1 shows the overview of the methodology for designing anddeveloping a DICOM-RT based ePR system and standardizing RTdata into DICOM-RT objects. Once the standardized data objects areintegrated, the next steps in the imaging informatics methodology isto develop the knowledge base, the data mining, and quantificationand visualization tools which ultimately become add-on features to aDICOM-RT based ePR system. This methodology can be utilized fordeveloping a knowledge base and clinical decision-making tools fora DICOM-RT ePR based system. Secondly, the integration of DICOMdata from both RT and radiology in a DICOM RT ePR server willbe the foundation for the integration of future treatment planningsystems for an efficient one-stop-shop source where clinicians cantrack and review their patient cases with decision-support tools anda knowledge base. The methodology steps will be discussed furtherin the following paragraphs.


550 Brent J Liu

Decision supportfor new brain

tumorpatients prior to

treatmentplanning

Imaging andinformatics RT DataID and extraction from RT Systems

Convert data intoDICOM-RT objects

when necessary

DICOM-RT ePR system w/

DICOM-RT dataobjects

Add-on features

Define knowledgebased on clinical

workflow scenarios

Development ofknowledge base

Developquantification andvisualization tools

Data mining tools for extracting

knowledge

Datacollection

Fig. 1. Amedical imaging informatics approach towards development of decision-support tools for the DICOM-RT based ePR system. The final results are add-onfeatures for the ePR system to provide decision-support for new patient cases. Thismethodology can be applied to different lesion types as well as treatment types toquickly develop new decision-support tools.

22.2.2 Workflow Model Development

One of the most important first steps for system integration of clin-ical image and information systems is to research the workflowmodel of the clinical operations. The example that will be describedin this chapter is patients with brain lesions who will be undergo-ing intensity-modulated radiation therapy (IMRT). The workflowrelated to these particular treatment cases should be studied todevelop the workflow model. A general clinical workflow modelfor IMRT of brain tumor cases was developed for the Departmentsof Radiology and Radiation Oncology, Saint John’s Health Center(SJHC), Santa Monica as shown in Fig. 2. Although this workflowmay be specific to SJHC, the workflow steps can be extended to otherinstitutions with further refinement. The treatment begins with thepatient diagnosed with brain lesion or multiple brain lesions. Thepatient meets with the physician(s) and determines whether to treatthe tumor(s) but also what type of radiotherapy will be performed.The patient is entered in an oncology information system and isscheduled for treatment. If conventional RT is prescribed, then a



Treatpatient?

Approve plan?

Patient consultw/ physician

Patient entered intooncology information

system

Physician(s)decides ontreatment

Retrieve CTand import in

TPS

Schedule pat.for simulator (if necessary)

Schedule CTAcquire CT andstudy and store

in PACS

Acquire reference images

(if necessary)

Oncologist reviews case anddefines initial plan

parameters (e.g., dose limits,critical structures,etc.)

Physicist computes plan

on TPS

Oncologistreviews plan

Oncologistmakes changes

to planPhysicist and

oncologistreview results

Radiation therapistexecutes

treatment plan

Physicistperforms QA and

setup on RT modality

Sign off ontreatmentsession

No

Yes

Fig. 2. Ageneral clinical workflow for the treatment of brain tumors with intensity-modulated radiation therapy (IMRT).

simulator image may be acquired. Otherwise, depending on thetreatment type, such as IMRT, a diagnostic CT will be acquired toplan the treatment. The radiologist and radiation oncologist reviewthe patient’s case and then the radiation oncologist defines the initialplan parameters such as dose limits and constraints, critical struc-tures, and tumor volume to be treated. The physics team then com-putes the plan based on these dose constraints on the correspondingTPS. Once the initial plan is computed, the oncologist reviews theresults and makes any necessary changes. This process can be iter-ative and the feedback loop is defined in Fig. 2 by a dashed lineregion. Once the treatment plan has been approved, the treatmentsession is executed by the radiation therapist, the correspondingRT plan data are stored in the treatment planning systems of theRT modalities and some results are also inputted into the oncologyinformation system or a record and verify system. Since there area variety of brain tumor types, and the treatment paths can dif-fer, it is important to develop a robust workflow model that canaccommodate the various treatment paths and identify points withinthe workflow that can be improved. Not only would this enhancethe design of the DICOM-based ePR system, but also serve as thefoundation for a methodology to build quantification and visualiza-tion tools for decision-support. In our example for this chapter, the


552 Brent J Liu

iterative feedback loop is identified as a potential area of improve-ment. The feedback loop represents the inverse treatment planningprocess and can be quite tedious if much iteration is necessary. Thisbecomes the area of focus where decision-support tools may benefitduring the decision-making. If more a priori knowledge and robustquantification and visualization tools can be included during thedecision-making process of the initial plan parameters, then it ispossible to reduce the iterative process.

22.2.3 DICOM-RT Data Model Developmentand Data Collection

The Digital Communication in Medicine (DICOM) standard hasbeen well established and widely successful for clinical imagingsystems in radiology, in particular Picture Archiving and Commu-nication System (PACS). Image data acquired from equipment fromdifferent vendors can readily communicate with each other andintegrate into a system through the DICOM standard. In 1997, theDICOM standard was extended to include radiotherapy informationand further updated in the latest version released in 2003.9,10 SevenDICOM radiotherapy (DICOM-RT) objects have been included bythe DICOM standards committee for the transmission and storageof radiotherapy images and related information11:

RT Image (1997) — includes all images taken using radiotherapyequipment such as conventional simulators, digitizers or elec-tronic portal imagers.

RT Dose (1997) — contains dose data such as dose distribution gen-erated by treatment planning system, Dose Volume Histogram(DVH), dose points etc.

RT Structure Set (1997) — contains information related to patientstructures, markers, isocenter, target volume, contours andrelated data.

RT Plan (1997) — refers to information contained in a treatmentplan such as beam angles, collimator openings and beam mod-ifiers etc.

RT Beams Treatment Record (1999) — records for external beamtreatment.



RT Brachy Treatment Record (1999) — records for brachytherapy.RT Treatment Summary Record (1999) — summary of a patient’s

radiation treatment.

Most RT vendors are at the various stages of implementing theseobjects, in particular the 1997 ratified objects, RT structure set, plan,image, and dose. The three record objects are still in their prelimi-nary stage of implementation by vendors. Figure 3 categorizes these

Diagnostic Radiology Radiation Therapy (DICOM Objects) (Seven DICOM RT objects)

Patient

Study

Series

RT plan

RT structure

set

RT dose

RT images

Three RT treatment

record objects

Images

Seven RT objects

Study

Simulator image DRR portal image

Dose data Isodose curve DVH dose points

Gantry angle collimator openingsbeam modifiers

Beams treatment recordbrachy treatment record treatment Summary record

Tumorcritical organs isocenter markers

Fig. 3. Data structure of diagnostic radiology and the seven radiation therapy(RT) objects. Digital Reconstructed Radiography (DRR), Dose Volume Histogram(DVH).


554 Brent J Liu

objects and their contents against those of diagnostic radiology. Theadvantages of having these DICOM objects in RT are obvious. First,information and images within an object can be transferred acrossthe boundary of different RT vendors with minimum efforts from theusers. Second, it allows the total integration of RT components fromvarious vendors. Third, the workflow of RT treatment can be closelymonitored and analyzed resulting in a better healthcare delivery tothe patient. Fourth, an individual patient’s RT treatment can be inte-grated under the scheme of ePR (Electronic Patient Record), a currenttrend of patient-oriented healthcare delivery system. The individualRT treatment ePR can be combined with other related informationincluding demographic, diagnostic, pharmacy, clinical laboratory,and others under the same format and standard. This will result in aportable ePR of the patient, a giant leap from the current hospital orhealthcare information system which is organization-oriented. TheDICOM-RT object information models can be utilized to develop thedata structure for the electronic patient record. To develop a concep-tual data model, the RT workflow must be reviewed to define thedata required. Additionally, clinical user input is needed as well.Along with the input sources mentioned above, a conceptual modelcan be developed for an RT electronic patient record.

A data survey should be performed to track and collect patientcases utilizing any related clinical oncology information systems aswell as diagnostic images from PACS. In our example, cases thatexhibit brain tumors were tracked to determine the treatment pathand outcome. The preliminary data collection survey was performedto determine the feasibility of data collection for the treatment ofbrain tumors. A sample data set from an IMRT TPS at Saint John’sHealth Center (SJHC), Santa Monica, CA will be presented and dis-cussed in the further sections.

22.2.4 DICOM-RT Data Conversion and System Integration

Based on the clinical workflow model as well as the guidance ofexpert users such as oncologists and physicists, a data model can bedeveloped to determine which data will be needed to convert into



DICOM-RT objects and which are already in the DICOM-RT format.The data model includes:

(1) Patient demographic data(2) CT images(3) Reference and portal images (e.g. simulator images, digitally

reconstructed radiographs — DRR)(4) Critical structure curves(5) Isodose curves(6) Dose limits and weighting factors(7) Dose volume histogram (DVH) curves(8) Radiation beams records

The data not in DICOM-RT format are converted into the sevenDICOM-RT objects described in Sec. 22.2.3. Integration of these dataobjects into the DICOM-RT ePR system is the next step. For theDICOM RT ePR system, a three-tier architecture was developed12:(1) The RT archive server manages, archives and distributes DICOMimages and DICOM-RT objects; (2) The RT web-based applicationserver processes patient planning and treatment data; and (3) theRT web-based client application presents the RT data. The databaseschema reflects this three-tiered system by physically representingthe data as well as providing data structures, file organizations andmechanisms for system operation as well as data storage. In thedesign of the RT workflow, there are two database schemas devel-oped; one for the RT archive server and the second for the RTweb-based ePR application server. Because there is more RT datapresentation at the web-based application server level, the latterdatabase schema is much more complex as compared to the RTarchive server. Based on the data model and the clinical workflowmodel, the data workflow was designed as shown in Fig. 4 for systemintegration.13−15

Data from the oncology information system and the IMRT TPSare converted into DICOM-RT objects and sent to the DICOM RTgateway. The diagnostic images are sent from the PACS Server intothe DICOM RT gateway as well. Once the DICOM-RT objects havebeen received by the DICOM RT gateway, they are sent to the archive


556 Brent J Liu

DICOM RT objectinput:oncology information systemtreatment planningsystem

linear accelerator etc

PACS Server

DICOM RTGat

DICOM RT Archive Server

Imaging Databaseand Archive

RT Web-based ePRApplication Server

Web Client-Review workstations forRadiation Oncologists

and Radiation Therapists

DICOM RT objectinput:

• oncology information system

• treatment planningsystem

• linear accelerator etc.

PACS server

DICOM RTgateway

DICOM RT archive server

imaging databaseand archive

RT web-based ePRapplication server

Web Client-review workstations forradiation oncologists

and radiation therapists

Fig. 4. The RT data workflow. (1) RT data input from various different RT sources;(2) diagnostic images from radiology PACS; (3) conversion into DICOM-RT objectsby the DICOM-RT gateway; (4) RT archive server stores, manages, and distributesRT data; (5) RT web-based ePR application server further manages and preparespatient planning and treatment information; (6) web-based client review worksta-tion displays RT-related data for clinician review (courtesy YY Law).

server. Adatabase schema is developed for the archive server so thatthe DICOM RT objects can be archived and distributed to the web-based application server. The archive server should be a continuousavailable (CA) server design with 99.999% uptime that has beenpreviously utilized for a variety of clinical applications.16,17 Integra-tion of the DICOM-RT ePR system within the clinical environmentincludes, in our example, evaluating the target feedback loop shownin Fig. 2. The iterative process of inverse treatment planning is anexample where additional knowledge and decision-support toolscan improve the overall decision-making process. Based on inputfrom both radiation oncologists and radiation therapists at SJHC,married with existing data and workflow models, a database schemaand user interface design was developed to meet the clinical needs.This is implemented in the web-based application server as well asthe web client.12

22.2.5 Knowledge Base Development

Knowledge is defined and quantified based on the expert’s ability,either the oncologist or physicist, to utilize data and other criteria



in evaluating, grouping, and defining certain clinical characteristicsthat are extracted from the standardized DICOM-RT objects storedwithin the ePR system. The knowledge base is designed in an object-oriented and modular fashion so that additional knowledge and newobject classes defined in the future can be easily integrated withoutaffecting the overall design. An example of quantified knowledgemodeling is the dose constraint relationship between the weightingfactor and the DVH curve of each critical structure and target tumorwhich can be loosely defined mathematically as

wi = f [DVH1 · · · DVHN],where the weighting factor i of a particular tumor or critical structurehas a value function relationship with all DVH curves from one to Nin the treatment plan. This relationship can be defined by analyzingthe DVH curves and the changes to each of them when a particularweighting factor is modified between iterations. The functional form“f” can be defined further when more clinical experience is gainedduring the course of research and development. Likewise, the rela-tionship between the DVH curves and the isodose curve lines oneach of the image slices can be loosely defined as

DVHi =M∑

j=1

isodosej,

where a particular tumor or critical structure’s DVH curve i is a sum-mation of all the 1 to M isodose curves within the diagnostic imageslices of the treatment plan. The two models above will depend onrudimentary quantified knowledge data elements. These data ele-ments can be derived from the knowledge base schema which canbe defined into class objects. A few of these are shown in Fig. 5 alongwith their attributes:

• Class Object (1) DVH• Class Object (2) Isodose curve• Class Object (3) Critical structure• Class Object (4) CT image


558 Brent J Liu

Then, for each of these classes, attributes can be defined as shown.For example, for the CT Image class object, there are the primarykey (PK) identifier and five attributes: critical structure curve; iso-dose curve; spatial coordinates of the image including x, y, andz-directions; Pointer to the image data; and DICOM header data.The relationships between each of the class objects are through theforeign keys (FK). For example, in Fig. 5, isodose curve FK1, and crit-ical structure FK2 are related to CT image object. This is because a CTimage would contain multiple isodose curves and multiple critical

1: DVH

PK DVH ID

Critical structure volumePlot points (Dose vs. Vol)Dose limitArea under curveOverdose area

2: Isodose Curve

PK Isodose curve ID

CoordinatesSpatial coordinatesPercentage dose

FK1 CT Image slice ID

4: CT Image

PK CT Image Slice ID

Critical structure curveIsodose curveSpatial coordinatesPointer to imageDICOM data

3: Critical structure

PK Critical structure ID

VolumeStructure type

FK1 DVH IDCoordinatesSpatial coordinatesArea

FK2 CT image Slice ID

Fig. 5. Entity-relationship example of a sample knowledge base for a clinical sce-nario to perform treatment plan assessment for IMRT. The classes defined haveattributes that are extracted from the standardized DICOM-RT data integrated inthe ePR system. Each class carries a primary key (PK) Identifier, and can containforeign keys (FK) which link it to another class object. For example, the (3) criti-cal structure class object contains FK1 and FK2 that link it to a (1) DVH curve aswell as the (4) CT image class objects. Each DVH curve is derived from a criticalstructure and CT images contain critical structure contours. The knowledge base isobject-oriented in design and modular to allow for additional or new knowledgeto be easily integrated. This knowledge is included in the database schema of theweb-based ePR application server.



structures. Another example is that the DVH class uses the criti-cal structure volume attribute to relate to the FK1 volume of thecritical structure object. Once the knowledge has been defined andknowledge base schema developed, the knowledge can be extractedand stored within the knowledge base.Asearch engine can be built toperform queries on the quantified knowledge for automatic extrac-tion of particular knowledge described and further illustrated in thenext section.

22.2.6 Data Mining for Knowledge and Developmentof a Quantification and Visualization Tool

Referring again to the clinical scenario described in this chapter inFig. 2, a quantification and visualization tool can be developed toautomatically mine the knowledge base for the information neededto assess a treatment. The tool design used as an example in thischapter and user interface was developed through the guidanceof the oncologist and physicist at SJHC. Figure 6 shows an illus-tration of a mockup for data mining utilizing quantification andvisualization tools to be developed for decision-support of treat-ment plan assessment. This proof-of-concept method shows howquantified knowledge can be mined for and then visually presentedas an example for further development of powerful and effectivedecision-support tools. This specific tool example eliminates thetedious manual procedure of first analyzing the DVH curves, whichis a 2D plot graph, and then having to toggle through a 3D volumeof CT image slices overlaid with multiple isodose curves to iden-tify locations and characteristics of regions within critical structuresthat are receiving radiation overdose. These areas are sometimescalled “hot spots.” The exact diagnostic CT image slice togetherwith the exact critical structure and tumor contours and isodosecurves representing hot spots can be automatically displayed as awarning and red flag to the oncologist during the review of thetreatment plan. In addition to the illustration presented in Fig. 6,important quantified knowledge measurements can also be dis-played. The results of the research and development based on this


560 Brent J Liu

Optic Chiasm Tumor

Optic Chiasm

DVH Marker

Tumor

Fig. 6. Illustration of an interactive decision-support tool for clinicians. The 2Dnature of the DVH curve (left) is automatically linked to specific image slices (right),isodose curves, critical structure and tumor contours which all are extracted from3D volumes instead of the manual 2D to 3D data registration currently performedby the oncologist and physicist. In this example, the DVH curve for a critical struc-ture volume, the optic chiasm, has been automatically displayed showing whereoverdose is occurring which is the region past 6 000cGY (See left: DVH marker).In addition, the DVH curves for the prescribed tumor volume (PTV) as well asthe clinical tumor volume (CTV) are displayed. Note that the two DVH curvesfor the tumor are nearly overlapping which is a common occurrence. A CT imageslice is automatically extracted and displayed superimposed with the optic chi-asm region and isodose curves showing overdose. In this special case CT slice, theoptic chiasm region (light-colored shaded region) overlaps the tumor to be treated(dashed black line contour) which can make the treatment planning complex andwhere the development of quantified knowledge and tools can be especially ben-eficial. The user can move the arrows (left, bottom) left and right adjusting theDVH marker across the DVH curve which links and displays pertinent diagnosticimages with both critical structure and superimposed isodose curves automati-cally. This is one example of the tools that can be developed based on quantifiedknowledge.

mockup design and the quantified knowledge measurements willbe further discussed in Sec. 22.3. Any design of what quantifiedknowledge to present and how it will be presented should be closelyguided by the oncologist and physicist. Some of this knowledge caninclude:

• Percent region of critical structure covered by an isodose curve• Shape models of overdose and prescription dose regions• Ratio of overdose to critical structure regions• Shape models of DVH curves



• Location models of dose of target tumor and critical structureregions

The decision-support tools can be used real-time by the expert usersanywhere since the ePR is web-based and portable. In the long term,as more knowledge data is collected from additional brain tumorpatients treated with IMRT, as in this example, the knowledge basewill be enriched accordingly.

22.3 RESULTS OF DEVELOPED QUANTIFIED KNOWLEDGEAND DECISION-SUPPORT TOOLS FOR AN EXAMPLEOF A BRAIN TUMOR PATIENT TREATED WITH IMRT

22.3.1 DICOM-RT ePR Timeline Overview Display

This section describes some of the results of an example of a braintumor case treated by IMRT utilizing the TPS for treatment plan-ning and integrated within the DICOM-RT based ePR system. Theend result is a comparison between what clinical information isdisplayed by a conventional RT information system provided bya manufacturer versus that of the richer database of the preliminaryDICOM-RT based ePR system which can provide more informationin the display. In addition, some preliminary development of theknowledge base, quantification, and visualization tools is presentedbased on the particular clinical scenario of assessing treatment plansof brain tumor patients. Figure 7 is a timeline overview displayshowing that a CT and MR diagnostic exam was acquired for a sam-ple patient. Referring to Fig. 7 (bottom), a conventional RT manage-ment information system or record and verify system only has theDICOM RT records but no DICOM RT plan, RT images, and DICOMimages. On the other hand, the DICOM-RT based ePR system is ableto display information extracted from all of the DICOM-RT objectsand can be expanded for more detailed views from the icons on thetimeline in the user interface (both bottom and upper sections). Thedata reconstructed in the DICOM-RT based ePR system were con-verted to DICOM-RT objects that can be further distributed to otherclinical areas and DICOM-compliant clinical systems while the data


562 Brent J Liu

RT ePR

ManagementSystem

Fig. 7. Timeline overview display of a patient in the DICOM-RT based ePR sys-tem. The RT ePR system has a richer database than the conventional RT informa-tion/management system. A given RT information management system has onlythe DICOM RT records (Bottom of figure), while the RT ePR is able to display theinformation extracted from all the DICOM objects including the DICOM RT plan,RT images, and DICOM images (courtesy YY Law).



from the TPS are proprietary and difficult to distribute throughoutthe healthcare enterprise. In addition, this standardized data can beused to develop knowledge based on clinical scenarios as well asdata mining tools to extract this knowledge for decision-support.

22.3.2 Development of a Visualization Toolwith Quantified Knowledge

As an example for this chapter in the medical imaging and informat-ics approach towards the development of decision-support tools, aclinical scenario where an oncologist needs to assess the isodose planof critical structures from a treatment plan for a brain tumor patienthas been identified and applied to this methodological approach.During treatment planning for brain tumor patients, the treatmentplan developed, usually by the physicist, must be approved bythe oncologist as shown earlier in the workflow in Fig. 2. Part ofthe clinical decision-making process for the physician is to ana-lyze the DVH curves of critical structure areas to evaluate whetherthe critical structures are receiving an overdose of radiation thatis clinically unacceptable. These curves only show dose values inrelation to the critical structure volume. The physician must thenevaluate the various isodose plans to first locate areas of overdosewithin the critical structures called “hot spots,” and then to deter-mine whether the plan is acceptable or whether it must be modifiedand recalculated. In order to make this clinical assessment, the oncol-ogist must navigate through multiple image slices showing multi-ple isodose curve lines as well as overlapping critical structures tomake the assessment. Navigation of all this knowledge, while cru-cial, is also extremely tedious and complex since there is no toolto quantify and visualize the direct relationship between the DVHcurves to the diagnostic images and the corresponding dose andcritical structure curves. Based on the methodology described pre-viously, a tool has been designed to automatically display the DVHcurve of a critical structure linked with the diagnostic image slice(s)that contain corresponding isodose curves and critical structureregions.


564 Brent J Liu

22.3.3 Development of a Web-Based GUI for Visualizationof Quantified Knowledge

Figure 8 shows a screenshot of an overview of a particular IMRTtreatment plan (TP). The letter A indicates tabs that display differentpage views. The first view is the TP Overview which shows the gen-eral overview showing the DVH curves and the DICOM CT images

B CA

D

E

Fig. 8. Screenshot showing an overview of a treatment plan. (A) Tabs that dis-play different page views. TP Overview — shows the general overview show-ing the DVH curves and the DICOM CT images with isodose curves overlaid. TPEvaluation Page — shows quantified knowledge. TPknowledge base search page —allows users to query for specific knowledge. TP comparison page — shows quan-tified knowledge for two different treatment plan iterations in comparison mode.(B) timeline display showing all treatment plans, both current and historical of aparticular patient. (C) drop-down window allowing the user to view different itera-tions within a current plan. (D) DVH curve of a particular plan with overdose regionshaded for the particular critical structure, in this case, the optic chiasm. (E) DICOMimages with isodose curves overlaid from the TPS.All Data is in DICOM-RT format.



with isodose curves overlaid. The TP evaluation page allows theuser to quickly assess a treatment plan for hot spots. The TP knowl-edge base search page allows users to query for specific knowledgeand will be developed in the future. The TP comparison page showsquantified knowledge for two different TP iterations in a comparisonmode. The letter B indicates a timeline display showing all TP’s, bothcurrent and historical, of a particular patient. The letter C indicatesa drop-down window which allows the user to view different itera-tions within a current plan. The letter D indicates the DVH Curve of aparticular plan with the overdose area shaded for the Optic Chiasm.The letter E indicates DICOM Images with isodose curves overlaidfrom the TPS. All data is in DICOM-RT format and standardized.The user can further view each of the DICOM images with isodosecurves by selecting one of the images. A pop-up window is gener-ated with a larger image view window. Tools such as Zoom, Pan,and Window/Level are included as well as the ability to toggle onand/or off particular isodose curves or critical structure curves toallow the user to properly review the plan.

Figure 9 shows the TP evaluation page with the DVH curve of aspecific critical structure, the optic chiasm, and the tumor with theoverdose area shaded under the optic chiasm curve. In the leftmostcolumn, only the image slices with overdose regions are extractedfrom the entire CT study and displayed with the regions highlighted.In addition, quantified knowledge such as percent area overdosedis displayed as well. The rightmost column shows only the imageslices where the tumor is not receiving the full clinically prescribedradiation dose. In this manner, the user can quickly assess the treat-ment plan to determine whether the critical structures are beingoverdosed while at the same time the tumor is being prescribed asmuch dose as possible without having to review the entire CT imagestudy.

Finally, Fig. 10 presents the TP comparison page view whichshows two iterations of a current treatment plan side by side eval-uation. Only the image slices with overdosed areas to the optic chi-asm are extracted and displayed with quantified knowledge. In thiscase, there is an improvement between iteration 1 and iteration 2 in


566 Brent J Liu

Fig. 9. TP evaluation page showing the DVH curves of a specific critical struc-ture and the tumor (leftmost column). The middle column shows only the imageslices with overdose regions extracted from the entire CT study and displayed withthe regions highlighted. In addition quantified knowledge such as percent areaoverdosed is displayed as well. The rightmost column shows only the image sliceswhere the tumor is not receiving the full clinically prescribed radiation dose. In thismanner, the user can quickly assess the treatment plan to determine whether thecritical structures are being overdosed while at the same time the tumor is beingprescribed as much dose as possible without having to review the entire CT imagestudy.

the difference in the shaded area of the DVH curves with less beingshown in iteration 2. In addition, there is less number of image slicesextracted with “hot spots” in iteration 2 as compared to iteration 1and the quantified knowledge also confirms this with the percentareas of overdose. This comparison mode, allows the user to quicklycompare between the results of one iteration and a subsequent itera-tion to assess any improvements in the treatment planning process.In addition, this comparison mode can be used to review previousapproved treatments within the knowledge database to help guidethe oncologist and physicist in developing a new treatment plan fora new patient.



Fig. 10. TP comparison page view showing two iterations of a current treatmentplan. Only the image slices with overdosed areas to the optic chiasm are extractedand displayed with quantified knowledge. In this case, there is an improvementbetween iteration 1 and iteration 2 both in the difference in the shaded area (less areain iteration 2 DVH) of the DVH curves as well as the number of image slices (twovs three) extracted with overdose regions and the quantified knowledge showingpercent area of overdose. Note that the third image (lower left) extracted in itera-tion 1 is cut off due to the limitations of the screen shot size and can be viewed inthe active GUI by scrolling the window.

The developed tools help in part to assist the user during thetedious manual procedure of first analyzing the DVH curves andthen having to toggle through a volume of CT image slices withmultiple isodose curves to review and assess the plan. Since data isalready mined, the exact diagnostic CT image slice together withthe exact structure and isodose curves can be automatically dis-played the moment the oncologist opens the case within the ePRsystem. The oncologist would then have the ability to continue tonavigate the presented data or view a different DVH curve if desired


568 Brent J Liu

to make a quicker assessment of the treatment plan for approval. Ifthe oncologist decides that changes are needed in the treatment plan,the decision-support tools can be used to perform a real-time con-sult with the physicist either at different locations or at the samelocation or even directly on the treatment planning system, sincethe ePR is web-based and portable. If there are historical patientsstored within the ePR system, the tools can display all the similarcritical structures (e.g. in the above example, the optic chiasm) ofsimilar treatment plans with the corresponding dose configurationsthat have been approved. This extracted a priori knowledge wouldhelp the clinician to decide on an initial plan for a new brain tumorpatient planning to be treated with IMRT and perhaps shorten theiterative process of the inverse treatment planning workflow.

22.4 DISCUSSION

In the previous sections, an imaging informatics methodology wasapplied to radiation therapy planning to develop quantified knowl-edge and decision support tools.As an example, a DICOM-RT basedePR system for managing patients with brain tumor cases was intro-duced with an example of patients from the radiation oncologydepartment, Saint John’s Health Center, Santa Monica, CA. Dataobtained for the example was a brain tumor case where the treat-ment was planned on the IMRT TPS. The richness of the clinicaldata available was shown in comparison to standard RT informa-tion management systems. The results show that with the availabil-ity of standardized DICOM-RT data, further knowledge base anddecision-support tools development can be realized to aid the clini-cians in critical decision-making processes.

Figure 11 shows the new knowledge-enhanced inverse treatmentplanning workflow with the ePR system with quantified knowl-edge integrated in dashed lines within the clinical feedback loopdescribed in the original clinical workflow in Fig. 2. This knowledge-enhanced inverse treatment planning approach may eliminate thefeedback loop and subsequent iterative steps of recomputing of a



Approve plan?

Physicistcomputes plan

on TPSYes

Oncologist or physicistoutlines critical

structure

Crit struct dataconverted to DICOM-RT object and sent to

ePR system

Data mining in ePRsystem of historic

knowledge of criticalstructure

Onc and Phys Review auto-retrieved knowledge using decision-support tools and

determine dose constraints

TPS results converted to DICOM-RT objects and sent

to ePR system

Oncologist review results onePR system w/ decision-

support tools and can also compare new results w/

retrieved historic knowledge

Fig. 11. Knowledge-enhanced inverse treatment planning: dashed lines showwhere workflow steps would be performed in the ePR System as compared tothe current feedback loop workflow in dash-lined rectangle shown in Fig. 2.

treatment plan since the first attempt was acceptable based on theprior knowledge. Because each plan is computationally complex andtime-consuming, a best practice first computed plan aided by pre-vious knowledge would greatly enhance the decision-making pro-cess and ultimately shorten the length of time before the patientundergoes treatment as well as better preserve normal tissue andquality of care. Future progress includes the complete developmentand collection of a suite of knowledge base and tools as well as aclinical evaluation of the decision-support tool development and itsimpact on the overall clinical workflow within the radiation oncol-ogy department.


The imaging and informatics methodology introduced for the devel-opment of decision-support tools based on standardized DICOM-RTdata within the ePR system represents a new frontier for image-assisted knowledge discovery within the realm of radiation ther-apy planning. As an example in this chapter of how crucial stan-dardized RT data can be, a clinical scenario was developed whereknowledge base was defined and quantification and visualizationtools were designed to extract the knowledge and display it for adecision-making process for a brain tumor case undergoing IMRT.By implementing this DICOM-RT based ePR system, both clinicalimage and related informatics data are integrated into a one-stop


570 Brent J Liu

source of pertinent clinical information necessary for making treat-ment decisions within the RT department and throughout the health-care enterprise. With the medical imaging informatics methodologyintroduced in this chapter, the decision-support and knowledge basedevelopment can be easily extended to various lesion types as wellas other inverse treatment planning methods.

References

1. Brem RF, Hoffmeister JW, Rapelyea JA, Zisman G, et al., Impact of breastdensity on computer-aided detection for breast cancer, Am J Roentgenol184: 439–444, 2005.

2. Dayhoff R, Siegel E, Digital Imaging Within andAmong Medical Facili-ties, in R Kolodner (ed.) Computerized Large Integrated Health Networks —The VA Success, 473–490, Springer Publishing, New York, 1997.

3. Liu BJ, Huang HK, Cao F, Zhou MZ, et al., A complete continuous-availability, PACS archive server, Radiographics 24: 1203–1209, 2004.

4. Liu BJ, Cao F, Zhou MZ, Mogel G, et al., Trends in PACS image storageand archive, Comput Med Imaging Graph 27: 165–174, 2003.

5. Palta JR, Frouhar VA, Dempsey JF, Web-based submission, archive,and review of radiotherapy data for clinical quality assurance: A newparadigm, in J Radiat Oncol Biol Phys 57(5): 1427–1436, 2003.

6. Zhou MZ, Huang HK, Cao F, Zhang J, et al. A RIS/PACS simulatorwith web-based image distribution and display system for education,Proceedings of the SPIE on CD-ROM, Medical Imaging 372–381, San Diego,CA, USA, 2004.

7. Doi K, MacMahon H, Giger ML, Hoffman KR (eds.), Computer-AidedDiagnosis in Medical Imaging, Elsevier Science Ltd., Chicago, 1998.

8. Connolly T, Begg C, Database Systems — A Practical Approach to Design,Implementation, and Management, 2nd edn., Addison Wesley, England,1998.

9. Digital Imaging and Communications in Medicine (DICOM), Supple-ment 11: Radiotherapy Objects, 1997.

10. Digital Imaging and Communications in Medicine (DICOM), Supple-ment 29: Radiotherapy Treatment Records and Radiotherapy Media Exten-sions, 1999.

11. DICOM Standard 2003, http://medical.nema.org/dicom/2003.html.12. Huang HK, PACS: Basic Principles and Applications, pp. 521, Wiley &

Sons, NY, 1999.



13. Law MYY, Huang HK, Concept of a PACS and Imaging Informatics-Based Server for Radiation Therapy, Comput Med Imaging Graph 27(1):1–9, 2003.

14. Law MYY, Huang HK, Chan CW, Zhang X, Zhang J, A DICOM-basedradiotherapy information system, Proceedings of the SPIE on CD-ROM,Medical Imaging, 309–317, San Diego, USA, 2004.

15. Law MYY, Huang HK, Zhang X, Zhang J, DICOM and imaginginformatics-based radiation therapy server, Proceedings of the SPIE onCD-ROM, Medical Imaging 160–167, San Diego, CA, USA, 2002.

16. Law MYY, Huang HK, Zhang X, Zhang J, The data model of a PACS-based DICOM radiation therapy server, Proceedings of the SPIE on CD-ROM, Medical Imaging, 128–129, San Diego, CA, USA, 2003.

17. Law MYY, A model of DICOM-based electronic patient record in radi-ation therapy, Comput Med Imaging Graph, 2004.


CHAPTER 23

Lossless Digital Signature EmbeddingMethods for Assuring 2D and 3D

Medical Image Integrity

Zheng Zhou, HK Huang and Brent J Liu

Medical image integrity, which assures that the original image is not acci-dentally or deliberately modified by unauthorized person, has becomecritical when medical images are stored in an archive or transmittedover public networks. A two-dimensional (2D) lossless digital signatureembedding (LDSE) method has been developed for assuring the imageintegrity by permanently embedding the digital signature (DS) of animage into image pixels. Experimental results show that the 2D LDSEmethod is effective for assuring image integrity. With the advent of mul-tidetectors and volume acquisition technologies, a CT, MR, or US exam-ination can generate hundreds to thousands of three-dimensional (3D)volumetric image sets, further aggregating the importance of the individ-ual image as well as the 3D volume integrity. The 2D LDSE method orother security technology such as DICOM transport layer security (TLS)is not effective and efficient for assuring the integrity of 3D image vol-umes. A novel 3D LDSE method has been developed for assuring theintegrity of large 3D image volumes. Experimental results with various3D medical images demonstrate that the method is effective and effi-cient for assuring the integrity of 3D volumetric images both for archiveand during transmission. In order to apply the 2D and 3D LDSE meth-ods to clinical diagnostic workflow, the integration of the LDSE methodswith a PACS has been developed. The 3D LDSE method has also beenintegrated with two relevant Integrating the Healthcare Enterprise (IHE)profiles, key image note profile and post-processing workflow profile,accordingly.

573


574 Zheng Zhou, HK Huang and Brent J Liu

23.1 INTRODUCTION

Image security is a critical issue when medical images with perti-nent patient information are transmitted over public networks.1−3

With integrated medical imaging systems being extensively usedin clinics for healthcare delivery, image security consideration isno longer limited to images in transit but also in storage. Gener-ally, medical image security can be characterized by three majorissues: privacy (or confidentiality), authenticity, and integrity.4 Pri-vacy seeks to protect image data from being accessible or disclosedto unauthorized individuals. Authenticity verifies that the sourceof an image is what it claims to be. Integrity assures that imagedata is not altered, destroyed, or deleted by unauthorized person.With current information technology and knowledge of its use,it is easy to alter a medical image without detection when theimage is in transit or in storage. The consequence of such alter-ation could influence the intended objectives, behavior, and func-tionalities of healthcare services, and even worse, could cause legalproblems.5,6 For these reasons, image integrity is one of the mostparamount concerns of current clinical imaging systems. Tradi-tional methods, such as encryption, firewall, virtual private net-work, and access control by user password, have been used toprotect the privacy and authenticity of image data. These meth-ods, however, are not effective for assuring the image integrity,because the image can still be altered or destroyed by an intruderwho does not need to have the knowledge of the content of theimage.

A lossless digital signature embedding (LDSE)7 method hasbeen developed for assuring the integrity of two-dimensional (2D)medical images in transit and in storage. Experimental resultsshow that the method is effective in the assurance of 2D med-ical image integrity. With the advent of multidetectors and vol-ume acquisition technologies, a CT, MR or US examination cangenerate various three-dimensional (3D) volumetric image setsconsisting of hundreds or even thousands of images. To performconventional DICOM TLS,8 SSL,9 or the 2D LDSE methods on each


Lossless Digital Signature Embedding Methods 575

individual image in the volume would be time consuming andinefficient. In order to overcome these, a novel 3D LDSE methodhas also been developed for assuring the integrity of 3D medicalimages.


The goal of LDSE is to provide robust integrity assurance to med-ical images in various application environments. In pursuing thisgoal, it is important to permanently embed the digital signature(DS) in the image pixels. A permanently embedded DS would pro-vide image integrity assurance for medical images during theirlifetime.

23.2.1 General LDSE Method

The LDSE method consists of two processes (Fig. 1):

(1) Sign and Embed Processes

a. Generate the DS of the image pixels with the image owner’sprivate key:

s = Sk,priv(I), (1)

Sign process

Embed process

Medical image

Signature embedded

image

Signature embedded

image

Extract process

Verifyprocess

Verification result

Sign and Embed processes

Extract and Verify processes

I s I*

I* s' v

I'

Fig. 1. Data flow of Sign and Embed, and Extract and Verify processes in the LDSEmethod. I: original image, s: signature of the original image I∗: signature embeddedimage, s’: recovered signature, I’: recovered image, v: verification result.



where s is the digital signature (DS) of the image, S denotesthe signature signing process,a k, priv is the owner’s privatekey, and I is the image.

b. Embed the bit stream of DS into the image pixels using losslessdata embedding approaches:

I∗ = I ⊕ s, (2)

where I∗ is the signature embedded image, ⊕ denotes thelossless data embedding process, and s is the DS.

(2) Extract and Verify processes

a. Extract the DS from the signature embedded image andrecover the image from the embedding process:

(s, I ′) = �I∗, (3)

where s is the DS, I ′ is the recovered image, � denotes thedata extraction process, I∗ is the signature embedded image.

b. Verify the extracted DS with the owner’s public key:

v = Vk, pub(I ′, s), (4)

where v is the verification result, V denotes the signature ver-ification process,b k, pub is the owner’s public key, I ′ is therecovered image, and s is the DS of I. If the verification resultis true, which means the image has not been altered, the imageintegrity is assured. If the verification is false, the image hasbeen altered.

23.2.2 2D LDSERS Algorithm10,11

23.2.2.1 Algorithm definition

Consider the original N × M medical image with pixel values in theset P = {0, . . . , 4095, or higher}. The algorithm starts by dividing the

aSignature signing process begins by computing a hash value of all pixels of the image usingcryptographic hash functions (e.g., SHA1) and follows by encrypting the hash value withpublic key encryption method.bSignature verification process begins by decrypting the DS to get the original hash value andthen compares this hash value to a second hash value computed from the recovered imageusing the same hash function used in signing process.



original image into disjoint groups of n adjacent horizontal pixels(p1, . . . , pn) e.g., (p1, . . . , p4), where p stands for the value of pixel andn is an integer greater than 1. A discrimination function ‘f ’, definedin (5), computes the correlation coefficients of each pixel group G =(p1, . . . , pn). The function ‘f ’ converts the vector G into a number f (G).

f (G) =n−1∑

1

|pi+1 − pi| (5)

An invertible operation F on G called “flipping” is also defined. Flip-ping of a given bit in a pixel is defined as “0”→“1” or “1” →“0”. Theflipping would change the value of pixel and the value change woulddepend on the bit locations in the pixel.Appendix I describes the flip-ping operation in more detail. F has the property that F(F(p)) = p forall p in G. Thus, there are three possibilities if f (F(G)) is comparedto f (G). These three possibilities are defined as three groups: R, Sand U.

Regular (R) group: if f (F(G)) > f (G)

Singular (S) group: if f (F(G)) < f (G)

Unusable (U) group: if f (F(G)) = f (G)

A new grouped image is formed with these three possible states inthe selected bit plane.

23.2.2.2 Embedding

The embedding (Fig. 2) starts with the scanning of image pixels tofind R and S groups. The U groups are skipped during scanning.For n = 4, G = (p1, . . . , p4), our experimental results with the currentmedical images used in clinical practice show that f (F(G)) > f (G)after the flipping operation F. This is because F makes G less corre-lated, where the adjacent pixels are usually correlated. The relation-ship of the four pixels in every found group can be converted to an“R” (R group) or “S” (S group) symbol. As a result, an “R” and “S”sequence of the image is formed. One bit is assigned to every “R” or“S” symbol in this sequence and the value in this bit would be “1” for“R” and “0” for “S”. Thus, the “R” and “S” sequence is converted to



An “R” and “S” sequence

CompressedRS bit stream

DS

scan

image

embed

lossless compression

RS bit stream

Assign “1” to an “R” and “0” to an “S” symbol in the “R” & “S” sequence

C1 C2

Fig. 2. Embed the DS in an MR image using 2D LDSERS. The U groups are notused. C1: counter 1 to record the length of the compressed RS bit stream. C2: counter2 to record the length of the DS.

a bit stream of 1s and 0s, which is called an “RS bit stream.” The RSbit stream is then losslessly compressed using adaptive arithmeticcoding.12 The RS bit stream extraction and compression processesare complete until:

lRS − (lRScomp + lDS) ≥ 0, (6)

where lRS denotes the binary length of the RS bit stream, lRScomp

denotes the binary length of the compressed RS bit stream, and lDS

denotes the binary length of the DS.Afterward, the bit stream of DS is appended to the compressed

RS bit stream to form a new bit stream. This new bit stream is thencompared with the RS bit stream bit by bit. If there is no differencein the bit value, no change is made. If there is a difference, the corre-sponding group of pixels (R or S group) is flipped. After all the bitsare compared, the embedding process is complete and the result isa signature embedded image.

Since the forming of R and S groups as well as the embedding areall reversible processes, the original image can be completely recov-ered after the DS is extracted. The extracting process starts with thesame scanning to find R and S groups from the signature embed-ded image. As a result, the embedded bit stream is reconstructedfrom the R and S groups. The bit stream is then broken down intothe compressed RS bit stream and the DS. The compressed RS bitstream is decompressed to recover the original R and S groups. Theoriginal R and S groups are compared to the extracted groups and



the corresponding group of pixels is flipped if there is any differ-ence. Since the flip operation is reversible, the original image pixelscan be completely recovered. The recovered DS is verified with therestored image. If verification results is true, there is no alteration ofthe image and the image integrity is assured.

23.2.3 General 3D LDSE Method

The general 3D LDSE method consists of two processes: signing andembedding and extracting and verifying.A3D volume of a single CTseries with n images is used as an example to illustrate the methodin the following sections. If there are multiple series in an exam, themethod can be applied to each series separately.

23.2.3.1 Signing and embedding

In order to make it more difficult to extract the embedded digitalsignature, randomization is utilized to rearrange the image order inthe CT volume before actual embedding. The random order is gen-erated based on a set of pseudo-random numbers rk computed bythe random number generator.13,14 After the rearrangement, all thepixels are arranged into a pixel stream starting from the first pixel ofthe first image “1” to the last pixel of the last image “2” as shown inFig. 3. The n is the random order. A hash value is computed for allpixels in the pixel stream using cryptography hash functions suchas SHA1.15 The hash value is then encrypted to form a digital signa-ture (DS) of the whole volume using public-key encryption methodsuch as RSA.16,17 Finally, the DS is embedded in pixels of all imageswithin the volume using a lossless embedding algorithm. Since theimages are still in random order, the DS is embedded according tothis order. The result of embedding is a signature embedded imagevolume. After embedding, the images within the volume are rear-ranged into the original order; therefore the LDSE method will notaffect clinical data flow.

The embedding algorithm used in the 3D LDSE method is dif-ferent from the 2D LDSE method in that it uses all images of the



Hashvalue

Digital signature

compute

encrypt

losslessembed

2

1

n

Fig. 3. The general 3D lossless digital signature embedding (LDSE) method.A single digital signature is generated for a CT volume for assuring the integrity ofthe volume. The original 3D volume set with 1, …, n images has been randomized(1, n, …, 2).

volume as a whole for embedding. In comparison, the 2D methodembeds the 2D image signature in each individual image within thevolume.

23.2.3.2 Extracting and verifying

When extracting and verifying, the same random seed is usedto reproduce the same random order in the signature embeddedvolume. Images in the 3D volume are then rearranged accord-ing to this random order. Since embedding is an invertible pro-cess, the DS can be extracted and the original volume can becompletely recovered. The extracted DS is then decrypted andverified with the hash value computed from the recovered vol-ume. The verified volume is rearranged into the original order forclinical use.

23.2.4 3D LDSERS Algorithm

To embed a digital signature in a 3D image volume is a com-plex problem. A 3D LDSERS (regular/singular groups) algorithmhas been developed extended from 2D LDSERS described inSec. 23.2.2.



1

n

2

y

xz

Fig. 4. The 3D LDSERS uses a Z-shape walking pattern to search the R and S groupsfor data embedding. Four voxels are used to form a group in order to increase thebit compression ratio to accommodate digital signature embedding.

23.2.4.1 Embedding

The embedding starts by searching R and S groups in the imagesof the CT volume. A Z-shape walking pattern (Fig. 4) is utilized tosearch the R and S groups, which consist of four voxels in each group,in the CT volume. Our experimental results showed that four worksbest for all tested image sets. A voxel is defined as px,y,z, where x rep-resents the horizontal line (or row), y the vertical line (or column),and z the image number in the randomized volume. For example,p1,1,1 represents the voxel in the first row and the first column ofthe first image. After the Z-shape walking, the extracted groupsof voxels are (p1,1,1, p1,2,1, p1,3,1, p1,4,1), (p1,1,n, p1,2,n, p1,3,n, p1,4,n), …,(p1,1,2, p1,2,2, p1,3,2, p1,4,2), (p1,5,1, p1,6,1, p1,7,1, p1,8,1), …. A discriminatefunction “f” is defined for computing the correlation coefficientsof the group of voxels.

f (pi,j,k, . . . , pi,j+3,k) =3∑

j=1

|pi,j+1,k − pi,j,k| (7)

The R and S groups are found in the volume of randomized imagesand converted into an RS bit stream by applying Eq. (5) and theflipping operation F defined in 2D LDSERS on the extracted groupsobtained from the Z-shape walking. The RS bit stream is then lossless



compressed. The walking and compress processes end until there issufficient space, which can be estimated before hand from character-istics of the image volume, in the RS bit stream to embed the DS. TheDS is converted to a DS bit stream, which is appended to the com-pressed RS bit stream to form a new bit stream. This new bit streamis then compared with the original RS bit stream bit by bit. If there isno difference in the bit value, no change is made. If there is a differ-ence, the corresponding group of voxels (R or S group) is flipped.After all the bits are compared, the embedding process is completeand the result is a signature embedded volume. After embedding,the 3D volume is rearranged to the original order.

23.2.4.2 Extracting

When extracting, the images in the CT volume is rearranged in thesame order as in the embedding process. The R and S groups are thenfound based on the same Z-shape walking pattern. The embeddedbit stream is reconstructed from the R and S groups. This bit streamis broken down into the compressed RS bit stream and the DS. Thecompressed RS bit stream is decompressed to recover the originalR and S groups. The original R and S groups are compared to theextracted groups and the corresponding group of voxels is flippedif there is any difference. Since the flip operation is reversible, theoriginal CT volume can be completely recovered. The verification ofthe DS has been described in Sec. 23.2.3.

23.2.5 From A 3D Volume to 2D Image(s)

In many clinical scenarios such as a referring physician retrievingimages for clinical review, only several 2D images from a 3D volumewould be needed instead of the entire volume. These several imagesare usually the significant images that radiologists had selected forreferring physician’s review. How to protect these several 2D imagesin transit and in storage becomes a new data integrity issue.

For example, assuming only the third image in the CT volumeis needed, and that 3D volume already has a signature embedded



using the 3D LDSERS algorithm described in Sec. 23.2.4. The proce-dure of how to protect the image integrity of this selected image isas follows:

(1) The method starts by recovering the original 3D CT volumeusing the extract and verify process of the 3D LDSERS algorithm.

(2) Once the verification result assures the integrity of the CT vol-ume, a copy of the third image is extracted.

(3) A digital signature of this single CT image is generated andembedded in the image pixels using the 2D LDSERS algorithmdescribed in Sec. 23.2.2. The signature embedded image is sentto the physician for review. If more than one single image isrequired, then step 3 is repeated for each image.

By using this method, the integrity of both the 3D volume and theextracted 2D images can be assured during the workflow wherea physician is retrieving specific images from a 3D volume. Thismethod can be directly applied to Integrating the Healthcare Enter-prise (IHE) key image note profiles to be discussed in Sec. 23.4.18

23.3 RESULTS

23.3.1 Data Collection

The 2D LDSERS algorithms was tested with four major modalitytypes of 2D images used in current clinical practice, including CR,CT, grayscale US, and MR. Although color US images were not eval-uated in the experiments, the LDSERS method can be used for colorUS images by embedding the digital signature in the three chromi-nance components of the color image. A total of 762 images, includ-ing 152 CR images, 204 CT images, 204 grayscale US images and 202MR images, have been collected. Most of the images collected havestandard spatial and density resolution: MR (256 × 256 × 12), CT(512 × 512 × 12), and US (640 × 480 × 8), and CR images varyingfrom 2010 × 1670 × 12 to 2510 × 2000 × 12.

Thirty image sets from three most common 3D imaging modali-ties, CT, US, and MR, were collected for evaluating the performance



of the 3D LDSERS algorithm. The maximum number of images inthese image sets was 176, while the minimum number was 10.

23.3.2 2D LDSERS Results

Examples of the tested images and their corresponding results areshown in Fig. 5. Figures 5(A) and (B) depict the signature embeddedCT chest image using the 2D LDSERS algorithm and the sub-tracted image between the signature embedded image and theoriginal image, Figs. 5(C) and (D) depict the signature embeddedUS OBGYN image and the corresponding subtracted image, andFigs. 5(E) and (F) depict the signature embedded CR hand image andthe corresponding subtracted image. The subtracted images wereobtained by subtracting the original image from the correspondingsignature embedded image. The subtracted image appears black ina regular window/level display. After window/level adjustments,the embedded data becomes visible. A horizontal strip shape pat-tern is observed in the subtracted images [e.g. Fig. 5(B)]. The stripshape shows that every bit embedding changes four adjacent pixelsin 2D LDSERS.

23.3.3 Time Performance of 2D LDSERS

The time performance of Sign and Embed processes as well as Extractand Verify processes of the 2D LDSERS algorithm have been com-puted and tabulated in Table 1. The process time of “Embed” or“Extract” was in hundredth seconds level for all four types of images.This demonstrates that the 2D LDSERS is efficient for assuring theintegrity of a single 2D image. However, the processing time for animage examination with hundreds of those images could still have alengthy overall time, which can be shortened by using 3D LDSERS.

23.3.4 3D LDSERS Results

The 3D LDSERS algorithm was evaluated in two steps. First, onedigital signature is generated for the entire volume set and the



(A) (B)

(C) (D)

(E) (F)

Fig. 5. Example of the 2D LDSERS results. (A) CT chest image with signatureembedded; (B) the subtracted image between the original CT chest image and (A)showing where the digital signature is embedded; (C) US OBGYN image withsignature embedded; (D) The subtracted image between the original US imageand (C); (E) CR hand image with signature embedded; (F) The subtracted imagebetween the original CR image and (E).



Table 1. Time Performance of the 2D LDSERS Method

Per image Average Process Time of the LDSE Method (seconds)

Sign Embed Extract Verify

MRI 0.013 0.018 0.019 0.013CT 0.019 0.041 0.029 0.016US 0.014 0.033 0.042 0.013CR 0.19 0.09 0.08 0.19

signature is embedded in the volume set using the 3D LDSERS algo-rithm. Second, the digital signature was extracted from the signatureembedded volume set and verified.

Figures 6–8 show three examples of an MR breast, an USOBGYN, and a CT reformatted coronal chest volume set from ourresults. Each figure depicts four consecutive images from each ofthese three volumes sets with a partial signature embedded, andthe subtracted images between the original and the corresponding

(A)

(B)

Fig. 6. Example of the 3D LDSERS results of MR breast volume. (A) four consec-utive images of the MR volume with a partial digital signature embedded; (B) thesubtracted images between the four original MR images and (A) showing wherethe digital signature is embedded.



(A)

(B)

Fig. 7. Example of the 3D LDSERS results of US OBGYN volume. (A) Four consec-utive images of the US volume with a partial digital signature embedded; (B) Thesubtracted images between the four original US images and (A) showing where thedigital signature is embedded.

(A)

(B)

Fig. 8. Example of the 3D LDSERS results of a reformatted CT coronal chest vol-ume. The images are reformatted and displayed from anterior to posterior. (A) Fourconsecutive images of the CT volume with a partial digital signature embedded;(B) the subtracted images between the four original CT images and (A) showingwhere the digital signature is embedded.



signature embedded images.An intuitive view of the pixels changedafter the data embedding process can be observed from the sub-tracted images in Figs. 6–8 (B). A horizontal strip can be observed inevery subtracted image of the volume set (e.g. Fig. 6 (B)). The stripshows that every 1/0 bit embedding process changes four adjacentpixels in the 3D LDSERS algorithm. As it can be seen, the portionof pixels being changed in every image is small, which means thatplenty of space is still available for embedding more data. This is oneof the advantages of the 3D over the 2D image embedding methods,7

because the embedded data can be distributed in the entire volumeinstead of just a single image.

23.3.5 Time Performance of 3D LDSERS

The time performance of the 3D LDSERS has been recorded and theresults tabulated in Table 2. The results demonstrate:

23.3.5.1 Sign or verify

• The process time to Sign or Verify increased steeply as the totalsize of the image sets increased. For instance, the time to sign is0.125 seconds for the MR set 5 (3.16 megabytes), whereas it is 2.516seconds for the MR set 14 (94.4 megabytes).

• The size of the image but not the number of images contained inan image set was a main factor in determining the process time ofthe digital signature. For instance, although the MR set 10 containsmore images than the MR set 1, the process time to sign for MRset 10 is much shorter than the MR set 1 that is larger than the MRset 10 in total size of images.

• Different digital signature algorithms will also affect the pro-cess time. Our experimental results show that SHA1withRSA16

has a faster process time for medical images than otherdigital signature algorithms, including SHA1withDSA16 andRIPEMD160withRSA.16



Table 2. Time Performance of the Image Volume Sets Using 3D LDSERS vs2D LDSERS

3D Volumesets (numberof images)

3D LDSERS 2D LDSERS

Sign + Embed(seconds)

Extract +Verify

(seconds)

Sign + Embed(seconds)

Extract +Verify

(seconds)

MR set1 (20) 0.25 + 0.14 0.19 + 0.25 0.34 + 0.55 0.34 + 0.34MR set2 (23) 0.11 + 0.05 0.05 + 0.11 0.42 + 0.51 0.40 + 0.38MR set3 (23) 0.12 + 0.06 0.06 + 0.11 0.38 + 0.38 0.53 + 0.38MR set4 (23) 0.30 + 0.06 0.05 + 0.30 0.42 + 0.58 0.66 + 0.39MR set5 (25) 0.12 + 0.06 0.06 + 0.12 0.38 + 0.53 0.62 + 0.43MR set6 (36) 0.46 + 0.06 0.05 + 0.48 0.65 + 0.62 1.05 + 0.62MR set7 (40) 0.59 + 0.05 0.05 + 0.58 0.78 + 0.65 0.67 + 0.66MR set8 (40) 0.59 + 0.06 0.05 + 0.61 0.74 + 0.66 0.65 + 0.66MR set9 (49) 0.22 + 0.06 0.06 + 0.20 0.70 + 0.94 0.88 + 0.77MR set10 (57) 0.09 + 0.06 0.05 + 0.09 0.71 + 0.88 0.89 + 0.88MR set11 (160) 1.69 + 0.37 0.51 + 1.67 2.51 + 4.30 3.03 + 2.65MR set12 (160) 1.73 + 1.11 1.26 + 1.67 2.49 + 4.34 3.04 + 2.43MR set13 (160) 1.69 + 0.52 0.64 + 1.67 2.59 + 4.09 2.51 + 2.55MR set14 (176) 2.52 + 0.34 0.19 + 2.42 3.21 + 3.27 2.89 + 2.97US set1 (30) 0.28 + 1.10 0.53 + 0.28 0.42 + 0.99 1.26 + 0.39US set2 (54) 0.48 + 0.86 0.39 + 0.76 0.82 + 1.48 1.89 + 0.85US set3 (38) 0.34 + 0.05 0.03 + 0.34 0.53 + 1.25 1.60 + 0.49US set4 (42) 0.39 + 0.05 0.05 + 0.37 0.59 + 1.38 1.76 + 0.55CT set1 (10) 0.17 + 0.06 0.06 + 0.17 0.19 + 0.41 0.29 + 0.16CT set2 (20) 0.33 + 0.06 0.08 + 0.31 0.38 + 0.82 0.58 + 0.32CT set3 (29) 0.44 + 0.06 0.06 + 0.44 0.55 + 1.19 0.84 + 0.46CT set4 (42) 0.62 + 0.06 0.06 + 0.61 0.80 + 1.72 1.22 + 0.67CT set5 (51) 0.73 + 0.19 0.14 + 0.73 0.97 + 2.09 1.48 + 0.81CT set6 (59) 0.84 + 0.08 0.06 + 0.83 1.12 + 2.42 1.71 + 0.94CT set7 (72) 1.00 + 0.06 0.08 + 1.00 1.37 + 2.95 2.09 + 1.15CT set8 (80) 1.12 + 0.06 0.08 + 1.11 1.52 + 3.28 2.32 + 1.28CT set9 (90) 1.28 + 0.20 0.16 + 1.25 1.71 + 3.69 2.61 + 1.44CT set10 (100) 1.42 + 0.19 0.11 + 1.39 1.84 + 3.42 2.17 + 1.63CT set 11 (69) 1.15 + 0.16 0.17 + 1.05 1.31 + 2.83 2.00 + 1.10CT set 12 (97) 1.37 + 0.06 0.05 + 1.33 1.84 + 3.98 2.81 + 1.55



23.3.5.2 Embed or extract

The process time to “Embed” or “Extract” is mainly determined bycorrelation of adjacent pixels of each image in an image set because ofthe concept of LDSERS algorithm. For images with a high correlationbetween adjacent pixels, the process time to “Embed” or “Extract”is short. For example, MR sets 11 and 12 contain the same number ofimages and the same total size of images, whereas the process timeto “Embed” for the MR set 12 is more than three times longer thanthe MR set 11.

23.3.5.3 3D LDSERS vs 2D LDSERS

(a) Sign or VerifyIn order to compare the time performance between the 3D LDSERSand the 2D LDSERS applied to all images in the volume, the 2DLDSERS algorithm was applied to every image in each image set inTable 2. The process time of 2D LDSERS has been measured and theresults are tabulated in Table 2 as well. The process time to “Sign”or “Verify” of every volume set using 3D LDSERS is faster than2D LDSERS. For instance, the process time to sign for MR set 2using 3D LDSERS is about 0.3 seconds less than 2D LDSERS. Thisis because only one digital signature was generated for a volumeset in 3D LDSERS resulting in a faster performance time for publickey encryption required in digital signature. For example, to gen-erate 10 digital signatures for 10 images in the volume would needten times public key encryption, whereas it only needs one encryp-tion using 3D LDSERS. This reduction saves process time, sincethe public-key encryption was a relatively slow process.16 The 3DLDSERS saves even more process time when the number of imagesin the volume increased. For instance, the process time to sign isabout 0.7 seconds less for the MR set 14 using 3D LDSERS than2D LDSERS.

(b) Embed or ExtractA more significant improvement in time performance occurs inthe “Embed” or “Extract” processes. The processing time using



3D LDSERS remained in the tenth to hundredth of seconds for mostvolume sets, whereas most of the process time using 2D LDSERS wasmore than one second for the same volume set shown in Table 2. Themaximum difference was more than 60 times less for the CT set 12by using 3D LDSERS than by 2D LDSERS.

(c) Embed or Extract versus Sign or VerifyThe time to “Embed” or “Extract” process was shorter than the timeto “Sign” or “Verify.” When the number of images in the volume setbecame larger, the former could be only about a tenth of the latter.For instance, the time to Embed for the CT set 10 is 0.188 secondsversus the time to Sign 1.422 seconds. These results show that thetime to “Embed” or “Extract” process becomes almost negligiblewhen the number of images increases in volume. All these resultsindicate that using the 3D LDSE method for 3D volumes is far moreefficient than using the 2D LDSE method.

23.4 APPLICATION OF 2D AND 3D LDSE INCLINICAL IMAGE DATA FLOW

23.4.1 Application of the LDSE Method in a Large MedicalImaging System Like PACS

The goal of the integration of the LDSE method with imaging sys-tems is to assure the integrity of an image right after it is generatedfrom an imaging modality. Thus, the LDSE Sign and Embed pro-cess should be positioned near the imaging modality as close aspossible. A PACS simulator19,20 has been developed as a test bedfor evaluating the system integration of the LDSE method witha PACS.

Currently, no network security, such as DICOM Transport LayerSecurity (TLS), has been applied in current clinical PACS; thereforethe communication between any two PACS components is not safe.Each of the PACS components is not safe either. The image integrityin every point after the image modality, therefore, has to be assuredusing the LDSE method. Figure 9 shows the ideal system integration



AMS aDICOMGateway

c

LocalDisk

PACS Controller

e

Archive

ViewingW.S.

Sign andEmbed

Extract andVerify

1

2

3 4 5

6

7 8

f

9

b d10

11

Clinical PACS

Fig. 9. System integration of the LDSE methods with the PACS simulator with-out Transport Layer Security (TLS). The PACS simulator consists of an acquisitionmodality simulator (AMS), a DICOM gateway, a PACS Controller, and viewingworkstations (W.S.). The connection between these components can be private orpublic networks. Black boxes represent the LDSE processes. a: signer (Sign andEmbed processes), b–f: verifier (Extract and Verify processes).

of the LDSE method with the PACS simulator. The data flow is asfollowing:

(1) Modality simulator passes the DICOM image to the signer “a,”which calls the LDSE Sign and Embed process to embed the DSof the image in the pixels. Once the DS is embedded, it becomesa permanent part of the image. This is the only place where thedigital signature is signed and embedded.

(2) “a” sends the signature embedded image to the verifier “b” usingthe DICOM communication protocol.

(3) “b” calls the LDSE Extract and Verify process to verify digitalsignature. If the signature is valid, “b” forwards the image to theDICOM gateway.

(4) DICOM gateway receives the signature embedded image andstores it in its local disk. It then reads the image file back fromthe local disk and passes it to the verifier “c.”

Steps 5–11 repeat steps 3 and 4 verifying the digital signature ineach imaging component and the communications between everytwo different components.



By verifying the signature from “b”–“f”, the image integrity iscompletely assured in transit and in storage within each componentuntil the image reaches the viewing workstation.

If DICOM TLS is applied in the PACS simulator, then the pro-tection of the image integrity in transit (e.g. verifiers “b”, “d”, and“f”) can be omitted. Besides these verifiers, other procedures arestill necessary for assuring the image integrity in the archive of eachcomponent. By combining the LDSE method with DICOM TLS, theimage integrity in PACS is completely assured. If an image is foundaltered during the transmission, the verifiers (e.g. “b”) would rejectthe image and ask the sending component to resend the image.

23.4.2 Integration of the 3D LDSE Methodwith the Two IHE Profiles

3D imaging modalities can greatly improve the quality of clinicaldiagnosis by providing additional features. For example, a seriesof reformatted CT coronal images generated from 3D postprocess-ing provides more information for diagnosis when it is reviewedtogether with original axial CT images. To integrate these featureswith clinical imaging systems like PACS seamlessly, however, novelclinical image data flow is required. Integrating the HealthcareEnterprise (IHE) has released two important profiles relevant to 3Dvolume image data. One is key image note profile and the otheris Post-processing workflow profile. In order to apply the 3D LDSEmethod in PACS, the 3D LDSE method must be able to integrate withthese two profiles. The integration of these two IHE profiles in PACSwith the 3D LDSE method has been developed. Integration of the3D LDSE method with these two IHE profiles is focused on how toprotect the integrity of all images involved in the workflow profilesanytime at anywhere without interrupting the clinical data flow.

23.4.3 Integration of the 3D LDSE Methodwith Key Image Note

As shown in Fig. 10, the image set of the exam stored in thearchive already has a digital signature embedded when the exam



3D modality

Image manager/

image archive

3D PostProcessing

Reformattedimages

Diagnosis W.S.

Review W.S.

1

2

3

4

5

6

Keyimage note

A new Sign and Embed

3D LDSE

Sign and Embed3D LDSE1

3

Fig. 10. Integrate the 3D LDSE method with the IHE 3D post-processing workflowand key image note profiles in a clinical PACS.

is generated in the 3D image modality. The lack of the protectionis when several flagged 2D images are sent to the viewing work-station for review. The 2D LDSE method described previously inSec. 23.2.2 can be used in this situation to embed the signature ofevery 2D image in the flagged image correspondingly; therefore,the physician at the viewing workstation can verify the integrity ofthe flagged images whenever they are viewed.

23.4.4 Integration of the 3D LDSE Method with 3DPost-Processing Workflow

As shown in Fig. 10, the integration of the 3D LDSE method withthe 3D post-processing workflow is as follows:

(1) The original exam is generated in a 3D imaging modality.Before the exam is stored in the archive, a signature of each seriesof the exam is embedded in the image set of the series using the3D LDSE method. The signature embedded exam is then sent tothe image server for archiving. Thus, the integrity of the orig-inal image exam is assured when the exam is in transit and inarchive.



(2) The signature embedded exam is sent to a 3D post-processingworkstation.

(3) When a series of reformatted images (e.g. a CT coronal reformat-ted series) is generated in the 3D post-processing workstation,a signature of this image set is also generated and embedded inthe image set using the 3D LDSE method that has been installedin the workstation.

(4) The signature embedded series of reformatted images is sentto the image server for archiving. It is important to notice thatno changes are made to the signature embedded original exambecause the new signature is only embedded in the reformattedimage series.

(5) & (6) Once the exam is retrieved to the diagnosis workstation orthe review workstation, the integrity of the exam can be verifiedanytime on demand. If there is a key image note, then only sev-eral significant images are retrieved to the review workstation.The integration method described previously in Sec. 23.2.5 canbe used to protect the integrity of these significant images.


We have proposed a novel LDSE method for assuring the integrityof 2D medical images. Experimental results demonstrated that the2D LDSERS algorithm is effective for assuring the integrity of animage individually or in a large medical imaging system like PACS.As more and more three dimensional (3D) imaging modalities areused for clinical diagnosis, an examination created by a 3D imag-ing modality could generate hundreds or thousands of images.To apply the 2D LDSE method for these large amount of imageswould be time consuming and inefficient. In order to improve theefficiency, the 3D LDSE method has been developed for assuring theintegrity of clinical 3D image data specifically. Experimental resultsof the 3D LDSE method show a significant improvement in the timeperformance when dealing with a large amount of images in the 3DCT, MR, or US volume sets compared with the 2D LDSE method.



This shows that the 3D LDSE method can be used to assure dataintegrity of 3D volumes in clinical imaging systems. The integrationof the 3D LDSE method with the IHE profiles related to 3D imageworkflow has also been investigated and developed.

APPENDIX I

Flipping Operation

An invertible operation “F” is defined as a flipping on individualbit(s) of an image pixel. Flipping of a given bit is always “0” → “1”or “1” → “0”. The absolute value change of the pixel by flipping,however, depends on the bit locations in the pixel. The followingshow some examples of the absolute value change based on theflipping operation on different bits of the pixel:

FLSB F1B … F7B

The bit flipped 0th (LSB) 1st bit 7th bit

If the pixel value isthe “left,” then it ischanged to the“right.”

0 → 1, 0 → 2, 0 → 128,1 → 0, 2 → 0, 128 → 0,2 → 3, 1 → 3, 1 → 129,3 → 1, 3 → 1, 129 → 1,…, …125 → 124,126 → 127,127 → 126, …, …,128 → 129,…,254 → 255, 253 → 255, 127 → 255,255 → 254, 255 → 253, 255 → 127,

7B LSB1B



When a flipping operation is performed on the LSB (see Col-umn 2) of a group of four pixels G (see Sec. II), the value ofthese four pixels changes accordingly. For example, assuming thatG = (127, 125, 128, 126), we would have F(G) = (126, 124, 129, 127)after the flipping operation. Thus, the original second and third pix-els (125, 128) become less correlated after flipping (124, 129). In otherwords, if Eq. (8) (see Sec. II) is applied in G, then f (F(G)) > f (G).

The number of pixels in the group G can be set to other numbersinstead of 4, such as 3 or 5. The numbers 3–6 have been tested in ourexperiments. Experimental results show that “4” works best for alltested medical images.

References

1. Berger SB, Cepelewicz BB, Medical-legal issues in teleradiology, Am JRoentgenoloy 166: 505–510, 1996.

2. Berlin L, Malpractice issue in radiology-teleradiology, Am JRoentgenoloy 170: 1417–1422, 1998.

3. Zhou XQ, Huang HK, Lou SL, Authenticity and integrity of digitalmammography images, IEEE Trans Medical Imaging 20(8): 784–791,2001.

4. Information processing systems, Open systems Interconnection, BasicReference Model-Part 2: Security Architecture, ISO 7498-2, 1989.

5. Hodge Jr JG, Lawrence GO, Jacobson PD, Legal issues concerning elec-tronic health information: Privacy, quality, and liability, J American Med-ical Association 282(15): 1466–1471, 1999.

6. James Jr AE, James III E, Johnson B, James J, Legal considerations ofmedical imaging, J Legal Medicine, 87–113, 1993.

7. Zhou Z, Lossless digital signature embedding for medical imageintegrity assurance, PhD dissertation Chapter 2, Univ of SouthernCalifornia, Los Angeles, CA, 2005.

8. Digital Imaging and Communications in Medicine (DICOM) Part 15,Security and System Management Profiles, 2004.

9. Secure Socket Layer (SSL), http://wp.netscape.com/eng/ssl3/draft302.txt.

10. Fridrich J, Goljan M, Du R, Lossless data embedding for all imageformats, in Proc SPIE Photonics West, Electronic Imaging 4675: 572–583,2002.

11. Fridrich J, Goljan M, Du R, Lossless data embedding — New paradigmin digital atermarking, EURASIP J Appl Sig Proc 2002: 185–196, 2002.



12. Nelson M, Arithmetic Coding + Statistical Modeling = Data Compres-sion. Available: http://dogma.net/markn/articles/arith/part1.htm,1991.

13. Lehmer DH, Mathematical methods in large-scale computing units,in Proc. 2nd Symposium on Large-Scale Digital Calculating Machinery,pp. 141–146, Cambridge, MA, 1949.

14. Park SK, Miller KW, Random number generators: Good ones are hardto find, Comm ACM, 31: 1192–201, 1988.

15. Secure hash standard, Federal Information Processing Standards Publica-tion, 180–181, 1995.

16. Menezes AJ, Oorschot PC, Vanstone SA, Handbook of Applied Cryptog-raphy, CRC Press, 1997.

17. Rivest RL, Shamir A, Adleman LM, A method for obtaining digitalsignatures and public-key cryptosystems, Communications of the ACM21: 120–126, 1978.

18. Integrating the Healthcare Enterprise (IHE) Technical Framework Vol-ume I: Integration Profiles, 2005.

19. Zhou Z, Huang HK, Cao F, Liu BJ, et al., Educational RIS/PACS Simu-lator, SPIE Medical Imaging 4: 139–147, 2003.

20. Law MYY, Zhou Z, New direction in PACS education and training,Computerized Medical Imaging and Graphics J 27: 147–156, 2003.


CHAPTER 24

The Treatment of Superficial TumorsUsing Intensity Modulated Radiation

Therapy and Modulated ElectronRadiation Therapy

Yulin Song and Maria Chan

Technological advances in diagnostic radiology with four-dimensional(4D) imaging have brought a new treatment modality in radiation oncol-ogy. While 2D X-ray imaging technology provided 2D radiation treatmentplanning, 3D volumetric imaging led to the development of 3D confor-mal radiation therapy (3DCRT) and intensity modulated radiation ther-apy (IMRT). Recently, 4D imaging resulted in a series of research activitiesin the radiation oncology community and eventually led to the forma-tion of the concept of 4D radiation therapy (4DRT). It is expected that thiscross-pollination between diagnostic radiology and radiation oncologywill continue in the future. This chapter presents principles and appli-cations of IMRT and Modulated Electron Radiation Therapy in treatingsuperficial tumors.

24.1 INTRODUCTION

In medicine, no other branch played a more significant and directrole than diagnostic radiology in the evolution of radiation oncol-ogy. From earlier day’s 2D X-ray films to today’s 4D imaging, eachtechnological advance in diagnostic radiology brought about a newtreatment modality in radiation oncology. 2D X-ray films gave birthto the 2D radiation treatment planning. 3D volumetric imaging led

599


600 Yulin Song and Maria Chan

to the development of 3D conformal radiation therapy (3DCRT)and intensity modulated radiation therapy (IMRT). Cone beam CT(CBCT) and other on-board imaging systems brought about image-guided radiation therapy (IGRT). Recently, 4D imaging resulted ina series of research activities in the radiation oncology communityand eventually led to the formation of the concept of 4D radiationtherapy (4DRT). It is expected that this cross-pollination betweendiagnostic radiology and radiation oncology will continue in thefuture. The state of the art medical imaging modalities, combinedwith computer-controlled medical linear accelerators (linac) andhigh precision photon multileaf collimators (MLC), have fundamen-tally changed the practice of radiation oncology. These, along withnewly developed IMRT and inverse treatment planning techniques,provide significant improvement in the delivery and control of exter-nal beam radiation through beam intensity modulation. IMRT hasbeen used extensively in the treatment of various cancers.1−5 By2003, IMRT was accepted as a mature radiation treatment modal-ity by the radiation oncology community.6−8 Both theoretical stud-ies and clinical investigations have shown that given a reasonableset of dose-volume constraints, dose limits, and carefully chosenbeam angles, it is possible to produce highly conformal dose distri-bution around the target, while sparing as much normal tissue aspossible.9−11 Highly conformal IMRT plans may reduce the risk oftreatment induced complications and sequelae and provide a poten-tial means of escalating the dose, and thus, improve local controland survival. Results from recent clinical studies are favorable andencouraging, compared to traditional 3DCRT.12−14 The advantagesof a photon beam are its high penetrating power and narrow beampenumbra. Therefore, photon beam IMRT is well suited for moredeep seated tumors. However, it is not the ideal beam modality fortreating superficial targets like breast cancer, certain types of headand neck cancer, skin cancer, and mesothelioma due to low skin doseand high exit dose. Consequently, conventional radiation therapyapproaches in these patients are often delivered with a combina-tion of electrons and photons or exclusively with electrons. Figure 1presents the comparison of dose distribution between electron and


The Treatment of Superficial Tumors Using IMRT & MERT 601

6 MeV Electrons 6 MV Photons

Right Breast

Left Breast

High Exit Dose

Fig. 1. Comparison of dose distribution between electron and photon beams fora chest wall case. It is clear that the dose distribution of modulated electron beamis well confined to the target volume while there is high exit dose for photon beam.

photon beams for a chest wall case. It is clear that the dose distri-bution of modulated electron beam is well confined to the targetvolume while there is a high exit dose for photon beam. Promis-ing alternative approaches are a combination of IMRT and electrons(IMRT+e) and modulated electron radiation therapy (MERT). In thischapter, we will present the basic principles of IMRT+e and MERTand dosimetric results of several case studies.

24.2 INTENSITY MODULATED RADIATION THERAPY+ELECTRONS

24.2.1 Basic Principles of IMRT+ e

Like IMRT, IMRT+ e treatment planning starts with patient simu-lation. Currently, most patient simulations are accomplished on aCT simulator, which is a whole body CT scanner designed specif-ically for radiation therapy simulation. Once the CT simulation



scan is acquired, the CT images are transferred to a virtual simu-lation workstation where four important pre-planning tasks will beaccomplished. The first task is the anatomic volume delineation orsegmentation. The targets, organs at risk (OAR), and other relevantnormal tissues will be delineated on the CT images by the radi-ation oncologist for the purpose of dosimetric statistics. Based onthese volumes, the planner will create other planning structures forthe purpose of inverse treatment planning optimization. To facili-tate target delineation, it is now more common to use CT/MRT orCT/PET fusion technique. The target volume is drawn either onMRT or PET images. It is automatically transferred to the corre-sponding CT images. For anatomical structures that differ greatlyin CT number from the surrounding tissues, such as the lungs andbones, this task can be accomplished by using various exiting seg-mentation algorithms. However, so far, there are no reliable seg-mentation techniques for target volume delineation. Target volumedelineation is still a tedious and time consuming process for radia-tion oncologists. The second task is the localization of the treatmentisocenter. Once the target volumes and anatomical structures aredelineated, the treatment isocenter will be placed with the help ofthe 3D rendering of the volumes. Normally, the ideal location forthe isocenter is the geometrical center of the target. This is becausethe beam parameters near the isocenter, such as symmetry and flat-ness, are the most desirable. In addition, by placing the isocenterat the geometrical center of the target, the resulting treatment planwill have less chances of violating the physical constraints of theMLC and, therefore, splitting the treatment fields. However, in somecases, it is necessary to place the isocenter at the edge of the treatmentfields to best match the adjacent fields. The third task is the determi-nation of a suitable set of treatment beam parameters. These includegantry angles, collimator angles, couch angles, field sizes, electronapplicator sizes, and shielding blocks. This is particularly impor-tant for the IMRT+e technique because the electron applicator maycollide with the patient, the immobilization device or the treatmentcouch for some beam angles. The virtual simulator allows the plan-ner to select these parameters through the beam’s eye view (BEV).



The BEV projects the target volumes and the other structures onto avirtual plane. The planner can choose the optimal beam parametersby observing the changes of the projected images. The fourth task isthe generation of digitally reconstructed radiographs (DRRs) alongwith the target and other relevant structures. At the completion ofthe virtual simulation, a DRR is generated for each treatment field.The DRRs will be used for pretreatment field verification. A DRR isa virtual radiographic projection of the 3D CT images onto a virtualplane in BEV. Each pixel in a DRR is obtained by summing the linearattenuation coefficients of those voxels traversed by a specific ray.It is similar to a conventional projection X-ray film but with a lowerspatial resolution.

The IMRT+e optimization stems from the conventional IMRTinverse planning optimization. The basic concept of the inverse plan-ning is to use a mathematical optimization algorithm to search anoptimal set of beam parameters or intensity maps that produce adose distribution closest to the prescribed one. In plain language,inverse planning is to look for the cause given the results. Mathe-matically, inverse planning is identical to CT image reconstruction.That is given a set of projections, what is the linear attenuation coef-ficient distribution that produces the projections. In IMRT inverseplanning, the results or the input parameters are prescribed dosedistributions, represented by a set of dose limits and dose-volumehistograms (DVH). Depending on the mathematical model used, theinput parameters could also be the desired probability distributions,such as tumor control probability (TCP) and normal tissue compli-cation probability (NTCP).15,16 The cause or the output parametersare a set of optimal beam intensity maps or weighing factors. Prior tooptimization, the number of beams and their angles are determinedmanually based on planner’s experience. Each beam is digitized intoa beamlet or a bixel map. The typical beamlet size is 1 × 1 cm2. Inour institution, we use a 1 × 1 mm2 beamlet size. A smaller beamletsize provides a finer and more accurate dose distribution but takesa longer time to optimize and deliver it. Thus, a trade off has to bemade between the two. Currently, most commercial treatment plan-ning systems offer the planner the option to select the beamlet size.



In inverse treatment planning, the plan evaluation can be madeby condensing the quality of a plan into a single value. This valueis referred to as the objective function and naturally depends onthe choice of criteria that was made to compute it. The objectivefunction is, therefore, a mathematical function that takes its input asthe dose distribution of both the evaluated plan and the prescrip-tion. Once we have determined the criteria that define the objectivefunction, we can easily compare two possible treatment plans bycomparing the values of their respective objective functions. Ideally,the objective function should be constructed by including all exitingradiological, biological, and dosimetric knowledge, relevant beamparameters, and physical constraints of the linac. However, in prac-tice, it is only a function of anatomic structures and beamlet weightsfor simplicity. Over the years, various objective functions have beenproposed. Several of them have been implemented clinically. Basedon their end points, objective functions can be classified as: (1) phys-ical dose based; (2) biologic based (TCP and NTCP); (3) equivalentuniform dose based (EUD); and (4) clinical knowledge based. Eachof these has its own advantages and disadvantages. Currently, thephysical dose based objective function is the most widely adoptedone. This is because (1) it reflects current clinical practice; radiationoncologists are accustomed to prescribe dose rather than a probabil-ity to an anatomic structure and plans are evaluated based on somedosimetric parameters; (2) it is intuitive and directly linked to theoptimization parameters; by observing the trend of DVHs duringthe optimization, the planner is able to tell whether this run couldproduce a desired plan or not; (3) the quadratic objective functionis guaranteed to reach its global minimum using fast conjugate gra-dient search algorithm. Here, we briefly describe the physical dosebased objective function.

The physical dose based objective function is normally con-structed with two types of term, representing the targets and theorgans at risk (OAR), respectively. For complex IMRT plans, the



objective function may contain other terms, such as terms repre-senting various dose tuning structures.

F = Ftarget−1 + Ftarget−2 + · · · + FOAR−1 + FOAR−2 + · · · (1)

The term for the target is given by a quadratic function:

Ftarget = 1Nt

Nt∑i=1

(Di − Dpresc)2

+ wt,min ·Nt∑i=1

(Di − Dmin)2 · �(Dmin − Di)

+ wt,max ·Nt∑i=1

(Di − Dmax)2 · �(Di − Dmax)

, (2)

where Nt is the number of dose calculation points in the target, Di

is the dose to point i, and Dpresc is the prescription dose. The sec-ond and third terms inside the brackets implement the target dosehomogeneity criteria. Dmin and Dmax are the desired minimum andmaximum target doses, and wt,min and wt,max are the weighting fac-tors or penalties corresponding to under- and overdosing. �(x) isthe Heaviside function, defined as:

�(x) ={

1 x ≥ 00 x < 0

. (3)

Similarly, the term for the OARs is given by:

FOAR = 1NOAR

wOAR,max ·NOAR∑i=1

(Di − Dmax)2 · �(Di − Dmax)

+ wOAR,dv ·Ndv∑i=1

(Di − Ddv)2 · �(Di − Ddv)

, (4)

where the first term inside the brackets implements a maximumdose constraint Dmax on the OAR and the second term implementsa dose-volume constraint. The relative penalty weights are given



by wOAR,max and wOAR,dv, respectively. NOAR is the number of dosecalculation points in the OAR and Ndv the number of dose calculationpoints whose dose must be kept below the dose-volume constraintdose Ddv. Here, the number of dose calculation points is essentiallythe total volume for s particular structure. The dose to point i, Di, isthe sum of dose contributions from all rays:

Di =Nr∑j=1

xjaij, (5)

where Nr is the number of rays, xj is the intensity of the j-th ray, aij

the dose deposited to the ith point per unit intensity of the j-th ray,and the product is summed over all j. The goal of optimization is tofind a set of ray intensities xj that minimizes the objective functiongiven by Eq. (1).

In order to evaluate a treatment plan, we need to assess “howfar” it is from the prescription. Therefore, some sort of optimizationneeds to be performed on the objective function. Equation 1 can beoptimized using various optimization algorithms. The simplest wayis to use an iterative approach: an initial plan is computed and eval-uated, adjustments are made, the plan is reevaluated, further adjust-ments are made, and so on until some convergence criterion is met.Currently, the most commonly implemented algorithms are itera-tive gradient-based search techniques and the simulated annealingmethod.17−19 The former, starting with an arbitrary plan, creates aseries of candidate plans that eventually converge to an optimalone. It provides a fast convergence and, generally, works efficientlyfor the quadratic types of objective function, particularly for largesystems. The latter, although slow in convergence, can avoid beingtrapped in local minima by assigning a small probability for accept-ing changes that increase the objective function. It has been shownthat the simulated annealing method is able to find the true globalminimum even there are local minima.

In theory, the above-described methodology is applicable notonly to photons, but also to other types of radiation, includ-ing electrons, protons or combination of electrons and photons.



However, since current commercial medical linear accelerators arenot equipped with electron MLC (EMLC), the hardware required forbeam intensity modulation, the beamlet based intensity modulatedelectron plans cannot be delivered. As a result, inverse treatmentplanning is routinely performed for photon beams only. Neverthe-less, to fully take advantage of the finite range and high surfacedose of electron beams, it is beneficial to combine electron beamswith intensity modulated photon beams for certain disease sites.This can be accomplished by optimizing IMRT photon beams overan exiting electron dose distribution. In this scenario, Di in Eq. (5)is replaced by Di + Di,electron, where Di,electron represents the existingelectron dose distribution. In practice, the electron component of theIMRT+e plan is computed first. Currently, Di,electron is manually opti-mized through a trial-and-error approach by the planner. Thus, it isnot modulated by the subsequent IMRT optimization. However, inprinciple, Di,electron could be computer optimized by using aperture-based optimization technique.20−22 In the following section, we willpresent several case studies using IMRT+e technique.

24.2.2 Clinical Applications of IMRT+e

24.2.2.1 Cancer of the orbit

Amale patient in his late 50s was diagnosed as Merkel cell carcinomaof the left upper eyelid (T3NxM0). The patient underwent a wideexcision with flap reconstruction under the care of an ophthalmicplastic surgeon. The specimen revealed a 1.8 cm Merkel cell carci-noma that was 8 mm in depth and invaded the dermis and subcutisand displayed lymphovascular invasion. Since the surgical marginwas as close as 1 mm, the patient, therefore, was recommended toreceive postoperative adjuvant radiation therapy to improve localcontrol. To spare the optic nerve, lens, retina, and parotid gland asmuch as possible, a complex 4-field IMRT+e plan was created forthe patient. The electron portion of the plan was a single enfaceanterior-posterior (AP) boost field using 6 MeV electrons. The fieldcovered the left eyelid only and was treated daily with 1 cm cus-tom bolus to a total dose of 5400 cGy over 30 fractions. The IMRT



portion of the IMRT+e plan consisted of three IMRT fields using 6MV photons, with beam angles of 35◦, 140◦, and 175◦, respectively.The IMRT fields were optimized by incorporating the exiting APelectron field into the optimization loop. A total of approximately20 000 rays were included in the optimization. The spatial resolutionof each ray was 0.2 cm in the leaf travel direction and 0.5 cm along theleaf width direction. The total computing time was about 1 to 2 min-utes for each run of optimization (∼100 iterations) using a PC-basedtreatment planning system, with a CPU of 3.2 GHz and 1 GB RAM.The IMRT fields covered the left eyelid and left lateral facial, cervical,and periparotid lymph nodes and were also treated to a total dose of5400 cGy over 30 fractions. The IMRT fields were delivered using thedynamic multileaf collimator (DMLC) technique. A lead eye shieldwas placed in the left orbit daily to protect the cornea. The patient tol-erated the treatment well by applying eye drops daily and bacitracinlubricant at night. He experienced no significant visual changes orside effects during the course of treatment.

Dosimetrically, the IMRT+e plan provided excellent target dosecoverage. The dose delivered to 95% planning target volume (PTV)or D95 was 101%. The PTV volume covered by 95% prescriptiondose or V95 was 99%. Figure 2 shows the dose distribution in a rep-resentative axial CT slice. Figure 3 shows the dose dsitribution ina coronal CT slice. In addition, the plan also offered adequate pro-tection to the OARs. The maximum dose to the cord was 1270 cGy.The mean dose to the contralateral parotid gland was 96 cGy. Themaximum dose to the brainstem was 1622 cGy. The ipsilateral opticnerve received a maximum dose of 2533 cGy, while the contralat-eral optic never received a maximum dose of 775 cGy. The con-tralateral lens and retina received a maximum dose of 310 cGyand 364 cGy, respectively. Figure 4 shows the DVHs of the targetand OARs.

24.2.2.2 Cancer of the scalp

Cancer of the scalp has traditionally been treated with electronbeams alone due to its superficial location. Electron beams provide



Fig. 2. The isodose distribution in a representative axial plane. The yellow solidline is the 1 cm bolus. The light blue solid line represents the PTV. The dark brownand red solid lines are the retinas and lenses. The light brown solid line represents100% isodose line.

sufficient surface dose in this setting while sparing the brain andoptic structures. However, extensive lesions of scalp present a chal-lenge to radiation therapy because of their large target volumes andmany surrounding OARs. To search for the best treatment option,several planning techniques have been developed and tested. Theseinclude electron and photon arc beams, energy and intensity modu-lated electron beams, and photon IMRT.23−30 However, each of theseapproaches has its pros and cons. With IMRT+e approach, the brain



Fig. 3. The isodose distribution in a representative coronal plane.

and optic structure doses can be further reduced and the techniqueis clinically implementable.

An 80-year-old male patient was diagnosed as poorly differen-tiated recurrent squamous cell carcinoma of the scalp. The lesioncovered the entire forehead scalp and both temporal surfaces andhad a total volume of approximately 103 cm3. During simulation, thepatient was positioned in a supine position and his previous surgi-cal scars were wired. After a suitable treatment position was deter-mined, the patient was immobilized using an Aquaplast facemaskto reduce voluntary head motion. After CT scanning, the physician



Fig. 4. The dose-volume histograms (DVHs) of the IMRT+e plan. A more verticalcurve for the target indicates a more conformal or tighter dose distribution, whilefor OARs, a more left curve indicates more sparing and better protection.

delineated the PTV and relevant OARs and prescribed 6300 cGy in35 fractions. A six-field IMRT+e plan was computed for the patient.The electron portion of the plan was a large single enface electronfield using 9 MeV. Our clinical experience indicates that this type offield arrangement was generally sufficient to meet the acceptable tol-erance levels for OARs. The electron field was manually weighted tocontribute about 50% of the prescribed dose. The electron field wastreated with a custom electron cutout and a 1.5 cm bolus. The IMRTportion of the IMRT+e plan employed five equally-spaced coplanarphoton beams using 6 MV. The D95 and V95 for the IMRT+e planwere 97% and 98%, respectively. The mean dose to the brain was13.6 Gy, showing good normal tissue sparing capability. The dosesto all optic structures were far below the clinical tolerance levels.The patient received low dose Carboplatin weekly for eight cycles,given concomitantly with radiation treatment. Figure 5 shows a typ-ical isodose distribution in an axial plane.



Fig. 5. The isodose distribution in a representative axial plane. The blue contourrepresents the PTV and the orange line is the prescription isodose line (100%).

Prior to the treatment, the IMRT+e plan was verified withionization chamber measurements for selected points and filmdosimetry for each treatment field. During the first treatment, anin vivo dosimetry was also performed by placing thermoluminescentdosimeters (TLD) and diodes to the external surface of the scalp.The measured data agreed with the calculated dose distributionwithin ±5%.

24.2.3 Summary

IMRT+e, a combination of IMRT fields and static electron fields,accounts for existing electron dose contributions during optimiza-tion. Our clinical experience with IMRT+e shows that it is straight-forward and available on existing linacs. It can improve target dose



coverage and normal tissue sparing, compared to non-IMRT tech-niques and photon-alone IMRT. In terms of planning time, IMRT+eis approximately 30 minutes longer than IMRT alone using ourin house planning system. The IMRT+e treatment time is about15 minutes longer than IMRT alone. Thus, in terms of time efficiency,IMRT+e is competitive with photon IMRT and is a viable alternativefor superficial tumors.

24.3 MODULATED ELECTRON RADIATION THERAPY

24.3.1 Basic Principles of MERT

In theory, superficial tumors can be best treated with modulatedelectron radiation therapy (MERT).31 Unlike photon IMRT, MERTcan provide both intensity and energy modulations. Conceptually,an MERT plan would consist of multiple electron beams with dif-ferent energies, depending on the target depth. Dose conformity inthe depth direction would be achieved by the use of different elec-tron energies. Dose conformity and uniformity in the lateral direc-tion would be achieved by electron beam intensity modulation byusing an electron-specific MLC (EMLC). In addition, compared tophoton beams, electron beams have negligible scatter radiation. Fur-thermore, because MERT plans mainly use normal incident electronbeams, they are less affected by patient’s respiration. Through bothintensity and energy modulations, MERT is capable of deliveringhighly conformal doses to targets with complex shapes and of spar-ing surrounding normal tissue, particularly, the distant OARs. Tra-ditionally, electron beams are shaped using cutouts and differentenergies at treatment depths may be achieved using variable inci-dent energies. However, it would be very time consuming to makecutouts for MERT plan delivery and the treatment time wouldbecome unacceptably long for routine clinical applications. Boluscan be used for missing tissue compensation and/or limited depthmodulation. However, 3D bolus requires sophisticated techniquesto build and it does not provide good intensity modulation. Unfortu-nately, because of severe electron in-air scattering, the conventional



photon beam MLC (PMLC) is not suitable for the delivery of MERTplans. An electron beam collimated by a PMLC has a relatively largepenumbra32−34 due to the location and the thickness of the PMLC.A PMLC is normally located at a large distance (40 cm ∼ 60 cm)from the patient skin surface. As an example, out of the most popu-lar medical linear accelerators, the distance from the leaf bottom tothe isocenter is 46.1 cm for Varian CLINAC 2100C, 62.7 cm for ElektaSL 75-5, and 62.1 cm for Siemens Digital Mevatron. When electronstravel such a long distance from the PMLC to the patient skin, thebeam penumbra will be broadened and the useful sharpness of thebeam edge will be smeared out due to the extended electron sourceand in-air multiple scattering. In addition, the thickness of a PMLCis optimized to minimize the leakage dose for photon beams. Theamount of electrons scattered from the end or side of an MLC leafis proportional to the area irradiated by the beam. For thick leaves,large areas of leaf ends and sides will be exposed to the electronbeam, and will therefore produce a large amount of scattered elec-trons into the beam, which will broaden the beam penumbra. Todeliver MERT plans effectively, a prototype EMLC was manufac-tured and tested at Stanford University in 2002, based on the resultsof Monte Carlo simulations and a conventional Varian 25 × 25 cm2

electron applicator.31

We now turn to the description of the optimization algorithm.In the following approach, the objective function is chosen to beroughly the sum of the squared differences between the prescriptionand the delivered doses at different measurement points in differ-ent structures. We naturally want to achieve a plan whose objectivefunction is as small as possible since a small objective function indi-cates that the delivered doses are close to the prescription ones. Inorder to derive the expression of the objective function, we first needto introduce some notations.

24.3.1.1 Notations for parameters and constraints

We begin with a CT dataset containing anatomic structures of inter-est that are manually contoured. There are A structures. Index a = 0



refers to the targets while other structures are indexed from 1 toA − 1. Each structure a contains Na dose calculation points, fora = 0, . . . , A−1. For simplicity, we also assume that the structures donot overlap, which means that every dose calculation point belongsto one and only one structure. As discussed earlier, beam parame-ters and energies are specified before the optimization. Each beamangle/energy combination is called a field; a port refers to a sin-gle beam geometry. Just like in beamlet-based optimization, eachfield is divided into beamlets, and we denote B the total number ofbeamlets. All the beamlets belong to a total number of G fields andevery beamlet only belongs to one single field). bg is the number ofbeamlets in field g, and we thus have:

B =G∑

g=1

bg. (6)

For each beamlet b (where b ∈ [1; B]) and each structure a, wedefine the dose kernel d(a)

b as the array of doses received by eachdose calculation point n in structure a when only beamlet b is givenweight 1 and all the other beamlets are given zero weights. Sincestructure a contains Na dose calculation points, we have d(a)

b ={d(a)

b,1, . . . , d(a)b,Na

}. The weight given to beamlet b is denoted as wb, andwe are trying to determine the set of weights that best matches theprescription. Once the dose kernels are computed, the doses receivedat each point in each structure at any time depend only on the beam-let weights at that time because they are only computed once priorto the optimization. The vector w = {w1, . . . , wB} containing all thebeamlet weights is called the state-vector. Since we know the dosekernels and the state vector, we can easily compute the accumulateddose at every dose calculation point. We denote D(a)

n as the totalaccumulated dose at point n in structure a. Thus, we have:

D(a)n =

B∑b=1

wbd(a)b,n (7)

which is simply the sum of the doses delivered by each individualbeamlet.



We now need to specify our conventions regarding prescrip-tion doses. A prescribed dose is specified for the target and thendose-volume constraints are defined for both the target and criti-cal structures. We denote D(0)

0 as the prescribed dose for the target.Besides this prescribed dose, there are two additional dose-volumeconstraints that apply to the target. The upper dose-volume con-straint is that no more than v(0)

1 % of structure 0 (i.e. the target) shouldreceive a dose greater than D(0)

1 , whereas the lower dose-volume con-straint is that no more than v(0)

1 % of structure 0 should receive a doseless than D(0)

1 . For each other structure a, there are Ca dose-volumeconstraints, each one of them is that no more than v(a)

c % of structurea should receive a dose greater than D(a)

c . Eventually, each constraintc that applies to structure a (including the target) is given a relativeweight α(a)

c .

24.3.1.2 Objective function and gradient

The objective function is the sum of terms that represent the targetprescription dose and other terms that represent violations to all theconstraints that apply:

F = F(0)c +

A−1∑a=0

Ca∑c=1

(F(a)c ). (8)

The term for the target prescription dose has the following quadraticform:

F(0)0 = α0

N0

N0∑n=1

[D(0)0 − D(0)

0 ( �w)]2. (9)

The terms that account for dose-volume constraints are morecomplicated since their computation requires some reordering of thedelivered doses in order to count the number of points that are inviolation of the constraint. To avoid technical details, we just accept



that the overall objective function has the following form:

F = α0

N0

N0∑n=1

[D(0)0 − D(0)

0 ( �w)]2 + α(0)1

N0

N1∑n=n′

1,(0)

[D(0)1 − D(0)

x(0)n

( �w)]2

+ α(0)2

N0

N1∑n=n1,(0)′

[D(0)2 − D(0)

x(0)n

( �w)]2

+A−1∑a=1

Ca∑c=1

α(a)

c

Na

Na∑n=n′

c,(a)

[D(a)c − D(a)

x(a)n

( �w)]2 , (10)

where n′1,(0) is the point index from which we have to penalize for

structure 0 for a low dose-volume constraint (all violating points arereordered by decreasing order), n′

c,(a) is the point index from whichwe have to penalize for structure a for a high dose-volume constraint.x(a)

n are the original indices of points, when n are the indices of thedose calculation points that are sorted in decreasing order. Similarly,x(a)

n represent the original indices of dose calculation points, whenn are the indices of the dose calculation points that are sorted inincreasing order.

It can be shown that the objective function can be further con-densed into:

F =A−1∑a=1

1Na

∑c,a

Na∑n=1

[ζ(a)c,nα

(a)c (D(a)

c − D(a)n ( �w))]2

=A−1∑a=1

1Na

∑c,a

Na∑n=1

[ζ(a)

c,nα(a)c

(D(a)

c −B∑

b′=1

wb′d(a)wb′ ,xn

)]2

, (11)

where∑

c,a is the sum over all constraints of structure a, including thetarget prescription dose constraint and all dose-volume constraints.Moreover, ζ(a)

c,n is either 0 or 1, depending on whether the point n instructure a should be incorporated in constraint c. This is determinedby sorting the computed delivered doses, counting the number ofviolating points beyond the tolerance.



Mathematically, ζ(a)c,n is defined as:

ζ(a)c,n =

1 if a = 0, and c = 0∑N0

m=n′1,(0)

δx(0)m

if a = 0, and c = 1

∑N0

m=n′1,(0)

δx(0)m

otherwise,

(12)

where δi,j is the Kronecker delta function, defined as:

δi,j ={

1 if i = j0 if i �= j

(13)

From the Eq. (8), the original gradient formula is obtained bydifferentiating with respect to each single beamlet weight.

∇F = ∇F(0)c +

A−1∑a=0

Ca∑c=1

(∇F(a)c ) (14)

It can be shown that the gradient of the objective function hasthe following form:

∂F∂wb

= −2 ·A−1∑a=1

1Na

Na∑n=1

d(a)wb,n

∑c,a

[ζ(a)c,nα

(a)c (D(a)

c − D(a)n ( �w))]. (15)

Once we have computed a gradient, we can move along itsdirection (departing from the initial state) and find a step size thatminimizes the objective function. This step is referred to as theline-minimization and comes down to finding step size t such thatF(w + t∇F) is minimal.

24.3.2 Clinical Applications of MERT

24.3.2.1 Cancer of the breast

Breast cancer is the leading incident cancer in the United States,affecting one in nine women over their lifetimes and account-ing for 32% of all newly diagnosed cancers in women. Nearly allbreast cancer patients receive radiation therapy as part of theirtreatments.35 Radiation therapy has been shown to be very effec-tive for all stages of localized breast cancer. Especially for early



stage (T1–T2) patients, breast conservation with lumpectomy fol-lowed by irradiation has been very successful.36,37 For advancedstage (T3–T4) patients, studies have shown that the addition of post-mastectomy chest wall irradiation could significantly reduce the riskof a local failure on the chest.35 Currently, breast radiotherapy fallsinto two major categories: post-lumpectomy intact breast irradia-tion and post-mastectomy chest wall irradiation. They all use photonbeams.38−41 The former employs a pair of photon tangential fields tocover the entire intact breast while avoiding as much lung and heartpossible, supplemented by an electron boost to the tumor bed. Thecommon planning techniques in this setting are opposed tangen-tial wedge pair, field-in-field approach, compensator-based inten-sity modulated radiation therapy (IMRT), and full-volume IMRT.The latter normally uses a pair of photon tangential fields to encom-pass the entire surgical scar or a combination of photon and elec-tron fields to spare lung and heart. In breast radiation therapy, elec-tron beam treatment is primarily used to supplement photon beamtreatment or applied at surgical scar or lymph nodes as a boostdose after photon treatment. Unfortunately, all photon beam treat-ments, including the state of the art IMRT, have serious drawbacks.This is due primarily to its intrinsic physics — high penetratingpower, slow attenuation in soft tissues, and low skin dose. First ofall, in opposed tangential wedge pair approach and even modernIMRT, a certain volume of the ipsilateral lung and, in the case ofthe left breast cancer, a small volume of the heart are inevitablyincluded in the tangential fields, resulting in high radiation dose tothese critical organs. This problem becomes even worse if a physi-cian wants to treat the internal mammary (IM) or axiliary lymphnode chain simultaneously. In this case, the so called “deep” tan-gential fields will have to be used, resulting in extra lung and, forleft sided lesions, additional heart volume being treated. Secondly,because of lack of electron build up in a photon beam, the skin sur-face receives a low dose. This problem is particularly pronouncedin patients who have undergone a modified radical mastectomy, thesurgical procedure that preserves the pectoralis minor muscle andmost skin of the chest wall.42 For these patients, the risk for chest wall



failure is high if post-mastectomy chest wall irradiation fails todeliver a full dose to the skin. Thirdly, for large sized breasts, thetangential field technique often fails to produce homogeneous dosedistribution in the entire breast because, intrinsically, the breast isnot a uniform structure. Fourthly, hot spots are often observed atthe entrance and exit points of the beams, in the nipple, and in thesuperior and inferior aspects of the breast. The excessive dose inthese areas can produce acute and chronic skin toxicities, resultingin unfavorable cosmetic outcomes. Fifthly, in opposed tangentialwedge pair treatment, the wedge in the medial field can result ina significant increase in scatter dose to the contralateral breast.43 Inaddition, the presence of a wedge inevitably increases the beam-ontime, permitting more scatter dose to reach the contralateral breast.This relatively lose dose of radiation may cause secondary malig-nancy in the uninvolved breast.44 Sixthly, studies showed that post-mastectomy chest wall irradiation reduced local failure by 60%–70%.Nevertheless, the overall survival was not improved.36 The cardiactoxicity induced by the high exit photon dose outweighed the bene-fits and many patients died from cardiac events. In recent years, thecombination of photon and electron beams is becoming the preferredtreatment to reduce cardiac toxicity.45,46 However, it takes a skillfulphysicist three to five days to manually create an acceptable plan.In many aspects, particularly the target dose conformity, the planis not ideal because it is not fully optimized. In addition, it is esti-mated that about 15%–30% of patients, particularly older patients,who undergo a lumpectomy or breast conserving therapy (BCT),don’t receive post-lumpectomy irradiation. This is due, in part, tosix weeks of daily treatments.47 All of these drawbacks ultimatelyaffect patients’ long term survival and quality of life.

With MERT approach, the above mentioned limitations can beeither eliminated or alleviated. Specifically, (1) it can significantlyincrease the skin dose and thus, greatly reduce the risk for chestwall failure; (2) it can greatly reduce radiation toxicity to lung andheart and thus, improve patient long term survival; (3) it can offermuch better target dose conformity and homogeneity and thus,produce favorable cosmetic outcomes; (4) it can greatly reduce the



scatter dose to the uninvolved breast and thus, minimize the riskof radiation-induced secondary malignancy; (5) it can reduce thetotal number of fractions and shorten the overall treatment timeand thus, encourage more patients to receive post-lumpectomy irra-diation; (6) it can significantly reduce the treatment planning timeand thus, increase the working efficiency and patient throughput.Figure 6 shows a prototype EMLC, which was designed based on theresults of Monte Carlo simulations. To reduce air scattering effect, weremoved the last scraper and its electronic circuitry of a conventional25 × 25 cm2 electron applicator. The EMLC was placed immediatelyat the bottom of the modified electron applicator and stabilized witheight screws.48 Figure 7 shows the comparison between a conven-tional photon tangential plan and a MERT plan. The photon beamplan used 6 MV photons and the MERT plan used 6 MeV electrons.It is evident that the MERT plan provided not only a much moreconformal dose distribution than the corresponding photon plan,

Fig. 6. A prototype EMLC. The EMLC was placed immediately at the bottom of amodified 25 × 25 cm2 electron applicator.



14 Gy 24 Gy 33 Gy 43 Gy

Fig. 7. A comparison between a conventional photon tangential plan and a MERTplan. The photon beam plan used 6 MV photons and the MERT plan used 6 MeVelectrons (Courtesy of Dr Todd Pawlicki, UCSD).

Fig. 8. A left breast case with the internal mammary (IM) lymph node chain. Inthis case, the deep tangential photon fields were used, resulting in excessive lungand heart volume being irradiated (Courtesy of Dr Todd Pawlicki, UCSD).

but also adequate protection to the lung. Figure 8 shows a left breastcase, where the internal mammary (IM) lymph node chain was alsotreated. In this case, the deep tangential photon fields were used,resulting in excessive lung and heart volume being irradiated. Thecorresponding MERT plan greatly reduced radiation dose to the lungand heart (Fig. 9).



00

20

40

60

80

100

20 40

Dose [Gy]

Vol

ume

[%]

Heart

Heart

Lt Lung

Lt Lung

60

Target

Fig. 9. The dose-volume histograms (DVHs) of the plans shown in Fig. 8 (Courtesyof Dr Todd Pawlicki, UCSD).

24.3.2.2 Cancer of the parotid gland

Each year, about 70 000 new head and neck cases were diagnosedin the USA49 and tumors of the parotid gland are the most fre-quently encountered salivary gland tumors, accounting for about3% of total head and neck cancers.50 Currently, the most widely usedtreatment is a combination of surgery and adjuvant postoperativeradiotherapy for the malignant salivary tumors.51,52 The minimallyrequired operation for tumors of the parotid gland is a superficialparotidectomy with careful identification and preservation of thefacial nerve.53 The parotid gland has two lobes, a superficial lobeand a deep lobe. Most tumors are located in the superficial lobe ofthe parotid gland and can be excised easily. Retrospective studieshave indicated that this combined modality treatment can reducethe local recurrence rate by 5%–40%.54,55



However, in certain situations, such as high surgical risk ofdamage to the facial nerve, advanced inoperable cancers, unfavor-able cosmetic outcome after surgery, lymph node metastases, anddeep lobe malignant tumors, radiotherapy should be the preferredtreatment.53 Because of its proximity to many critical structures, suchas the oral cavity, brainstem, auditory apparatus, spinal cord, opti-cal nerves, and the lenses of the eyes, parotid cancer treatment usingradiation still remains a very challenging task. Currently, the mostcommonly used radiotherapy techniques for the treatment of theparotid cancers are: (1) an ipsilateral wedged pair of 6 MV photonbeams oriented at oblique angles to encompass the entire parotidbed; (2) an ipsilateral field treated with electron beams (12, 16 or20 MeV), and (3) a combination of high energy photon and elec-tron beams (6 MV+12 or 16 or 20 MeV) with proper weighting.56−58

However, all these techniques have drawbacks. The first techniquegives a low radiation dose to the contralateral parotid gland andhigh doses to the oral cavity, brainstem, cochlea, optical nerves, thelenses of the eyes, and spinal cord. In addition, because relativelyhigh energy photon beams are used, the slow build up effect ofthe photon beams results in a low skin dose. This is not accept-able for the treatment of the majority of the parotid cancers becausemost parotid cancers are located in the superficial lobe of the parotidgland. Although electron beams may be the best option in terms ofnormal tissue sparing, it is impossible to achieve depth dose confor-mity with a single electron energy only. The third technique typicallyemploys a high-energy electron beam (12 MeV–20 MeV) and a single6 MV photon beam. However, good matching of photon and elec-tron beams is not easy to achieve and poor matching may produceareas of high inhomogeneity within the tumor dose distribution.

Recently, there has been wide interest in using photon beamIMRT to treat head and neck cancers and parotid cancers inparticular.57,59−64 A common feature of the head and neck cancersis their complex geometry. IMRT has shown potential to pro-duce a highly conformal dose distribution around the concave-shaped target volumes and a steep dose gradient near the OARs.



These characteristics can spare radiosensitive normal structures andreduce complication rates.65−67 In addition, there has been an effortto exploit IMRT to improve local-regional tumor control throughdose escalation. Although the photon beam IMRT is a powerfulmodality for treating the parotid tumors that extend deeply into tis-sue, it is not suitable to treat very shallow targets due to the low sur-face dose and large depth of photon beam penetration. For photonbeam IMRT, the slow attenuation of photon beams can still delivera high dose to the contralateral parotid gland.

Several research groups have used scanned beam systems(MM50 racetrack microtron, Scanditronix Medical AB, Uppsala,Sweden) to improve dose distributions for superficial targets usingintensity- and energy-modulated high-energy electron beams.68,69

The MM50 racetrack microtron has two major advantages. First ofall, it can provide high-energy electrons up to 50 MeV. Secondly, thesame multileaf collimator (MLC) can be used for both photons andelectrons. Treatment planners could easily make plans that consist ofboth photon and electron beams. The plans could be delivered usingthe existing linear accelerators without any further capital invest-ment. However, the use of photon MLC for electron collimationhas severe limitations. It has been shown that a source-to-surfacedistance (SSD) of 70 cm was needed to produce a clinically accept-able field when using the photon MLC and the beams collimatedby the photon MLC were inferior to applicator fields in penumbraand uniformity.70 Others have tried proton beams in high precisionradiotherapy for targets close or distal to OARs.71 It is well knownthat clinical proton beams can deliver highly conformal and uniformhigh dose (the spread-out-Bragg peak) in the target with a sharpfall off dose in the OARs and a small lateral penumbra. However,because of prohibitively high capital cost of proton facilities, protonbeam therapy is only available at a few large academic centers in theworld today.

An attractive alternative approach to the treatment of parotidtumors is MERT. Figure 10 shows the comparison of the MERTand IMRT isodose distributions for a right parotid cancer case. The



95%

80%

70%

50%

30%

20%

110%

95%95%

80%80%

70%70%

50%50%

30%30%

20%20%

110%110%

Fig. 10. Comparison of MERT and photon IMRT plan isodose distributions for aright parotid cancer case. The isodose distributions on the left were created witha 5-field IMRT plan using 6 MV photons. The isodose distributions on the rightwere created with a MERT plan, consisting of five electron energies and three beamangles. The isodose curves were normalized to 55.0 Gy.

isodose distributions on the left were created with a 5-field IMRTplan using 6 MV photons. The isodose distributions on the right werecreated with a MERT plan, consisting of five electron energies andthree beam angles. The total number of fields for the MERT was, thus,15. In both plans, the isodose curves were normalized to 55.0 Gy.Both plans showed good target conformity for higher isodose lines.However, for the lower isodose lines, the photon beam IMRT plannot only exhibited a relatively poor conformity, but also penetratedmuch deeper regions than the corresponding MERT plan. This wasclearly shown in the axial slice of the IMRT plan, in which the 30%and 50% isodose lines covered a significant amount of normal tissue.On the contrary, the MERT plan showed a superior critical structuresparing.



24.4 SUMMARY

The rapid dose fall off of electron beams makes MERT an attrac-tive treatment modality for shallow targets. In addition, comparedto photon beams, electron beams have negligible scatter radiation.Furthermore, because MERT mainly uses normal incident electronbeams, it is not sensitive to patient’s respiration compared to tan-gential photon beams. However, difficulties associated with elec-tron beam treatment are increased beam penumbra at depth andhigher uncertainties in the dose calculation in the regions aroundthe interface between soft tissue and bone or lung. However, thesedrawbacks will not outweigh MERT’s benefits. Based on our ini-tial experience with MERT, we believe that MERT will be a viablealternative approach to superficial tumors in the near future. Asthe development of the computer-controlled EMLC is underway,we believe that the widespread routine implementation of thisnovel technique for superficial tumors should be further investi-gated.

24.5 FUTURE TRENDS

Currently, the most popular inverse treatment planning techniquesfall into two distinct categories: (1) beamlet-based inverse treat-ment planning and (2) aperture-based inverse treatment plan-ning. In beamlet-based treatment planning, each field (a beamangle/electron energy combination in case of MERT planning) isdivided into a matrix of beamlets whose weights are to be opti-mized independently. The optimization therefore yields an inten-sity map that is discretized into desired intensity levels and thenconverted into a sequence of leaf apertures before being delivered.However, there are several problems associated with beamlet-basedinverse treatment planning. First of all, optimized intensity maps areoften converted into a series of discrete levels in an attempt to makethe leaf sequencing easier. This step also introduces some quanti-zation errors and therefore results in loss of treatment plan quality.



Secondly, leaf sequencing is constrained by hardware-related fac-tors and therefore often requires a large number of complex fieldshapes or small monitor unit (MU) apertures to deliver a givenintensity map, thereby decreasing the overall delivery efficiency.Thirdly, since the leaf sequencing step is excluded from the intensitymap optimization process, therefore, all the delivery-related effects,such as leakage, the tongue-and-groove design, and head scatter arenot taken into account when choosing an intensity map.

Aperture-based optimization is designed to reduce the complex-ity of IMRT and MERT treatment plans and is flexible enough toeasily include delivery related effects. In particular, aperture opti-mization requires no leaf sequencing and apertures are guaranteedto satisfy hardware constraints. It is, therefore, a promising alterna-tive to beamlet-based MERT and IMRT. In addition, although con-siderable efforts have been made to realize beamlet-based MERTthrough EMLC,48,72 their applications are still limited by physicsand engineering constraints. Electron beam planning has to be donethrough a time consuming manual trail-and-error procedure. On thecontrary, aperture-based MERT plans can be delivered with elec-tron cutouts rather than an EMLC. At present time, it is, there-fore, the only practical approach to the delivery of MERT plans.However, to achieve an optimal target dose distribution and maxi-mize patient survival, it is of paramount importance to have a clini-cally applicable and fully optimized aperture-based MERT planningsystem.


The MERT study was supported in part by grants DAMD17-00-1-0443 (Yulin Song and Steve Jiang) and DAMD17-00-1-0444 (ToddPawlicki) from the US Department of Defense. We would like toexpress our sincere thanks to Drs Arthur Boyer, C-M Ma, Lei Xing,Todd Pawlicki, Steve Jiang, and Michael C Lee for many useful dis-cussions and supports.



References

1. Mundt AJ, Roeske JC (eds), Intensity modulated radiation therapy:A clinical perspective, BC Decker, Lewiston, NY, 2004.

2. Dobbs HJ, Radiation therapy for breast cancer at the millennium,Radiot & Oncol 54: 191–200, 2000.

3. Barbiere J, Hanley J, Song Y, Dhawan A, et al., Concise matrix analy-sis of point-based prostate targeting for intensity modulated radiationtherapy, Technol Cancer Res Treat 6(1): 1–10, 2007.

4. Zelefsky MJ, Fuks Z, Hunt M, Yamada Y, et al., High-dose intensitymodulated radiation therapy for prostate cancer: Early toxicity andbiochemical outcome in 772 patients, Int J Radiat Oncol Biol Phys 53:1111–1116, 2002.

5. Chan MF, Chui CS, Song Y, Burman C, et al., A novel radiation therapytechnique for malignant pleural mesothelioma combing electrons withintensity-modulated photons, Radiot Oncol 79(2): 218–23, 2006.

6. IMRT Collaborative Working Group, Intensity-modulated radiother-apy: Current status and issues of interest, Int J Radiat Oncol Biol Phys51: 88–914, 2001.

7. AAPM IMRT Sub-committee, Guidance document on delivery, treat-ment planning, and clinical implementation of IMRT: Report of theIMRT subcommittee of the AAPM radiation therapy committee, MedPhys 30: 2089–2115, 2003.

8. Palta JR, Mackie TR, (eds), Intensity-modulated radiation therapy:The state of the art, American Association of Physicists in Medicine(AAPM), Medical Physics Monograph, No 29, Medical Physics Pub-lishing, Madison, WI, 2003.

9. Webb S, Intensity-modulated radiation therapy, Institute of PhysicsPublishing, Bristol, UK, 2001.

10. Xing L, Lin Z, Donaldson SS, et al., Dosimetric effects of patient dis-placement and collimator and gantry angle misalignment on intensitymodulated radiation therapy, Radiot Oncol 56: 97–108, 2000.

11. Chao KS, Majhail N, Huang CJ, et al., Intensity-modulated radiationtherapy reduces late salivary toxicity without compromising tumorcontrol in patients with oropharyngeal carcinoma: A comparison withconventional techniques, Semin Radiat Oncol 12(1 Suppl 1): 20–25, 2002.

12. Leibel SA, Fuks Z, Zelefsky MJ, Hunt M, et al., Technological advancesin external-beam therapy for the treatment of localized prostate cancer,Semin Oncol 30: 596–615, 2003.

13. Eng TY, Thomas CR Jr, Herman TS, Primary radiation therapy for local-ized prostate cancer, Urol Oncol 7: 239–257, 2002.



14. Lee N, Xia P, Akazawa P, et al., Intensity modulated radiotherapy inthe treatment of nasopharyngeal carcinoma: An update of the UCSFexperience, Int J Radiat Oncol Biol Phys 53: 12–21, 2002.

15. Mohan R, Wang X, Jackson A, et al., The potential and limitations ofthe inverse radiotherapy technique, Radiot Oncol 32: 232–248, 1994.

16. Wang XH, Mohan R, Jackson A, et al., Optimization of intensity-modulated 3D conformal treatment plans based on biological indices,Radiot Oncol 37: 140–152, 1995.

17. Chui CS, Spirou SV, Inverse planning algorithms for external beamradiation therapy, Med Dosim 26: 189–197, 2001.

18. Spirou SV, Chui CS, A gradient inverse planning algorithm with dose-volume constraints, Med Phys 25: 321–333, 1998.

19. Kirkpatrick S, Gelatt C, Vecchi M, Optimization by simulated anneal-ing, Science 220: 671–680, 1983.

20. Earl MA, Afghan MK, Yu CX, Jiang Z, et al., Jaws-only IMRT usingdirect aperture optimization, Med Phys 34(1): 307–314, 2007.

21. Ahunbay EE, Chen GP, Thatcher S, Jursinic PA, et al., Direct aper-ture optimization-based intensity-modulated radiotherapy for wholebreast irradiation, Int J Radiat Oncol Biol Phys 67(4): 1248–1258, 2007.

22. Bedford JL, Webb S, Direct-aperture optimization applied to selectionof beam orientations in intensity-modulated radiation therapy, PhysMed Biol 52(2): 479–498, 2007.

23. Akazawa C, Treatment of the scalp using photon and electron beams,Med Dosim 14: 129–131, 1989.

24. Tung SS, Shiu AS, Starkschall G, et al., Dosimetric evaluation of totalscalp irradiation using a lateral electron-photon technique, Int J RadiatOncol Biol Phys 27: 153–160, 1993.

25. Kinard JD, Zwicker RD, Schmidt-Ullrich RK, et al., Short communi-cation: Total craniofacial photon shell technique for radiotherapy ofextensive angiosarcomas of the head, Br J Radiol 69: 351–355, 1996.

26. Klein EE, Modulated electron beams using multisegmented multileafcollimation, Radiot Oncol 48: 307–311, 1998.

27. Karlsson MG, Karlsson M, Zackrisson B, Intensity modulation withelectrons: Calculations, measurements, and clinical applications, PhysMed Biol 43: 1159–1169, 1998.

28. Yaparpalvi R, Fontenla DP, Beitler JJ, Improved dose homogeneity inscalp irradiation using a single set-up point and different energy elec-tron beams, Br J Radiol 75: 670–677, 2002.

29. Locke J, Low DA, Grigireit T, et al., Potential of tomotherapy for totalscalp treatment, Int J Radiat Oncol Biol Phys 52: 553–559, 2002.

30. Bedford L, Childs PJ, Hansen VN, et al., Treatment of extensivescalp lesions with segmental intensity-modulated photon therapy, IntJ Radiat Oncol Biol Phys 62: 1549–1558, 2005.



31. Song Y, Boyer A, Pawlicki T, et al., Modulated electron radia-tion therapy-emerging technology, in Intensity Modulated RadiationTherapy — A Clinical Perspective, pp. 327–336, BC Decker, Lewiston,NY, 2004.

32. Ma CM, Pawlicki T, Lee MC, Jiang SB, et al., Energy-and intensity-modulated electron beams for radiotherapy, Phys Med Biol 45: 2293–2311, 2000.

33. Hyodynmaa S, Gastafsson A, Brame A, Optimization of conformalelectron beam therapy using energy- and fluence-modulated beams,Med Phys 23: 659–666, 1996.

34. Lief EF, Larsson A, Humm JL, Electron dose profile shaping by modu-lation of a scanning elementary beam, Med Phys 23: 33–44, 1996.

35. LichterAS, Cancer of the Breast, in StevenALeibel, Theodore LPhillips(eds), Textbook of Radiation Oncology, pp. 1299–1333, Phillips, Elesvier,2004.

36. Early Breast Cancer Trialists’ Collaborative Groups: Favorable andunfavorable effects on long-term survival of radiotherapy for earlybreast cancer: An overview of the randomized trials, Lancet 355: 1757,2000.

37. Vicini FA, Douglas A, et al., Breast cancer overview, in Intensity Mod-ulated Radiation Therapy — A Clinical Perspective, 386–396, BC Decker,Lewiston, NY, 2004.

38. Song Y, Peng P, Boyer AL, et al., Concurrent boost using forwardmultiple-segment planning and step-and-shoot delivery: Anovel tech-nique of breast-conserving radiation therapy, Int J Radiat Oncol Biol Phys57: S368, 2003.

39. Chui CS, Hong L, Hunt M, et al., A simplified intensity modulatedradiation therapy techniques for the breast, Med Phys 29: 522–529, 2002.

40. Hong L, Hunt M, Chui C, et al., Intensity modulated tangential beamirradiation of the intact breast, Int J Radiat Oncol Biol Phys 44: 1155–1164,1999.

41. William SS, Xing L, Boyer AL, et al., Intensity modulated treatment ofbreast cancer with inclusion of superclavicular and internal mammarylymph nodes, Int J Radiat Oncol Biol Phys 42: 370, 1998.

42. Patey DH, Dyson WH, The prognosis of carcinoma of the breast inrelation to the type of operation performed, Br J Cancer 2: 7, 1948.

43. Muller-Runkel R, Kalokhe UP, Scatter dose from tangential breast irra-diation to the uninvolved breast, Radiology 175–873, 1990.

44. Boice J, Harvey F, et al., Cancer in the contralateral breast after radio-therapy for breast cancer, N Engl J Med 326: 781, 1992.

45. Jansson T, Lindman H, Nygard K, et al., Radiotherapy of breast cancerafter breast-conserving surgery: An improved technique using mixed



electron-photon beams with a multileaf collimator, Radiot & Oncol 46:83–89, 1998.

46. Li JG, Xing L, Boyer Al, et al., Matching photon and electron fields withdynamic intensity modulation, Med Phys 26: 2379–2384, 1999.

47. Farrow DC, Hunt WC, Samet JM, Geographic variation in the treatmentof localized breast cancer, N Engl J Med 326: 1097–1101, 1992.

48. Song Y, Jiang SB, Lee MC, Ma CM, et al., A Multileaf collimator formodulated electron radiation therapy (MERT) for breast cancer, TheDepartment of Defense Breast Cancer Research Program Meeting ProceedingsVolume I, pp. 17–25, 2002.

49. Parker S, Tong T, Bolden S, Wingo P, Cancer Statistics, CA Cancer J Clin47: 5–27, 1997.

50. Seifert G, Brocheriou C, Cardesa A, Eveson JW, WHO internationalclassification of tumors. Tentative histological classification of salivarygland tumors, Pathol Res Pract 186(5): 555–581, 1990.

51. Garden AS, El-Naggar AK, et al., Postoperative radiotherapy for malig-nant tumors of the parotid gland, Int J Radiat Oncol Biol Phy 37: 79–85,1997.

52. Spino IJ, Wang CC, Montogmery WW, Carcinoma of the parotid gland.Analysis of treatment results and patterns of failure after combinedsurgery and radiation therapy, Cancer 71: 2699–2705, 1993.

53. Lenhard Jr RE, Osteen RT, Gansler T, Clinical oncology, AmericanCancer Society 297–329: 2001.

54. North CA, Lee DJ, Piantedosi S, et al., Carcinoma of the major salivaryglands treated by surgery plus postoperative radiotherapy, Int J RadiatOncol Biol Phy 18: 1319–1326, 1990.

55. Sykes AJ, Logue JP, Slevin NJ, et al., An analysis of radiotherapy inthe management of 104 patients with parotid carcinoma, Clin Oncol 7:16–20, 1995.

56. Yaparpalvi R, Fontenla DP, Tyerech SK, et al., Parotid gland tumors:A comparison of postoperative radiotherapy techniques using threedimensional (3D) dose distributions and dose-volume histograms(DVHS), Int J Radiat Oncol Biol Phy 40(1): 1319–1326, 1990.

57. Nutting CM, Rowbottom CG, Cosgrove VP, Henk JM, et al.,Optimization of radiotherapy for carcinoma of the parotid gland:A comparison of conventional, three-dimensional conformal, andintensity-modulated techniques, Radiot Oncol 60: 163–172, 2001.

58. Garden AS, El-Naggar AK, Morrison WH, Callender DL, et al., Post-operative radiotherapy for malignant tumors of the parotid gland, IntJ Radiat Oncol Biol Phy 37(1): 79–85, 1997.



59. Boyer AL, Geis P, Grant W, Carol M, Modulated beam conformal ther-apy for head and neck tumors, Int J Radiat Oncol Biol Phy 39: 227–236,1997.

60. Bragg CM, Conway J, Robinson MH, The role of intensity-modulatedradiotherapy in the treatment of parotid tumors, Int J Radiat Oncol BiolPhy 52(3): 729–738, 2002.

61. Wu Q, Manning M, Schmidt-Ullrich R, Mohan R, The potential forsparing of parotids and escalation of biologically effective dose withintensity-modulated radiation treatments of head and neck cancers:A treatment design study, Int J Radiat Oncol Biol Phy 46(1): 195–205,2000.

62. Cozzi L, Fogliata A, Lomax A, Bolsi A, A treatment planning compari-son of 3D conformal therapy, intensity modulated photon therapy andproton therapy for treatment of advanced head and neck tumors, RadiotOncol 61: 287–297, 2001.

63. Vineberg KA, EisbruchA, Coselmon MM, McShan DL, et al., Is uniformtarget dose possible in IMRT plans in the head and neck?, Int J RadiatOncol Biol Phy 52(5): 1159–1172, 2002.

64. Rowbottom CG, Nutting CM, Webb S, Beam-orientation optimizationof intensity-modulated radiotherapy: Clinical application to parotidgland tumors, Radiot Oncol 59: 169–177, 2001.

65. Boyer AL, The physics of intensity-modulated radiation therapy, PhysToday 38–44: 2002.

66. Boyer AL, et al., Intensity-modulated radiation therapy: Current statusand issues of interest, Int J Radiat Oncol Biol Phy 51(4): 880–914, 2001.

67. Xing L, Cotrutz C, Hunjun S, Boyer AL, et al., Inverse planning forfunctional image-guided IMRT, Phys Med Biol 47: 3567–3578, 2002.

68. Karlsson MK, Karlsson MG, Zackrisson B, Intensity modulation withelectrons: Calculations, measurements and clinical applications, PhysMed Biol 43: 1159–1169, 1998.

69. Korevaar EW, Huizenga H, Lof J, Stroom JC, et al., Investigation of theadded value of high-energy electrons in intensity-modulated radio-therapy: Four clinical cases, Int J Radiat Oncol Biol Phy 52(1): 236–253,2002.

70. Klein EE, Li Z, Low DA, Feasibility study of multileaf collimated elec-trons with a scattering foil based accelerator, Radiot Oncol 46: 189–196,1996.

71. Suit HD, Protons to Replace Photons in External Beam RadiationTherapy?, Clin Oncol 15(1), S29–S31, 2003.

72. Lo A, Song Y, Boyer A, Pawlicki T, et al., Combing IMRT and MERT forbreast-conserving radiation therapy, RSNA, 2003.


CHAPTER 25

Image Guidance in Radiation Therapy

Maria YY Law

Radiation therapy is one of the modalities for treating cancers. It has beenimage-based since its early stage of implementation. Medical images areused in radiation therapy for tumor diagnosis, tumor delineation, treat-ment planning and treatment verification. Computerization and advanceddevelopment in medical imaging have revolutionarized the practice ofradiation therapy. The invention of computer-controlled multileaf collima-tors leads to the development of conformal radiation therapy and intensitymodulated radiation therapy which enables the irradiated target volumeto be more conformed to the shape of the tumor, sparing more surroundingnormal tissue and thus decrease the probability of toxic effects to normaltissue. At the same time, a higher dose can be delivered to the tumorfor better tumor control. However, systematic and random errors due topatient positioning and organ motion occur in a course of radiation ther-apy. They need to be monitored and accounted for by more stringent imageguidance methods at times of tumor delineation, treatment planning andtreatment delivery. This chapter describes the concept of image guidancein radiation therapy and reviews the different technologies adopted forimage-guided radiation therapy.

25.1 INTRODUCTION

Radiation therapy is a treatment modality that uses ionizing radia-tion for treating a majority of cancers (malignant tumors). The ulti-mate goal of radiation therapy is to give as high a dose as possibleto the tumor but as little dose as possible to the surrounding nor-mal tissue. More accurate targeting of a tumor allows better tumorcontrol. On the other hand increased dose to a non-tumor bearing

635


636 Maria YY Law

organ e.g. increased dose into the lung when treating a breast cancer,will increase the toxicity into the lung. Failure of achieving the goalwould either lead to failure in tumor control or long-term compli-cations in the surrounding tissues.

Basically radiation therapy consists of two major procedures,treatment planning and treatment delivery. Medical images areinvolved in many of the steps and have to a large extent, contributedto the evolution of treatment techniques in radiation therapy.

Treatment planning involves delineation of tumor target vol-ume and critical structures (Fig. 1) nearby as well as arrangingthe radiation beams to achieve the best radiation dose distributionthat will kill the tumor but spare the surrounding normal tissue asmuch as possible. It begins with imaging in the treatment simula-tor or CT simulator. The former produces 2D radiographic projec-tional images while the latter, like an ordinary CT scanner, producescross-sectional images. CT images taken in treatment positions canbe input into treatment planning system (TPS) for radiation beamplanning. The beam arrangement will then be displayed in the TPS

Fig. 1. Delineation of target volume (red) and organs at risk of a prostate can-cer case for treatment planning. The cross-sectional CT images (upper left) canbe reconstructed to different planes: coronal plane (bottom left) and sagittal plane(right).


Image Guidance in Radiation Therapy 637

Fig. 2. DRR (digitally reconstructed radiograph) reconstructed from CT imagesfor 2D treatment planning.

workstation with the resultant radiation dose distribution superim-posed on the tumor and the surrounding structures. The graphicaldisplay of the tumor, critical structures around and the radiationdose distribution helps radiation oncologists to decide on the suit-ability of the treatment plan for the patient. Digitally reconstructedradiographs (DRR) can be generated from the set of CT images in theTPS for 2D treatment planning (Fig. 2). Simulator images or DRRsare planning images that also serve as reference images for set upverification later.

In treatment delivery, it is important to verify that the radiationbeam is delivered as planned. Conventional verification involves theacquisition of 2D radiographic images on film or by electronic portalimaging device (EPID) from the megavoltage (MV) beam of the lin-ear accelerator with the patient in treatment position. These imagesare called portal or verification images. Bony landmarks, which arethe internal structures most visible on megavoltage images, are visu-ally aligned with those on the planning or reference images to deter-mine if the treatment portals match accurately with the treatmentplans. Discrepancy between the images requires corrections or even


638 Maria YY Law

replanning on some occasions. It can be seen that radiation therapyhas been an image-guided treatment modality from its early stage.

Computerization in medicine and digitization in medical imag-ing have revolutionarized radiation therapy and allowed the moreintensive use of images. Different image processing techniques aredeployed to aid in the visualization of the treatment plans. Seg-mentation is used to delineate the tumor and neighboring impor-tant body structures. For example, in planning a prostate tumor,other than delineating the tumor, the rectum behind the tumor orthe bladder nearby need to be segmented for planning of the radia-tion beams. Surface or volume rendering are important methods forseeing the target tumor or body structures or even the level of radi-ation dose within the body. The dose-volume histogram (DVH) isan important tool for appraisal of a radiation treatment plan (Fig. 3).The combination of image processing methods provides the vol-ume of segmented structure(s) or tumor and the treatment planning

Fig. 3. Dose-volume histogram (DVH) for treatment plan evaluation.



algorithm provides the dose at different points within the irradi-ated volume. Together, they result in the formulation of the dose-volume histogram for medical decision on the appropriateness ofthe treatment plan. To improve the accuracy of tumor delineation,multimodality images such as CT and MRI or CT and PET can bevariously aligned, registered and be mapped to each other or fusedto obtain the maximum information from the combined images.PET/CT fusion images have shown to enhance tumor target volumedefinition and the registration algorithm is well established.1 Imageacquisition and image processing such as segmentation, registrationand visualization are crucial to all procedures in radiation therapy.

25.2 IMAGE-GUIDED RADIOTHERAPY

A course of radiation treatment is normally delivered in many frac-tions of small doses. In the course of treatment, geometric uncer-tainties may arise between fractions (interfractional) or even withina fraction (intrafractional). Examples of such uncertainties includeorgan motion, variable filling of digestive or urinary organs, setuperrors, tumor shrinkage and change in weight of patients (as thecourse may last for several weeks). To avoid marginal miss of thetumor, it is necessary to have a generous margin around the grosstumor volume (Fig. 4). The width of the margin depends on the esti-mated extent of tumor invasion, planning and setup errors, possible

Irradiated volume

Gross target volume(GTV)

Treated volume

Clinical target volume(CTV)Planning target volume(PTV)

Fig. 4. Target volume definition (ICRU Report 50).


640 Maria YY Law

organ motion and the radiosensitivity of adjacent normal tissue.The wider the margin means the more surrounding normal tis-sue would be irradiated. In order to reduce complication to nor-mal tissue, it is ideal to keep the margin to its minimum but atthe same time to ensure that there is no geographical miss on thetumor. The evolvement of imaging technologies such as CT, MR,PET, advancement in planning software and computer-controlledradiation treatment delivery such as computerized collimator sys-tems have all contributed to the delivery of a treatment that can betightly conformed to the tumor, while the surrounding normal tissuecan be spared more from the unwanted radiation.

The recent development of 3D conformal radiation therapy(3DCRT) and intensity modulated radiation therapy (IMRT) pro-duces shaped radiation dose that closely conforms to the tumordimensions. This means a tighter margin is allowed around thetumor target volume so that a higher radiation dose can be deliv-ered (dose escalation) for tumor killing with reduced risk of com-plications to surrounding normal tissue. For example, using IMRT,a higher dose can be given to the prostate tumor which leads tobetter tumor control while the neighboring rectum received a mini-mal radiation dose and thus lowers the risk of rectal complications.These techniques call for a high degree of accuracy and precisionof tumor delineation and beam targeting. These stringent require-ments set off the scene for more sophisticated image guided tech-nologies to cope with possible geometric uncertainties in radiationtherapy. The technologies include image-based tumor localizationmethods in treatment planning as well as patient positioning devicesand radiation delivery guiding tools used in treatment delivery.Image-guided radiation therapy (IGRT) “attempts to correct randomand systematic errors associated with: daily organ motion, dailypatient setup, patient immobilization, transfer of treatment plan-ning information, delivery of treatment plan, corrections made topretreatment delivery and internal organ motion between the timeof planning and actual treatment delivery”.2 Its goal is “to man-age both interfractional and intrafractional motion to improve theaccuracy of treatment delivery”.3 In short, IGRT is the use of images



to visually confirm that the radiation treatment plan is accuratelyreproduced during radiation treatment. It is best adopted in situ-ation where error is likely to result in large adverse consequencesuch as 3DCRT and IMRT and where large setup uncertainties areexpected as in obese patients as well as for diseases in certain sitesof the body e.g. head and neck, lung, liver or prostate.

25.3 IMAGE-GUIDED TECHNOLOGIES FORRADIATION THERAPY

25.3.1 Imaging for Radiation Therapy Planning

Treatment planning starts with tumor volume delineation, theaccuracy of which is vital to the subsequent procedures. Advancedimaging technologies contribute to the precision in target volumedelineation which is crucial in reducing or eliminating uncertainties.

25.3.1.1 Multimodality imaging

CT scans are the most commonly used modality for tumor delin-eation as the dose calculation algorithms of TPSs are based on theelectron density of CT images. Many radiation oncology depart-ments have their own CT scanners with simulation functions. Thepatients are scanned in their immobilized treatment positions. TheCT images acquired will be transferred in the DICOM format tothe TPS, where the tumor is delineated. DRRs can be reconstructedfor 2D visualization as in a conventional KV simulator and serveas the planning or reference images. KV CT has high spatial res-olution (512 × 512 pixels), shows excellent bony structures andprovides relative electron density information for radiation dosecalculation.1 Other imaging modalities can be registered to CTscans for enhancing tumor volume delineation. MRI shows betterdiscrimination for soft tissue and often shows the extent of the tumorbetter. Magnetic resonance spectroscopy (MRS) and PET (positronemission tomography) can provide functional information unavail-able from CT images. They show the spatial extent of the tumor


642 Maria YY Law

with its physiological movement about the tumor and normal tis-sue. Coregistration of images (image fusion) in the same settingfrom different imaging modalities e.g. CT and MRI, CT and PETwill enhance target volume delineation.4 Software for image regis-tration is available in most TPSs nowadays. To improve the accuracyof delineating deforming organs e.g. liver or tumor in moving struc-tures such as lung tumor, deformable registration techniques wouldbe helpful. Multimodality anatomic imaging registration PET-CTdoes not only help in target delineation, but also in disease appraisaland assessment of therapy response.

25.3.1.2 Imaging for organ motion

Organ motion is a major source of error in beam targeting. A tumorin the lung and organs in the upper abdomen can move up to threecentimeter with respiration.3,5,6 Other internal organs such as pan-creas, liver or prostate also move due to digestion, respiration orvariation in filling from day to day. If the tumor is moved out ofthe planned range of the radiation beam, it will not receive the fullradiation dose as planned and what is worse is a high dose will bedelivered to the adjacent normal tissue instead. The situation willdefeat the purpose of radiation therapy.

In tumors that are subject to motion, it is important to accountfor the motion. Methods include measuring the motion by differ-ent imaging modalities; reducing motion e.g. by applying com-pression on abdomen; eliminating motion during treatment e.g. bybreath-holding; increasing the PTV thus providing a wider marginto accommodate the potential error; tracking the tumor positionby continuous monitoring of internal anatomy or surrogates; andradiation beam gating to synchronize the beam with the movingtarget.

Organ motion can be visualized and assessed in fluoroscopy,which is a function in conventional KV treatment simulators orincorporated in recent MV treatment units. Traditional CT onlyshows the respiratory motion as artifacts and the tumor volume isshown to be distorted or larger than what it is. As such, information



in the fourth dimension — time is needed. 3D CT has to be extendedto incorporate the fourth dimension, so that the changing positionof the tumor over time can be visualized.

Multidetector CT scanners can show the temporal changes frombreathing. The respiratory signal can be tracked and recorded e.g.by a camera over an external marker placed on the abdomen or bymonitoring the chest expansion with a pneumatic bellow. CT datacan be acquired over the whole respiratory cycle and then sortedbased on the phase of the respiratory cycle. The series of CT imagesover several points in the respiratory cycle are then reconstructed.Images through the respiratory phases from end-inspiration to end-expiration can be scrolled through and be exported to TPS in DICOMformat for geometric and dosimetric computation.

Other methods to counteract the artifacts caused by organmotion in radiation therapy are immobilization of the organs byrepeated breath holds, respiration gating and 4D tumor tracking. Thetumor is irradiated at the predetermined breathing interval when thetumor is at a constant position. Breath-hold techniques can be eitheractive or passive. Deep inspiration breath-hold, active breathing con-trol (inducing shallow breathing to minimize tumor motion) andself-held breath-hold are some of such techniques which have beenreported in treating liver cancer,7,8 Hodgkin’s disease9 and breastcancer. One of the most important steps of planning in IGRT is theaccurate registration of the tumor position at the same phase of res-piration as it is in treatment planning, hence such details need to betaken care of during planning.

Respiration gating is to match the radiation beam to the respi-ratory pattern. For monitoring the treatment, IGRT is coupled withrespiratory control. The patient’s breathing cycle is monitored by agating system that uses an infrared camera to track a marker that isplaced either on the patient’s chest or abdomen. The radiation beamis triggered to turn on and off as the tumor moves in and out ofthe preset gating window respectively (Figs. 5 and 6). For planningpurpose, 4D-CT images are acquired either prospectively or retro-spectively. In the former, CT images are only acquired at one specificphase of the patient’s normal breathing cycle. In the latter, multiple


644 Maria YY Law

Fig. 5. Respiratory cycle of patient is tracked and recorded (upper curve) and iscoupled with IGRT. Radiation beam is enabled at expiratory phase (lower curve).

volumetric CT images are acquired and then be sorted according tothe patient’s respiratory phases. The 4D-CT data can then be syn-chronized with the respiratory phase to show the exact location ofthe tumor at the specific respiratory phase.3

In 4D tumor tracking, fiducial markers are inserted into theirradiated site as an aid to visualization. CT images are taken forplanning and the coordinates of the markers are registered. Beforetreatment starts, the marker coordinates are traced for one to twominutes (at a rate of 30 times per second) and recorded. Registra-tion of the marker coordinates at the planning image and those attreatment will indicate if misalignment exists, and in which case,the treatment couch will be adjusted accordingly. Real-time tumor-tracking starts where radiation beam will only be turned on if thepositions of the fiducial markers are within the gating window andturned off if they are not in the planed position.10,11 While Shirato’sgroup used only one set of CT for treatment planning, Keall andcolleagues12 developed a method of 4D treatment planning thatinvolves a series of eight 3D CT image sets acquired at different res-piratory phases for DMLC (dynamic multileaf collimator)-based res-piratory motion tracking. Deformable image registration was used



Fig. 6. Upper and middle: Beam off to spare normal tissue. Lower: Beam on whentarget volume is in the range of the radiation beam (Courtesy of BrainLab AG,Heimstetten, Germany).


646 Maria YY Law

in mapping CT sets from the peak-inhale respiratory phase to thoseof subsequent phases. Treatment planning was done on each of theeight CT sets with the MLC aperture conforming to the PTV (plan-ning target volume). Dose distribution from each CT set was thenmapped back to the CT of the peak-inhale phase for analysis. Thegoal of the 4D planning is to estimate the extent of the tumor motionand decide on appropriate measures to ensure accurate beam tar-geting at treatment.

25.3.2 Imaging for Treatment Delivery

Another important IGRT technology is devising accurate methodsfor treatment verification and radiation dose delivery during treat-ment. Conventionally, for treatment verification, 2D portal imageswhether on film or by EPID are taken before the radiation dose isdelivered and at regular interval e.g. weekly thereafter, to be com-pared with the planning or reference images to confirm that the setupaccurate as planned. Though many EPID systems offer computer-assisted tools for registration of anatomical features with the plan-ning images and quantitative alignment analysis, errors if detectedare corrected retrospectively because adjustments can only be madeuntil the portal images are processed and reviewed, which is usu-ally after the treatment is delivered. To improve treatment accuracy,it requires immediate error correction (before the treatment is deliv-ered) and more frequent imaging such as daily.

25.3.2.1 MV/KV 2D imaging

MV EPIDs that generate digital images have been implemented forover a decade. An active matrix flat panel detector is installed oppo-site to the radiation source of the linear accelerator. Portal imagesof the irradiated area with patient in treatment position will be gen-erated. Manual matching or template aided matching is normallyused to determine the accuracy of the setup. However, the radiolog-ical properties of MV X-rays do not produce images of good con-trast and is an issue when very high precision treatment verificationis required. Also due to the higher radiation dose, MV imaging



precludes its daily use. On the other hand, KV X-rays generatehigher contrast images with lower radiation dose and thus allowmore frequent imaging. This has motivated the incorporation of KVX-ray tube systems in the treatment room that can be used for flu-oroscopy, radiography and cone beam CT. Examples are the On-Board Imager™ (OBI) (Varian Medical System, Palo Alto, USA);XVI (X-ray Volume Imaging) system (Elekta®, Stockholm, Sweden);and ARTISTE system (Siemens AG, Munich, Germany). They areinstalled orthogonally to the MV treatment beam. The details of thecone beam CT will be described later while other KV X-ray sys-tems for treatment verification purpose will be detailed in the nextparagraph.

25.3.2.2 Room mounted KV fluoroscopic imaging system

For real-time tumor/position tracking, room mounted KV fluoro-scopic units (two or more X-ray tubes paired with image intensi-fiers) are implemented10,11,13 Such technique requires the insertionof markers in the soft tissue tumor to aid visualization. Planning CTwill be acquired with the markers in situ and transferred to the spe-cific planning software for dose planning. Positions of the markersin the reference images are registered with those in the fluoroscopy.In other similar systems, DRRs can also be reconstructed from plan-ning CT for computer comparison with the real-time X-ray imagesacquired during treatment by registering the positions of the mark-ers (Fig. 7). Deviations, as compared with the DRR are displayed andthe computer-driven controller of the linear accelerator or treatmentcouch will automatically correct for the deviations up to a predeter-mined range. Another similar room mount KV X-ray system is theExacTrac® X-ray 6D which uses two KV X-ray tubes with opposedamorphous silicon panel imaging detectors to acquire 2 orthogo-nal planar X-ray images (Fig. 8). The Cyberknife (Accuray, Sunny-vale, CA USA), a 6 MV linear accelerator with a moving roboticarm, capable of 6 degree of freedom movement, is another exam-ple of image guided system that is coupled with dynamic trackingsoftware Synchrony.14,15 Its localization system is also based on 2


648 Maria YY Law

Fig. 7. (A) Fiducial markers; (B) Fiducial markers seen on KV radiographs;(C) Corresponding fiducial markers in DRR.

X-ray images generated by 2 X-ray tubes in orthogonal directions.By comparing the fiducial positions in these two images with thoseof the DRRs from the planning CT, the translation and rotation errorscan be calculated and adjustment can be made to the treatmentcouch. The tracking in the Cyberknife is a continuous procedureat predetermined intervals throughout the treatment.

25.3.2.3 Integrated CT/linear accelerator

Reduction in geometric uncertainties requires the visualizationof internal organs at treatment. CT images nowadays can be



Fig. 8. ExacTrac X-ray 6D (Courtesy of BrainLAB AG, Heimstetten, Germany).The stereoscopic X-ray image sets generated allow detection of tumor movement3-dimensionally.

reconstructed three or four dimensionally for radiation therapy. CTscan of patient in treatment position is the basis for radiation treat-ment planning and considered as the reference for treatment veri-fication. A CT dataset obtained during treatment and reconstructedto similar views provides a straightforward comparison of the treat-ment delivered with the intended treatment plan. Therefore, moreemphasis is now placed on introduction of facilities that can generateCT images in the radiation therapy treatment room.

To start with, bringing a CT scanner to the treatment room isthe easiest solution. The CT scanner is mounted on rail in the treat-ment room and moves over the treatment couch of the linear accel-erator. Patients can remain in the same position for imaging andtreatment. The close proximity of the imaging CT and the treat-ment unit enables fine adjustments for changes in the size, shapeor location of tumors and surrounding tissues by using CT imagestaken just before treatments are delivered. The first CT-on-rails was


650 Maria YY Law

installed in Japan in 1996. Nowadays commercial products such asPRIMATOM™ (Siemens Medical Solutions Inc) and EXaCT Target-ing™ (Varian Medical Systems Inc and GE Medical Systems) areavailable.

25.3.2.4 Helical MV CT

Tomotherapy is delivered by a 6 MV linear accelerator that uses aring gantry geometry like that of a CT scanner. With the slip ringtechnology of a diagnostic scanner, the unit is capable of contin-uous rotation around the the patient. Intensity modulated radia-tion therapy is given through a fan-beam multileaf collimator fromall angles around the patient slice by slice. CT imaging technol-ogy is incorporated in the gantry for precise localization of tumor.The patient’s anatomy can be reviewed the moment prior to treat-ment and the size, shape and intensity of the radiation beam can beadjusted to the precise location of the patient’s tumor. The Hi ArtSystem® (Tomotherapy Incorp, Middleton, WI USA) is a commer-cial tomotherapy unit. The energy fluence actually delivered to thepatient can be computed and superimposed on a CT representationto be compared with the planned. Hence dose-guidance in additionto image-guidance is possible with tomotherapy. MV energy has thedisadvantages of low inherent soft tissue contrast and poor detectionefficiency of X-ray detectors.16 Quantitative analysis of contour vari-ations has shown to be inferior to KVCT for prostate delineation.17

25.3.2.5 Cone beam CT (CBCT)

In contrast to conventional fan-beam CT which acquires images sliceby slice and in several rotations, CBCT captures a larger volumeof tissue in one scan with a physically larger detector. It involvesthe reconstruction of 3D volumetric data from 2D projections. Itworks for either MV or KV X-ray beams. Example of MV Conebeam can be found in MVision™ Megavoltage Cone Beam (MVCB)(Siemens, AG, Munich, Germany). A maximization of mutual infor-mation algorithm is used to automatically register the MV CBCT



data set with the planning CT.18 MV CBCT shows superior imagequality in the presence of materials with high atomic number such asdenture or hip prosthesis. However, generally speaking, the qual-ity of the anatomical features reconstructed from MV CBCT maynot be good enough for the accuracy desired for image guidance.Also it requires 1–2 cGy of radiation dose per scan. To improve theimage quality for better visualization in tracking the tumor and ver-ifying treatment setup, a KV X-ray source with large area amor-phous silicon flat-panel digital detector is now mounted on the drumof a medical linear accelerator with two independent robotic armsorthogonal to the treatment beam. This is in addition to the elec-tronic portal imaging devise (EPID) which is positioned oppositeto the MV beam. The same centre of rotation is shared by the twobeams. Examples of KV cone beam CT are the On-Board Imager™(OBI) (Varian Medical System, USA) (Fig. 9), X-ray Volume Imag-ing (XVI) system (Synergy® Elekta Oncology, Stockholm, Sweden)and ARTISTE™ system (Siemens AG, Munich, Germany). They pro-vide KV CT technologies for acquisition of data from cone-shapedbeam of X-rays rather than from the conventional fan-beam. Imagesare taken at each degree of rotation. A total of 360 or more pro-jections are collected over a thirty seconds to two minutes interval.The whole volume is reconstructed in one operation producing highresolution isotropic images instead of the conventional sliced imagewhich are stacked after reconstruction.19 Cone beam images havebeen used for body parts such as bladder, lung and prostate. Afteracquisition, software tools such as tile display or color wash displaymust be provided for processing of the images and for registrationof the CBCT images with the planning CT images either manually orautomatically (Fig. 10). Shifts in the x-, y- and x-direction and in therotational direction need to be shown to indicate the corrections thatshould be applied to the patient setup. CBCT has shown to revealsetup error, anatomical deformation and physiological changes.1

Other than longer acquisition times, registration is still a tediousand time-consuming procedure that deters the wide implementa-tion of IGRT. Much effort is required to improve the efficiency ofthe volumetric image registration. Also, wide field scatter and the


652 Maria YY Law

Fig. 9. A linear accelerator with an on-board imaging system orthogonal to theMV treatment beam.

slow gantry motion often introduces motion artifacts greater thanconventional CT scanners.20

To test its accuracy, KV CBCT-based setup corrections werecompared with orthogonal MV portal imaged-based correctionsfor patients with prostate cancer treated by external beam. Thetwo methods correlated well using three intraprostate gold fiducialmarkers but less well for matching of soft tissue,21 the visualizationof which is being explored actively.

The role of CBCT has been extended to online planning andtreatment delivery in a single step at the treatment unit for palliativespine metastases.21,22 The cupping artifacts were greatly reduced byusing image corrections and the accuracy of CBCT numbers wasimproved. Bony landmarks were sufficient for tumor definition.



Fig. 10. CBCT/CT registration in tile display (Courtesy of Tuen Mun Hospital,Hong Kong).

Dose placement was in agreement as planned. IGRT using CT veri-fication has proven to be a time efficient alternative to conventionalverification practices and has the potential to further improve patientoutcomes through better target volume localization.

25.3.2.6 Ultrasound-guided radiation therapy

Ultrasound (US) is a noninvasive, flexible and inexpensive imagingmodality with no extra radiation dose to patients. It has a higherdifferential capability for soft tissues than X-ray imaging modalities.US IGRT targeting system can provide fast localization of a treatmenttarget on a daily basis.

Such a targeting system combines US and a 3D tracking systemwith an interface to track tumor targets. An infrared optical camera


654 Maria YY Law

system is mounted on the ceiling or wall to track the position ofthe US transducer, which is attached to a tracking arm. The camerasare calibrated to recognize the isocenter of the treatment machineand thus the system allows the US images to be correlated with themachine coordinates. Real time US images in the axial and sagittalplanes are acquired with patient at the time of treatment delivery.Contours of the tumor volume or surrounding critical structuresfrom the planning CT, can be exported to the system in DICOM-RT format, and be superimposed onto the US images. The trans-ducer and couch positions are calculated and correlated with theregistered US/CT offsets in the X, Y, Z axes. 3D couch shift requiredto realign the patient to the correct position in the three principalaxes will be indicated. Two such commercial US systems are theBATCAM™ (B-mode Acquisition and Targeting device with opti-cal camera) (NOMOS, Cranberry Township, Pa) and the SonArraySystem (Varian Medical Systems, Palo Alto, Calif ).

The US method has been investigated to be a feasible tool thatcan significantly improve the residual mean 3D setup error vectorfrom 11 mm to 4.6 mm for vascular structures close to tumors.23 Theresult was confirmed by repeat CT scans after repositioning. Hasanand colleagues24 compared using the BAT™ transabdominal ultra-sound system and fiducial markers with BrainLAB Exac Trac™ sys-tem for IGRT in a group of prostate cancer patients and found nodifference in the acute toxicities. However, the US method is limitedto abdominal and pelvic regions where no air cavities or less amountof bone is present. Interuser variation of the contour alignment wasalso found significant.25,26 Potential for intrafractional treatment islimited.

25.3.3 Strategies for Error Correction

Setup variations can be separated into random and systematiccomponents.27 Random component is the daily variation includingrandom patient movement and periodic movement such as breath-ing. Systematic component, on the other hand, is the deviation



between planning positions and actual treatment positions andrepeats everyday. This is probably due to equipment inaccuracy,inaccurate outlining of target and the difference between patientpositioning in CT planning and actual treatment in linear accelera-tor. The goal of image guidance is to improve treatment accuracy bycorrecting setup errors before the radiation treatment is delivered.This means imaging at the treatment room prior to each treatmentand automatic or manual repositioning if misalignment exceeds thepredefined tolerance upon comparison between planning CT andtreatment CT. This is an online approach with immediate interven-tion based on predefined action level but requires more effort andtime at treatment delivery.

The offline approach is to acquire images without immediateintervention. In this approach, verification images are taken for thefirst three to five days of treatment, and statistical analysis is thendone for the systematic and random components of the setup errors.In contrast to online approach in which corrections are made forrandom errors, offline analysis only corrects for systematic errors,which is considered to have greater impact on tumor control prob-ability because it may cause under dosage to the tumor volume ateach fraction of the treatment. Corrections are made only at subse-quent fraction. In offline approach, the treatment plan can be reop-timized to account for individual patient variations. A new plan canbe developed based on the average position of the tumor or criti-cal organs as estimated from the treatment images. Such practice iscalled adaptive radiation therapy (ART) and can be repeated severaltimes during the course of radiation therapy.28,29 Treatment param-eters such as field margin, position of the MLC leaves, collimatorrotation and treatment dose may be adjusted to help achieve a safedose escalation. ART started as an offline method but can be broughtonline, which means based on the deviations obtained from the vol-umetric images, a new plan can be generated just prior to treatmentdelivery. Online replanning will significantly improve dose confor-mality and is the ultimate goal of IGRT.30 ART is particular useful incases of tumor regression during a course of radiation therapy.


656 Maria YY Law

25.3.4 Frequency of Imaging

IGRT for treatment delivery requires a change in the work pattern.Instead of taking images weekly during the treatment course, IGRTrequires imaging to be done as frequently as daily or for everyfraction. Also errors should be corrected immediately rather thanretrospectively.

The frequency of image guidance on patients with head and neckcancer treated by a helical tomotherapy unit was evaluated.31 Sys-tematic setup errors are generally reduced with more frequent imag-ing but for fractions that were not imaged, random setup errors arenot reduced. If every other treatment was imaged, about 11% of alltreatments still subject to 3D setup errors of at least 5 mm. The resid-ual setup errors could be reduced with increasing frequency of imageguidance. The use of daily electronic portal imaging for improv-ing precision in radiation therapy of prostate cancer was studied.32

Interfraction prostate bed motion, setup error and total positionalerror were analyzed by comparing the location of intraprostate goldseed fiducials on the electronic portal images with those on the DRRsfrom the planning CT. Among the errors, the total positioning errors>5 mm were found in 14.1%, 38.7% and 28.2% of all treatment frac-tions in the left-right, superior-inferior and anterior-posterior axesrespectively. This shows that daily imaging and immediate correc-tions are necessary for reduction of setup errors.

The online correction is normally based on a fixed action level.For example, correction should be done for errors >3 mm. Togetherwith the daily imaging for verification, the workload is considerableand the procedure is costly.33 Increasing the action levels to say 5 mmwould reduce the workload but might not bring dose benefits tothe organs at risk. Instead Keller and colleagues34 proposed onlinecorrection strategies that aim for compliance with original treatmentplan intent using dose volume histogram (DVH) and equivalentuniform dose (EUD) score. The new correction strategies were foundto comply effectively with the initial treatment plan intent and couldbe tailored to individual patient. If IGRT were to be used for allconventional fractionation that requires 30–40 fractions for a full



course, the tradeoffs between benefits and cost would be a subjectfor debate. Considerations are now given to hypofractionation, i.e.using fewer fractions to save the setup time.33,35

25.4 CLINICAL RESULTS

Dose escalation to tumor will result in a rise in tumor control proba-bility. A tighter margin around the target volume means a reductionin the volume of irradiation to normal tissue which will reduce theprobability of normal tissue complications. With image guidance in astudy on prostate cancer treatment, Ghilezan and colleagues36 foundthat the target dose could be increased by 13% on average. Thoughadvantage varies for individual patients, substantial dose escala-tion was possible in 32% of the patients. In a study by Ramsey andcolleagues,37 image-guided ART was used in a group of seven lungcancer patients to adjust the planning target volume (PTV) weeklybased on the previous week’s CT images used for image-guidedsetup. The gross target volume (GTV) was reduced by 60%–80%and the ipsilateral lung volume receiving 20 Gy can be reduced toan average of 21%. Redpath and Muren38 found that use of IGRT inurinary bladder cancer treatment leads to significant reduction (from30 mm to 16 mm) in the required margin for full volume coverage.Online CBCT guidance reduces the random errors in setup in partialbreast irradiation when compared with the conventional method ofusing skin marks.39 Preliminary geometric benefits support reduc-tion in PTV margins in IGRT cases.40 Whether such benefits can betranslated to improvement in treatment outcome is an issue to beaddressed by clinical trials.

25.5 FUTURE WORK

Research studies are ongoing evaluating the appropriateness ofthe image-guided technologies. The image-guided tools, whetherequipment, accessories or software programs, should be able toreview accurately the extent of errors that is likely to occur during


658 Maria YY Law

treatment and make automatic corrections. The accuracy of any newtool should be measured against the standard tools.

Current IGRT tool development is more focused on geometricprecision of tumor and margins around it. Accuracy of the toolshas proved acceptable. Biological and functional imaging may helpprovide the necessary information, not only for enhancing targetdefinition and for treatment planning, but also to adapt the radia-tion dose distribution within the tumor so as to increase the tumorcontrol probability. Image registration techniques are needed to linkbiological imaging with scans obtained at treatment. Methods fordose verification are also needed from IGRT tools to ensure that thedelivered dose is the planned/prescribed dose.41

Image registration techniques for rigid transformation are wellestablished. In IGRT, because of its varying imaging conditions andthe multimodality imaging, the need for deformable image registra-tion is on the increase and becomes a fundamental tool for imageanalysis. Organ deformation during respiration can be modeled42

and organ motion or 4D planning19 can be incorporated into theTPS. Currently, a robust and efficacious algorithm is still lackingthough numerous existing methods are being validated. It is hopedthat deformable registration can be a standard in radiation therapytreatment planning systems.

Thorson and Prosser19 suggested storing the imaging data ofpatients with similar disease and technique for retrospective sys-tematic analysis for better prediction of organ motion or status. Forexample, the bladder status at the time of treatment can be predictedfrom large amount of imaging data of the specific patient group forestablishing treatment margin and doses. This predictive approachwould provide guidelines for future IGRT planning.

25.6 SUMMARY

IGRT uses images for precise tumor and organ delineation specificfor each patient and for the estimation of organ motion. Image acqui-sition, segmentation, registration and visualization are performed at



treatment planning. MR images can be registered to CT images andallow better target delineation. Before treatment, the target locationor treatment setup is verified by different imaging technologies, themore popular one being CBCT. Registration of planning images withverification images provides the magnitude of deviations from thetreatment plan intent for correction to be made or for replanning. Forexample, DRR from planning CT is registered with “live” 2D X-rayimages taken at treatment using bony anatomy or fiducial markersor CT/CBCT 3D registration. The translation needed to register theimage pairs is used to calculate a 3D setup error for the patient. Forevaluation of target motion, 4D CT images are needed for planningand verification of beam gating treatment. The DRRs from 4D plan-ning CT are to be correlated with the DRRs of CBCT on treatmentunit for verification.Astrong need for image registration at planningand treatment is indicated in IGRT.

References

1. Xing L, Thorndyke B, Schreibman E, Yang Y, et al., Overview of image-guided radiation therapy, Med Dosimet 31: 91–112, 2006.

2. Stinson, Image-guided radiation therapy, Radiat Therapist 15: 139–156,2006.

3. Huntzinger C, Munro P, Johnson S, Miettinen M, et al., Dynamic tar-geting image-guided radiotherapy, Med Dosimet 31: 113–125, 2006.

4. Heron DE, Smith RP,Andrade RS,Advances in image-guided radiationtherapy — the role of PET-CT, Med Dosimet 31: 3–11, 2006.

5. Dawson LA, Sharpe MB, Image-guided radiotherapy: Rationale, ben-efits, and limitations, Lancet Oncol 7: 848–858, 2006.

6. Langen KM, Jones DTL, Organ motion and its management, Int J RadiatOncol Biol Phys 50: 265–278, 2006.

7. Dawson LA, Eccles C, Bissonnette JP, Brock KK, Accuracy ofdaily image guidance for hypofractionated liver radiotherapy withactive breathing control, Int J Radiat Oncol Biol Phys 62: 1247–1252,2005.

8. Balter JM, Brock KK, Litzenberg DW, McShan DL, et al., Daily targetingof intrahepatic tumors for radiotherapy, Int J Radiat Oncol Biol Phys 52:266–271, 2002.

9. Stromberg JS, Sharpe MB, Kim LH, Kini VR, et al., Active breath-ing control (ABC) for Hodgkin’s disease: Reduction in normal tissue


660 Maria YY Law

irradiation with deep inspiration and implications for treatment, Int JRadiat Oncol Biol Phys 48: 797–806, 2000.

10. Shirato H, Shimizu S, Kitamura K, Nishioka T, et al., Four-dimensionaltreatment planning and fluoroscopic real-time tumor tracking radio-therapy for moving tumor, Int J Radiat Oncol Biol Phys 48: 435–442,2000.

11. Shirato H, Shimizu S, Kunieda T, Kitamura K, et al., Physical aspects ofa real-time tumor-tracking system for gated radiotherapy, Int J RadiatOncol Biol Phys 48: 1187–1195, 2000.

12. Keall PJ, Joshi S, Vedam SS, Siebers JV, et al., Four-dimensional radio-therapy planning for DMLC-based respiratory motion tracking, MedPhys 32: 942–951, 2005.

13. Shirato H, Suzuki K, Sharp GC, Fujita K, et al., Speed and amplitude oflung tumor motion precisely detected in four-dimensional setup andin real-time tumor-tracking radiotherapy, Int J Radiat Oncol Biol Phys64: 1229–1236, 2006.

14. Kuo JS, Yu C, Petrovich Z, Apuzzo MLJ, The cyberknife stereo-tactic radiosurgery system: Description, installation, and an ini-tial evaluation of use and functionality, Neurosurg 53: 1235–1239,2003.

15. Cerszten PC, Ozhasoglu C, Burton SA, Vogel WJ, et al., Cyberknifeframeless stereotactic radiosurgery for spinal lesions: Clinical experi-ence in 125 cases, Neurosurg 55: 89–99, 2004.

16. Groh BA, Siewerdsen JH, Drake DG, Wong JW, et al., A performancecomparison of flat-panel imager-based MV and KV cone beam CT, MedPhys 29: 967–975, 2002.

17. Song WY, Chiu B, Bauman GS, Lock M, et al., Prostate contouring uncer-tainty in megavoltage computed tomography images acquired with ahelical tomotherapy unit during image-guided radiation therapy, Int JRadiat Oncol Biol Phys 65: 595–607, 2006.

18. Amies C, Bani-Hashemi A, Celi J, Grousset G, et al., A multi-platformapproach to image guided radiation therapy (IGRT), Med Dosimet 31:12–19, 2006.

19. Thorson T, Prosser T, X-ray volume imaging in image-guided radio-therapy, Med Dosimet 31: 126–133, 2006.

20. Moore CJ, Amer A, Marchant T, Sykes JR, et al., Developments in andexperience of kilovoltage X-ray cone beam image-guided radiotherapy,Br J Radiol 79: S66–S78, 2006.

21. Létourneau D, Wong R, Moseley D, Sharpe MB, et al., Online planningand delivery technique for radiotherapy of spinal metastases usingcone-beam CT: Image quality and system performance, Int J RadiatOncol Biol Phys 67: 1229–1237, 2007.



22. YamadaY, Lovelock M, Bilsky MH, Image-guided intensity-modulatedradiation therapy of spine tumours, Current Neurology and NeuroscienceReports 6: 207–211, 2006.

23. Fuss M, Salter BJ, Cavanaugh SX, Fuss C, et al., Daily ultrasound-basedimage-guided targeting for radiotherapy of upper abdominal malig-nancies, Int J Radiat Oncol Biol Phys 59: 1245–1256, 2004.

24. Hasan IM, Reddy C, Mahadevan A, Comparison of acute toxicitieswhen utilizing image guidance for prostate cancer external beamradiation — Ultrasound vs. fiducial markers, J Radiat Oncol Biol Phys66(3): S539–S591, 2006.

25. Langen KM, Pouliot J, Anezinos C, Aubin M, et al., Evaluation ofultrasound-based prostate localization for image-guided radiotherapy,Int J Radiat Oncol Biol Phys 57: 635–644, 2003.

26. Fuller CD, Thomas CR, Wong A, Cavanaugh SX, et al., Image-guidedintensity-modulated radiation therapy for gallbladder carcinoma,Radioth Oncol 81: 65–72, 2006.

27. Bortfeld T, van Herk M, Jiang SB, When should systematic patientpositioning errors in radiotherapy be corrected? Phy Med Biol 47: N297–N302, 2002.

28. Yan D, Vicini F, Wong J, Martinez A, Adaptive radiation therapy, PhyMed Biol 42: 123–132, 1997.

29. Yan D, Ziaja E, Jaffray D, Wong J, et al., The use of adaptive radiationtherapy to reduce setup error: A prospective clinical study, Int J RadiatOncol Biol Phys 41: 715–720, 1998.

30. Burgess L, Zhang T, Liang J, Wu Q, et al., Image guided radiotherapy byonline plan re-optimization: Studies of dosimetric benefits by treatmentsimulations, Int J Radiat Oncol Biol Phys 66(3): S629–S630, 2006.

31. Zeidan OA, Langen KM, Meeks SL, Manon RR, et al., Evaluation ofimage-guidance protocols in the treatment of head and neck cancers,Int J Radiat Oncol Biol Phys 67: 670–677, 2007.

32. Schiffner DC, Gottschalk AR, Lometti M, Aubin M, et al., Daily elec-tronic portal imaging of implanted gold see fiducials in patients under-going radiotherapy after radical prostatectomy, Int J Radiat Oncol BiolPhys 67: 610–619, 2007.

33. Ling CC, Yorke E, Fuks Z, From IMRT to IGRT: Frontierland or Never-land? Radiother Oncol 78: 119–122, 2006.

34. Keller H, Jaffray DA, Rosewall T, White E, Efficient online setup cor-rection strategies using plan-intent, Med Phys 33: 1388–1397, 2006.

35. Song WY, Schaly B, Bauman G, Battista JJ, et al., Evaluation ofimage-guided radiation therapy (IGRT) technologies and their impacton the outcomes of hypofractionated prostate cancer treatments:A radiobiologic analysis, Int J Radiat Oncol Biol Phys 64: 289–300, 2006.


662 Maria YY Law

36. Ghilezan M, Yan D, Liang J, Jaffray D, et al., Online image-guided inten-sity modulated radiotherapy for prostate cancer: How much improve-ment can we expect? A theoretical assessment of clinical benefits andpotential dose escalation by improving precision and accuracy of radi-ation delivery, J Radiat Oncol Biol Phys 60: 1602–1610, 2004.

37. Ramsey CR, Langen KM, Kupelian PA, Scaperoth DD, et al., A tech-nique for adaptive image-guided helical tomotherapy for lung cancer,Int J Radiat Oncol Biol Physics 64: 1237–1244, 2006.

38. Redpath AT, Muren LP, CT-guided intensity-modulated radiotherapyfor bladder cancer: Isocentre shifts, margins and their impact on targetdose, Radiother Oncol 81: 276–283, 2006.

39. White EA, Lee G, Blackburn H, Nageeti T, et al., ConeBeam CT guidancefor daily setup of patients receiving accelerated partial breast irradia-tion, J Radiat Oncol Biol Phys 66(3): 104, 2006.

40. Wong JR, Uematsu M, Chen T, Merrick S, et al., Correction to tar-get (PTV) underdose and rectal overdose_review of 1762 CT scansobtained during image- guided radiation therapy using an in roomCT-on-rail with a linear accelerator for the treatment of prostate can-cer, J Radiat Oncol Biol Phys 66(3): S317–S318, 2006.

41. Wertz H, Boda-Heggemann J, Walter C, Dobler B, et al., Image-guidedin vivo dosimetry for quality assurance of IMRT treatment for prostatecancer, J Radiat Oncol Biol Phys 67: 288–295, 2007.

42. Schreibmann E, Chen GTY, Xing L, Image interpolation in 4D CT usinga BSpline deformable registration model, Int J Radiat Oncol Biol Phys64: 1537–1550, 2006.


CHAPTER 26

Functional Brain Mapping andActivation Likelihood Estimation

Meta-Analysis

Angela R Laird, Jack L Lancaster and Peter T Fox

In recent years, the increasing richness of data generated by fMRI andPET brain mapping studies has encouraged the growth of meta-analysisresearch. In response to this progress, a new method of quantitative, voxel-based meta-analysis, termed activation likelihood estimation (ALE), hasbeen developed and applied in a number of cognitive and perceptualdomains. Here, the method is discussed and findings from a meta-analysisof the Stroop task are highlighted.

26.1 META-ANALYSIS OF THE FUNCTIONAL BRAINMAPPING LITERATURE

Research in human functional brain mapping (HFBM) using func-tional magnetic resonance imaging (fMRI) or positron emissiontomography (PET) has increased at an astonishingly fast rate overthe past ten years, and this activity has generated a deluge ofpublished articles in the field. As a consequence, there exists anextremely rich resource available and suitable for large-scale datamining and meta-analysis of data designed to localize activationpatterns of various behavioral paradigms. This list of paradigmsincludes, but is not limited to, tasks such as delayed match to sam-ple, Stroop, mental rotation, saccades, semantic discrimination, andfinger tapping. While any single functional neuroimaging study can

663


664 Angela R Laird, Jack L Lancaster and Peter T Fox

highlight the neural activity that occurs in response to a unique com-bination of task implementation, imaging parameters, and scanningenvironment, combining the data obtained from multiple, indepen-dent studies gives a measure of the robustness of the observed acti-vation patterns.

There are a number of imaging standards in HFBM, but there aretwo in particular that allow for quantitative meta-analysis of fMRIand PET data. First, nearly all published studies include the analysisstep of spatial normalization in which individual subject brains arewarped and transformed into a standard brain space, referenced toa brain atlas. Second, it has become very common for researchers toreport locations of brain activation in response to a stimulus or taskas stereotactic (x, y, z) coordinates, reflecting the centers of mass ofthe activated brain regions. These two standards, one an analysisstandard and the other a reporting standard, have encouraged thegrowth of a new category of meta-analysis possible with functionalneuroimaging data.

Meta-analysis, generally defined as the post hoc combination ofindependently performed studies to better estimate a parameter ofinterest, has been utilized for decades in many medical fields.1−3

Traditional meta-analyses often merge nonsignificant results to testfor significance in pooled data. In human functional brain mapping,function-location meta-analysis has emerged as an analysis tool inwhich statistically significant effects from published studies are com-bined to create predictive models of neural systems.4,5

Function-location meta-analysis must be distinguished from tra-ditional literature review. The most common method of literaturereview in HFBM is to construct a table or figure that summarizesthe activation patterns of a given group of studies. This can bedone either by plotting stereotactic coordinates of activation on astandard brain, organizing the coordinates into a bar graph thatis segregated by cortical and subcortical regions, or by creating atable that individually lists these foci in text format. These methodsare widely used for finding agreement among studies with simi-lar experimental contrasts and are well accepted.6−12 However, as


Functional Brain Mapping and Activation Likelihood Estimation Meta-Analysis 665

opposed to meta-analysis, these reviews do not involve any quan-titative analysis of the patterns of brain activations, yield no formalestimate of probability, and are difficult to visually interpret.

26.2 ACTIVATION LIKELIHOOD ESTIMATION (ALE)

In 2002, Peter Turkeltaub13 presented a new and quantitative meta-analysis method, termed activation likelihood estimation, or ALE.13

In this first ALE publication, the method was presented, applied in ameta-analysis of single word reading PET studies, and verified in anfMRI reading task. Around the same time, Chein et al.14 published ameta-analysis of working memory studies using an analysis methodtermed aggregated Gaussian-estimated sources (AGES), which fol-lows the same general procedure detailed by Turkeltaub et al.13 Thesimultaneous development by two groups of the same voxel-basedmeta-analytic tool is strongly indicative of the timeliness and utilityof this form of meta-analysis. For simplicity, we henceforth refer tothis method as an ALE meta-analysis.

In ALE, each x, y, z coordinate of activation is thought of not as asingle point of activation, but rather as the center of a Gaussian prob-ability distribution. While this is a rough approximation to the real-life complexity of three-dimensional clusters of activation in brainspace, Turkeltaub’s results were surprisingly robust and introduceda new era of meta-analysis research in functional neuroimaging. Inan ALE meta-analysis, three-dimensional coordinates in stereotac-tic space are collected and filtered from a number of similar stud-ies. These coordinates are typically published relative to Talairachspace15 or Montreal Neurological Institute (MNI) space16 and mustbe spatially renormalized to a single template. This transformationhas generally been performed using the mni2tal transform.17 How-ever, a recent study has shown that the mni2tal transform is notoptimal and has recommended best-fit coordinate transforms for usewith different brain templates (ICBM-152 and MNI-305) and differ-ent software packages (FSL and SPM2).18 Once all the included fociin the meta-analysis refer to locations in a single stereotactic space,the ALE analysis begins.



26.2.1 The ALE Statistic

Each reported coordinate (focus) is modeled by a three-dimensionalGaussian distribution, defined by a user-specified FWHM (fullwidth at half maximum). If Xi denotes the event that the ith focus islocated in a given voxel, then the probability of Xi occurring at voxelx, y, z is

Pr(Xi) = exp(−d2i /2σ2)

(2π)3/2σ3 · �V (1)

where di is the Euclidean distance from the center of the voxel tothe ith focus, σ is the standard deviation of the Gaussian distri-bution, and Pr(Xi) satisfies 0 ≤ Pr(Xi) ≤ 1. The Gaussian prob-ability density is multiplied by �V = 8 mm3 (corresponding tovoxel dimension of 2mm × 2mm × 2 mm) in order to obtain theprobability estimate for the entire voxel volume, instead of its cen-tral point. If X denotes the event that any foci are located withina given voxel, then Pr(X) is defined as the union of all Pr(Xi),where Pr(Xi) is shown in Eq. (1). This value, Pr(X), is defined as theALE statistic and quantifies the likelihood of activation at a givenvoxel and task, as determined by the chosen set of studies from theliterature.

26.2.2 Permutation Tests

The ALE statistic is computed at every voxel in the brain. In orderto make a valid assessment of the significance of the results, a non-parametric procedure for testing the statistic images was developedusing a permutation test.19 To test the null hypothesis that the fociare uniformly spread throughout the brain, x random foci are gen-erated, where x equals the number of foci included in the ALEmeta-analysis. The corresponding ALE values for these random fociare computed. This process of randomization and computation ofrelabeled statistics is repeated 1 000–10 000 times, depending on thedesired precision of the test. The set of ALE values calculated from



the random foci forms the null distribution of the test statistic. Awhole-brain histogram is computed in which the null hypothesis ofuniformly distributed foci is rejected for voxels with an ALE valuegreater than the critical threshold. The critical threshold is definedas the 100(1 − α)th percentile of the permutation distribution, whereα refers to the desired level of significance.

26.2.3 Modifications to the ALE Approach

When ALE was introduced in 2002, a discussion of its limita-tions and areas in need of further development were provided.13

In response to this discussion, two areas of interest were subse-quently developed and tested.20 First, the permutation test proposedby Turkeltaub et al. was improved in order to more accuratelyderive null distributions for the ALE statistic using a correctionfor the multiple comparisons problem that controls for the falsediscovery rate.21,22 Second, a reliable method testing for the dif-ferences between two ALE meta-analyses was established. Thesemodifications to the ALE method are currently distributed withan image-based graphical user interface as part of the BrainMapdatabase project (http://brainmap.org). BrainMap is a free, commu-nity database of published functional neuroimaging results in theform of Talairach or MNI coordinates,23,24 and is committed to con-tinued support and development of advanced meta-analysis tech-niques, including ALE.

26.3 ALE META-ANALYSES OF HUMAN COGNITIONAND PERCEPTION

In May 2005, as a result of a virtual workshop on meta-analysistechniques,25 the journal, Human Brain Mapping, published the“Special Issue on Meta-Analysis in Functional Brain Imaging.”This issue included three methodology articles on ALE and theanalysis of meta-analysis networks20,26,27 and twelve ALE meta-analyses of human cognition and perception. Specifically, nine ALEmeta-analyses were presented on various cognitive tasks such as



the Stroop task,28 switching tasks,29 the Wisconsin Card-Sortingtask,30 the n-back working memory task in healthy subjects31 andschizophrenic subjects,32 object naming,33 phonological processingof Chinese characters,34 reading in Western and Eastern languages,35

and fluent vs stuttered speech production.36 In addition, three meta-analyses were published in the special issue on perceptual processes,including audition,37 pain perception,38 and vision.39 Presentedbelow are the highlights of the meta-analysis of the Stroop task.

26.3.1 Meta-Analysis of Stroop Interference Studies

In the Stroop task, subjects view color names presented in varyingink colors and are asked to name the color of the ink, while ignoringthe word.40 In the congruent condition, the color names match theirdisplayed ink color. In the incongruent condition, the words are pre-sented in non-matching ink colors (e.g. “blue” presented in red ink).The Stroop task is widely used to study inhibition and attentionalcontrol since correct performance in color naming often competeswith the relatively automatic tendency to perform word reading.

An ALE meta-analysis of all published neuroimaging studiesinvestigating the Stroop effect was performed to identify the regionsof concordance across the published set of Stroop papers in order tomore fully understand the detection of conflict and response selec-tion in the human brain.28 To reach this objective, a comprehensiveliterature search was carried out using Medline to determine thefMRI and PET Stroop studies that published Talairach or MNI coor-dinates of activation locations. From this set of studies, the includedcontrasts (Incongruent — Control) were filtered to eliminate non-standard task variations (counting Stroop, emotional Stroop), andonly include group activation data from normal subjects. This filter-ing isolated 19 Stroop studies (13 fMRI and 6 PET) with 19 contrasts,containing a total of 205 foci. A plot of these foci is presented on astandard glass brain in Fig. 1(A).

This group of Stroop coordinates was then segregated byresponse modality. The studies were parsed into two differentgroups based on use of a button press response (manual Stroop;



Fig. 1(A). Selected contrasts from the Stroop literature yielded a total of 205 foci,which are viewed in Talairach space in the BrainMap database java-based appli-cation Search & View. In this image, each color identifies a paper within the Brain-Map environment and the number displayed along with each focus refers to theexperiment within the corresponding paper; the circles can be changed to differentsymbols for identification purposes. Pooling the results of 19 experiments onto asingle brain resulted in a diffuse pattern of activation across all lobes, with someclustering visually evident in the frontal lobes.

six studies) or a covert or overt speech response (verbal Stroop;thirteen studies). Three different ALE maps were computed for allStroop studies, Stroop studies that required an overt or covert ver-bal response, and Stroop studies that required a manual response[Fig. 1(B)].

The ALE meta-analysis of all Stroop studies revealed high ALEvalues in the limbic, frontal, and parietal lobes. The verbal Stroopmap revealed regions of high ALE values in the left inferior frontalgyrus (IFG) near BA 44 and bilateral insula, two regions com-monly involved in articulation. In contrast, the manual Stroop maprevealed a parietal involvement more extensive than seen in the ver-bal Stroop and an absence of concordance in the speech production



Fig. 1(B). ALE meta-analyses of the Stroop task were performed on renormalizedTalairach coordinates from all studies, from studies that utilized a verbal speechresponse, and from studies that utilized a manual button press response.ALE valueswere computed at each voxel in the brain using a FWHM of 12 mm. Statisticalsignificance was determined using a permutation test of randomly generated focifor 5000 permutations, corrected for multiple comparisons using the false discoveryrate.20,22 Thresholded ALE maps are viewed at a significance level of p < 0.05. Onthe right, the ALE map of the pooled Stroop foci is viewed in axial slices. On the left,the ALE map of verbal (red) and manual (blue) Stroop foci is viewed as a compositeimage (overlap = yellow) on a 3D brain surface.

areas observed in the verbal Stroop (BA44 and insula). Clearly, whilethe Stroop task is essentially a verbal task and it is reasonable toassume that some form of covert vocalization occurs during the man-ual Stroop, it can be seen in Fig. 1(B) that the two response modalitiesdisplay different activation patterns due to a stronger emphasis onvocalization and articulation in the verbal as opposed to manualStroop task. When the manual and verbal Stroop ALE results areviewed in a composite image, regions of overlap were observed inthe anterior cingulate, left inferior parietal lobule, and bilateral infe-rior frontal junction. The inferior frontal junction is located betweenthe precentral gyrus and the inferior frontal gyrus, and is known



to be involved during tasks of cognitive control.29,41 Based on theseresults, these regions have been isolated as major components of thenetwork for response conflict resolution in the Stroop task.

26.4 ANALYSIS OF META-ANALYSIS NETWORKS(RDNA AND FSNA)

As described above, ALE can be used to identify the networkinvolved in a given paradigm or behavioral domain; however, theALE methodology does not include a technique to evaluate the rela-tionships between nodes in the identified network. In response tothis, Neumann et al.26 published a method of investigating inter-regional connectivity based on replicator dynamics, a strategy basedon the dynamics of competitive growth that is well established insocial and biological sciences. Neumann et al. proposed that thisreplicator dynamics network analysis (RDNA) be used to isolate cor-tical networks that are activated most often together across multiplestudies. The replicator dynamics approach can be used to identifysubordinate networks within a larger network (e.g. to separate aperceptual subsystem from a motor subsystem in a cued-responseparadigm). This function is based on analysis of a co-occurrencematrix, in which each element indicates how often a given pair ofactivation maxima is found to be coactivated in a given study.26,42

Co-occurrence networks determined by ALE meta-analysis areassumed to be the summation of subnets. The fractional contributionof each subnet affects co-occurrence of the whole network.

In an RDNAanalysis, theALE method is first used to identify theregional nodes of activation from individual coordinates in multiplestudies. Next, the occurrence of each of these nodes in the includedstudies is recorded. Third, the co-occurrence matrix is computedfor the activation nodes. Last, the replicator process is applied toidentify the dominant network.

Neumann et al.26 presented an RDNA analysis of the Strooptask to illustrate their new method. In this example, the ALEmeta-analysis identified 15 activation nodes. The replicator process



isolated five of these nodes to be the dominant network, includ-ing the presupplementary motor area, left inferior frontal sulcusextending onto the middle frontal gyrus, bilateral anterior cingulate,and the left inferior frontal junction. The replicator process assignedthe highest connectivity weight to the right anterior cingulate node,which was the second largest node and showed the second highestnumber of co-occurrences. The highest number of co-occurrenceswas found for the inferior frontal sulcus, which was assigned thesecond highest connectivity weight, but was the smallest node inthe network. These results demonstrate that connectivity weight isdetermined by the relationship between different activation nodesand is a function of co-occurrence, the extent of the ALE clusters,and the magnitude of the ALE scores.

The network analysis technique based on replicator dynamics(RDNA) presented by Neumann et al.26 introduced the first appli-cation of meta-analysis data to network analysis Lancaster et al.27

examined both RDNA and a similar method known as fractionalsimilarity network analysis (FSNA). Whereas the RDNA methodused by Neumann et al. was applied to determine the dominantsubset of nodes, the FSNA method determines the complete subsetsof the data using binary pattern matching. Lancaster et al. chose tostudy both RDNA and FSNA on the pooled Stroop data set (19 stud-ies with 205 foci) from the meta-analysis performed by Laird et al.28

This dataset was similar to that used by Neumann et al., but includedsix additional studies. This pooled Stroop dataset was first analyzedusing ALE, and yielded 13 nodes (p < 0.01). RDNA on this data setreported a dominant network of only two nodes (anterior cingulateand left inferior frontal junction), which contrasted from the five-node network identified as dominant in Ref. 26. However, modifyingRDNAto return multiple maximal cliques, resulted in finding a five-node maximal clique consistent with the five-node network reportedby Neumann et al.26 Applying FSNA to the same Stroop data setrevealed several important segregations of the data. The two cingu-late clusters were parsed into different subnets. This is consistentwith the previous determination of somatotopy within the cingu-late motor area;28 however, in the case of FSNAthis parcellation into



different subnets was done using the pooled Stroop data, and not byperforming separateALE analyses based on response modality. BothRDNA26 and FSNA27 have proved to be interesting extensions of theALE meta-analysis method, and it is hoped that further investiga-tion of these techniques will yield critical information concerningmeta-analysis networks of cognition and perception.


The utility of the ALE meta-analysis method is well established,and ALE has proved capable in illustrating differences in task stim-ulus or response modalities,28,31,38 baseline conditions,33 and nor-mal vs diseased subject groups.32,36,43 However, the true potential ofconnectivity analysis of meta-analysis networks remains yet to bediscovered. While establishing these function-location relationshipsand uncovering areas of functional dissociation within the cortex hasbeen a primary focus of research, more investigators are progressingfrom simple identification of network nodes towards studying theinteractions between brain regions Neumann et al.26 and Lancasteret al.27 provided a path forward in this direction using their respec-tive methods of replicator dynamics network analysis (RDNA) andfractional similarity network analysis (FSNA). Future work in thisarea will certainly involve probing network connection from meta-analysis data, perhaps using this information to inform networks forstructural equation modeling44,45 or dynamic causal modeling.46,47

References

1. Conn HO, Interpretation of data from multiple trials: A critical review,J Intern Med 241: 177–183, 1997.

2. Mosteller F, Colditz GA, Understanding research synthesis (meta-analysis), Annu Rev Public Health 17: 1–23, 1996.

3. Petitti DB, Meta-analysis and endocrinology, Endocrinol Metab ClinNorth Am 26: 31–44, 1997.

4. Fox PT, Lancaster JL, Mapping context and content: The BrainMapmodel, Nature Rev Neurosci 3: 319–321, 2002.



5. Fox PT, Parson LM, Lancaster JL, Beyond the single study: Function/location meta-analysis in cognitive neuroimaging, Curr Opin Neurbiol8: 178–187, 1998.

6. Barch DM, Braver TS, Akbudak E, Conturo T, et al., Anterior cingulatecortex and response conflict: Effects of response modality and process-ing domain, Cereb Cortex 11: 837–848, 2001.

7. Becker JT, MacAndrew DK, Fiez JA, Acomment of the functional local-ization of the phonological storage subsystem of working memory,Brain Cogn 41: 27–38, 1999.

8. Buckner RL, Petersen SE, What does neuroimaging tell us about therole of prefrontal cortex in memory retrieval? Sem Neurosci 8: 47–55,1996.

9. Bush G, Luu P, Posner MI, Cognitive and emotional influences in ante-rior cingulate cortex, Trends Cogn Sci 4: 215–222, 2000.

10. Fiez JA, Raife EA, Balota DA, Schwarz JP, et al., A positron emissiontomography study of the short-term maintenance of verbal informa-tion, J Neurosci 16: 808–822, 1996.

11. Owen AM, The functional organization of working memory processeswithin human lateral frontal cortex: The contribution of functional neu-roimaging, Eur J Neurosci 9: 1329–1339, 1997.

12. Picard N, Strick PL, Motor areas of the medial wall: A reviewof their location and functional activation, Cereb Cortex 6: 342–353,1996.

13. Turkeltaub PE, Eden GF, Jones KM, Zeffiro TA, Meta-analysis of thefunctional neuroanatomy of single-word reading: Method and valida-tion, Neuroimage 16: 765–780, 2002.

14. Chein JM, Fissell K, Jacobs S, Fiez JA, Functional heterogeneity withinBroca’s area during verbal working memory, Physiol Behav 77: 635–639,2002.

15. Talairach J, Tournoux P, Coplanar stereotaxic atlas of the human brain,Thieme, New York, 1988.

16. Collins DL, Neelin P, Peters TM, Evans AC, Automatic 3D intersubjectregistration of MR volumetric data in standardized Talairach space,J Comput Assist Tomogr 18: 192–205, 1994.

17. Brett M, (1999), The MNI brain and the Talairach atlas, CambridgeImagers. http://www.mrc-cbu.cam.ac.uk/Imaging/mnispace.html.

18. Lancaster JL, Tordesillas-Gutierrez D, Martinez M, Salinas F, et al., Biasbetween MNI and Talairach coordinates analyzed using the ICBM-152brain template, Hum Brain Mapp, 2007.

19. Good P, Permutation tests: A practical guide to resampling methodsfor testing hypotheses, Springer-Verlag, New York, 1994.



20. Laird AR, Fox PM, Price CJ, Glahn DC, et al., ALE meta-analysis: Con-trolling the false discovery rate and performing statistical contrasts,Hum Brain Mapp 25: 155–164, 2005.

21. Benjamini Y, Hochberg Y, Controlling the false discovery rate: A prac-tical and powerful approach to multiple testing, J R Stat Soc Ser B 57:289–300, 1995.

22. Genovese CR, Lazar NA, Nichols TE, Thresholding of statistical mapsin functional neuroimaging using the false discovery rate, Neuroimage15: 870–878, 2002.

23. Laird AR, Lancaster JL, Fox PT, BrainMap: The social evolution of ahuman brain mapping database, Neuroinformatics 3: 65–78, 2005.

24. Fox PT, Laird AR, Fox SP, Fox PM, et al., Lancaster JL, Experimentaldesign taxonomy of the BrainMap database: Description and valida-tion, Hum Brain Mapp 25: 185–198, 2005.

25. Fox PT, Laird AR, Lancaster JL, Coordinate-based voxel-wise meta-analysis: Dividends of spatial normalization.Areport of a virtual work-shop, Hum Brain Mapp 25: 1–5, 2005.

26. Neumann J, Lohmann G, Derrfuss J, von Cramon DY, Meta-analysis offunctional imaging data using replicator dynamics, Hum Brain Mapp25: 165–173, 2005.

27. Lancaster JL, Laird AR, Fox PM, Glahn DC, et al., Automated analysisof meta-analysis networks, Hum Brain Mapp 25: 174–184, 2005.

28. LairdAR, McMillan KM, Lancaster JL, Kochunov P, et al.,Acomparisonof label-based review and activation likelihood estimation in the Strooptask, Hum Brain Mapp 25: 6–21, 2005.

29. Derrfuss J, Brass M, Neumann J, von Cramon DY, Involvement of theinferior frontal junction in cognitive control: Meta-analyses of switch-ing and Stroop studies, Hum Brain Mapp 25: 22–34, 2005.

30. Buchsbaum BR, Greer S, Chang W-L, Berman KF, Meta-analysis of neu-roimaging studies of the Wisconsin card sorting task and componentprocesses, Hum Brain Mapp 25: 35–45, 2005.

31. Owen AM, McMillan KM, Laird AR, Bullmore E, The n-back work-ing memory paradigm: A meta-analysis of normative functional neu-roimaging studies, Hum Brain Mapp 25: 46–59, 2005.

32. Glahn DC, Ragland JD, Abramoff A, Barrett J, et al., Beyondhypofrontality: A quantitative meta-analysis of functional neuroimag-ing studies of working memory in schizophrenia, Hum Brain Mapp 25:60–69, 2005.

33. Price CJ, Moore CJ, Morton C, Laird AR, Meta-analysis of picture nam-ing: The effect of baseline, Hum Brain Mapp 25: 70–82, 2005.



34. Tan L-H, Laird AR, Fox PT, Neuroanatomical correlates of phonolog-ical processing of Chinese characters and alphabetic words: A meta-analysis, Hum Brain Mapp 25: 83–91, 2005.

35. Bolger DJ, Perfetti CA, Schneider W, A cross-cultural effect revisited:Universal structures plus writing system variations, Hum Brain Mapp25: 92–104, 2005.

36. Brown S, Ingham RJ, Ingham JC, Laird AR, et al., Stuttered and fluentspeech production: An ALE meta-analysis of functional neuroimagingstudies, Hum Brain Mapp 25: 105–117, 2005.

37. PetacchiA, LairdAR, Bower JM, The cerebellum and auditory function.An ALE meta-analysis of functional neuroimaging studies, Hum BrainMapp 25: 118–128, 2005.

38. Farrell MJ, Laird AR, Egan GF, Brain activity associated with painfullyhot stimuli applied to the upper limb:Ameta-analysis, Hum Brain Mapp25: 129–139, 2005.

39. Grosbras M-H, Laird AR, Paus T, Cortical regions involved in gazeproduction, attention shifts and gaze perception, Hum Brain Mapp 25:140–154, 2005.

40. Stroop JR, Studies of interference in serial verbal reactions, J ExpPsychol 18: 643–662, 1935.

41. Derrfuss J, Brass M, von Cramon DY, Cognitive control in the posteriorfrontolateral cortex: Evidence from common activations in task coor-dination, interference control, and working memory, Neuroimage 23:604–612, 2004.

42. Bomze IR, Pelillo M, Stix V,Approximating the maximum weight cliqueusing replicator dynamics, IEEE Trans Neural Netw 11: 1228–1241, 2000.

43. Fitzgerald PB, Oxley TJ, LairdAR, Kulkarni J, et al.,An analysis of func-tional neuroimaging studies of dorsolateral prefrontal cortical activityin depression, Psych Res: Neuroimaging 148: 33–45, 2006.

44. McIntosh AR, Gonzalez-Lima F, Structural equation modeling and itsapplication to network analysis in functional brain imaging, Hum BrainMapp 2: 2–22, 1994.

45. McIntoshAR, Gonzalez-Lima F, Structural modeling of functional neu-tral pathways mapped with 2-deoxyglucose: Effects of acoustic startlehabituation on the auditory system, Brain Res 547: 295–302, 1991.

46. Penny WD, Stephan KE, Mechelli A, Friston KJ, Modeling functionalintegration: A comparison of structural equation and dynamic causalmodels, Neuroimage 23: S264–S274, 2004.

47. Friston KJ, Harrison L, Penny W, Dynamic causal modeling, NeuroImage 19: 1273–1302, 2003.


CHAPTER 27

Dynamic Human Brain Mappingand Analysis: From Statistical

Atlases to Patient-SpecificDiagnosis and Analysis

Christos Davatzikos

This chapter describes methodologies for measuring spatiotemporal pat-terns of brain structure in brain images, a problem that arises very oftenin monitoring disease progression and treatment responses, from a seriesof scans. A 4-dimensional shape transformation is used to map images toa stereotactic coordinate space, in order to standardize the coordinates ofanatomical structures across different individuals and remove interindi-vidual variability. A statistical atlas is then constructed from a set of datathat has been mapped to the same stereotactic space, and reflects the vari-ation of brain structure across individuals of the population used to con-struct the statistical atlas; the transformation that was used to map theimages to the stereotactic space is also measured, as it often constitutes akey morphometric measurement reflecting morphological characteristicsof the respective individual relative to a standardized template brain thatresides in the stereotactic space. Individual patient scans are then com-pared against one or more statistical atlases, in order to diagnose diseaseor predict likelihood of disease progression. This statistical comparison istypically performed via pattern classification systems, which are trainedto recognize spatiotemporal patterns of brain structure that are highlycharacteristic of a disease of interest.

677


678 Christos Davatzikos

27.1 INTRODUCTION: THE CONCEPTOF STATISTICAL ATLASES

The explosive expansion of imaging in the 1990s and 2000s hasopened up tremendous opportunities for studying the structure andphysiology of the human brain, as well as the ways in which struc-ture and function are affected by a variety of diseases and disorders.Although earlier studies typically involved a few dozens of imageseach, many current studies involve hundreds, and thousands of par-ticipants, often with multiple scans each. Large databases are there-fore constructed rapidly, incorporating rich information of brainstructure and function in normal and diseased states. The analysis ofsuch a wealth of information is becoming increasingly difficult, with-out the availability of advanced statistical image analysis methods.

The conventional type of analysis of brain images has reliedon manual tracings of regions of interest (ROI).1−18 For example,the volumes of a limited number of brain regions can be measuredby delineating the boundaries of these regions and measuring vol-umes. Overlaying structural and functional images via registrationtechniques further allows us to obtain measurements of functionalactivity within these ROIs. These methods typically require thatthe reliability and repeatability of manual tracings across differentraters, but also within the same rater at different times, be establishedfirst. However, methods based on manually defined ROIs are lim-ited in many ways. First, they rely on the need for a priori knowledgeof the regions that are affected by a disease, so that respective ROIscan be defined, and therefore they might fail to discover new find-ings. Although a good hypothesis might be available in the begin-ning of a morphometric study, one would typically want to discovernew knowledge that, by definition, is not part of the hypothesis. Asan example selected from the neuroimaging of dementia literature,although the role of hippocampal and entorhinal cortical atrophyin early prediction of Alzheimer’s Disease (AD) is widely accepted,relatively little is known about the potential involvement of otherbrain regions, which could help construct more sensitive methodsfor detection of and differentiation among different types dementia.The complete investigation of the role of all brain structures in a


Dynamic Human Brain Mapping and Analysis 679

disease and its diagnosis would be prohibitively labor-intensive foran adequately large sample size, if manual methods are employed.Moreover, inter- and intrarater reliability issues would become cru-cial limiting factors, particularly in longitudinal studies in whichit is extremely difficult to maintain intra- and interrater reliabilityover time. Second, the spatial specificity of ROI-based methods islimited by the size of the ROIs being measured, which is typicallyrather coarse. Aregion that might be affected by disease may be onlypart of a predefined ROI, or it might span two or more ROIs, whichinevitably washes out the results and reduces the statistical power ofthe measurement method. Alternative methods, such as stereology,are also limited in a similar way. Although in principle, one coulddefine the size of the ROIs measured to be as small as desired, inorder to increase spatial specificity, this would decrease rater reli-ability for measurement methods that are based on human raters.Finally, manual ROI tracing is severely limiting in many modernstudies, for which it is not unusual to include over a thousand scansper study.

In order to address the limitations of ROI-based approaches,image analysis methods based on shape analysis have been studiedin the literature during the past 15 years.19−34 One very promisingapproach for morphometric analysis is based on shape transforma-tions, and the associated methods are often called unbiased, orhypothesis-free methods. A shape transformation is a spatial mapthat adapts an individual’s brain anatomy to that of another. Theresulting transformation measures the differences between the twoanatomies with very high spatial specificity, ultimately the speci-ficity allowed by the image voxel size. More generally, a template ofanatomy is first selected, which serves as a measurement unit. Theshape transformation that maps other brains to the template is deter-mined via some image analysis algorithm, and it is used as a meansof quantifying the individual anatomies. Interindividual compar-isons are then performed by applying standard statistical methods tothe respective shape transformations. Voxels that display significantgroup differences or longitudinal changes are grouped into regions.Therefore, there is no need to define ROIs in advance. Instead, the



Fig. 1. Using a shape transformation for morphometric measurements. (A) (Topleft)Atemplate of the cross-section of the corpus callosum, a brain structure connect-ing the two hemispheres. (Top middle and right) Two individual shapes. (Bottom)Respective color-coded maps of the determinant of the Jacobian of the shape trans-formation mapping the template to the two shapes. Contraction is colored greenand expansion is colored red. Voxel-wise comparison of these images reveals localshape differences of the respective shapes. (B) Seminal work by D’Arcy Thompsonin 1917 using shape transformations to make comparisons among species.

ROIs are determined retrospectively via the voxel-wise statisticalanalysis of the shape transformations. The concept of this approachis shown in Fig. 1(A), which is based on some of the second author’searlier work on the corpus callosum.21

Although this approach gained widespread attention only inthe past decade, it has its roots in the seminal work by D’ArcyThompson,35 who studied differences among species by measur-ing deformations of coordinate grids from images of one species toimages of another (see Fig. 1(B)). At that time, very limited manualdrawing methods were available to D’Arcy Thompson, somethingthat imposed limits on the spatial specificity of this approach. Theapproach was later adopted by the landmark-based morphometricsliterature19 and further extended by pattern theory36 and later workon diffeomorphisms,20 among several other investigators. One ofthe first applied studies was performed by our group by focusingon sex-differences in the corpus callosum.21,37

In addition to allowing the mophometric analysis of brainstructure, the availability of shape transformations from ananatomical brain template to various brain images provides anadditional important benefit. Through the inverse of such shapetransformations, information defined on the domains of individualbrain images can now be mapped onto the domain of the template.



Thus the template acts as a stereotactic space where structural, func-tional, and pathological information from large databases of brainimages can be collected and used for the construction of statisticalatlases of normal and pathological variability. Through the use ofmultivariate statistical analysis methods, it is also possible to dis-cover correlates between all variables stored in the atlas and to linkthis information to normal, pathological and aging processes.

The process of mapping image data into a stereotactic space iscalled spatial normalization. It leads to the construction of a statisti-cal atlas, i.e. an atlas that reflects the spatiotemporal characteristicsa certain type of brain images over certain populations. For exam-ple, a statistical atlas of the typical regional distribution of gray andwhite matter (GM, WM) in the brain can be constructed by spa-tially normalizing a number of brain images of healthy individualsinto the stereotaxic space, and measuring the average and standarddeviation of the amount of GM and WM in each brain region. Thisatlas can also become more specific, for example to the age, sex, andother characteristics of the underlying population. A scan from anyindividual can then be compared against the atlas, after it under-goes the same spatial normalization procedure, in order to deter-mine whether the individual is or is not within the normal range ofvariation, and if he/she is not, how and where he/she differs.

27.2 SPATIAL NORMALIZATION AND THECONSTRUCTION OF A STATISTICAL ATLAS

As described above, of fundamental importance in the process ofconstructing a statistical atlas is that of finding a shape transfor-mation that maps one brain image to another, and specifically tothe template, and vice versa. Many methods have been proposedin the literature for obtaining shape transformations between atemplate shape and anatomical shapes in images. This is usuallyachieved through a method for deformable image registration. Thegoal of the deformable registration between two brain images isto find the transformation that maps every point in the first image



to its matching point in the second image. Matching points shouldcorrespond to the same anatomical feature or structure. The shapetransformation found is usually required to be a diffeomorphism —a differentiable and invertible mapping between the domains ofthe two images. In mathematical terms, let D1 be the domain ofthe first image, and let D2 be the domain of the second image. Thesought shape transformation is a differentiable and invertible map-ping ϕ : D1 → D2 such that for every point x ∈ D1, the pointϕ(x) ∈ D2 corresponds to the same anatomical feature or structureas that of point x.

Finding shape transformations from 3D medical images requiresthe use of an automated or semi-automated method for findingthe deformation map ϕ. Algorithms based on maximizing the sim-ilarity between an image treated as template and other images inthe study have been widely used for solving this deformable reg-istration problem.20,29,38−44 These methods assume that if a shapetransformation renders two images similar, it implies anatomicalcorrespondence between the underlying anatomies. This is a rea-sonable assumption, but it can easily be violated in practice, sincetwo images can be made similar via shape transformations that donot respect the underlying anatomical correspondences. For exam-ple, one can simply flow gray matter into gray matter, white mat-ter into white matter, and CSF into CSF, thereby creating imagesthat look alike, since these three tissue types have similar inten-sity distributions throughout the brain, without the underlyingshape transformations reflecting true anatomical measures, since,for example, it could morph the precentral gyrus to the postcentralgyrus.

An important issue with intensity-based transformations is thatof inverse consistency. In particular, if we attempt to match Image1to Image2, then Image2 to Image1, we should get shape transfor-mations that are the inverses of each other. This condition is notnecessarily met in practice, especially by image similarity measures.Therefore, techniques that specifically impose inverse consistencyhave also been examined in the literature.38,45,46



Somewhat related to image intensity matching are methods opti-mizing information theoretic criteria, in order to find appropriateshape transformations. The main advantage of these methods overimage similarity methods is that they can potentially be used acrossdifferent imaging modalities, i.e. when tissue intensities are differentin two images to be matched. The most popular criterion of optimal-ity has been mutual information,40,47,48 which is maximized whenthe “predictability” of the warped image based on the template ismaximized, and which tends to occur when the different tissue typesin two images are well registered.

A different class of algorithms is based on some form of featurematching.22,42,49−54 A number of features, such as edges or curvesor surfaces, are typically extracted from the images via an imageanalysis algorithm, or simply drawn manually, and are then used todrive a 3D deformable registration method, which effectively inter-polates feature correspondence in the remainder of the brain. Relatedare medialness models,26 which use the medial axes of anatomi-cal shapes as features, instead of boundaries themselves. Feature-based methods pay more attention to the biological relevance of theshape matching procedure, since they only use anatomically dis-tinct features to find the transformation, whereas image matchingmethods seek transformations that produce images that look alike,with little warranty that the implied correspondences have anatom-ical meaning. However, the latter approaches take advantage ofthe full dataset, and not only of a relatively sparse subset offeatures.

A method that has been previously developed by our groupattempts to bridge between these two extremes by developingattribute vectors that aim at making each voxel a feature,46,55,56 andit was called Hierarchical Attribute Matching Mechanism for Elas-tic Registration (HAMMER). HAMMER is a hierarchical warpingmechanism that has two key characteristics. First, it places emphasison determining anatomical correspondences, which in turn drive the3D warping procedure. In particular, we have used feature extrac-tion methods whose goal is to determine a number of parameters



form the images, which can characterize at least some key anatomi-cal features as distinctively as possible. In Ref. 46, we used geomet-ric moment invariants (GMIs) as a means for achieving this goal.GMIs are quantities that are constructed from images that are firstsegmented into GM, WM and CSF, or any other set of tissues of inter-est. They are determined from the image content around each voxel,and they quantify the anatomy in the vicinity of that voxel. GMIs ofdifferent tissues and different orders are collected into an attributevector, for each voxel in an image. Ideally, we would like for eachvoxel to have a distinctive attribute vector; of course, this is not pos-sible in reality. Figure 2 shows a color-coded image of the degree ofsimilarity between the GMI-based attribute vector of a point on theanterior horn of the left ventricle and the attribute vectors of everyother point in the image. The GMI attribute vector of this point,as well as of many other points in the brain, is reasonably distinc-tive, as Fig. 2 shows. HAMMER was constructed to solve an opti-mization problem that involves finding a shape transformation thatmaximizes the similarity of respective attribute vectors, while beingsmoothed by a standard Laplacian regularization term (a detaileddescription can be found in Ref. 46). We have recently explored moredistinctive attribute vectors, aiming at constructing even more dis-tinctive morphological signatures for every image voxel. Toward

Fig. 2. The point marked by a cross has a relatively distinctive GMI-based attributevector. The color-coded image on the right shows the degree of similarity betweenthe attribute vector of the marked (by crosses) point and the attribute vector of everyother point in the brain. 1 is maximum similarity and 0 is minimum similarity.



this goal, we used wavelet-based hierarchical image descriptions oflarge neighborhoods centered on each image voxel.57,58

A second key characteristic of HAMMER addresses a funda-mental problem encountered in high-dimensional image matching.In particular, the cost function being optimized typically has manylocal minima, which trap an iterative optimization procedure intosolutions that correspond to poor matches between the templateand the individual. This is due, in part, to the ambiguity in findingpoint correspondences. For example, if many candidate points in anindividual image have similar attribute vectors to that of a partic-ular template voxel, then this introduces an ambiguity that resultsin local minima of the corresponding energy function. In contrast,consider the situation in which there are a few anchor points forwhich correspondence (the value of the shape transformation) canbe determined rather unambiguously, perhaps because each anchorpoint’s attribute vector is very different for all but its correspondinganchor point. In that case, the shape transformation on all other (non-anchor) points could be determined via some sort of interpolationfrom the anchor points. This problem would not have local minima.Of course, the cost function being minimized would be only a lower-dimensional approximation, compared to a cost function involvingevery single voxel in the image. HAMMER is based on this fact, andforms successive lower-dimensional cost functions, based initiallyonly on key anchor points, and gradually involving more and morepoints. More points are considered as a better estimate of the shapetransformation is obtained, and potential local minima are avoided.Anchor points are defined based on how distinctive their attributevectors are.

A third feature of HAMMER is that it is inverse-consistent, interms of the driving correspondences. This means that if the indi-vidual is deformed to the template, instead of the converse, the map-ping between any two driving points during this procedure wouldbe identical. This feature is a computationally fast approximation tothe problem of finding fully 3D inverse consistent shape transforma-tions originally proposed by Christensen.59 Representative resultselucidating HAMMER’s performance are shown in Figs. 3 and 4.



Fig. 3. Results using HAMMER warping algorithm. (A) 4 representative sectionsfrom MR images of the BLSA database (B) Representative sections from the imageformed by averaging 150 images warped by HAMMER to match the templateshown in (C). (D1–D4) 3D renderings of a representative case, its warped con-figuration using HAMMER, the template, and the average of 150 warped images,respectively. The anatomical detail seen in (B) and (D4) is indicative of the registra-tion accuracy. The red crosses in (D3–D4) are identically placed, in order to allowvisualization of point correspondences.

Fig. 4. Representative example of automated definition of regions of interest, bywarping a pre-labeled atlas (left) to an individual’s MR images (the warped atlasis shown on the right as a color-coding of a volume rendering of the target brain).This automated ROI definition makes it possible to apply the method to studieswith large sample sizes in a streamlined way.



27.3 STATISTICAL ATLASES OF THE SPATIALDISTRIBUTION OF BRAIN TISSUE: MEASURINGPATTERNS OF BRAIN ATROPHY

Morphometric analysis based solely on the shape transformationthat maps a template to an individual anatomy is affected by errorsin determining the shape transformation. If the warping mecha-nism used by a particular method is not able to perfectly matchthe anatomy of each individual with the anatomy of the template,then subtle structural characteristics are lost and never recoveredin subsequent stages of the analysis. These errors can be signifi-cant obstacles in studying subtle differences between two or moreindividuals or time points. In order to address this problem, wedeveloped a mass-preserving framework for shape transformations,which is relatively more robust, for reasons that are explained below.Our approach is shown schematically in Fig. 5.

In the mass-preserving framework of RAVENS60−62 (RegionalAnalysis of Volumes Examined in Normalized Space), if the shapetransformation applies an expansion to a structure, the density of thestructure decreases accordingly to warranty that the total amount

Fig. 5. Schematic representation of the mass-preserving framework of theRAVENS analysis. A shape transformation (A) that causes contraction of thestructure as it maps it to a stereotaxic space increases the tissue density withinthe structure, so that the total amount of tissue is preserved. The transformation(B) is different (e.g. it might correspond to a result with greater error in the shapetransformation). However, the total amount of tissue is preserved under both trans-formations, (A) and (B). For example, integrating the tissue density within the out-lined regions gives exactly the same result, and equal to the area of the outlinedregion in the original shape. This property is lacking in direct measurements of theshape transformation.



of tissue is preserved. Conversely, tissue density increases duringcontraction. Consequently, tissue density in the templates (stereo-taxic) space is directly proportional to the volume of the respectivestructure in its original form. Therefore, regional volumetric mea-surements and comparisons are performed via measurements andcomparisons of the respective RAVENS density maps. One RAVENSmap is generated for each tissue of interest, typically GM, WM andCSF. In Ref. 60, we validated RAVENS on 24 MR images having syn-thetic atrophy. Specifically, we randomly selected standard SPGRimages of 12 BLSA subjects, and we outlined the precentral andsuperior temporal gyri in all of them. We then introduced a uni-form 30% volumetric contraction in these two outlined gyri, therebygenerating another 12 images with synthesized atrophy in them.Figure 6 (top) shows cross-sections of a typical image before and

Fig. 6. (Top left) Representative slices from the level of the precentral gyrus, withsimulated atrophy indicated by the arrows (left is before and right is after uni-form 30% atrophy within the gyrus was applied). (Bottom left) Regions detectedby the RAVENS analysis, overlaid on the average WM RAVENS maps of the 24individuals. The two detected regions were exactly where atrophy was simulated(Right).



after contraction of the precentral gyrus (segmented images areshown).

We then used RAVENS to determine the 24 respective braintissue density maps, and applied a point-wise statistical analy-sis to them via paired t-tests. Regions of statistically significantdifferences between the two sets of 12 are shown in Fig. 6 (bot-tom), overlaid on the average WM RAVENS map of the 24 sub-jects (used for reference). The highlighting of the two regions inwhich atrophy was introduced shows the spatial specificity ofRAVENS. In Ref. 60, we also compared the sensitivity of RAVENSwith the widely used VBM approach of the SPM package,63 and wefound that RAVENS performed significantly better in this validationstudy.

27.4 MEASURING DYNAMIC PATTERNSOF BRAIN ATROPHY

With a growing interest in longitudinal studies, which are impor-tant in studying development, normal, aging, early markers ofAlzheimer’s disease, and response to various treatments, amongstothers, securing longitudinal stability of the measurements is ofparamount importance. However, in a longitudinal morphometricstudy, we would typically measure the shape transformation dur-ing each time point, and then examine longitudinal changes in theshape transformation. This approach is valid in theory, but limitedin practice. This is because small error measurements are dramati-cally amplified when we calculate temporal differences. Althoughtemporal smoothing can be applied retrospectively to shape mea-surements, it is far better if temporal smoothing is actually incorpo-rated into the procedure for finding the shape transformation, whenthe image information is available to the algorithm, rather than ret-rospectively adjusting a noisy shape transformation. The issue oflongitudinal measurement robustness is particularly important inmeasuring the progression of a normal older adult into mild cog-nitive impairment, which makes it important to have the ability todetect subtle morphological changes well before severe cognitive



decline appears. To further illustrate the difficulties that the cur-rent 3D method is facing, in Fig. 7 we have shown some represen-tative longitudinal volumetric measurements from single subjectsas well as from averages obtained from 90 older individuals oversix years.

2 4 6 82000

2200

2400

2600

2800Individual Volume

"hippocampal formation" right

1 2 3 4 5 62000

2200

2400

2600

2800Average Volume

"hippocampal formation" right

2 4 6 81400

1500

1600

1700

1800

1900

2000Individual Volume (AE)

"parahippocampal gyrus" left

1 2 3 4 5 61400

1500

1600

1700

1800

1900

2000Average Volume

"parahippocampal gyrus" left

(AA)

Fig. 7. Example illustrating the problems faced when applying a 3D atlas warp-ing method independently to each time-point in a longitudinal study. (Left) Plots ofvolumetric measurements from two representative BLSA participants and 2 struc-tures, using 3D HAMMER (right hippocampal formation and left parahippocampalgyrus). (Right) Analogous plots showing average volumes of these two structures,obtained by averaging the volumetric measurements of 90 BLSA participants foreach of six years. Considerable variation is apparent. For example, the standarddeviation around the baseline is about 5% for the left hippocampus of subject AD.Although a difference of 5% cannot be appreciated by visual inspection, it canadversely affect the accuracy of longitudinal measurements.As should be expected,variation of the average hippocampal volume is much lower (less than 1%), becauseof the averaging over 90 individuals.



In order to address this issue and be able to obtain longitudinallystable measurements, we have developed an approach to finding theshape transformation in 4D, with the 4-th dimension being time.64

The formulation is readily reduced to a 3D problem, if only cross-sectional data is available. We should note that a step towards thisproposed direction was proposed in Ref. 33, in which the image atone time-point was used as the template for shape reconstruction inanother frame. However, that approach still measures longitudinaldifferences independently for different time-points, and therefore itdoes not apply temporal smoothing other than by using the sameanatomy of a different time-point as the template.

The 4D warping approach of Ref. 64 simultaneously estab-lishes longitudinal correspondences in the individual as well ascorrespondences between the template and the individual. This isdifferent from the 3D warping methods, which aim at establishingonly the inter-subject correspondences between the template andthe individual in a single time-point. Specifically, 4D-HAMMERuses a fully automatic 4-dimensional atlas matching method thatconstrains the smoothness in both spatial and temporal domainsduring the hierarchical atlas matching procedure, thereby producingsmooth and accurate estimations of longitudinal changes. Mostimportantly, morphological features and matches guiding this defor-mation process are determined via 4D image analysis, whichsignificantly reduces noise and improves robustness in detect-ing anatomical correspondence. Put simply, image features thatare consistently recognized in all time-points guide the warpingprocedure, whereas spurious features, such as noisy edges, appearinconsistently at different time-points and are eliminated. In Ref. 64,this 4D approach was found to yield both stable and accurate longi-tudinal measurements, compared to 3D warping.

Figure 8 shows an example of measuring a dynamic patternof brain atrophy in 116 healthy elderly individuals, all participantsto the Baltimore Longitudinal Study of Aging.65,66 Red and yellowareas indicate brain regions that displayed the most significant lon-gitudinal atrophy, over a four year period.



Fig. 8. Regions displaying significant longitudinal grey matter atrophy over a fouryear period. Estimates of longitudinal atrophy were determined by segmentationinto GM, WM and CSF, then applying the mass-preserving RAVENS methodologydescribed in the text, which deforms each individual’s brain into alignment witha template brain, while preserving tissue mass by converting it to density. Voxel-based analysis of the resultant tissue density maps is equivalent to voxel-basedvolumetric analysis and therefore of atrophy quantification.

27.5 FROM THE STATISTICAL ATLAS TO THEINDIVIDUAL DIAGNOSIS

The voxel-based morphometric analysis methods described abovehave enjoyed wide-spread acceptance in the past decade, since theydo not rely on any a priori hypotheses regarding the structures tobe measured, but rather apply unbiased analyses of the entire set ofdata on a voxel-by-voxel basis. Accordingly, they highlight regionsin which there is statistically significant difference between twogroups, for example. However, the existence of significant differ-ences in certain brain regions does not necessarily imply that vol-umetric measurements of those regions are sufficient to diagnosedisease. For example, say that normal control older subjects differfrom patients developing mild cognitive impairment (MCI) in thevolumes of the hippocampus and the entorhinal cortex (ERC), butvolumes of normal and MCI individuals are highly overlapping.In this case, diagnosis based solely on volumes of the hippocam-pus and the ERC could be unreliable. In the recent years, inter-est in integrating voxel-wise mophometric measurements into toolsthat can be used for diagnosis has gained interest.27,67,68 One of themotivating factors behind these developments is the complex andspatiotemporally distributed nature of the changes that most dis-eases cause, particularly in the brain. For example, the anatomical



structures that carry most discriminative power are likely to dependon the stage of the disease, as the disease progressively spreadsthroughout various brain regions,69 but also on age and other demo-graphic and genetic factors,70 since disease is to be distinguishedfrom complex and progressively changing background normal vari-ations in anatomy and function that may depend on demographicand/or genetic background. Moreover, disease might cause changesof the image characteristics beyond those measured by volumet-rics, such as for example brightening or darkening of an MR imagedue to demyelination, deposition of minerals, or other macro- ormicrostructural changes caused by disease. Vascular disease alsocauses well known MR signal changes, for example in the whitematter of the brain (e.g. brightening of T2-weighted signal). It isthus becoming clear that multiple modalities and multiple anatomi-cal regions must be considered jointly in a multivariate classificationfashion, in order to achieve the desirable diagnostic power. More-over, regions that are relatively less affected by disease should also beconsidered along with known to be affected regions (which, for theexample of Alzheimer’s disease might include primarily temporallobe structures, in relatively early disease stages), since differentialatrophy or image intensity changes between these regions are likelyto further amplify diagnostic accuracy and discrimination from abackground of normal variation.

The approach described in Ref. 67, examines spatiotemporal pat-terns of regional brain atrophy, by hierarchically decomposing aRAVENS map into images of different scales, each of which captur-ing the morphology of the anatomy of interest at a different degreeof spatial resolution. The most important morphological parame-ters are then selected and used in conjunction with a nonlinearpattern classification technique to form a hyper-surface, the high-dimensional analog to a surface, which is constructed in a way that itoptimally separates two groups of interest, for example normal con-trols and patients of a particular disease. Effectively, that approachdefines a nonlinear combination of a number of volumetric measure-ments from the entire brain, each taken at a different scale that typi-cally depends on the size of the respective anatomical structure and



Fig. 9. ROC curves of classifiers for female subjects (Left) and for male subjects(Right). The numbers around the curves are the correct classification rates (%).The circled points on the curves correspond to the optimal classification resultssuggested by SVM, i.e. using zero score as threshold.

the size of the region that is most affected by the disease. This nonlin-ear combination of volumetric measurements is the best way, accord-ing to the respective optimality criteria, to distinguish between thetwo groups, and therefore to perform diagnosis via classification ofa new scan into patients or normal controls. Figure 9 shows the ROCcurve obtained by a high-dimensional nonlinear classification sys-tem applied to a population of healthy controls and age-matchedschizophrenia patients.71

27.6 SUMMARY AND CONCLUSION

Modern techniques for computational neuroanatomy have enabledthe neuroimaging and related scientific communities to transcendthe limitations of traditional methods of analysis of image data,which typically involved the definition of a number of ROIs thatstems directly from a predefined hypothesis of regions tha tareexpected to display certain abnormal structural of functional char-acteristics. The new methodologies quantitatively examine complexspatiotemporal patterns of structure and function with very highspatial specificity, and without the need to know a priori which



regions to look at, since the entire set of data is analyzed simul-taneously. These methods are based on the concept of a statisticalatlas that is constructed by spatially normalizing, or warping, a num-ber of individual scans to a standardized coordinate system, thestereotactic space. Statistics of the regional distribution of brain tis-sue as well as functional activity can be determined this way, andcomparisons between two or more groups can be performed on aregion by region basis, in order to identify regions that display sta-tistically significant differences. Therefore, regions affected by dis-ease, for example, are identified from the analysis itself, and notfrom an a priori hypothesis, which might or might not be optimallyformed.

A fundamental building block in this entire process is the spa-tial transformation that maps the brain scan of an individual to thatof another, and ultimately to the template residing in the stereotac-tic space. A variety of algorithms that achieve this goal have beenreported in the literature, most often relying on image matching cri-teria. Unfortunately, two brain images can be made identical in aninfinite number of ways, most of which don’t imply anatomicallycorrect correspondence. We presented the HAMMER methodology,which attempts to overcome this limitation of other methods, byrelying on a rich set of image attributes collected from the vicin-ity of each voxel, which collectively form an anatomical signatureof that voxel. Image matching then is achieved by matching theseanatomical signatures.

Measuring dynamic patterns of brain atrophy, and most impor-tantly mapping them to the stereotactic space, can be better achievedvia 4D shape transformation algorithms, as opposed to the conven-tional 3D approaches that operate on each individual scan whenmapping it to the template. We described the 4D HAMMER method-ology, which has been used in several longitudinal studies investi-gating dynamic patterns of brain atrophy. By coregistering all scansin a temporal sequence of scans of the same individual, thus con-structing 4D images, 4D HAMMER simultaneously estimates thepattern of deformation (growth or atrophy) within the same indi-vidual, and optimally maps it to the stereotactic space. It has been



found to achieve more stable measurements of the dynamic changesin the brain.

Finally, we discussed the issue of individual scan classification,which has been receiving increasing attention in the recent yearsand is in contrast to the common group analysis. We argued that,although statistically significant differences in structure and func-tion between two groups can be found for a large enough samplesize, they are often of no diagnostic or predictive value, becauseof statistical overlap between the groups. That is, the hippocampiof people at early stages of AD are generally smaller than those ofhealthy elderly, however given an individual’s hippocampal vol-ume, we cannot be sure whether this person is at early AD stages ornot. We presented a high-dimensional nonlinear pattern classifica-tion approach that aims to overcome this limitation, by identifyingcomplex spatiotemporal patterns of brain structure and function,rather than examining each brain region one at a time. Correlationsamong measurements from different brain regions tremendouslyhelp identify patterns of brain atrophy that are highly discriminatoryof disease, and help achieve clinically sufficient sensitivity and speci-ficity. Methods for individual patient analysis are likely to receivea great deal of attention in the near future, as they are necessarysteps in translating scientific findings to clinically useful tools fordiagnosis and prognosis.


This work was supported in part by the National Institutes of Healthgrant R01AG14973.

References

1. Bobinski M, de Leon MJ, Convit A, De Santi S, et al., MRI of entorhinalcortex in mild Alzheimer’s disease, Lancet 353(9146): 38–40, 1999.

2. Convit A, De Leon MJ, Tarshish C, De Santi S, et al., Specific hippocam-pal volume reductions in individuals at risk for Alzheimer’s disease,Neurobiology of Aging 18(2): 131–138, 1997.



3. Cuenod CA, Denys A, Michot JL, Jehenson P, et al., Amygdala atrophyin Alzheimer’s disease. An in vivo magnetic resonance imaging study,Archives of Neurology 50(9): 941–945, 1993.

4. De Santi S, de Leon MJ, Rusinek H, Convit A, et al., Hippocampal for-mation glucose metabolism and volume losses in MCI and AD, Neuro-biology of Aging 22(4): 529–539, 2001.

5. de Toledo-Morrell L, Sullivan MP, Morrell F, Wilson RS, et al.,Alzheimer’s disease: In vivo detection of differential vulnerability ofbrain regions, Neurobiology of Aging 18: 463–438, 1997.

6. Dickerson BC, Goncharova I, Sullivan MP, Forchetti C, et al., MRI-derived entorhinal and hippocampal atrophy in incipient and verymild Alzheimer’s disease, Neurobiology of Aging 22(5): 747–754, 2001.

7. Du AT, Schuff N, Amend D, Laakso MP, et al., Magnetic resonanceimaging of the entorhinal cortex and hippocampus in mild cognitiveimpairment and Alzheimer’s disease, Journal of Neurology Neurosurgeryand Psychiatry 71(4): 441–447, 2001.

8. Frisoni GB, Beltramello A, Weiss C, Geroldi C, et al., Linear measures ofatrophy in mild Alzheimer disease, American Journal of Neuroradiology17(5): 913–923, 1996.

9. Jack CR Jr, Petersen RC, Xu YC, Waring SC, et al., Medial temporalatrophy on MRI in normal aging and very mild Alzheimer’s disease,Neurology 49(3): 786–794, 1997.

10. Jack CR, Petersen RC, XYC, O’Brien PC, et al., Prediction of ADwith MRI-based hippocampal volume in mild cognitive impairment,Neurology 52(7): 1397–1403, 1999.

11. Killiany RJ, Moss MB, Albert MS, Sandor T, et al., Temporal loberegions on magnetic resonance imaging identify patients with earlyAlzheimer’s disease, Archives of Neurology 50(9): 949–954, 1993.

12. Killiany RJ, Gomex-Isla T, et al., Use of Structural Magnetic ResonanceImaging to Predict Who Will Get Alzheimer’s disease, Annals of Neu-rology 47(4): 430–439, 2000.

13. Krasuski JS, Alexander GE, Horwitz B, Daly EM, et al., Volumes ofmedial temporal lobe structures in patients with Alzheimer’s diseaseand mild cognitive impairment (and in healthy controls), BiologicalPsychiatry 43(1): 60–68, 1998.

14. Laakso MP, Soininen H, Partanen K, Helkala EL, et al., Volumes of hip-pocampus, amygdale and frontal lobes in the MRI-based diagnosis ofearly Alzheimer’s disease: Correlation with memory functions, Jour-nal of Neural Transmission. Parkinson’s Disease and Dementia Section 9(1):73–86, 1995.

15. Laakso MP, Hallikainen M, Hanninen T, Partanen K, et al., Diagnosisof Alzheimer’s disease: MRI of the hippocampus vs delayed recall,Neuropsychologia 38(5): 579–584, 2000.



16. Lehericy S, Baulac M, Chiras J, Pierot L, et al., AmygdalohippocampalMR volume measurements in the early stages of Alzheimer’s disease,American Journal of Neuroradiology 15(5): 929–937, 1994.

17. Rosen AC, Prull MW, Gabrieli JD, Stoub T, et al., Differential associ-ations between entorhinal end hippocampal volumes and memoryperformance in older adults, Behavioral Neuroscience 117(6): 1150–1160,2003.

18. Xu Y, Jacky Jr CR, O’Brien PC, Kokmen E, et al., Usefulness of MRImeasures of entorhinal cortex versus hippocampus in AD, Neurology54(9): 1760–1767, 2000.

19. Bookstein FL, Principal warps: Thin-plate splines and the decomposi-tion of deformations, IEEE Transactions on Pattern Analysis and MachineIntelligence 11(6): 567–585, 1989.

20. Miller MI, Christensen G,AmitY, Grenander U, Mathematical textbookof deformable neuroanatomies, Proceedings of the National Academy ofSciences 90: 11944–11948, 1993.

21. Davatzikos C, Vaillant M, Resnick S, Prince JL, et al., A computerizedapproach for morphological analysis of the corpus callosum, Journal ofComputer Assisted Tomography 20: 88–97, 1996.

22. Sandor S, Leahy R, Surface based labeling of cortical anatomy usinga deformable atlas, IEEE Transactions on Medical Imaging 16(1): 41–54,1997.

23. Thompson PM, MacDonald D, Mega MS, Holmes CJ, et al., Detectionand mapping of abnormal brain structure with a probabilistic atlasof cortical surfaces, Journal of Computer Assisted Tomography 21(4):567–581, 1997.

24. Ashburner J, Hutton C, Frackowiak RSJ, Johnsrude I, et al., Identify-ing global anatomical differences: Deformation-based morphometry,Human Brain Mapping 6(5–6): 348–357, 1998.

25. Golland P, Grimson WEL, Kikinis R, Statistical shape analysis usingfixed topology skeletons: Corpus callosum study, Lecture Notes in Com-puter Science 1613: 382–387, 1999.

26. Pizer S, Fritsch DS, Yushkevich PA, Johnson VE, et al., Segmenta-tion, registration and measurement of shape variation via image objectshape, IEEE Transactions on Medical Imaging 18(10): 851–865, 1999.

27. Golland P, Grimson WEL, Shenton ME, Kikinis R, Deformation analysisfor shape based classification, Lecture Notes in Computer Science 2082:517–530, 2001.

28. Rexilius J, Warfield SK, Guttman CRG, Wei X, et al., A novel nonrigidregistration algorithm and applications, Proceedings of MICCAI’98,pp. 202–209, 1999.



29. Christensen G, Rabbitt RD, Miller RI, 3D brain mapping using adeformable neuroanatomy, Physics in Medicine and Biology 39: 609–618,1994.

30. Joshi S, Pizer S, Fletcher PT, Thall A, et al., Multiscale 3D deformablemodel segmentation based on medial description, Lecture Notes in Com-puter Science 2082: 64–77, 2001.

31. Szekely G, Kelemen A, Brechbuhler C, Gerig G, Segmentation of 2Dand 3D objects from MRI volume data using constrained deformationsof flexible Fourier contour and surface models, Medical Image Analysis1: 19–34, 1996.

32. Styner M, Gerig G, Medial models incorporating object variability for3D shape analysis, Lecture Notes in Computer Science 2082: 502–516, 2001.

33. Freeborough PA, Fox NC, Modeling brain deformations inAlzheimer’sdisease by fluid registration of serial 3D MR images, Journal of ComputerAssisted Tomography 22: 838–843, 1998.

34. Collins L, Peters TM, Dai W, Evans AC, Model-based segmentationof individual brain structures from MRI data, in Proceedings of SPIEConference on Visualization in Biomedical Computing, 1992.

35. Thompson DW, On Growth and From, Cambridge University Press,1917.

36. Grenander U, Tutorial in pattern theory: A technical report, BrownUniversity, 1983.

37. Davatzikos C, Resnick SM, Sex differences in anatomic measures ofinterhemispheric connectivity: Correlations with cognition in men butnot in women, Cerebral Cortex 8: 635–640, 1998.

38. Christensen GE, Johnson HJ, Consistent image registration, IEEE Trans-actions on Medical Imaging 20(7): 568–582, 2001.

39. Collins DL, Neelin P, Peters TM, Evans AC, Automatic 3D intersubjectregistration of MR volumetric data in standardized Talairach space,Journal of Computer Assisted Tomography 18: 192–205, 1994.

40. Rueckert D, Sonoda LI, Hayes C, Hill DLG, et al., Non-rigid registrationusing free-form deformations: Application to breast MR images, IEEETrans on Med Imaging 18: 712–721, 1999.

41. Thirion JP, Non-rigid matching using deamons, in Proceedings of IEEEConference on Computer Vision and Pattern Recognition, 1996.

42. Ferrant M, Warfield S, Guttman CRG, Mulkern RV, et al., 3D imagematching using a finite element based elastic deformation model, inProceedings of MICCAI’98, pp. 202–209, 1997.

43. Friston KJ, Ashburner J, Frith CD, Poline JB, et al., Spatial registra-tion and normalization of images, Human Brain Mapping 2: 165–189,1995.



44. Chung MK, Worsley KJ, Paus T, Cherif C, et al., A unified statisti-cal approach to deformation-based morphometry, Neuroimage 14(3):595–606, 2001.

45. Christensen GE, Consistent linear-elastic transformations for imagematching, Lecture Notes in Computer Science: Proc of MICCAI’98 1613:224–237, 1999.

46. Shen D, Davatzikos C, HAMMER: Hierarchical attribute matchingmechanism for elastic registration, IEEE Transactions on Medical Imaging21(11): 1421–1439, 2002.

47. Viola P, Wells WM III, Alignment by maximization of mutual infor-mation, in Proceedings of the International Conference in Computer Vision,Los Alamitos, CA, 1995.

48. Well I, William M, Viola P, Kikinis R, Multimodal volume registrationby maximization of mutual information, in Medical Robotics and Com-puter Assisted Surgery, Wiley, New York, pp. 55–62, 1995.

49. Davatzikos C, Spatial normalization of 3D images using deformablemodels, Journal of Computer Assisted Tomography 20: 656–665, 1996.

50. Thompson P, Toga AW, A surface-based technique for warping three-dimensional images of the brain, IEEE Transactions on Medical Imaging15: 402–417, 1996.

51. Wang Y, Staib LH, Elastic model-based non-rigid registration incorpo-rating statistical shape information, Lecture Notes in Computer Science:MICCAI’98 1496: 1162–1173, 1999.

52. Wang Y, Peterson BS, Staib LH, 3D brain surface matching based ongeodesics and local geometry, Computer Vision and Image Understanding89(2–3): 252–271, 2003.

53. Rangarajan A, Chui H, Bookstein FL, The softassign procrustes match-ing algorithm, in Information Processing in Medical Imaging, 15th Inter-national Conference, Springer, Poultney, VT, 1997.

54. Chui H, Rangarajan A, A new point matching algorithm for non-rigidregistration, Computer Vision and Image Understanding 89(2–3): 114–141,2003.

55. Shen DG, Davatzikos C, Very high resolution morphometry usingmass-preserving deformations and HAMMER elastic registration,Neuroimage 18(1): 28–41, 2003.

56. Shen D, 4D image warping for measurement of longitudinal brainchanges, in Proceedings of the IEEE International Symposium on BiomedicalImaging, Arlington, VA, 2004.

57. Xue Z, Shen D, Davatzikos C, Correspondence detection using wavelet-based attribute vectors, in Medical Image Computing and Computer-Assisted Intervention (MICCAI), Springer-Verlag, Heidelberg, Montreal,Canada, 2003.



58. Xue Z, Shen D, Davatzikos C, Determining correspondence in 3D MRbrain images using attribute vectors as morphological signatures ofvoxels, IEEE Trans on Medical Imaging, 2004.

59. Johnson H, Christensen G, Landmark and intensity-based consistentthin-plate spline image registration, in Proceedings of the Conference onInformation Processing in Medical Imaging, LNCS, 2001.

60. Davatzikos C, Genc A, Xu D, Resnick SM, Voxel-based morphometryusing the RAVENS maps: Methods and validation using simulatedlongitudinal atrophy, Neuroimage 14(6): 1361–1369, 2001.

61. Goldszal AF, Davatzikos C, Rham D, Yan M, et al., An image processingprotocol for the analysis of MR images from an elderly population,Journal of computer Assisted Tomography 22(5): 827–837, 1998.

62. Davatzikos C, Mapping of image data to stereotaxic spaces: Applica-tions to brain mapping, Human Brain Mapping 6(5–6): 334–338, 1998.

63. Ashburner J, Friston KJ, Voxel-based morphometry: The methods,Neuroimage 11(6): 805–821, 2000.

64. Shen D, Davatzikos C, Measuring temporal morphological changesrobustly in brain MR images via 4-dimensional template warping, Neu-roimage 21(4): 1508–1517, 2004.

65. Resnick SM, Goldszal A, Davatzikos C, Golski S, et al., One-year agechanges in MRI brain volumes in older adults, Cerebral Cortex 10:464–472, 2000.

66. Resnick SM, Pham DL, Kraut MA, Zonderman AB, et al., Longitudinalmagnetic resonance imaging studies of older adults: Ashrinking brain,The Journal of Neuroscience 23(8): 3295–3301, 2003.

67. Lao Z, Shen D, Kue Z, Karacali B, et al., Morphological classificationof brains via high-dimensional shape transformations and machinelearning methods, Neuroimage 21(1): 46–57, 2003.

68. Gerig G, Styner M, Lieberman J, Shape versus size: Improved under-standing of the morphology of brain structures, in MICCAI 2001, Klu-ver, Utrecht, the Netherlands, 2001.

69. Braak H, Braak E, Bohl J, Bratzke H, Evolution of Alzheimer’s diseaserelated cortical lesions, Journal of Neural Transmission. Supplementum 54:97–106, 1998.

70. Moffat SD, Szekely CA, Zonderman AB, Kabani NJ, et al., Longitu-dinal change in hippocampal volume as a function of apolipoproteinE genotype, Neurology 55(1): 134–136, 2000.

71. Fan Y, Shen D, Gur RC, Gur RE, et al., COMPARE: Classification ofmorphological patterns using adaptive regional elements, IEEE Trans-actions on Medical Imaging, in press, 2006.


CHAPTER 28

Diffusion Tensor Imaging BasedAnalysis of Neurological Disorders

Tianming Liu and Stephen TC Wong

Diffusion weighted imaging (DWI) and diffusion tensor imaging (DTI)allow in vivo investigation of molecular motion of tissue water at amicroscopic level in cerebral gray matter (GM) and white matter (WM).Quantitative analysis of GM diffusivity and/or WM fiber integrity is ofsignificant interest and promises to have a clinical impact on the investi-gation of many neurological diseases. This chapter briefly reviews severalselected DWI/DTI studies in neurological disorders. Then, we introducean automated framework for analysis of GM diffusivity in 76 standardanatomic subdivisions of gray matter in order to facilitate studies of neu-rodegenerative and other gray matter neurological diseases. The com-putational framework includes three enabling technologies: (1) automaticparcellation of structural MRI GM into 76 precisely defined neuroanatomicsubregions (“76-space”), (2) automated segmentation of GM, WM andCSF based on DTI data, and (3) automatic measurement of the averageapparent diffusion coefficient (ADC) in each segmented GM subregion. Weapplied this computational framework of 76-space GM diffusivity analysison normal brains and patient brains with Creutzfeldt-Jakob disease.

28.1 INTRODUCTION

Diffusion weighted imaging (DWI) and diffusion tensor imaging(DTI) allow in vivo measurement of the diffusivity of water moleculesin living tissues.1,2 Although the diffusivity of water molecules isgenerally represented as a Brownian motion, the microstructure ofliving tissues imposes certain constraints on this motion, whichresults in an anisotropic diffusion measured by DWI/DTI.1–3 The

703


704 Tianming Liu and Stephen TC Wong

measured diffusion can be approximated by an anisotropic Gaus-sian model, which is parameterized by the diffusion tensor in eachvoxel2 to create the tensor field. Diffusion tensor measure provides arich data set from which a measurement of diffusion anisotropy canbe obtained through the application of mathematical formulas andcalculation of the underlying eigenvalue.3–6 Mori7 recently providesan excellent tutorial on the principles of DTI and its applications toneuroscience. Chapter 12 of this book also reviews recent advance-ments in diffusion tensor MR imaging.

DTI provides insights into the nature and degree of white mat-ter (WM) injury or disruption that occurs in many neurological dis-eases. It yields quantitative measures of the integrity of WM fibertracts derived from the intrinsic directionality of water diffusion inhuman brain. It has been in wide use for the investigation of WMabnormality associated with various progressive neuropathologiesand sheds light on detection, diagnosis, treatment, and follow up ofassociated neurological disorders.7–11

In spite that these DTI studies on WM are very useful in theinvestigation of the abnormality occurring on fiber pathways con-necting remote computation centers of various gray matter (GM)regions, many neurodegenerative and neurological diseases, includ-ing Alzheimer’s disease,12,13 Parkinson’s disease14 and multiplesclerosis,15,16 primarily involve the GM. Water diffusivity in GM isnearly isotropic, and scalar diffusivity quantified by the apparentdiffusion coefficient (ADC) in DWI/DTI reflects pathologic changein a number of neurodegenerative and neurological diseases.11 Thereare increasing interest in the community to apply DWI/DTI in study-ing GM diffusivity in neurological disorders.11,17–19

28.2 BACKGROUND AND LITERATURE REVIEW:APPLICATION OF DWI/DTI IN STUDYINGNEUROLOGICAL DISORDERS

A comprehensive literature review of application of DWI/DTI inneurological diseases is beyond the scope of this chapter. Never-theless, we would like to briefly review several selected DWI/DTI


Diffusion Tensor Imaging Based Analysis of Neurological Disorders 705

studies in neurological disorders. Then, we focus on our recent workon automated whole brain GM diffusivity analysis for selected neu-rological disorders.

28.2.1 Aging and Neurodegenerative Diseases

DWI/DTI has been widely applied in studies of normal aging20

and neurodegenerative diseases.13,14,16,21 For example, the DTI stud-ies in Ref. 20 revealed that age related declines in WM fractionalansiotropy (FA) in healthy adults are equivalent in men and womenand are linear from around age 20 years onwards. The study alsodemonstrated that age related declines in WM integrity are asso-ciated with similar declines in interhemispheric transfer, especiallydependent on frontal systems.

Several GM diffusivity studies have recently been reported in theliterature for Alzheimer’s disease.17,22 Kantarci et al. demonstratedstatistically significant differences in mean diffusivity between theAD group and the control group in a number of brain regions,most notably the hippocampus, and in temporal, cingulate, andparietal white matter.23 The same research group reported a speci-ficity of 80% and a sensitivity of 57% for distinguishing patientswith AD from control subjects by using the hippocampal ADCalone and found out that higher baseline hippocampal diffusivitywas associated with a greater risk of progression to AD in amnes-tic mild cognitive impairment (MCI) patients.17 DTI has also beenapplied to study the WM integrity in AD. For example, it wasshown in Ref. 18, that in AD patients, FA was bilaterally decreasedin the WM of the temporal lobe, the frontal lobe and the sple-nium, compared with those regions in controls. These DTI studiesreveal abnormalities in the frontal and temporal WM in early ADpatients.

Applications of DWI/DTI in other neurodegenerative diseases,such as Parkinson’s disease and multiple sclerosis have also gen-erated interesting findings. For example, in Ref. 21, statistical para-metric mapping (SPM) was applied in a DWI study to objectivelylocalize focal changes of structural neuronal integrity. This SPM



study identified significant increases of diffusivity in the regionsof both olfactory tracts in Parkinson’s patients. In Ref. 16, DWI/DTIquantification techniques were employed to characterize the localmicrostructure of brain tissues in multiple sclerosis patients. Itwas reported that MS-associated disease progression results inregions characterized by increased water diffusivity and decreasedanisotropy, and those changes generated different patterns in MSpatients presenting different courses of the disease. For a review ofapplication of DWI/DTI in aging and neurodegenerative disorders,the readers could refer to Ref. 11.

28.2.2 Neurodevelopment and Neurodevelopmental Disorders

DTI has become a popular tool in neurodevelopmental studies,as the imaging technique is able to delineate the axonal organiza-tion of the whole brain.7,55 For example, the DTI studies in Ref. 24revealed more widespread changes in the microstructure with mat-uration than previous reports that suggested a continuation of themicrostructural development through adolescence. In Ref. 25, theDTI studies demonstrated that in the time courses of WM develop-ment, the limbic fibers develop first and association fibers last, andcommissural and projection fibers are forming from the anterior tothe posterior part of the brain.

DWI/DTI has also found important applications in the studyof neurodevelopmental disorders. For instance, DTI studies havedemonstrated a correlation between WM microstructural integrityand reading ability in dyslexic and normal reader adults26 and inchildren of varying reading ability.27–29 It was shown in Ref. 30,that children with attention-deficit/hyperactivity disorder (ADHD)have FA declines in the area of right premotor, right striatal, rightcerebral peduncle, left middle cerebellar peduncle, left cerebellum,and left parietooccipital. In Ref. 31, a DTI study was performed onsubjects with high functioning autism and controls matched for age,handedness, IQ, and head size. Significant differences in fractionalanisotropy, mean diffusivity, and radial diffusivity between groupsin corpus callosum and subregions (genu, body and splenium)



were reported. DTI has also been applied in the study of Fragile Xsyndrome, e.g. the report in Ref. 32. The preliminary result in Ref. 32,indicated that regionally specific alterations of WM integrity occurin females with fragile X. A review of application of DTI in neurode-velopment and neurodevelopmental disorders is referred to Refs. 33and 34.

28.2.3 Neuropsychiatric Disorders

In recent years, there have been increasing DTI studies in neuropsy-chiatric disorders, such as schizophrenia, alcoholism, depressionand bipolar disorder. In particular, DTI studies have been per-formed to study WM integrity in schizophrenia, e.g. it is shownin Ref. 35, that diffusion anisotropy was decreased in the fornix,the corpus callosum, and a couple of other regions in schizophre-nia patients. Kubicki et al. postulated that the reason might be dueto the loss of coherence of WM fiber tracts, due to changes inthe interconnecting fiber tracts, or due to changes in myelination.DWI/DTI has been deployed to study the diffusivity in schizophre-nia, e.g. the studies of Ref. 36, reported increased diffusivity in thefrontotemporal regions of schizophrenic patients. In Ref. 37, sig-nificantly elevated ADC measures in temporal, parietal, and pre-frontal cortical regions in the schizophrenia group, especially in themedial frontal gyrus and anterior cingulated, were reported. Thestudy demonstrated that ADC measurement provided an alterna-tive strategy for studying altered prefrontal thalamic circuitry inschizophrenia.

DTI has also been applied to study other neuropsychiatric dis-orders. It is notable that the DTI studies of alcoholism in Ref. 38,revealed FA deficits in genu and splenium of the corpus callo-sum and centrum semiovale. It was reported that the alcoholicshad abnormally high WM diffusivity values in the genu and cen-trum. In Ref. 39, the disruption of neural circuits in the frontal lobesand limbic structures in late life depressed patients is investigated.The correlation between the degree of microstructural abnormalitiesof WM and clinical symptom severity in late life depression was



also examined. In the DTI studies in Ref. 39, a significant WM FAreduction was found in the frontal and temporal lobes of depressedpatients. Another example of neuropsychiatric disorders examinedby DTI is the bipolar disorder. Adler et al. studied WM tractsof adolescents in their first episode of mania to address whetherabnormalities are present in early bipolar disorder by using DTI.40

Bipolar adolescents showed significantly decreased FAonly in supe-riorfrontal white matter tracts. But there was no significant ADCmeasurements in any regions examined. A review of application ofDTI in neuropsychiatric disorders is referred to Ref. 41.

28.2.4 Neurooncology and Neurosurgical Planning

DTI has also found their utility in neurosurgical planning.42 Forexample, preoperative WM fiber tracking is useful for proceduresinvolving deep seated lesions adjacent to the corticospinal tract. InRef. 43, it was reported that fiber tracking showed that the corti-cospinal tract was displaced anterolaterally from the medial sidein a patient with a paraventricular cavernous angioma manifestingas hemiparesis caused by haemorrhage. The paraventricular lesionwas completely removed without damaging the corticospinal tractby using a transcortical, transventricular approach. For this patient,however, preoperative conventional MRI failed to determine theanatomical relationship between the paraventricular lesion and thecorticospinal tract.

In addition to neurosurgical planning, DTI has been used in neu-roimaging follow up studies to determine the effectiveness of ther-apeutic treatment of brain tumor patients. As an example, DWI isfound to be a means to characterize and differentiate morphologicfeatures, including edema, necrosis, and tumor tissue, by measuringdifferences in ADC.27 It is reported in Ref. 44 that diffusion tensortractography allows for visualization of the exact location of tumorsrelevant to eloquent tracts and benefits the neurosurgical planningand postoperative assessment. Finally, a review of the application ofDTI in neurosurgery and follow ups of treated brain tumor patientscan be found in Ref. 42.



28.3 PROCEDURES AND METHODS: AUTOMATEDWHOLE BRAIN GM DIFFUSIVITY ANALYSIS

To date, quantification of DWI/DTI derived diffusivity measures inneurological disease research has relied on manual “Region of Inter-ests” (ROI) analysis.11,17,23 ROI analysis is subject to several notabledrawbacks. First, manual ROI delineation is subject to inter-raterand intra-rater variations and is thus inherently difficult to repro-duce. Also, the location, number and size of ROIs must be selectedbefore analysis, leading to site selection bias and to difficulty in com-paring the results of different studies.45 In addition, time consumingmanual labeling is not scalable for studies involving a wealth of neu-roanatomic structures from large datasets. To manually parcellatethe cortex of a single brain image dataset into well defined neu-roanatomic regions, e.g. the 76 cytoarchitectonic spaces proposed inRef. 19, would take a trained neuroanatomist dozens of hours ormore. Thus, manual ROI methods are impossible to apply to largenumbers of patients required for studies of causality or therapeuticresponse in neurological disorders. Since ROI selection reflects thea priori hypothesis, this technique also has limited potential to iden-tify new and unexpected pathological correlations and structuralfunctional relationships. To address the aforementioned issues, werecently proposed an automated approach for whole brain GM dif-fusivity analysis.19,54

28.3.1 Overview of the Computational Framework

The computational framework of 76-space analysis of GM diffusiv-ity is composed of seven steps, as summarized in Fig. 1.

28.3.1.1 SPGR space

The first two steps automatically segment the SPGR brain image intodistinct tissues of CSF, GM, and WM tissues,46 and the GM is fur-ther parcellated into 76 fine-detailed neuroanatomic regions usingthe high-dimensional hybrid registration method.46 This enabling



SPGR image

WM/GM/CSFmap

ADC image FA image

CSF/non-CSFmap

WM/non-WMmap

DWI/DTI images

GM ADC map

Bra

in A

tlas

(SP

GR

, Lab

els)

76 GM regions

2

3

1 4 5

6

1. Tissue classification using SPGR image2. Atlas-based warping3. Preprocessing and co-registration4. Tissue classification using ADC5. Tissue classification using FA

6. Multichannel fusion7. 76-space analysis7

SPGR space DWI/DTI space

Fig. 1. The computational framework of 76-space analysis.

technology makes it possible to analyze the diffusivity of a wealth ofGM structures of large amounts of normal and pathological brains.

28.3.1.2 DWI/DTI space

The third step performs preprocessing in the DWI/DTI space, e.g.the eddy current correction, ADC/FA image generation, reslicing,and coregistration of the DWI/ADC/FA images with the SPGRimage. Owing to the problems such as EPI geometric distortion,47

partial volume effect, image reslicing errors, and the inaccuracy ofcoregistration algorithm, the warped B0/ADC/FA images wouldnot be in exact anatomic correspondences with SPGR image. Theinaccurate alignment of ADC/FA images with the SPGR image pre-vents us from directly applying the GM parcellation in the SPGRspace to theADC/FAimages, in that the GM in SPGR space may cor-respond to heterogeneous tissues e.g. CSF and WM, in the DWI/DTIspace. To overcome this problem, step 4 and step 5 segment the braininto CSF/GM/WM tissues in the DWI/DTI space by utilizing thetissue contrasts existing in ADC image and FA image. Afterwards,step 6 combines the results of tissue segmentation from both theSPGR space and the DWI/DTI space, and takes the intersection



of the GM maps in both spaces. This AND-like operation of GMmaps results in a GM-intersection map, which is classified as GM byboth SPGR image segmentation and DWI/DTI image segmentation.Finally, we apply the neuroanatomic parcellation of GM-intersectionmap obtained in the SPGR space to the ADC image, and per-form the 76-space analysis of GM diffusivity, represented by step 7in Fig. 1.

28.3.2 GM Parcellation in SPGR Space

We employed a hybrid volumetric and surface warping method46 toautomatically segment the brain SPGR image into a variety of neu-roanatomic structures. After the high-dimensional hybrid registra-tion and atlas-based warping, the subject brain image is segmentedinto a variety of neuroanatomic structures, as shown in Fig. 2(D).Meanwhile, we apply automatic tissue segmentation on the sub-ject SPGR image [Figs. 2(C) and 2(F)], and use the resulted GM map[Fig. 2(G)] to mask the automatically labeled SPGR image [Fig. 2(D)],generating the labeled GM map [Fig. 2(E)]. Finally, we remove othernon-GM tissues by looking at the GM neuroanatomy table (Table 1in Ref. 19), and obtained the 76 GM structures [Fig. 2(H)] for thefollowing step of 76-space analysis. Notably, to ensure that the atlasGM is mapped to the subject GM, the GM masking procedures ofstep 6 and step 7 in Fig. 2 are conducted to eliminate erroneous GMmappings caused by registration inaccuracy, which is akin to theANIMAL+INSECT merging strategy in Ref. 48.

28.3.3 Tissue Classification in DWI/DTI Space

28.3.3.1 Motivation

Preprocessing: For the DWI/DTI images, first we perform pre-processing.19 Then, we coregister the b0 image, ADC image and FAimage with the structural SPGR image using the linear multimodal-ity registration method of Oxford FSL Flirt.49 To obtain better coreg-istration accuracy, we further apply the registration method, AIR,



1. Atlas construction

1

2

(A) SPGR atlas

(C) SPGR subject

(F) Subject tissue map (G) Subject GM (H) 76-space map

(D) Labeled subject (E) Labeled GM

(B) Labeled atlas3

6

74

5

2. Registration

3. Atlas warping

4. Tissue classification

5. GM extraction

6. Subject GM masking

7. Atlas GM masking

Fig. 2. Procedures for automatic parcellation of GM into 76 regions in SPGR space.



from UCLA50 to generate a better coalignment between ADC/FAimages and the SPGR image.

Although the Oxford FSL Flirt and UCLA AIR algorithmscan produce reasonably good coregistration results, a couple ofresources (described in Sec. 28.3.1) render the possibility that GMregions in the SPGR space may include heterogeneous tissues, e.g.CSF and WM, in the DWI/DTI space. Measurements of GM diffu-sivity may fail to reveal real changes occurring in the GM tissue, ifwe directly apply the GM segmentation in the SPGR space into theDWI/DTI space. In other words, small errors in the alignment of theADC image and the SPGR image may place CSF tissue of the ADCimage onto GM tissues on the structural SPGR image. Given thatthe GM is a laminar with a thickness of about 3.0 mm, and the ADCvalues of the CSF are more than twice as high as the GM values,small errors in coregistration may lead to significant deviation ofthe measured ADC value in the GM region.

Figure 3 illustrates the problem of heterogeneous tissues.Figure 3(D) is the GM map segmented from the SPGR image[Fig. 3(A)]. After applying this SPGR GM segmentation to maskADC image [Fig. 3(B)], we have the GM ADC map [Fig. 3(E)]. Ashighlighted by the yellow arrows in Fig. 3(E), it is clear that certainCSF tissues are included in the GM ADC map. These erroneouslyincluded heterogeneous CSF tissues would greatly affect the accu-racy of the ADC measurement in those regions. In an alternativemethod of confirmation, we overlay the boundaries of GM seg-mented in SPGR image onto the ADC image, as shown in Fig. 3(H).Clearly, SPGR GM boundaries (red) are crossing the CSF in the ADCimage. It has been shown that, on average, 15% of GM segmented inthe SPGR image would be overlapped on CSF on the ADC image,even after linear and nonlinear coregistration of ADC and SPGRimage in the preprocessing step.19 Similarly, as shown in Figs. 3(C),3(F) and 3(J), direct application of the SPGR GM segmentation toADC image would also erroneously include heterogeneous WM tis-sues into the GM ADC map, which would slightly decrease the mea-sured ADC values in GM, as the ADC value of WM is slightly lowerthan that of GM. The yellow arrows emphasize the WM areas that



(A) (B) (C)

(D) (E) (F)

(G) (H) (I)

ANDAND

Fig. 3. Illustration of the problem of heterogeneous tissues when directly applyingSPGR GM segmentation to DWI/DTI space. The “AND” above means taking theintersection of two maps.

are erroneously included into the GM ADC map. On the average,17% of GM in SPGR image would be placed on WM tissues on theADC map.19

28.3.3.2 Tissue classification based on ADC and FA images

Other than segmenting tissues into three classes, we classify tissuesinto two classes: CSF and non-CSF in ADC images, and WM and



non-WM in FA images. We apply a Hidden Markov Random Field(HMRF) model and the Expectation-Maximization (EM) algorithmfor the two-class segmentation, which is akin to that in Ref. 51.

28.3.4 Multichannel Fusion

We now have tissue classification results obtained from three chan-nels: SPGR image, ADC image, and FA image. The SPGR channelhas the complete segmentation of CSF, GM, and WM [Fig. 4(B)],

(A) SPGR image (B) Tissue maps (C) GMAND 2

AND 1

AN

D 4

AND 3

(D) Intersection of (C) and (H)

(E) ADC image (F) Tissue maps (G) Non-CSF (H) CSF

(I) FA image (J) Tissue maps (K) Non-WM (L) WM

(M) GM 76-space (N) GM ADC map (O) GM-intersection (P) Intersection of (C) and (L)

Fig. 4. Multichannel data fusion. The “AND” means taking the intersectionof two.



whereas the ADC channel has the segmentation of CSF and non-CSF [Fig. 4(F)] and the FA channel has segmentation of WM andnon-WM [Fig. 4(J)]. The multichannel fusion is to take the inter-section of the SPGR GM map [Fig. 4(C)], the ADC non-CSF map[Fig. 4(G)], and the FA non-WM map [Fig. 4(K)] by performingan AND-like operation, thus generating a GM-intersection map asshown in Fig. 4(O). This AND-like operation ensures that the GM-intersection map is the consensus of all three channels and sub-stantially removes heterogeneous tissues in the GM ADC image.To demonstrate the heterogeneous tissue removal by multichannelfusion, Fig. 4(D) shows the overlap of the SPGR GM map [Fig. 4(C)]and ADC CSF map [Fig. 4(H)]. These large areas of overlapped het-erogeneous tissues could give rise to significant increases of GMdiffusivity measurement. Likewise, Fig. 4(P) is the overlap of SPGRGM map [Fig. 4(C)] and FA WM map [Fig. 4(L)]. These large areasof overlapped heterogeneous tissues could result in the decrease ofthe GM diffusivity measurement, since the ADC value of WM isslightly lower than that of GM. Using the method in Sec. 28.3.2, weautomatically parcellate the GM into 76 spaces [Fig. 4(M)]. Then, theGM-intersection map [Fig. 4(O)], the labeled GM map [Fig. 4(M)],and the original ADC map [Fig. 4(E)] are combined together, andfinally, we have the GM ADC image [Fig. 4(N)] for 76-space anal-ysis of GM diffusivity, e.g. for each labeled neuroanatomical GMregion in Fig. 4(M), we measure its average ADC value based onFig. 4(N).

28.4 RESULTS AND FINDINGS

28.4.1 GM Diffusivity Study of Normal Brains

We applied the automated whole brain GM diffusivity analysismethod to fifteen normal brains. The ages of the fifteen normal con-trols are between 29 and 51. All of them are male. Figure 5 showsthe color-coded GM ADC distribution, where the ADC values ofGM structures are mapped onto GM/WM surface of an atlas. The



0.5

1.0

Fig. 5. Color-coded GM diffusivity of normal controls. ADC scale is10−3 mm2/sec.

visualization shows that the parietal lobe has higher ADC valueswhile the temporal lobe has lower ADC values. Also, we observedthat deep GM structures have much lower ADC values. Figure 5exhibits that there is no visible difference between the ADC on theright and left hemispheres.

28.4.2 GM Diffusivity Study of Creutzfeldt-Jakob Disease

We have measured the ADC values of the seventy-six GM struc-tures of four Creutzfeldt-Jakob disease (CJD) patients and foundeight GM structures have significant differences (p-value < 0.05)between CJD and normal brains. Our results show that basal gangliaare frequently involved in these CJD cases, which is in agreementwith other research reports.52 Specifically, the averageADC values ofputamen, thalamus, and globus palladus of CJD patients are muchlower than those of normal brains. The average ADC values of CJDpatients’ right and left putamen dropped 29% and 26% respectively,compared to those of normal brains. The ADC droppings are con-firmed by expert manual tracing, as shown in Fig. 6. Although theCJD patients and normal controls in this study are not age-matchedand gender-matched, we believe that the comparison results aremeaningful since these changes are far larger than reported as age-related or gender-related ADC variation (Helenius).53



0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

12 48 59 60 61 62 63 64Structure Index

AD

C

Normal brain Algorithm result Manual labeling 1 Manual labeling 2

(A)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

23 24 48 59 60 61 62 63 64 67 68Structure Index

AD

C

Normal brain Algorithm result Manual labeling 1 Manual labeling 2

(B)

Fig. 6. Confirmation of algorithm results by manual ROI analysis. ADC scale is10−3 mm2/sec. (A) The first case. (B) The second case.

28.5 DISCUSSIONS AND CONCLUDING REMARKS

We have demonstrated that better measurement of GM diffusivitycould be achieved by removing heterogeneous tissues via multi-channel fusion.19 However, a basic assumption here is that we couldaccurately measure GM diffusivity although we use only the inter-section of the GM obtained in SPGR space and that in DWI/DTIspace. In our future work, we will investigate how much the removalof heterogeneous tissue would deviate the diffusivity measurementfrom the true one.



As confirmed in this work,ADC values are lower in patients withCJD than in normal brains. This fact should enhance the separation ofCSF from other tissues, rather than impair the power of ADC imagesto separate CSF and non-CSF tissues. However, in certain neurologi-cal diseases, e.g. Alzheimer’s disease, there might be increased ADCvalues observed in some regions.23 To what extent the ADC changesdue to the presence of diseases would cause problems in the tissuesegmentation in ADC images needs further investigation.

We proposed a new computational framework for automatedwhole brain analysis of GM diffusivity for study of normal brainsand neurological diseases. The framework has been applied tostudy data from normal volunteers and CJD patients, and pro-duced meaningful results. In the future, we will further improvethe computational framework, and apply the method to other neu-rological diseases, including Alzheimer’s disease and Parkinson’sdisease.


This research work is supported by a grant to Dr Stephen TC Wongfrom Harvard Center for Neurodegeneration and Repair (HCNR),Harvard Medical School. The normal control datasets are from theNIH sponsored NAMIC (National Alliance of Medical Image Com-puting) data-repository and are provided by the Laboratory of Neu-roscience, Department of Psychiatry, Boston VA Healthcare Systemand Harvard Medical School. We would like to thank Ms Yi-ru Linof HCNR Center for Bioinformatics for manual labeling of selecteddatasets, Dr Geoffrey Young of the Harvard Medical School for shar-ing the CJD DTI datasets, and Dr Noor Kabani of the Montreal Neu-rological Institute for sharing the brain atlas.

References

1. Le Bihan D, Molecular diffusion nuclear magnetic resonance imaging,Magn Reson Q 7: 1–30, 1991.

2. Basser et al., 1994.



3. Bammer R, Basic principles of diffusion-weighted imaging, Eur J Radiol45: 169–184, 2003.

4. Moseley ME, Kucharczyk J, Mintorovitch J, et al., Diffusion-weightedMR imaging of acuter stroke: Correlation with T2-weighted and mag-netic susceptibility-enhanced MR imaging in cats, AJNR 11: 423–429,1990.

5. Le Bihan D, 1991.6. Basser PJ, Jones DK, Diffusion tensor MRI: Theory, experimental design

and data analysis — Atechnical review, NMR Biomed 14: 456–467, 2002.7. Mori S, Zhang J, Principles of diffusion tensor imaging and its appli-

cations to basic neuroscience research, Neuron 51: 527–539, 2006.8. Moseley, 2002.9. Horsfield MA, Jones DK, Applications of diffusion-weighted and dif-

fusion tensor MRI to white matter diseases — A review, NMR Biomed15(7–8): 570–577, 2002.

10. Stahl R, Dietrich O, Teipel S, et al., Assessment of axonal degenerationon Alzheimer’s disease with diffusion tensor, MRI Radiologe 43(7): 566–575, 2003.

11. Sundgren PC, Dongl Q, Gómez-Hassan D, et al., Diffusion tensor imag-ing of the brain: Review of clinical applications, Neuroradiology 46(5):2004.

12. Hanuy H, Sakurai H, Iwamoto T, et al., Diffusion-weighted MR imagingof the hippocampus and temporal white matter inAlzheimer’s disease,J Neurol Sci 156: 195–200, 1998.

13. Rose SE, Chen F, Chalk JB, et al., Loss of connectivity in Alzheimer’sdisease: An evaluation of the white matter tract integrity with colorcoded MR diffusion tensor imaging, J Neurol Neurosurg Psychiatry 69:528–530, 2000.

14. Adachi M, Hosoya T, Haku T, et al., Evaluation of the substantia nigra inpatients with Parkinsonian syndrome accomplished using multishotdiffusion-weighted MR imaging, AJNR Am J Neuroradiol 20: 1500–1506,1999.

15. Cercignani M, Iannucci G, Rocca MA, et al., Pathologic damage inMS assessed by diffusion-weighted and magnetization transfer MRI,Neurology 54: 1139–1144, 2000.

16. Goldberg-Zimring D, Mewes AU, Maddah M, et al., Diffusion ten-sor magnetic resonance imaging in multiple sclerosis, J Neuroimaging15(4 Suppl): 68S–81S, 2005.

17. Kantarci K, Petersen RC, Boeve BF, et al., DWI predicts future progres-sion to Alzheimer’s disease in amnestic mild cognitive impairment,Neurology 64(5): 902–904, 2005.



18. Naggara O, Oppenheim C, Rieu D, et al., Diffusion tensor imaging inearly Alzheimer’s disease, Psychiatry Res 146(3): 243–249, 2006.

19. Liu T, Young G, Huang L, et al., 76-space analysis of grey matter diffu-sivity: Methods and applications, Neuroimage 31(1): 51–65, 2006.

20. Sullivan EV, Pfefferbaum A, Diffusion tensor imaging and aging, Neu-rosci Biobehav Rev 30(6): 749–761, 2006.

21. Scherfler C, Schocke MF, Seppi K, et al., Voxel-wise analysis of diffusionweighted imaging reveals disruption of the olfactory tract in Parkin-son’s disease, Brain 129(Pt 2): 538–542, 2006.

22. Bozzali M, Franceschi M, FaliniA, et al., Quantification of tissue damagein AD using diffusion tensor and magnetization transfer MRI, Neurol-ogy 57(6): 1135–1137, 2001.

23. Kantarci K, Jack CR, Xu YC, et al., Mild cognitive impairment andAlzheimer’s disease: Regional diffusivity of water, Radiology 219:101–107, 2001.

24. Snook L, et al. 2005.25. Huang H, Zhang J, Wakana S, et al., White and gray matter develop-

ment in human fetal, newborn and pediatric brains, Neuroimage 33(1):27–38, 2006.

26. Klingberg T, Hedehus M, Temple E, et al., Microstructure of tem-poroparietal white matter as a basis for reading ability: Evidence fromdiffusion tensor magnetic resonance imaging, Neuron 25: 493–500, 2000.

27. Beaulieu C, Plewes C, Paulson LA, et al., Imaging brain connectivity inchildren with diverse reading ability, Neuroimage 25: 1266–1271, 2005.

28. Deutsch GK, Dougherty RF, Bammer R, et al., Children’s reading perfor-mance is correlated with white matter structure measured by diffusiontensor imaging, Cortex 41: 354–363, 2005.

29. Niogi SN, McCandliss BD, Left lateralized white matter microstructureaccounts for individual differences in reading ability and disability,Neuropsychologia 44: 2178–2188, 2006.

30. Ashtari M, Kumra S, Bhaskar SL, et al., Attention-deficit/hyperactivitydisorder: A preliminary diffusion tensor imaging study, Biol Psychiatry57(5): 448–455, 2005.

31. Alexander AL, Lee JE, Lazar M, et al., Diffusion tensor imaging of thecorpus callosum in Autism, Neuroimage, 2006.

32. Barnea-Goraly N, Eliez S, Hedeus M, et al., White matter tract alter-ations in fragile X syndrome: Preliminary evidence from diffusion ten-sor imaging, Am J Med Genet B Neuropsychiatr Genet 118(1): 81–88, 2002.

33. Snook L, Paulson LA, Roy D, et al., Diffusion tensor imaging ofneurodevelopment in children and young adults, Neuroimage 26(4):1164–1173, 2005.



34. Wozniak JR, Lim KO, Advances in white matter imaging: A review ofin vivo magnetic resonance methodologies and their applicability to thestudy of development and aging, Neurosci Biobehav Rev 30(6): 762–774,2006.

35. Kubicki M, Park HJ, Westin CF, et al., DTI and MTR Abnormalities inSchizophrenia: Analysis of white matter integrity, Neuroimage, 2005.

36. Shin Y-W, Kwon JS, Ha TH, et al., Increased water diffusivity in thefrontal and temporal cortices of schizophrenic patients, Neuroimage30(4): 1285–1291, 2006.

37. Rose SE, Chalk JB, Janke AL, et al., Evidence of altered prefrontal–thalamic circuitry in schizophrenia:An optimized diffusion MRI study,Neuroimage 32(1): 16–22, 2006.

38. Pfefferbaum A, Sullivan EV, Disruption of brain white mattermicrostructure by excessive intracellular and extracellular fluid in alco-holism: Evidence from diffusion tensor imaging, Neuropsychopharma-cology 30(2): 423–432, 2005.

39. Nobuhara K, Okugawa G, Sugimoto T, et al., Frontal white matteranisotropy and symptom severity of late-life depression: A magneticresonance diffusion tensor imaging study, J Neurol Neurosurg Psychiatry77(1): 120–122, 2006.

40. Adler CM, Adams J, DelBello MP, et al., Evidence of white mat-ter pathology in bipolar disorder adolescents experiencing their firstepisode of mania: A diffusion tensor imaging study, Am J Psychiatry163(2): 322–324, 2006.

41. Lim KO, Helpern JA, Neuropsychiatric applications of DTI —Areview,NMR Biomed 15(7–8): 587–593, 2002.

42. Field AS, Alexander AL, Diffusion tensor imaging in cerebral tumordiagnosis and therapy, Topics in Magn Reson Imaging 15(5): 315–324,2004.

43. Niizuma K, Fujimura M, Kumabe T, et al., Surgical treatment of par-aventricular cavernous angioma: Fibre tracking for visualizing the cor-ticospinal tract and determining surgical approach, J Clin Neurosci 2006.

44. Yu CS, Li KC, Xuan Y, Diffusion tensor tractography in patients withcerebral tumors: A helpful technique for neurosurgical planning andpostoperative assessment, Eur J Radiol 56(2): 197–204, 2005.

45. Bilgili Y, Unal B, Effect of region of interest on interobserver variancein apparent diffusion coefficient measures, AJNR Am J Neuroradiol 25:108–111, 2004.

46. Liu T, Shen D, Davatzikos C, Deformable registration of cortical struc-tures via hybrid volumetric and surface warping, Neuroimage 22(4):1790–1801, 2004.



47. Jezzard P, Balaban RS, Correction for geometrical distortion in echoplanar images from Bo field variations, Magn Reson Med 34: 65–73,1995.

48. Collins DL, ZijdenbosAP, Baaré WFC, et al., Evans:ANIMAL+INSECT:Improved cortical structure segmentation, IPMI 210–223: 1999.

49. Jenkinson M, Smith SM,Aglobal optimisation method for robust affineregistration of brain images, Medical Image Analysis 5(2): 143–156, 2001.

50. Woods RP, Grafton ST, Watson JDG, et al., Automated image registra-tion. II: Intersubject validation of linear and nonlinear models, Journalof Computer Assisted Tomography 22(1): 153–165, 1998.

51. Zhang Y, Brady M, Smith S, Segmentation of brain MR imagesthrough a hidden Markov random field model and the expectation-maximization algorithm, IEEE Trans Med Imaging 20(1): 45–57, 2001.

52. Young GS, Geschwind MD, Fischbein NJ, et al., Diffusion weightedimaging (DWI) and FLAIR in Creutzfeldt-Jakob disease (CJD): Highsensitivity and specificity for diagnosis, American Journal of Neuroradi-ology, 2006.

53. Helenius J, Soinne L, Perkio J, et al., Diffusion-weighted MR imagingin normal human brains in various age groups, AJNR Am J Neuroradiol23(2): 194–199, 2002.

54. Tianming Liu, Hai Li, Kelvin Wong, et al., Brain tissue segmentationbased on DTI data, Neuroimage in press, 2007.

55. Bernard S Chang, Tami Kazir, Tiaming Liu, et al., A structural basis forreading fluency: Cortico-cortical fiber tract disruptions are associatedwith reading disability in a neuronal migration disorder, Neurology, inpress, 2007.


CHAPTER 29

Intelligent Computer AidedInterpretation in Echocardiography:Clinical Needs and Recent Advances

Xiang Sean Zhou and Bogdan Georgescu

Cardiovascular disease is a leading cause of death worldwide, and cardiacultrasound is the most cost efficient modality for real-time assessment ofcardiac function. However, the poor quality of ultrasound images andthe operator-dependency during both image acquisition and interpreta-tion have severely limited its capability for fast and accurate analysis anddiagnosis, thus prompting the need for intelligent computer assisted bor-der detection that can mimic a sophisticated consensus reasoning, whileachieving the critically required consistency, reproducibility, and robust-ness. Several representative approaches are discussed in the context of leftventricular border detection, including active shape/appearance/motionmodels, graph cuts, and level set methods. In particular, We introducea non-parametric, learning-based deformable segmentation framework,with a CBIR exemplar-based inference module combined with a discrim-inative classification-based detection module in a hierarchical, coarse-to-fine setting. The combination achieves both high computational efficiencyand strong shape modeling power. Another advantage of the CBIR-basedapproach is that since similar shapes from the annotated training databaseare extracted and used during the segmentation process, they are avail-able to the clinicians to help the diagnosis process. Based on motion track-ing of the detected borders, fully automatic cardiac wall motion analysisbecomes feasible.

725


726 Xiang Sean Zhou and Bogdan Georgescu

29.1 INTRODUCTION: CARDIAC IMAGING USINGULTRASOUND

What make the topic of cardiac ultrasound immediately interestingand extremely important are the facts that: (1) cardiovascular dis-ease is the leading cause of death resulting in one third of all deathsglobally1 and (2) echocardiography (ultrasound heart exam) is theeasiest, most portable, and most accessible means for real-time eval-uation of heart function.2

Echocardiography provides multiple ways to image the heart,either from outside the body (“transthoracic echocardiography”,where imaging probe is placed on the surface of the chest), or fromwithin the body (e.g. “transesophageal echocardiography”, wherethe probe is sent through the mouth to the esophagus to take a “closerlook” from behind the heart at, e.g. valve anatomy or clot in leftatrium). The majority of exams are done the former way because itis noninvasive.

It is possible to acquire signals in one-dimension (1D) in time(“M-mode,” where “M” stands for “motion”), 2D in time (“B-mode,”where we see a “video”), or 3D in time (where multiple 3D vol-umes are acquired throughout the heart cycle). The ultrasound wavecan be used to image either the anatomy of the heart by register-ing the reflected sound wave (i.e. “echo”) from tissue structures, orthe velocity of the blood or tissue by exploiting the frequency shiftof the reflected sound wave using the Doppler equation (ChristianDoppler, 1842). For example, blood flow direction and velocity areoften color-coded and overlaid in a B-mode video for clear visualiza-tion of abnormalities such as mitral regurgitation (i.e. valve leakage).To learn more, please refer to a classic book by Feigenbaum et al.3

We will focus on transthoracic B-mode images as our applicationdomain because of its prevalent use and rich information content.

As shown in Fig. 1, B-mode echocardiography enables real-time continuous visualization of an arbitrary 2D slice of the heart.The echocardiography examination defines a set of standard imag-ing slices (called “views”) that cover a full spectrum of possiblediagnoses and ensure consistency and comparability among exams.


Intelligent Computer Aided Interpretation in Echocardiography 727

Fig. 1. Ultrasound imaging of the human heart. (A) The anatomy of a humanheart, rotated and flipped to match a typical echocardiography view: left ventricle ison the upper right. (B) Upper row: images extracted from an apical 4-chamber viewsequence, from end-diastole when the left ventricle (LV) is the largest (first image) toend-systole when LV is the smallest (third image). The last image contains a Dopplerwindow in which the blood flow pattern into LV is visualized by color overlay.Lower row: from left, images extracted from sequences of apical 2-chamber view,apical long axis view, parasternal long axis view, and parasternal short axis view,respectively.

Although it requires multiple views to faithfully reconstruct sucha complex structure like the heart, some views capture much moreinformation than others. For example, the apical 4-chamber view (orA4C — first row in Fig. 1(B)) reveals, as its name implies, all fourheart chambers, two atrioventricular valves, and the two septa.

Among the four heart chambers, the left ventricle (LV) — theupper right chamber in Fig. 1(A) surrounded by thick muscle lay-ers called myocardium — is the largest and most powerful. LV isresponsible for the most demanding task of pumping oxygen-richblood to the whole body, from head to toe. Constantly under thehighest pressure, LV is also the most disease-prone of the four heartchambers. In most cases, assessment of LV function is the first steptoward the detection and diagnosis of cardiovascular diseases.

Apical 4-chamber view, alone or combined with the apical2-chamber view (or A2C — the first on the second row of Fig. 1(B) —this is acquired with a 60◦ rotation from the A4C view), can provideenough information for an initial, clinically meaningful assessment



of left ventricle (and left atrium) function. Some basic parametersinclude volumes as a function of time, global and regional wallmotion, and the LV ejection fraction (i.e., the fraction of blood vol-ume pumped out at each heart beat).

In this chapter, we will focus our attention on automatic estima-tion of these basic and useful parameters.

29.2 CLINICAL BACKGROUND: A NEED AND ANOPPORTUNITY FOR INTELLIGENT COMPUTERAIDED INTERPRETATION

Until very recently, quantitative assessment of LV function (e.g. ejec-tion fraction and wall motion) has been based on “eyeballing” (i.e.by looking at the image) and “guesstimate.”

Less frequently, the clinician may manually trace out the innerborder of the LV (i.e. endocardial border). Then, there are well estab-lished formula to estimate the blood pool volumes.3 Once we havethe maximum and the minimum volumes within a heart cycle, theejection fraction is simply the difference over the maximum.

However, due to the relatively poor definition of the endocar-dial border, complex anatomy, fast motion of the heart, and the factthat B-mode images are only a slice through this 3D moving target,the interpretation of echocardiography is a very difficult task evenfor the highly trained and highly experienced. And the interobservervariability can be high in many cases (Fig. 2). Even for the same expert,experiments have shown that the tracing for the same case over timemay vary noticeably.

The reason behind such inter- and intraobserver variability isthree-fold: ambiguity, subjectivity, and human error.

– Ambiguity: the image itself may be ambiguous — key pieces ofinformation may have been lost or buried in heavy noise andthat an observer can only guess. Guess of course carries intrin-sic variability. This scenario happens quite often in ultrasoundimages because of so called acoustic drop-out: when the tissue inter-face is parallel to the ultrasound beam, we get little or no echo.



Fig. 2. Interobserver variability in identifying the endocardial border: The twocontours in each image were manually traced by two trained and very experiencedsonographers. Discrepancies often occur at the apex and along the lateral wall.

Experience can statistically reduce the bias and variance of one’sguess.

– Subjectivity: even when the information is clearly preserved in theimage, different schools of training and personal experiences willaffect one’s belief or judgment as to where the border should be.Figure 3 was annotated by an expert with high confidence, butwe know that the endocardium pointer is not agreed by anotherexpert.

– Human error: when human gets tired, bored, or distracted, wemake errors — we do this often, and some do more than others…

All three factors above point to a need and an opportunityfor automatic computer aided interpretation of cardiac ultrasoundimages. Indeed,

– We can build computer models of the heart shape and appearancein ultrasound so that when information is missing either in spaceor in time, an optimal or informed guess can be computed based onavailable information.

– We can train a computer program to learn from multiple expertsa most-widely acceptable consensus — if such exists — so thatreproducibility and consistency can be assured across patients or



Fig. 3. Annotation of various structures by an expert. Some border locations aresubjective — for example, the exact endocardial border may be debatable. Notethat, however, in many cases ambiguity in one frame can be resolved by looking atthe motion across multiple frames.

for the same patient over time. Clinical workflow can be improvedas well by eliminating the time needed to manually trace theborder.

– Of course, you can run a computer program a million times with-out worrying about it getting bored or tired.

29.3 CHALLENGES: CAN A COMPUTER DO IT?

We have argued that computer can potentially be very helpful in thisdomain. But can it achieve a level of accuracy and consistency thatwill add value in a clinical setting? Before we answer this question, letus dig deeper into the actual task itself and see what are the problemsand challenges:

It turns out that there is no easy answer to the question of “whereis the inner border of the heart?” First of all, there are two pieces ofmuscles coming out of the inner wall called papillary muscles which



connect through strings called chordae tendineae to the mitral valveleaflets.Although these muscles may turn up in a 2D image (Fig. 1(B)first and last image on the lower row), the border tracing shouldcut them at their roots and include them in the blood pool as rec-ommended by the American Society of Echocardiography,4 becauseotherwise the aforementioned volume estimation formula will sig-nificantly underestimate the blood pool volume.2 The challenge isthat papillary muscles, when they do appear in the image, appearin different sizes, shapes, and orientations. No simple rules can dealwith the complexity and variability. Traditional edge or gradient-based segmentation algorithms would not work either.

Secondly, the inner wall of the LV is not smooth at all: thereare many small “bumps” and “strings” and “folds” and “holes”… ,and all of these change throughout the heart cycle. One school ofthoughts is that when the LV is fully expanded (at the end-diastolicphase), the small bumps separate and folds unfold. Therefore, theborder tracing should cut through the roots of those bumps in thesame way we cut the papillary muscles. Whereas when the LV fullycontracts (at the end-systolic phase), all bumps and strings collapseand squeeze out the blood in between. Then, the border tracingshould go through the tip of these bumps and folds (i.e. follow thestrongest edge in the image) to ensure more accurate estimation ofthe blood pool volume.

If a computer program cannot “see” such subtleties or “mimic”with consistency such level of sophisticated reasoning, it will not helpreducing human bias or variabilities. In other words, the computerprogram must have some level of “intelligence” built in. The trickis, of course, to perform “intelligently” in a robust and generalizablefashion.

29.4 EXISTING SOLUTIONS: FROM SIMPLETHRESHOLDING TO OPTIMIZATIONAND POPULATION MODELS

In this section, we will discuss several image processing and patternrecognition tools for shape detection, segmentation and tracking,with application in echocardiography.5



29.4.1 Thresholding and Edge Detection

Early attempts on automatic or semiautomatic (e.g. user clicks atthe center of the left ventricle) chamber segmentation in echocar-diography used simple thresholding and edge detection methods(e.g. Ref. 6, direct thresholding of the RF signal). The advantage isfast computation and intuitive interpretation. But there are seriouslimitations. They cannot correctly handle dropout regions, artifacts,or tissues inside the ventricle such as trabeculations and papillarymuscles; and the resulting contour is noisy and highly dependent onimage quality and gain settings.Asurvey of many earlier approachesis available in Ref. 7.

29.4.2 Energy Minimization and Optimization

There have been much research on image segmentation that involvesan optimization process, that balances multiple factors and con-straints at once. The most notable methods include snakes or activecontours models,8,9 level sets formulations,10 and graph theoreticapproaches.11,12

In graph theoretic approach, an image is modeled as a graph,where each pixel becomes a vertex and neighboring pixels are con-nected by edges. The edge weight w encodes similarity — for exam-ple, one could make w as a function of the image intensity differencenormalized by standard deviation. Using this graph model, seg-menting an image becomes a cut of the graph into two sets of ver-tices, while minimizing some normalized edge weights in the cut.11

Both level set and graph cut approaches optimize a global objec-tive function that depends on local image information (e.g. gradient)and some global constraints. Convergence at a local instead of globalminimum is among the most common pitfalls. Location-dependentshape and appearance priors are not intrinsically incorporated andare difficult to enforce. For example, prior knowledge such as thatthe papillary muscle might be present in the image at a certain loca-tion, or might not, is difficult to incorporate into these frameworks.There are attempts to incorporate global shape models into bothmethods,10,13 but they are still preliminary and restrictive.14 In the



cases where an active shape model is used, the inherent drawbackof such a model also applies (see next section).

29.4.3 Model-Based Methods

To take into account complex shape (e.g. of the LV) and appear-ance priors (e.g. of papillary muscles), one has to incorporate suchconstraints into the border detection process. The easiest and mostinfluential shape and appearance models assume Gaussian distri-bution in a reduced-dimensional space, called active shape modeland active appearance model.15−18

Active shape model can be better understood with the concep-tual separation of shape and preshape.19 In the 2D case, we are con-cerned with sets of k labeled points in a 2D Euclidean space wherek ≥ 2, and a set of invariant transforms. A set of k points will becalled a preshape. Any two preshapes will be regarded as having thesame shape if either of them can be transformed into the other. Witha common reference, the assemblage of all possible shapes forms theshape space.

Assuming all LVs in the shape space follow a Gaussian distribu-tion, one can perform PCA(principal component analysis) to find anorthogonal linear subspace, in which an average shape and selected“eigen-shapes” can faithfully represent the population. If all LVs arewarped to the average shape, PCA can be applied on the warpedimages to learn the eigen-images (much the same way like the learn-ing of the eigen-faces20). This technique can be extended to 2D + timeor 3D.21 For a new case, the segmentation is typically done by over-laying a model on top of the image and iteratively morph the modelto the image according to allowable transformations in both shapeand appearance.

The learned model can be used to “regularize” noisy mea-surements as well. For example, during detection or tracking ofthe border, the measured contour will in general depart from themodel subspace due to noise. A transformation T (from preshape toshape space), followed by an orthogonal projection (into the modelspace), followed by the inverse transform T−1, will do the trick.



However, if the measurement error along the border is heteroscedas-tic (both inhomogeneous and anisotropic — in echocardiography,more often than not this is the case!), both the transformation andthe projection steps should be revised to optimally exploit the noisecharacteristics.22,23

Because of the strong global shape constraint, influence fromlocal artifacts and signal dropout can be suppressed in many cases.The appearance model can incorporate more complex patterns thanedges. Another attractive feature of this method is that as long asthe iteration converges to the global minimum, the result alwayslooks good (just the way a LV should look like) and smooth. Thedrawbacks of these models include the limitation of the global linearassumption which is often violated in real world scenarios. Althoughnormal hearts all look alike, each diseased heart is diseased in its ownway. For example, small aneurysm (small bulge on the boundary)will always be ignored if not present in the training data or if notcaptured by the eigen-shape. Since aneurysm can appear anywherein the heart, it is very difficult, if not impossible, to efficiently model itlinearly. This observation applies to other disease conditions as well,although maybe to a lesser degree. Given that in the clinical worldpeople are much more interested in detecting and characterizingdiseases, this limitation can prove fatal.

29.5 A NEW PARADIGM: LEARNING A DEFORMABLESEGMENTATION

If parametric models (e.g. Gaussian) become too restrictive, can welearn directly from examples in a non-parametric way? This sectiondiscusses a framework to learn a deformable segmentation directly fromexamples.

This framework is rather “unconventional” because at no pointin the whole process do we do thresholding or edge detection orpixel clustering, nor perform any contour evolution (as in snake oractive contour), nor optimize any objective or energy functions. It isa pure learning-from-example, non-parametric approach. An active



shape model is used at a coarse level only as a loose constraint — it isloose in the sense that the final result will not reside in the subspacedefined by the shape model.

The complete segmentation of the LV endocardial borders isachieved in three steps: The first step is to localize the LV. The sec-ond step is to find the correct shape for it. The third step propagatescontours in time.

In the first step, the problem of localizing the LV is cast as a detec-tion problem solved using a classification approach.24 The differencehere is that the object is highly deformable. Therefore, the properdefinition, alignment, and cropping of the “object” are critical inensuring learnability and robust performance.

In the second step, given the detected location for LV, we inferthe shape of endocardial border by learning the correlation betweenappearance and shape in the training set. We discuss three alterna-tive ways to achieve this goal.

The last step is concerned with motion tracking to achieve fullsegmentation of all frames and temporal analysis of the LV function,including the estimation of ejection fraction, volume-time curve, andglobal and regional cardiac wall motion.

29.5.1 Learning to Localize the Left Ventricle

Given a sufficient number of annotated images of an object, i.e. withthe object cropped out in each image, one could formulate the prob-lem of object detection as a binary classification problem. The twoclasses would be LV (positive) and non-LV (negative) image patches,where the non-LV patches can be extracted from the same set ofimages, cropped in a similar way as the true LV patch but at adifferent location, rotation, scale, or aspect ratio. At run-time, thealgorithm will scan the image and send image patches to the classi-fier. The patch with the highest score will be assigned as containingthe LV.

The classifier can be implemented efficiently using the AdaBoostalgorithm.24,25 Due to the highly deformable nature of the LV, a keyissue is the alignment of the positive examples, i.e. how to transform



Fig. 4. The trade-off between shape estimation error and search space dimension.

all the LVs into a canonical reference patch, so that: (1) we achievemaximal similarity (thus learnability) across LVs; (2) at run time,we can efficiently “reverse” the transformations to find the LV. Themore sophisticated is the transform, the easier for the classifier tolearn — with the extreme case of non-rigidly morph all LVs into onetemplate — but then at run time, we have to try all possible morphsin order to reach the template which is too expensive. This trade-offis illustrated in Fig. 4.

To balance learnability and run-time speed, we detect in this steponly rigid transformations (plus several discrete aspect ratios), andleave the non-rigid learning to the second step.

Another important issue is the varying uncertainties along thecontour and their influence on the alignment process. Intuitively,since LV contours have higher uncertainty at the apex and lateralwall in general, we align the LVs in a way that relies more on otherparts of LV. This can be done with a weighted Procrustes alignmentprocess with the weights, W , reflecting feature stability and local-ization confidence. An iterative process is employed to minimize aweighted least squares criterion. More detailed analysis and treat-ments can be found in Refs. 22, 23 and 26.

29.5.2 Learning Local Deformations

In the second step, we perform the actual segmentation of the LV.Instead of using the traditional segmentation frameworks (Sec. 29.4),we reformulate the deformable segmentation problem as a mapping



problem, i.e. learning a direct mapping from the image space to the shapespace. Normally, this direct formulation, would be too ambitious andsurely intractable due to the infinite possibilities of global and localtransformations. However, in our case, the first step (Sec. 29.5.1)has removed all the global transformations and what is left is onlyto learn the local deformations. For this task, we propose threealternative solutions, all of which are based on non-parametriclearning.

29.5.2.1 A CBIR approach to shape inference

To avoid the limitations of Gaussian or Gaussian mixture models, wemaintain a non-parametric, sample-based representation of the jointdistribution of image appearance and associated shape. The shapeinference step is then carried out in a similar way as in content-based image retrieval (CBIR): we find k most similar images in thetraining set and compose a new shape based on the k associatedshapes.

The trick is in defining the similarity (or distance) measure orfeatures in the image space. We could use the original images ortheir eigen-representation20 as the features to find similar images;or the weak classifier features selected by the AdaBoosting processfor distinguishing the target shape from the background. But Ideally,we need to select those features that will best discriminate amongdifferent shapes. To find such features, we first cluster all trainingshape into C clusters. Then we select features that maximize theclass separability criterion: S = trace(S−1

w Sb), where Sw is the sum ofwithin-class scatter matrices and Sb the between-class scatter matrix.We apply a forward sequential feature selection approach, and ateach step a feature yielding the largest increase in the separationcriterion is included into the subset of selected features. The selectionprocess continues until there is no significant increase.

Then the distance measure in the image space is defined by thediscriminating metric distance:

d(f1, f2) = (f1 − f2)��(f1 − f2), (1)



where f1,2 are feature vectors for two image patches and:

� = S−1/2w

(S−1/2

w SbS−1/2w + εI

)S−1/2

w = S−1/2w (S∗

b + εI)S−1/2w , (2)

which spheres the space with respect to Sw and then it stretchesthe space in the null-space of S∗

b . The parameter ε rounds theneighborhood.27 With this distance metric, we obtain the inferenceresult as a kernel-weighted average of the shapes from the k-nearestneighbors, where the kernel weighting is inversely related to thedistance.

Experiments clearly show the advantage of the selected fea-tures over naive alternatives such as direct use of pixel intensity.26

Figures 5(A–C) compares the automatic segmentation results to con-tours drawn by an expert. The difficulties of the problem are evidentin Fig. 5(A) where the input images are affected by strong noise,unclear border definition, signal dropouts, and imaging artifacts.Two more segmentation results are shown in Fig. 5(D). Note thepoor quality of the image and large variations in appearance of theleft ventricle.

Fig. 5. Left ventricle endocardial border detection results: (A) input images;(B) border detection results; (C) expert drawn contours; (D) detection results intwo other cases.



As a by-product, after the segmentation, we could also showthe retrieved similar image patches from the annotated trainingdatabase. The doctor can then use these similar cases to aid the diag-nosis of the case in question.

29.5.2.2 Learning a ranking over deformations

If we take a seemingly bold assumption that all possible local shapedeformations can be enumerated, we can arrive at an alternativeway of learning such deformations by ranking. The resulting methodassumes no prior model on shape or appearance, thus can learnlocal non-Gaussian variations in both shape and appearance (thinkpapillary muscles).28

The idea is to learn a ranking of deformations as a function ofimage features extracted from each warped image. Assuming thatthere are a total of N warping templates, in the training stage, wewarp each training data N times, and rank the resulting warpedimages according to their distances in the shape space to the meanshape. The RankBoost29 algorithm is then employed to learn thisranking function based on image features.

At testing time, the input image patch is warped N times; andwith features extracted from each of these warped images, thetrained ranker then ranks these warped images. The top k candi-dates are then combined — based on kernel weighting as before —and then back-warped as the shape estimate for the input image.

The advantages of this approach as compared to the previousare: (1) it eliminated the need for a forceful shape clustering step;and (2) the feature selection step is embedded in the boosting step,which is potentially better than the naive class separation criterion.The drawbacks include the assumption that shape deformation canbe enumerated, and the growing computation as N grows.

29.5.2.3 Learning a regression function from appearance to shape

Another alternative to directly bridge the image space and the shapespace is through boosting regression.30



Since the dimensionality of the shape space is an exponen-tial term in the feature selection step in boosting regression, adirect implementation of deformable shape regression would beintractable in our case. The key idea here is an incremental feature selec-tion scheme where a multidimensional regression problem is brokeninto multiple dependent one-dimensional regression problem.

The obvious advantage of this algorithm is its speed: becausewhat is learned is a regression function, the run-time computationis minimum — there is no distance computation with the trainingset, nor any image warping required. However, one limitation isthat we probably need a much larger training set in order to learnsuch a powerful regression function; and more thorough analysisand testing are still required.

29.5.3 A Coarse-to-Fine Detection Hierarchy

A coarse-to-fine detection hierarchy can dramatically increase theflexibility of an example-based segmentation framework.As an anal-ogy, assuming that we have a total of M face images in a database,and if by averaging every three faces we get a new face, we canget a total of C3

M new face images. Now if we get eyes from oneset of three faces but mouth and nose from another set of threefaces, we can then synthesize (C3

M)2 new face images! This is theintuition behind the coarse-to-fine detection hierarchy, where weiteratively refine the detection and segmentation results of previ-ous sections. Each local refinement step is a complete repetition ofthe two-step process: (1) learning the location of the parts, (2) learn-ing their deformable segmentation. The only difference is that theobject is no longer the LV, but a part of the LV. With the local refine-ment step, the segmentation results become even more adaptableto never-before-seen shapes and local shape variations. This frame-work is illustrated in Fig. 6.

29.5.4 Motion Analysis: Ejection Fraction, Volume-TimeCurve, and Wall Motion

Echocardiography is a set of images taken in real time to capture thecardiac anatomy as well as motion. So far we have only discussed



Fig. 6. An illustration of the coarse-to-fine detection and segmentation scheme:(A) global shape detection and segmentation; (B) localization of parts; (C) segmen-tation of parts; (D) refined and fused global shape segmentation, with similar casesaccording to both global and local measures.

the analysis and segmentation of single images. Nevertheless, theanalysis across time is very important and can reveal import infor-mation regarding the heart function, such as it global function interms of ejection fraction,2 or local function as measured by regionalwall motion which can be used for early diagnosis of cardiovasculardiseases.

Our motion tracking algorithm23 focuses on the estimation andoptimal exploitation of the heteroscedastic noise along the LV bor-der [Fig. 7(A)]. The idea is to obtain a most reasonable estimateof the motion even when local information is completely miss-ing (e.g. dropouts) or partially missing (e.g. due to the apertureproblem31,32). We achieve this goal by fusing information acrosstime, space, and a prior knowledge. The resulting robust track-ing of the endocardial border can be visualized in different waysthat are suitable for the detection of wall motion abnormalities[Figs. 7(B, C)].



Fig. 7. Left ventricle endocardial border motion analysis: (A) Ellipses showinganisotropic and inhomogeneous uncertainties for feature localization and motionestimation; (B) a motion trajectory view showing a normal heart with strong pump-ing action; (C) a motion trajectory view showing hypokinesis — reduced pumpingcapability of the heart.

29.6 CONCLUSION

There is a clear need for computer assisted processing and interpre-tation of cardiac ultrasound images. With the exponential growth incomputer technology — both hardware and software, and the everadvancement of pattern recognition and machine learning theoriesand methodologies, computer and computational tools will have agreat potential to help the clinicians and society at large in improv-ing the heart health of the overall population. However, with thenew 3D transducer technology becoming more mature, and moreaccepted in clinical practice, the challenges are growing as well. So,have you got the heart for it?

References

1. Reinhardt E, The atlas of heart disease and stroke, UN Chronicle, 2005.2. Oh JK, Seward JB, Tajik AJ, The Echo Manual, Lippincott Williams &

Wilkins, Philadelphia, 1999.3. Feigenbaum H, Armstrong WF, Ryan T, Feigenbaum’s Echocardiogra-

phy, 6th ed., Lippincott Williams & Wilkins, 2005.4. Schiller NB, Shah PM, Crawford M, et al., Recommendations for quan-

tification of the left ventricle by two-dimensional echocardiography,J Am Soc Echocardiography 2: 358–367, 1989.



5. Bosch JG, Automated contour detection in echocardiographic images,Doctoral thesis, Leiden University, 2006.

6. Perez JE, Waggoner AD, Barzilai B Jr, et al., On-line assessment ofventricular function by automatic boundary detection and ultrasonicbackscatter imaging, J Am Coll Cardiol 19: 313–320, 1992.

7. Sher DB, Revankar S, Rosenthal S, Computer methods in quantitationof cardiac wall parameters from two-dimensional echocardiograms: Asurvey, Int H Cardiac Imaging 8: 11–26, 1992.

8. Kass M, Witkin A, Terzopoulos D, Snakes: Active contour models, IJCV1: 321–331, 1989.

9. Chalana V, Linker DT, Haynor DR, Kim Y, A multiple active con-tour model for cardiac boundary detection on echocardiographicsequences, IEEE Trans Medical Imaging 15: 290–298, 1996.

10. Chen Y, Thiruvenkadam S, Tagare HD, et al., On the incorporation ofshape priors into geometric active contours, in Proc IEEE Workshop onVariational and Level Set Methods in Computer Vision, pp. 145–152, 2001.

11. Shi J, Malik J, Normalized cuts and image segmentation, IEEE TransPattern Anal Machine Intell 22: 888–905, 2000.

12. Boykov Y, Jolly MP, Interactive organ segmentation using graph cuts,in Proc Medical Image Computing and Computer-Assisted Intervention,pp. 276–286, 2000.

13. Leventon M, Grimson E, Faugeras O, Statistical shape influence ingeodesic active contours, in Proc IEEE Conf on Computer Vision and Pat-tern Recognition, Hilton Head, SC, 2000.

14. Slabaugh G, Unal G, Graph cuts segmentation using an elliptical shapeprior, in Proc IEEE Int’l Conf Image Proc.

15. Cootes T, Taylor C,Active shape models–“smart snakes”, in Proc BritishMachine Vision Conference, pp. 266–275, 1992.

16. Bosch JG, Mitchell SC, Lelieveldt PF, Nijland F, et al., Automaticsegmentation of echocardiographic sequences by active appearancemotion models, IEEE Trans Medical Imaging 66: 1374–1383, 2002.

17. Jacob G, Noble J, Behrenbruch C, et al., A shape-space-basedapproach to tracking myocardial borders and quantifying regional left-ventricular function applied in echocardiography, IEEE Trans MedicalImaging 21: 226–238, 2002.

18. Paragios N, Jolly MP, Taron M, Ramaraj R, Active shape models andsegmentation of the left ventricle in echocardiography, in Lecture Notesin Computer Science 3459, pp. 131–142, 2005.

19. Kendall DG, Barden D, Carne TK, Le H, Shape and Shape Theory, JohnWiley & Sons, Ltd., Chichester, 1999.

20. Turk MA, Pentland AP, Face recognition using eigen-face, in Proc IEEEConf on Computer Vision and Pattern Recognition, Hawaii, pp. 586–591,1991.



21. Mitchell S, Bosch JG, Lelieveldt BPF, et al., 3D active appearancemodels: Segmentation of cardiac MR and ultrasound images, IEEETrans Medical Imaging 21: 1167–1178, 2002.

22. Zhou XS, Comaniciu D, Xie B, Cruceanu R, et al., A unified frame-work for uncertainty propagation in automatic shape tracking, in ProcIEEE Conf on Computer Vision and Pattern Recognition, pp. 872–879,Washington, DC, 2004.

23. Zhou XS, Comaniciu D, Gupta A, An information fusion frameworkfor robust shape tracking, PAMI 27: 115–129, 2005.

24. Viola P, Jones M, Rapid object detection using a boosted cascade ofsimple features, in Proc IEEE Conf on Computer Vision and Pattern Recog-nition, Hawaii, 2001.

25. Freund Y, Schapire R, A short introduction to boosting, J Japan Soc forArtif Intel 14: 771–780, 1999.

26. Georgescu B, Zhou XS, Comaniciu D, Gupta A, Database-guided seg-mentation of anatomical structures with complex appearance, in ProcIEEE Conf on Computer Vision and Pattern Recognition, pp. 429–436, SanDiego, CA, 2005.

27. Hastie T, Tibshirani R, Friedman J, The Elements of Statistical Learning,Springer Verlag, 2001.

28. Zheng Y, Zhou XS, BGSZ Comaniciu, D, Example based non-rigidshape detection.

29. Freund Y, Iyer R, Schapire RE, Singer Y, An efficient boosting algorithmfor combining preferences, in Int’l Conf Machine Learning, pp. 170–178,1998.

30. Zhou S, Georgescu B, Zhou XS, Comaniciu D, Image based regressionusing boosting method, in Proc Intl Conf on Computer Vision, pp. 541–548Beijing, China, 2005.

31. Kanazawa Y, Kanatani K, Do we really have to consider covariancematrices for image features?, in Proc Intl Conf on Computer Vision,pp. 586–591, Vancouver, Canada, 2001.

32. Irani M,Anandan P, Factorization with uncertainty, in Proc 6th EuropeanConf on Computer Vision, pp. 539–553, Dublin, Ireland, 2000.


CHAPTER 30

Current and Future Trendsin Radiation Therapy

Yulin Song and Guang Li

Radiation oncology, commonly known as radiation therapy (RT), is a spe-cialized application of medical imaging used to treat cancers as well asseveral benign diseases, such as non-cancerous tumors, heart disorders,and thyroid problems. It is estimated that nearly two thirds of all cancerpatients will receive RT as part of their treatments. In terms of deliverytechnique, RT can be divided into external beam radiation therapy (EBRT)and brachytherapy. To maximize patient survival, it is now more commonto combine RT with surgery, chemotherapy, and/or hormone therapy. Thischapter discusses some recent developments and future trends in RT.

30.1 INTRODUCTION

Radiation oncology uses ionizing radiation to destroy malignanttumor cells as the means of local tumor control; in the meanwhileit aims to spare surrounding critical organs and normal tissues.1,2

Its objective, therefore, is different from radiology, which uses radi-ation for medical imaging and diagnosis. Utilizing radiation to killcancer cells dates back about a century soon after the X-ray wasdiscovered in 1895. It has been proven to be an effective treatmentfor both invasive and noninvasive cancers.3,4 In certain situations,such as palliative treatment, radiation therapy (RT) may be the onlyeffective treatment option. Radiation is also used to treat severalbenign diseases, such as noncancerous tumors, heart disorders, andthyroid problems. It is estimated that nearly two thirds of all cancer

745


746 Yulin Song and Guang Li

patients will receive RT as part of their treatments. Recent statisticsreleased by the American Association for Therapeutic Radiologyand Oncology (ASTRO) show that nearly one million patients weretreated with radiation therapy in 2004.5 Cancer patients made about23.4 million RT treatment visits to 2 010 hospitals and freestandingradiation therapy centers in the same time period in the USA. Thestatistics also show that the most common types of cancer in theUSA are breast cancer, prostate cancer, and lung cancer, accountingfor 56% of all cancers treated with RT.

In terms of treatment intent, radiation therapy can be broadlydivided into definitive (or curative) and palliative treatments. It isemployed as palliative treatment in cases where cure is not possible.The primary goal is to achieve local disease control. The secondarygoal is to relieve symptoms and improve survival and quality oflife. The precise treatment intent (curative, adjuvant, neoadjuvant,therapeutic, or palliative) and optimal treatment approach (exter-nal beam, brachytherapy, or combination of both) depend on manyfactors, including the tumor type, stage, histological grade, loca-tion, as well as the patient’s age and existing medical conditions.In any case, RT can be used as either primary or adjuvant ther-apy. To maximize patient survival, it is now more common tocombine RT with surgery, chemotherapy, and/or hormone ther-apy. In terms of delivery technique, RT can be divided into exter-nal beam radiation therapy (EBRT) and brachytherapy, as shownin Fig. 1. The contemporary forms of EBRT include 3D confor-mal radiation therapy (3DCRT),6 intensity modulated radiationtherapy (IMRT),7 image-guided radiation therapy (IGRT),8 adap-tive radiation therapy (ART),9,10 and more recently, 4D radiationtherapy (4DRT).

In EBRT, the radiation oncologist prescribes radiation dose anddelineates the target volume and the organs at risk (OAR) on treat-ment planning CT images. The medical physicist then designs andcomputes the treatment plan using a radiation treatment plan-ning system (TPS). Depending on the type and complexity of thetreatment, the plan can be computed using either forward or inversetreatment planning.11 This is to ensure that the final treatment plan


Current and Future Trends in Radiation Therapy 747

Abbreviations:

RT: Radiation therapy

EBRT: External beam radiation therapy

3DCRT: 3-dimentional radiation therapy

IMRT: Intensity modulated radiation therapy

MERT: Modulated electron radiation therapy

IGRT: Image-guided radiation therapy

ART: Adaptive radiation therapy

4DRT: 4-dimentional radiation therapy

SRS: Stereotactic radiosurgery

SBRT: Stereotactic body radiation therapy

LDR: Low dose rate brachytherapy

HDR: High dose rate brachytherapy

Fig. 1. Classification of radiation therapy by means of delivery.

provides not only adequate dose coverage to the target, but alsonecessary protection to the nearby critical organs. Approved andsigned treatment plan is delivered on a computer-controlled medi-cal linear accelerator (Linac),12 from which high energy X-ray beams(4 MeV–25 MeV) are delivered to the target through a beam-shapingdevice called multileaf collimator (MLC). The shape of radiationfield matches the projection of the target in the beam’s eye view(BEV). The treatment fields are delivered sequentially using a recordand verify (R&V) system. Using MLC, the beam intensity can be



modulated in space to deliver a desired 3D dose distribution. Toreduce the treatment time, most treatment planning systems use10 intensity levels per bixel or beamlet. New generation of Linacis equipped with image-guidance devices, such as the amorphoussilicon-based electronic portal imaging device (EPID), kV flat-paneldigital imager, and cone beam CT (CBCT). Using these on site imag-ing devices, it is possible to minimize the uncertainty in initialpatient setup and to reassess the target location during treatmentfor compensating patient respiratory motion. This provides a poten-tial opportunity for dose escalation to the target. It has been shownthat extracranial stereotactic body radiation therapy (SBRT) withhypofractionated treatment schedule provides higher tumor localcontrol rate, resulting from the ablative dose to the tumor at hightreatment accuracy with image guidance.

In brachytherapy, as in the case of EBRT, the radiation oncol-ogist delineates the target volume on CT or ultrasound images.A treatment plan is computed to determine the optimal number ofradioactive seeds, their locations and strengths, and the treatmenttime, in order to deliver the prescribed dose and/or dose distri-bution. Depending on the dose rate, brachytherapy can be dividedinto low dose rate (LDR) and high dose rate (HDR) brachytherapy.Unlike EBRT, brachytherapy is a one-day procedure in most cases.During the procedure, radioactive seeds are implanted inside andat the peripheral region of the tumor either permanently or tem-porarily. Comparing with EBRT, brachytherapy has several distinctadvantages. The major advantage is that it can deliver sufficientlyhigh dose to the target and very low dose to the normal tissues. Iflow-energy γ-emitters or high-energy β-emitters are used, the doseto the surrounding normal tissues decreases very rapidly with dis-tance. Therefore, the radiation can be essentially confined within afew millimeters of the target.

The current rapid advances in radiation therapy primarily resultfrom the use of latest diagnostic imaging technologies in bothradiation treatment planning and radiation treatment delivery, inaddition to the development of therapeutic techniques, including



IMRT, SBRT, proton therapy, and 4DRT. The incorporation of mul-tiple imaging modalities, including PET/CT, MRI, magnetic reso-nance spectroscopic imaging (MRSI), 4DCT, multislice CT (MSCT),and cone beam CT (CBCT), into radiation therapy has ensured moreaccurate target volume delineation, target localization, and treat-ment delivery. IGRT and IMRT are today’s standard for state-of-the-art radiation treatment.7,8,13,14 It is fair to say that medical imagingis the cornerstone of the modern radiation therapy. In this chapter,we present the current status of radiation therapy, as well as sev-eral important trends in radiation therapy, including SBRT, protontherapy, and 4DRT.

30.2 GENERAL PROCESS OF RADIATION THERAPY

Radiation therapy is a complex and time consuming process. Itinvolves many different specialties. Thus, a successful radiationtreatment requires a careful planning and team effort. In the orderof workflow, it consists of (1) patient positioning; (2) patient immo-bilization; (3) CT simulation; (4) image fusion; (5) target volumedelineation; (6) treatment planning; (7) quality assurance; and(8) treatment delivery. Depending on the complexity of the tech-nique employed, a specific treatment may not use all these steps.Generally, an RT treatment contains three essential components:patient simulation (steps 1–3), treatment planning (steps 4–6), andtreatment delivery (steps 1–2 and 7–8). The major tools used in theseRT components are a CT simulator, a treatment planning system(TPS), and a linear accelerator (Linac) unit.

30.2.1 Patient Positioning

Radiation therapy begins with CT simulation, in which patient mustbe positioned in a way that the disease site can be easily accessedand treated using preferable radiation beams. Thus, a good patientposition must meet the following criteria. First, it must avoid irradi-ation of critical organs and extensive normal tissues. For example,



in treating breast cancer, a prone position is preferable to a supineposition for patients with large or pendulous breasts or history oftobacco use in order to maximize the sparing of the lungs and theheart and to minimize hot spots.4 Secondly, the position must becomfortable enough for patient to lie still for the entire treatmentsession. In particular, IMRT usually uses a large number of treat-ment fields so it may take 15 to 20 minutes to deliver a plan. Moreimportantly, a small patient movement during the treatment couldresult in overdosing the critical organs and underdosing the target.Thus, it is crucial for patient to remain comfortable for the durationof treatment. Thirdly, the position must be easy to setup. In a busyclinical department, each patient is only scheduled for 10 minutes–15 minutes of treatment. Lastly, the patient position must be repro-ducible for each fraction of treatment. Orthogonal lasers in bothsimulation and treatment rooms are routinely used to align themarks on patient skin for reliable and reproducible positioning. Mostrecently, the patient setup can be further improved using an on siteimager in an IGRT procedure.

30.2.2 Patient Immobilization

Once a suitable patient position has been decided, it needs to befixed to the treatment couch or the supporting board using a cus-tomly made immobilization device. The goal is to minimize volun-tary patient movement during the course of treatment. However,it should be pointed out that such a patient immobilization devicedoes not prevent internal organ motion with respect to the rela-tively fixed skeletal system. Various immobilization systems havebeen developed over the years,15 including Styrofoam cast, thermo-plastic materials, vacuum bean bags, dental molds and bite blocksetc. A good immobilization device should be light, rigid, radiotrans-parent, and easy to fabricate. In addition, it should not induce anyCT artifacts. Sometimes, the effective thickness of an immobiliza-tion device is comparable to that of a bolus, so it may be necessaryto make a beam entrance port on the immobilization device to reducethe skin dose.



30.2.3 CT Simulation

CT simulation consists of two separate processes: CT scanning andvirtual simulation. The planning CT image is acquired by scanningthe patient in the immobilized treatment position using a CT sim-ulator (Fig. 2), which contains: (1) a large bore CT scanner withflat table top; (2) virtual simulation software; and (3) a laser posi-tioning system for marking the beam portals on the patient’s skin.A large-bore (85 cm) scanner facilitates patient positioning, fits largepatient with immobilization devices, and produces larger field ofview (FOV) of 60 cm without dramatically sacrificing image quality.All CT simulators have a flat table top, mimicking that of a medicallinear accelerator to improve patient positioning reproducibility. Inaddition, because the table cannot move laterally, the sagittal laser is

Fig. 2. Aphoto of Philips Brilliance Big Bore CT scanner (Philips Medical Systems,Cleveland, OH, USA). The scanner has an 85 cm bore, providing a maximum of60 cm field of view (FOV). The detector width (collimation) is 24 mm (16 × 1.5).



movable laterally away from the mid longitudinal axis to mark refer-ence points. In addition to skin marks, three radio-opaque markers(1 mm lead beads) aligned with lasers should be used to provide aninternal reference point in the patient CT image. The 3D CT imagewill also be used for tissue inhomogeneous correction in radiationdose calculation using the TPS.

The second process in CT simulation is called virtual simula-tion, which produces digitally reconstructed radiographs (DRR),mimicking the radiographic films taken by a physical simulator.Figure 3 shows a typical screen of the virtual simulation software.There are three tasks in the virtual simulation using the TPS. Thefirst is to delineate anatomic volumes, including target volumes,critical organs, and other relevant normal tissues. The second isto localize the treatment isocenter, which is usually placed at the

Fig. 3. A representative screen of the Philips virtual simulation software TumorLOC (Philips Medical Systems, Cleveland, OH, USA).



Fig. 4. A DRR projected at a beam angle of 0◦ for a typical prone prostate IMRTplan. The red solid line represents the planning target volume (PTV). The open areais part of the pelvis that needs to be treated. The other areas are blocked by the MLC.

geometrical center of the target. In some cases, however, it is nec-essary to place the isocenter at the edge of the treatment fields tobest match the adjacent fields. The third is to determine treatmentbeam parameters through the beam’s eye view (BEV) using DRRs,including gantry angles, collimator angles, couch angles, field sizes,wedges, and shielding blocks. Figure 4 shows a DRR for a typicalprone prostate IMRT plan.

30.2.4 Image Registration

Image registration or image fusion is a process in which two or moreimages of the same or different imaging modalities are correctly



Fig. 5. A photo of GE Discovery PET/CT scanner (GE Medical Systems, USA).

overlaid according to the underlying anatomical structures. Bothanatomical images, such as CT and MRI, and functional images,such as PET and MRSI, can be employed in radiation treatmentplanning. Particularly, the hybrid PET/CT scanners (Fig. 5) havegreatly improved the accuracy of the image registration. In addi-tion, PET/CT also provides a more accurate attenuation correctionmap for standardized uptake value (SUV) quantification. The fusedimage provides comprehensive views of patient anatomy as well asphysiological, metabolic, and functional activities, providing moreinformation of the lesion for target delineation.

Clinically, image fusion can be performed using several TPStools, including manual fusion by matching three orthogonal pla-nar (3P) views of two volumetric images, or automatic voxel-basedimage registration by maximizing the mutual information of twovolumetric images. However, for all automatic image fusion tools



available in the clinic, the registration result must be verified visu-ally to assure a clinically acceptable result. Very often, the medicalphysicist will make a slight adjustment based on medical knowledgeusing the 3P-based visual fusion method. However, the 3P-basedvisual fusion method has some shortcomings, including large varia-tions from interobservers and even intraobserver at different times,potential global misalignment due to partially visual representationof the volumetric images, and prolonged processing time. Recently,a 3D volumetric image registration method has been reported toprovide an accurate visual-based manual registration method toovercome the shortcomings of the 3P-based fusion method in orderto provide adequate accuracy, which matches the automatic imageregistration.16

Frequently, medical images are deformed due to patient motionand changes over time. Thus deformable image registration meth-ods have been developed, but employed in clinical research only.There are two major issues that have strong impacts on the clinicaluse of deformable registration: (1) prolonged computing time, usu-ally hours; and (2) difficulties to validate the deformable registrationresult.17,18 However, it is a future trend to improve deformable regis-tration and employ it in the clinic since any time dependent motionor change causes image to deform. It is essential to 4DRT, wherecompensating deformable changes is its primary objective.

30.2.5 Target Delineation

The volumes of the target, critical organs, and surrounding nor-mal tissues are delineated by the radiation oncologist based on theCT images or multimodality images, which are acquired to visu-alize the lesion and normal anatomy. Functional images, such asPET images, provide the tumor metabolic activity and have recentlybeen employed in radiation therapy. The rapid advancements in18F-labled fluorodeoxyglucose (18FDG) positron emission tomogra-phy (18FDG-PET) have made 18FDG-PET a valuable noninvasivetool for the characterization and staging of cancer, detection of dis-tant metastases, and diagnosis of recurrent disease. Based on the



3D images, the treatment target is contoured as the gross tumorvolume (GTV), which is expanded to the clinical target volume(CTV) to include microscopic tumor extension. The CTV is furtherenlarged to become the planning target volume (PTV) by account-ing for the patient setup error and organ motion (Fig. 6). The motionenvelope of the GTV forms the internal target volume (ITV), whichis applied in most RTs that intend to compensate patient motion.Figure 7 shows a GTV delineated on a PET image for a lung cancercase. The GTV automatically appears on the corresponding treat-ment planning CT image.

GTV

ITV

CTV

PTV

Fig. 6. Schematic illustration of GTV, ITV, CTV, and PTV. Margins are not drawnin scale.

Fig. 7. GTV delineated on a PET image for a lung cancer case. The red solid linerepresents the PET-GTV. The image on the left is the treatment planning CT andthe image on the right is PET. The PET image was acquired with a GE DiscoveryPET/CT scanner.



In addition, the surrounding critical organs and normal tissuesshould also be contoured, so that they can be spared during thetreatment planning. For instance, the critical organs in a head andneck cancer case may include the spinal cord, parotid glands, eyes(lens), optical chiasm and nerves, oral cavity, and temporomandibu-lar joints (TMJ) etc. These critical organs can be contoured automati-cally using special image segmentation algorithms that have recentlybecome available in some commercial TPS.19 Although the auto-matically generated contours are never perfect, they have greatlyfacilitated the planning process by reducing the workload requiredfor manual contouring. Figure 8 shows the structure definitions ofa right lung cancer case for IMRT inverse treatment planning. Byassigning proper dose constraints to INRIND and OUTRIND, these

Fig. 8. Structure definitions of a right lung cancer case for IMRT inverse treatmentplanning. The tuning structures INRIND and OUTRING are used to fine tune thedose outside the PET-PTV and thus, to eliminate the undesirable hot spots in theseareas.



two tuning structures can be used to fine tune the dose outside thePET-PTV and thus, to eliminate the undesirable hot spots in theseareas.

30.2.6 Treatment Planning

With the target delineated and critical organs contoured, the treat-ment can be planned with a radiation dose prescription by aradiation oncologist. There are two different approaches: forwardplanning and inverse planning. A forward planning starts from set-ting up radiation fields, including the number of beams, their direc-tions, shapes, and intensity weights, based on the patient’s anatomyand the planner’s knowledge/experience. Different initial beamsetups produce different plans. It then proceeds to the calculation ofradiation dose, isodose lines, and dose-volume histograms (DVH).Based on the dose distribution, including the coverage of the targetand the sparing of critical organs and normal tissues, especially crit-ical organs, the plan can be evaluated and refined. This process mayiterate many times manually to find a clinically acceptable plan.

An inverse planning, after the number of beams and their ori-entations and energies are selected, starts from specifying doseconstraints, including the maximum and minimum acceptable tar-get doses, the maximum tolerable dose to certain percentage ofa critical organ (desired DVH features), and the weighting fac-tors (penalty factors) as the priority of concerns for each struc-ture. Using these predetermined dose and dose-volume constraints,an inverse planning program can automatically generate a planthat satisfies target coverage and normal tissue sparing by mini-mizing a quadratic cost function. After the beam fluence map iscalculated for each field, an MLC leaf sequence file is generatedand will be used to control the MLC for the delivery of this fieldthrough the R&V system. In general, an IMRT plan may havemore fields than a 3DCRT plan and, thus, takes longer time todeliver. Figure 9 shows the isodose distribution for a lung cancerIMRT plan.



Fig. 9. The isodose distribution in a sagittal plane for a lung cancer IMRT plan.The brown solid line represents 100% isodose line. The dotted red and brown linesare the PTV and ITV, respectively.

30.2.7 Pretreatment Quality Assurance (QA)

Prior to treatment delivery, certain QA procedure should be per-formed. There are two kinds of QA procedures. One is the pretreat-ment patient setup verification. This is to ensure that the patient isin the same position on the treatment couch as on the CT simulatortable. The other is the MLC leaf sequence verification. The goal isto ensure that the MLC leaf motion file can deliver the same dosedistribution as planned.



For patient setup verification, it can be performed either usingportal films or using an electronic portal imaging device (EPID). Forlocalization purpose, a high speed film of diagnostic quality is oftenused with intensifying screen. However, it takes long time for filmprocessing and so often the verification is done post-treatment. TheEPID is a more preferable approach because it provides an instantimage, which can be reviewed and signed digitally by the radia-tion oncologist prior to the treatment delivery. The EPID image foreach treatment field is compared with the corresponding treatmentplanning DRR for the setup verification. For both film and EPID,they produce MeV X-ray images, which have low spatial resolution,poor contrast, and low signal-to-noise ratio (SNR) and, therefore, itis difficult to perform visual side-by-side comparison by the radia-tion oncologist based on bony landmarks. The verification has lim-ited accuracy and excludes soft tissue motion. Recently, more on-siteimaging devices have become available in the treatment room to per-form an IGRT procedure. Two orthogonal kV X-ray images or even aCBCT image can be acquired on the treatment site. These images canthen be automatically registered online with the planning CT image.The alignment of these images yields a set of translational position-ing parameters, which indicate how much the treatment couch mustbe shifted in order to align the treatment position to the simulationposition, ensuring a treatment delivery as planned.

Prior to an IMRT treatment, the planned leaf sequences must beverified to ensure that they can produce the planned beam fluencemaps. The dosimetric measurements can be done using either a cubicor a cylindrical phantom. Film dosimetry and ion chamber measure-ments are performed to provide a relative dose distribution and anabsolute point dose at a certain depth. The plan verification can beeither performed for each individual field or for a composite fieldthat integrates all fields. The exposed film can be digitized usinga densitometer to produce the relative dose distribution, which isfurther converted to a dose distribution by applying a calibrationcurve or an absolute point dose. The measured dose distribution isthen compared with the planned dose distribution. The differenceshould be smaller than 3% in dose in low dose gradient regions and



Fig. 10. Comparison of planned (left) and measured (right) dose distributions foran IMRT head and neck field. The dosimetry was performed using MapCheck, acommercial IMRT QA device.

less than 2 mm in distance at high dose gradient regions. Figure 10shows a comparison of planned and measured dose distributionsfor an IMRT head and neck field.

Recently, 3D dosimeters have been developed and applied forclinical dose verification.20−22 Anthropomorphic phantom contain-ing chromic molecules in gelatin matrix can absorb radiation dose toinitiate polymerization with physical feature changes and the chem-ical reaction is proportional to the radiation dose deposited in anygiven point or voxel inside the phantom. Such 3D representation ofdose distribution can be imaged using 3D color densitometer imag-ing or MRI imaging since the polymerization also changes MR relax-ation time, which is semilinearly proportional to the absorbed doseinside the 3D dosimeter.

30.2.8 Treatment Delivery

In order to deliver the accurate dose as planned, the Linac unitmust be calibrated precisely (one monitor unit (MU) = 1.000 cGy,for instance) and maintained through regular QA procedures rec-ommended by the American Association of Physicist in Medicine(AAPM). Before therapists can press the button to treat the patient,the verified and signed treatment plan, monitor unit calculation, andassociated MLC leaf sequences must be transferred to the treatment



console via the R&V system. Most EBRT photon plans employ aSAD (source-axis distance) treatment setup, in which the tumor isplaced at the machine’s isocenter, so that patient’s position remainsunchanged for different treatment fields. For multiple lesion sites,the couch position can be shifted so that the beam isocenter relatedto the patient anatomy is shifted from one site to another. For elec-tron beam setup, the treatment plan always uses SSD technique(source-skin distance). In either case, patient setup must be preciselydone, as discussed above.

Different treatment delivery can be done using different Linacunits, such as helical tomotherapy unit and robotic cyberknife unit.These two newer Linac units integrate IGRT capability and the treat-ment delivery is more IGRT oriented. The tomotherapy delivers ahelical arc beam focusing on the target, so the dose to the surround-ing normal tissues is “diluted” because of the large number of beamsused.23 The cyberknife, which takes frequent 2D X-ray images toguide the robotic Linac arm to aim at the correct location, can com-pensate the patient motion to certain degree.24

Aregular radiation treatment is scheduled for multiple fractions,such as 20–30 fractions. Between each treatment, normal tissues thathave been irradiated can be self-repaired with a better efficiencythan the malignant tissue. This would allow a reduction of radiationtoxicity to normal tissue, while the damage to the tumor remainsunchanged. However, the entire treatment usually lasts four weeks–six weeks, a shortcoming for radiation therapy.

30.3 STEREOTACTIC BODY RADIATIONTHERAPY (SBRT)

Prudent investigations have been conducted on SBRT withhypofractionated (3–5 fractions) dose schedule to treat primary andoligometastasis lesions. Clinical trials over more than a decade haveshown that the SBRT has its advantage over the conventional RT(15–30 fractions). A better rate of local control has been reported intreating lung, liver, spine, pancreas, kidney, and prostate cancers.25

Even with similar clinical outcome, patients are more willing to have



a fewer visits to the radiation oncology clinics as their preference.This emerging radiation treatment method is heavily relying on theadvances of imaging and therapeutic technologies, which ensureadequate dose coverage to the target and minimize irradiation tothe surrounding critical organs and normal tissues. Ultimately, theratio of tumor control probability (TCP) and normal tissue compli-cation probability (NTCP) has been improved using the SBRT withimage guidance and respiratory motion control.

30.3.1 Comparison of SBRT with StereotacticRadiosurgery (SRS)

SBRT treats extracranial lesions indexed by noninvasive, removablebody frame or by bony landmark (frameless). It is a logical extensionof SRS, which treats cranial lesion using invasive, rigid head frameto index tumor location. In many aspects, SBRT resembles SRS.26

First, SBRT uses an indexing frame in both CT simulation and treat-ment delivery, similar to SRS. Second, a very large dose per fractionis prescribed to patient in both cases, although a single fraction isoften used in SRS. The ablative dose is so large that it causes damagebeyond repair. Third, because of the highly conformal requirement,any tumor with spreading infiltrative microscopic extension into itssurrounding normal tissue is not a good candidate for SBRT treat-ment; the same also holds true for SRS. However, there are some dis-tinct differences between these two RTs. First, SBRT treats extracra-nial lesions while SRS treats cranial lesions. Body is a deformableanatomy, in which involuntary motions, such as respiratory, cardiac,digestive, and muscular motions can cause anatomy deformation,while the head (brain) is a relatively more rigid anatomy. Second,SBRT employs a few (3–5) ablative fractions while SRS uses sin-gle ablative fraction. More than one fraction being used in SBRTis due to the large uncertainty of target localization in the body,caused primarily by respiratory motion, in comparison to that inthe brain. More fractions permit higher tolerance on the uncertaintyin the target localization through averaging. Last, different patientimmobilization and target indexing methods are employed betweenSBRT and SRS.



SBRT also utilizes tactics and concepts from other RTs, including3DCRT, IMRT, and IGRT. During an extracranial SBRT, advancedimaging is used to localize the target and to track its motion if4DRT techniques are employed.27−29 Also stricter requirement inpatient immobilization should be engaged in order to minimize bothinterfractional and intrafractional uncertainties in target localiza-tion. The objective is to deliver a safe and ablative dose to patientwith minimal radiation toxicity to normal tissues. Nevertheless,SBRT is not a subcategory of any existing RT, but carries uniquecharacteristics of its own to stand in parallel with other RTs.25,30,31

30.3.2 Hypo-Fractioned, Ablative RT for Extra-Cranial Lesion

It has been reported that 3–5 fractions have been employed to delivera total dose of 60 Gy to lung and liver cancer patients. As discussedabove, the 8 Gy–20 Gy per fraction is well beyond the normal tis-sue tolerance. Therefore, it is imperative to immobilize the patientduring the treatment and minimize the respiratory motion as muchas possible, in order to ensure that such ablative dose does not falloutside of the PTV with a small margin. A concern has been raisedabout whether SBRT is a sword with double edge; a prudent imple-mentation of SBRT is essential to achieve cancer local control in theclinic.30,31

More attentions to normal tissue sparing should be providedin order to minimize potential toxicity caused by SBRT. Based onthe physiological function of different organs, normal tissue can becategorized as parallel functioning tissues and serially functioningtissues. If a parallel tissue is damaged, the redundancy of such tissuecan make up the need without a severe toxicity effect. If a section of aserial tissue is damaged, however, all downstream function may bedisrupted, causing severe normal tissue toxicity. Potential damageto any serially functioning tissues and organs is considered to bethe biggest obstacle for implementing SBRT. Toxicity of 15%–18%for lung and liver cancer has been reported for SBRT treatment.31

However, acute toxicity is generally considered acceptable, whilelate toxicity may need further evaluations as more clinical follow-updata become available.25



Fig. 11. AVarian Clinac equipped with on-board imaging systems (Varian MedicalSystems, Palo Alto, CA). The horizontal one is the CBCT and the vertical one is the2D portal imager.

In order to deliver such biologically potent radiation dose tothe target, two major target localization uncertainties in treatmentdelivery must be minimized, namely the patient setup error andpatient motion error. The former can be reduced using one of thecommon IGRT approaches, including on site CBCT image (Fig. 11)that can be registered to the planning CT image for a much moreaccurate patient setup than the conventional approach. The latter canbe reduced by using patient immobilization and respiratory control,as discussed next.

30.3.3 Body Immobilization and Respiratory Control

The target localization for SBRT treatment has been assessed to be5 mm–10 mm, depending on the technique employed and the site oftreatment, where patient motions can make different contributions



to the uncertainty of target localization.25,30 For treating spine lesionusing SBRT, an accuracy of ±3 mm can be achieved since the spineis least affected by respiratory motion.

The most commonly used patient immobilization device is adouble vacuum bag system, in which one vacuum bag filled withStyrofoam pellets is laid under the patient while another with-out filling is laid on top of the patient. A soft Styrofoam layeris placed at patient abdomen for shallow breath. By this means,the patient is sandwiched and any significant motion caused bymuscular contraction can be avoided. The bottom vacuum bean-bag, which is locked on to the treatment couch, preserves patient-specific body contour molding and can be reused in the followingtreatment fractions, reducing the variation of patient setup betweenfractions.

The patient respiratory motion control can be performed usingone of the three methods: (1) respiratory damping; (2) respira-tory gating; or (3) motion tracking. Respiratory damping can beachieved using a belt to compress patient’s upper abdomen region,so that the motion of the diaphragm is limited and patient is forcedto breathe using thorax. The chest wall motion tends to move inthe opposite directions and its impact on tumor is minimal dueto the motion canceling effect. The advantage of the respiratorycontrol is that the radiation beam can be on at any time. Respi-ratory gating uses an external surrogate to indicate the phase ofrespiratory cycle, so that radiation beam can be turned on only atcertain breathing phase or at certain breathing amplitude (vidal vol-ume). The internal target can be localized based on the planningCT image that is acquired at the same respiratory phase or ampli-tude. Motion tracking requires image guidance in real time and thebeam adaptation to the moving target. Fluoroscopy imaging andfrequent 2D digital X-ray imaging have been used in the clinic. Theconventional Linac is too heavy to move back and forth with an ade-quate speed; whereas, light-weighted accelerators, such as roboticcyberknife and helical tomotherapy, are more suitable to performthis task.



30.3.4 Extremely High Conformal Dose Planningand Delivery

A large number of MV X-ray photon beams, such as 10–12 nonop-posing beams, should be used to create an extremely high conformaldistribution with a sharp dose falloff outside the PTV in all direc-tions. The nonopposing beam requirement minimizes the entranceand exit dose in the normal tissues. In order to avoid damaging theserially functioning tissues or organs, such tissues must be regardedas organ at risk and carefully delineated prior to radiation dosecalculation.

Using regular Linac, however, potential collisions between thepatient and accelerator head limit the selection of the beam angles,similar to the SRS case. However, using robotic cyberknife and heli-cal tomotherapy, such concerns are largely eased or eliminated. Inthe former method, there are a few thousands of candidate beamangles that can be selected based on the patient anatomy and onlineimaging guidance. In the latter method, an arc beam with beamaperture adjusted to the target shape in the beam’s eye view (BEV)is used for highly conformal dose delivery. In addition, proton beamtherapy has also been used in SBRT for high dose conformal therapydue to its unique depth dose curve governed by the law of physics,which will be discussed next.

30.4 PROTON AND HEAVY-ION RADIATION THERAPY

Proton and heavy-ion beam radiation therapy has shown a verypromising dose distribution, which can be beneficial in sparing crit-ical structures that are adjacent to the lesion, such as the spinalcord in spine lesion cases. The localized dose peak at the end ofits path and sharp dose falloff afterward (with no exit dose) pro-vide a dose distribution that other radiation modalities cannot com-pete with. Figure 12 shows a comparison of depth dose curves ofphotons and protons. Therefore, although a proton radiation unit isvery expensive since a cyclotron is used to accelerate protons, theadvantage of the unique dose distribution permits a treatment that



Tumour

Photons

Protons

15MV Photonsvs

SOBP Protons

00

1

2(B)

(A)

00

20

40

60

80

100

120

140

160

180

5 10 15 20

Depth [cm]

Rel

ativ

e D

epth

Dos

e [%

]

25 30 35 40

10Penetration depth in water [cm]

Rel

ativ

e do

se

20 30

Fig. 12. A comparison of depth doses (range and intensity) for 15 MV photonsand spread-out Bragg peak (SOBP) protons (A), which are composed of energy-modulated protons (B) using modulation wheel or energy-selected from the accel-erator (courtesy of Dr Alfred R Smith, MD Anderson Cancer Center, 2006).



other radiation modalities cannot.32 More than 15 years of clinicaldata from Loma Linda have shown that proton beam radiationtherapy is a superb RT to deliver conformal radiation dose and sparenormal tissue from radiation exposure.33 Currently, several clinicalproton therapy centers in the US are treating cancer patients, includ-ing Loma Linda University Medical Center, Massachusetts GeneralHospital, University of California San Francisco, MPRI at IndianaUniversity, MD Anderson Cancer Center, and University of FloridaProton Therapy Institute. Several other proton beam therapy centershave been recently built or will be built in the coming years, includ-ing Hampton University, Northern Illinois University, University ofPennsylvania, and MIT. Worldwide, heavy ion beam therapy, such ashelium (α-particle), lithium and carbon ions, is also clinically appliedin some countries, including Germany and Japan. Depending on thereference of comparison, these particles sometimes mentioned aslight ions comparing with other heavy metal ions, heavier ions com-paring with proton, or heavy-ions or particles comparing with elec-trons and photons that are commonly used in radiation therapy.34−36

30.4.1 Advantage of the Bragg Peak: Sparing CriticalNormal Tissue

The well-known Bragg peak in the depth dose curve of proton beamspermits protons to deliver extremely high conformal dose to thelesion target. The Bragg peak is a narrow dose peak at the end ofthe beam trajectory, in which the gradually attenuated and slowedparticles give up all their energy and are absorbed at the end of theirfinite range. Because the proton range is determined by its energy,thus by modulating the particle beam energy, the location of theBragg peaks can be adjusted, from patient surface to a depth. Theenergy for proton beam therapy is usually from 230 MeV to 250 MeV,providing large proton ranges to reach any deep-seated lesion. Clin-ically, a particle energy modulator (via different beam attenuations)has been used to broaden the Bragg peak through energy diversifi-cation to produce a uniform dose distribution within the PTV. Forproton beam, there is little dose tail at the distal direction from the



tumor. Therefore, proton beam does not have exit dose as photonbeam does, gaining significant normal tissue sparing.37

For heavier ions, such as helium, lithium, beryllium, and carbon,even more pronounced Bragg peak can be observed and utilizedfor radiation therapy. Because these ions are heavier than proton,it takes a higher-energy cyclotron to accelerate the heavy-ions inorder to have the same penetration depth as proton due to theirlarger masses. For instance, the carbon beam energy for RT is about350 MeV–400 MeV. However, the high energy releasing process ofheavy ions involves nuclear fragmentation, which contaminates thebeam, increases scatters, and results in a higher residue dose atdepths after the Bragg peak. Thus, slight exit dose does exist, butit is still much lower than the entrance dose.35 Therefore, the protonbeam RT and heavy-ion beam RT can provide a desired radiationdose distribution with no or little exit dose, so that significant nor-mal tissues can be spared from the radiation exposure. Such highlyconformal dose distribution is unique to proton and heavy-ion beamRT, making it superior to its competing therapeutic modalities, suchas photon/electron EBRT.

For proton and heavy-ion beam RT, special biophysical modelshave been applied to estimate therapeutic relative biological effec-tiveness (RBE), which has a complex dependency on dose, beamenergy, linear energy transfer (LET), atomic number, and cell/tissuetype. The calculated RBE has been used in heavy-ion treatment plan-ning and showed no sign of significant over- or underestimation.34

Proton and heavy-ion particles have very different interactions withmatters from those of photon and electron beams, and thereforepossess different biological impacts on both cancerous and normaltissues.

30.4.2 Advantage of the Radiobiological Efficacy:Overcoming Tumor Hypoxia

In general, the biological potency of proton (1H) and heavy-ion (4He,7Li, 9Be, 12C,) beams are increasing as their atomic number increases.The RBE for proton is about 1.1 and carbon beams can be as high



as 4–5, due to higher LET and beam quality, compared with photon(RBE = 1).34 However, higher RBE does not gain anything if thereis no other mechanism that can differentiate normal tissue from themalignant tissue. For carbon beams, increasing dose per fractiontends to lower the RBE for both the normal tissues and tumor, but atdifferent rate: normal tissue is at a higher descending rate than thetumor. Therefore, a gain of higher RBE to tumor can be achieved athigher dose per fraction for carbon beam therapy.35

In addition, tumor hypoxia (low oxygen tension) is a commonphenomenon in cancerous tissues, which are highly heterogeneousand circulation inefficient. For photon beam RT, it is well knownthat the cell survival curve is heavily dependent on the cell oxy-genation status. The oxygen enhancement ratio (OER) is high forphoton beams (OER ≈ 3), while it is lower for carbon beams (OER ≈1.6–2.0), meaning that the radiation damage by the heavy-ion beamdoes not depend upon the tissue oxygenation during treatment asmuch as photon beam does. Thus, radiation-resistant lesions due totumor hypoxia, such as uterine cervical cancer, can be treated usingcarbon beams clinically with improved local control rate.38 Notethat proton beam has the same OER value as photon beam, so pro-ton beam therapy does not provide any radiobiological advantageon this aspect as the heavy-ion beam RT does.

30.4.3 Cost Disadvantage and Technical Challenges

The major disadvantage of particle therapy is the high capital cost inboth initial setup and operational maintenance because high energycyclotrons or synchrotrons are employed to accelerate the positivelycharged particles. However, since about 2000, proton beam therapyhas reached a pivot point that it has been considered to be financiallyviable and technically feasible. This was owing to the impressiveclinical outcomes, various financial revenues (via reimbursementand investment), feasible hospital-based proton facilities, and inter-ested vendors in the proton beam RT market place.32

For most proton beam RT units, a rotating gantry has beeninstalled to perform isocentrical radiation therapy, in which patient



position is fixed regardless the proton beam field setting. However,many carbon-ion RT units are still delivered with fixed beam lines.A currently installed carbon/proton therapy facility at HeidelbergUniversity in Germany is going to have a rotating gantry.35

The particle energy modulation can be done either passively, asdiscussed above using beam modulators, or actively using spot orraster scanning. The former technique is the mostly applied method,while the latter may encounter problem when patient motion istaken into consideration because the same tissue within the mar-gin of PTV can be scanned more than once due to patient motion,resulting significant overdose. Therefore, motion tracking must beperformed if the treatment delivery is via beam scanning.35 Evenfor passive beam modulation method, in order to deliver high pre-cision dose distribution in particle beam RT to patient reproducibly,it requires an improved accuracy in interfractional patient setup andin intrafractional patient motion control. Otherwise, the gain in theparticle beam RT will be diminishing if these inter and intrafractionaluncertainties are high. As we have discussed in SBRT section, theseissues can be handled using one of the IGRT tactics. However, thereal solution to such complex clinical issues is what will be discussednext: the 4DRT techniques.37

30.5 FOUR-DIMENSIONAL RADIATION THERAPY (4DRT)

No matter how well a radiation dose distribution can be plannedusing a particular radiation modality in 3D treatment planning andtreatment delivery, patient motion, if not compensated properly, willhave negative impacts on the RT outcome. Clinically, patient motion,which can only be suppressed to a certain degree, must be taken intoaccount in both treatment planning and treatment delivery.

Involuntary and voluntary patient motions have been recentlyrecognized as an important source of uncertainty that may haveresulted in unrealized target underdose and unnecessary normal tis-sue overdose. The most significant involuntary motion is respiratorymotion, which can cause as large as 20 mm of diaphragm positionchange within a normal breathing cycle. The most important



voluntary motion is head motion, which is stochastic in nature andcan be effectively reduced by using a head immobilization device.Moreover, between treatment fractions, the patient could experi-ence weight loss/gain, tumor shrink/growth or different diges-tive/urinary fillings. These changes should be compensated andadapted in both the treatment planning and treatment deliveryin 4DRT.

30.5.1 The Concept of 4DRT

4DRT introduces the time dimension into the 3D conformal radiationtherapy in order to compensate for patient motion/changes occur-ring either during a single fraction (intrafractional) or between suc-cessive fractions (interfractional). By accurately localizing the targetduring the treatment, margins can be reduced, resulting in reduc-tion in radiation dose to normal tissues. 4DRT adopts the conceptsfrom both IGRT 8 and ART.9,10 The current 4DRT concept focuses onrespiratory motion and deformable change, but it does not excludeslower temporal changes.

Acomplete 4DRT should include 4D imaging, 4D treatment plan-ning, and 4D treatment delivery. Multimodality 4D imaging, includ-ing 4DCT, 4DMRI, 4DPET/CT, and 4DSPECT, may provide usefulclinical information, but 4DCT images are the most applicable imag-ing technique in 4DRT. Ideal 4D treatment planning should includeautomatic deformable image registration (inter-/intramodalities)and segmentation (contouring) to adapt to anatomic changes in the4D image, automatic field adjustment based on the image changes,and adaptive dosimetry calculation with respect to the initial plan-ning 4DCT image. An optimal 4D dose delivery system shouldinclude the image-guided patient setup, real-time target tracking (orgating), as well as radiation beam control with feedback communica-tion using dynamic multileaf collimator (DMLC), helical tomother-apy, or robotic cyberknife, so that the beam can track the movingtarget and the treatment margin can be reduced. Currently, thereis a technical gap between the ideal 4DRT concept and its clinicalimplementation.



30.5.2 Potential Advantage of 4DRT

Current 3DCRT planning is primarily based on an initial 3D plan-ning CT image, which often represents an arbitrary, single pointin time and doses not statistically reflect the respiratory motion.This approach is limited by two major errors: patient motion andpatient setup. Intrafractional motions (respiratory, cardiac, diges-tive, and muscular) can cause significant anatomical deformation,including shift and change of the GTV, the CTV, and their surround-ing normal tissues. Interfractional motions (daily fillings in diges-tive or urinary tracks, tumor shrinkage or growth, and weight lossor gain) can cause further deviation in the patient anatomy withrespect to the initial snapshot CT image. In addition, patient setupin the treatment position may deviate from the planned position andcan result in additional error using standard skin markers and laseralignment in the treatment room. In order to compensate for theseuncertainties inherent in the treatment plan, the current approach isto allow the planning target volume (PTV) to grow beyond the CTVwith a sufficiently large margin to cover both the motion and setupuncertainties.

With 4D image guidance, the ITV, defined by the union of 3Dtumor motion trajectory, or GTV motion envelope, can be obtained.27

The PTV derived from 4DCT image provides a more conformal tar-get volume, compared to the PTV derived from 3DCT image, wherethe population-based margin is often larger than necessary to ensureenough tumor coverage. This reduced PTV permits more normal tis-sue sparing, even with 3D treatment delivery. 4D treatment deliveryallows further sparing of normal tissue by adapting the treatmentfields to the moving target through real-time respiratory motiontracking or gating. In addition, on site X-ray and 3DCBCT allowspatient imaging in treatment position, reducing the patient setuperror.39,40 The on site X-ray imaging or fluoroscopy can be employedto track internal tumor position in real-time or periodically. There-fore, 4DRT can provide high precision conformal radiation therapy,minimizing the normal tissue complication probability and permit-ting possible dose escalation to the target.



30.5.3 4D Medical Imaging

Two general 4D imaging methodologies have been developed formotion-correction and motion analysis. Prospective 4D imagingacquires image projections using respiratory-gated image acqui-sition in reference to an external motion indicator, producing asingle motion-free 3D image at the selected respiratory phase.In contrast, retrospective 4D imaging acquires image projectionsat all respiratory phases and sorts them into appropriate phasebins based on the external motion indicator, producing a series ofmotion-free 3D images at different phases of the breathing cycle.External or internal fiducial markers are necessary for monitor-ing patient motion concurrently with 4DCT imaging because theyprovide a time-stamped indication of the motion stage (amplitudeor phase), especially if the 4D images are used in 4D radiationtherapy. With this tracking information, image acquisition can beprospectively gated and the acquired images can be retrospectivelysorted into image bins reflecting the different respiratory phases.Commonly used respiratory motion tracking devices includeoptical tracking, spirometer, Bellows pressure sensor, and nasalthermometer.

Using multi-detector-row CT (MDCT), image projections ofmultiple slices can be acquired simultaneously. After sorting ofthe projections into corresponding breathing phases, multiple 3DCT images can be reconstructed, representing motion-free andphase-resolved 3D images. Using parallel, multichannel MRI scan-ner, a volumetric torso image can be acquired within 1 seconds–1.5 seconds, so a sequential MRI image set or a 4DMR imagecan be obtained. Using gating technique, 4DPET, 4DPET/CT, and4DSPECT images can also be acquired and reconstructed. These4D images provide valuable temporal information for patientmotion tracking. In a longer time interval, daily CBCT acquiredfor patient setup can also be used to monitor patient anatomicalchanges for necessary adaptive treatment planning and treatmentdelivery.



30.5.4 4D Treatment Planning

From the treatment planning viewpoint, use of a full 4DCT image(a series of 3DCT images) has not yet been accepted in the cur-rent clinical practice. This is because (1) there is lack of clinicallyapproved, automated planning tools, including deformable imageregistration, multiple target contouring, adaptive dose calculation,and motion control plan delivery mechanism; and (2) 4DRT has notyet provided convincing clinical evidence on its promising treat-ment outcome. These two requirements may take a long time tofulfill. Thus, 4DRT planning is currently in its infancy.41

In order to utilize 4DCT image information into the existing treat-ment planning system, three approaches were studied and appliedin 4D radiation therapy planning, including (1) slow CT (equiva-lent to averaged 4DCT); (2) two CT sets at respiratory extreme posi-tions; and (3) a single midventilation CT chosen from 4DCT basedon diaphragm motion. These studies suggest that a more statisticallyvalid CT image should be used for treatment planning, rather than aCT at an arbitrary respiratory phase, together with an ITV with moreprecise margin covering the moving CTV. 4DMRI imaging can alsoplay a role in the 4DRT planning. For the single-slice MRI image, bothsagittal and coronal images are acquired to provide target motionin three orthogonal directions. Recently, volumetric 4DMRI imagewas reported to help 4DRT treatment planning. Although 3DPETand 3DSPECT images have recently been applied to radiation treat-ment planning to delineate the active tumor volume, 4DPET and4DSPECT images have not been reported being utilized in 4DRTtreatment planning.

30.5.5 4D Treatment Delivery

Patient setup uncertainty tolerances are generally within ±5 mm fora standard radiation therapy, within ±3 mm for extracranial bodystereotactic radiotherapy, and within ±1 mm for stereotactic radio-surgery. The 2D/3D/4D imaging of the patient in the treatment posi-tion should improve setup accuracy, compared with conventionalpatient setup using skin marks and external lasers, which can only



provide a coarse initial positioning. These on site images can be usedto register with the planning CT image, using rigid or deformableimage registration. Clinically, rigid registration with approximatesetup parameters is often used, providing an uncertainty within thetolerance.

Using external motion tracking and a planning 4DCT image, aninconsistency may occur between the actual and the planned targetlocalization due to breathing irregularities. This can be minimizedthrough patient respiratory coaching. The most direct method oftarget tracking is to implant fiducial markers into or around the tar-get. Both passive and active markers have been used. The passivemarkers are often gold seeds that can be monitored using X-ray flu-oroscopy in real-time or frequent acquisition of digital 2D X-rayimages. Active markers are electromagnetic sensors, such as theCalypso beacon transponders, which can provide real-time local-ization information. Currently, therapists must monitor the fiducialposition visually and turn beam off manually if it moves out-side the tolerance region. Automatic tracking system seems neces-sary by integrating the products from different venders. Althoughreal-time treatment delivery to the target guided by the target track-ing feedback has been proposed,42 it has not been feasible in most ofthe clinics. It is largely dependent on the future advances in systemintegration. Current treatment systems, however, are fundamentallycapable of delivering a 4D dose distribution. The key is the combi-nation or integration of the individual 4DRT components to form aclinically feasible system.

30.6 SUMMARY

It is expected that in the next decade that the conventional photonand electron radiation therapy should still be the predominant treat-ment modality in the radiation oncology clinic, together with otherconventional brachytherapy. The SBRT is going to be more appliedroutinely in clinic, as a thorough clinical evaluation of the hypofrac-tionated, ablative dose delivery to extracranial lesions will become



available and indicative of improved local control with tolerablerate of late effect. The proton and heavy-ion beam RT will play abigger role in radiation therapy despite the cost disadvantage. Theclinical benefits based on a decade of experience have been demon-strated, largely owning to the law of Physics in the proton behav-ior in radiation dose deposit, which non-arguably spares significantamount of critical normal tissues. In order to ensure these precisedose delivery methods, SBRT, proton RT, and 4DRT will attract broadattention from virtually any other RT methods because patient setupand motion control must be handled properly in all RT procedures.Therefore, there should not be lack of interest in pursuing further toreinforce the initial attempts in 4DRT.

There are other minor trends in RT, including further improv-ing EBRT, such as using both IMRT and electron beams (IMRT+e)to improve the uniformity of dose distribution for some clinicalcases43,44 and modulated electron radiation therapy (MERT) to sparethe distant critical organs.45 Also, moving the RT facilities, such assmall mobile Linac, into the surgical operation room to performintraoperative RT (IORT) is emerging. The rationale behind suchcombined cancer treatment modalities is simple and time is neededto appropriately evaluate the benefit of IORT.46−48 In addition, theuse of multimodality imaging for target delineation, especially func-tional imaging, such as PET and fMRI, will become routine eventhough the use of PET imaging has not yet shown any obviousclinical benefit statistically. Certainly, a lot of attentions are still onthis subject for proper characterization and delineation of the target.There is no doubt about its diagnostic and treatment evaluation val-ues, especially for detection of distant metastases and diagnosis ofrecurrent disease.

References

1. Rutqvist LE, Pettersson D, Johansson H, Adjuvant radiation therapyversus surgery alone in operable breast cancer: Long-term follow upof a randomized clinical trial, Radiot & Oncol 26: 104–10, 1993.

2. Dobbs HJ, Radiation therapy for breast cancer at the millennium, Radio-therapy & Oncology 54: 191–200, 2000.



3. Mundt AJ, Roeske JC, Intensity Modulated Radiation Therapy: A Clin-ical Perspective, BC Decker, Lewiston, NY, 2004.

4. Goodman KA, Hong L, Wagman R, Hunt MA, et al., Dosimetric anal-ysis of a simplified intensity modulation technique for prone breastradiotherapy, Int J Radiat Oncol Biol Phys 60: 95–102, 2004.

5. The American Association for Therapeutic Radiology and OncologyStatistics, 2004.

6. Vijayakumar S, Chen GT, Implementation of three dimensional con-formal radiation therapy: Prospects, opportunities, and challenges, IntJ Radiat Oncol Biol Phys 33: 979–983, 1995.

7. Palta JR, Mackie TR (eds.), Intensity-modulated radiation therapy:The state of the art, American Association of Physicists in Medicine(AAPM), Medical Physics Monograph, No 29, Medical Physics Pub-lishing, Madison, WI, 2003.

8. Jaffray D, Kupelian P, Djemil T, Macklis RM, Review of image-guidedradiation therapy, Expert Rev Anticancer Ther 7: 89–103, 2007.

9. Yan D, Vicini F, Wong J, Martinez A, Adaptive radiation therapy, PhysMed Biol 42: 123–132, 1997.

10. Court LE, Tishler RB, Petit J, Cormack R, et al., Automatic online adap-tive radiation therapy techniques for targets with significant shapechange: A feasibility study, Phys Med Biol 51: 2493–2501, 2006.

11. Webb S, Optimizing radiation therapy inverse treatment planningusing the simulated annealing technique, Int J Imaging Syst Tech6: 71–79, 1995.

12. Metcalfe P, Kron T, Hoban P, The physics of radiotherapy: X-rays fromlinear accelerators, Medical Physics Publishing, Madison, WI, 2004.

13. Ling CC, Leibel SA, Fucks Z, et al., A practical guide to intensity-modulated radiation therapy, Medical Physics Publishing, Madison,WI, 2003.

14. Bortfeld T, IMRT: A review and preview, Phys Med Biol 51: R363–R379,2006.

15. Bentel GC, Patient positioning and immobilization in radiation oncol-ogy, New York, McGraw-Hill, 1998.

16. Li G, Xie H, Ning H, Capala J, et al., A novel 3D volumetric voxelregistration technique for volume-view-guided image registration ofmultiple image modalities, Int J Rad Onc Biol Phys 63: 261–273, 2005.

17. Pluim JPW, Maintz JBA, Viergever MA, Mutual-information-basedregistration of medical images: A survey, IEEE Trans Med Imaging 22:986–1004, 2003.

18. Crum WR, Hartkens T, Hill DLG, Non-rigid image registration: Theoryand practice, Br J Radiol 77: S140–S153, 2004.



19. Ragan D, Starkschall G, McNutt T, et al., Semiautomated four-dimensional computed tomography segmentation using deformablemodels, Med Phys 32: 2254–2261, 2005.

20. Guo P,Adamovics J, Oldham M,Apractical three-dimensional dosime-try system for radiation therapy, Med Phys 33: 3962–3972, 2006.

21. Wuu CS, Xu Y, Three-dimensional dose verification for intensity modu-lated radiation therapy using optical CT based polymer gel dosimetry,Med Phys 33: 1412–1419, 2006.

22. Islam KT, Dempsey JF, Ranade MK, Maryanski MJ, et al., Initial eval-uation of commercial optical CT-based gel dosimeter, Med Phys 30:2159–2168, 2003.

23. Mackie TR, History of tomotherapy, Phys Med Biol 51: R427–R453, 2006.24. Gibbs IC, Frameless image-guided intracranial and extracranial radio-

surgery using the Cyberknife robotic system, Cancer Radiother 10:283–287, 2006.

25. Timmerman RD, Kavanagh BD, Cho LC, Papiez L, et al., Stereotacticbody radiation therapy in multiple organ sites, J Clin Oncol 25: 947–952,2007.

26. Kavanagh BD, Timmerman RD, Sterotactic raiosurgery and stereotacticbody radiation therapy: An overview of technical considerations andclinical applications, Hematol Oncol Clin N Am 20: 87–95, 2006.

27. Jin JY, Ajlouni M, Chen Q, Yin FF, et al., A technique of using gated-CT images to determine internal target volume (ITV) for fractionatedstereotactic lung radiotherapy, Radiother Oncol 78: 177–184, 2006.

28. Wurm RE, Gum F, Erbel S, Schlenger L, et al., Image guided respiratorygated hypofractionated stereotactic body radiation therapy (H-SBRT)for liver and lung tumors: Initial experience, Acta Oncol 45: 881–889,2006.

29. Hodge W, Tome WA, Jaradat HA, Orton NP, et al., Feasibility reportof image guided stereotactic body radiotherapy (IG-SBRT) withtomotherapy for early stage medically inoperable lung cancer suingextreme hypofractionation, Acta Oncol 45: 890–896, 2006.

30. Timmerman RD, Forster KM, Cho LC, Extracranial stereotatic radiationdelivery, Sem Radiat Oncol 15: 202–207, 2005.

31. Sampson MC, KatzA, Constine LS, Stereotactic body radiation therapyfor extracranial oligometastases: Does the sword have a double edge?Sem Radiat Oncol 16: 67–76, 2006.

32. Smith AR, Proton therapy, Phys Med Biol 51: R491–R504, 2006.33. Slater JD, Clinical applications of proton radiation treatment at Loma

Linda University: Review of a fifteen-year experience, Tech Cancer ResTreat 5: 81–89, 2006.



34. Kramer M, Weyrather WK, Scholz M, The increased biological effec-tiveness of heavy charged particles: From radiobiology to treatmentplanning, Tech Cancer Res Treat 2: 427–436, 2003.

35. Schilz-Ertner D, Tsujii H, Particle radiation therapy using proton andheavier ion beams, J Clin Oncol 25: 953–964, 2007.

36. Brahme A, Recent advances in light ion radiation therapy, Int J RadiatOncol Biol Phys 58: 603–616, 2004.

37. Suit H, Goldberg S, Niemierko A, Trofimov A, et al., Proton beamsto replace photon beams in radical dose treatments, Acta Oncol 42:800–808, 2003.

38. Nakano T, Suzuki Y, Ohno T, et al., Carbon beam therapy overcomesthe radiation resistance of uterine cervical cancer originating fromhypoxia, Clin Cancer Res 12: 2185–2190, 2006.

39. Forrest LJ, Mackie TR, Ruchala K, Turek M, et al., The utility of mega-voltage computed tomography images from a helical tomotherapysystem for setup verification purposes, Int J Rad Oncol Biol Phys 60:1639–1644, 2004.

40. Lu W, Olivera GH, Chen Q, et al., Deformable registration of theplanning image (kVCT) and the daily images (MVCT) for adaptiveradiation therapy, Phys Med Biol 51: 4357–4374, 2006b.

41. Keall PJ, Mageras GS, Malter JM, et al., The management of respiratorymotion in radiation oncology report ofAAPM Task Group 76, Med Phys33: 3874–3900, 2006.

42. Keall PJ, Joshi S, Vedam SS, Siebers JV, et al., Four-dimensional radiationtherapy planning for DMLC-based respiratory motion tracking, MedPhys 32: 942–951, 2005.

43. Mu X, Olofsson L, Karlsson M, Sjogren R, et al., Can photon IMRT beimproved by combination with mixed electron and photon techniques?Acta Oncol 43: 727–735, 2004.

44. Chan MF, Chui CS, Song Y, Burman C, et al., Combination of electronand photon intensity modulated radiotherapy: A novel radiation ther-apy technique for the management of malignant pleural mesothelioma,Radiot and Oncol 79(2): 218–223, 2006.

45. Song Y, Boyer AL, Pawlicki T, Jiang S, et al., Modulated electron radi-ation therapy: Emerging technology, in Intensity Modulated RadiationTherapy — A Clinical Perspective, BC Decker, New York, 2004.

46. Calvo FA, Meirino RM, Orecchia R, Intraoperative radiation therapy,first part: Radionale and techniques, Crit Rev Oncol Hemat 59: 106–115,2006.

47. Calvo FA, Meirino RM, Orecchia R, Intraoperative radiation therapy,Part 2: Clinical results, Crit Rev Oncol Hemat 59: 116–127, 2006.

48. Willett CG, Czito BG, Tyler DS, Intraoperative radiation therapy, J ClinOncol 25: 971–977, 2007.


CHAPTER 31

IT Architecture and Standards for aTherapy Imaging and Model

Management System (TIMMS)

Heinz U Lemke and Leonard Berliner

Appropriate use of information and communication technology (ICT) andmechatronic (MT) systems is considered by many experts as a significantcontribution to improve workflow and quality of care in the operatingroom (OR). This will require a suitable IT infrastructure as well as commu-nication and interface standards, such as DICOM and suitable extensions,to allow data interchange between surgical system components in the OR.A conceptual design of such an infrastructure, i.e. a therapy imaging andmodel management system (TIMMS) will be introduced in this chapter.

A TIMMS should support the essential functions that enable andadvance image, and in particular, patient model guided therapy. Withinthis concept, the image centric world view of the classical PACS technol-ogy is complemented by an IT model-centric world view. Such a view isfounded in the special modeling needs of an increasing number of mod-ern surgical interventions as compared to the imaging intensive workingmode of diagnostic radiology, for which PACS was originally conceptu-alised and developed.

A proper design of a TIMMS, taking into account modern softwareengineering principles, such as service oriented architecture, will clarifythe right position of interfaces and relevant standards for a surgical assistsystem (SAS) in general and their components specifically. Such a systemneeds to be designed to provide a highly modular structure. Modules maybe defined on different granulation levels. A first list of components (e.g.high and low level modules) comprising engines and repositories of anSAS, which should be integrated by a TIMMS, will be introduced in thischapter.

783


784 Heinz U Lemke and Leonard Berliner

31.1 INTRODUCTION

Since the OR and image-based interventional suites are the mostcost-intensive sector in the hospital, the optimization of workflowprocesses has become of particular concern for healthcare providers,managers, and administrators. The understanding and managementof workflows should become an integral part in the planning andimplementation of complex digital infrastructures supporting diag-nostic and interventional procedures (i.e. interventional radiology,minimal interventional surgery, computer assisted surgical proce-dures and image guided therapy (IGT)).

Examples of workflow and OR infrastructure related issues are1:

(1) Inefficient, ineffective and redundant processes.(2) Inflexible “systems” of operation.(3) Ergonomic deficiencies which hinder the workflow.(4) Data (text, 1D, 2D, 3D, 4D) presentations not adequate, e.g.

intraoperative and perioperative.(5) Soft knowledge (info + action strategy) presentation not

available.(6) Scheduling (and tracking/RFIDing) of patients, personnel,

operating rooms, equipment etc. not facilitated or coordinated(often the seeds of “busted” schedules).

(7) Too long set up times for image-guided and robotic surgery.(8) Lack of consistent working practices/guidelines or workflows

(the hospital as a high risk and high velocity “production” envi-ronment is not scripted enough, there is too much diversity ofbehavior).

(9) No standardized integration of surgical devices and systems.(10) Lack of quantified information on workflow and error

handling.(11) Communication across disciplines not adequate, e.g. between

radiology and surgery.

Possible solutions are:

(1) Improve situational awareness.


IT Architecture and Standards for a TIMMS 785

(2) Ensure availability of real time information regarding (peri)operative processes to respond to best practices and variancesin actual patient care.

(3) Develop standard interfaces to integrate seamlessly ICT and MTsystems into the OR by taking account of the special needs ofimaging and modelling tools within the surgical workflow.

This leads to the concept of an ICT supported OR which may benamed surgical PACS (S-PACS) or more specifically a “therapy imag-ing and model management system” (TIMMS). A TIMMS2 shouldsupport the essential functions that enable and advance image, andin particular, patient model guided therapy. Within this concept,the image centric world view of the classical PACS technology iscomplemented by an IT model-centric world view. Such a view isfounded in the special modelling needs of a number of modern sur-gical interventions as compared to the imaging intensive workingmode of diagnostic radiology, for which PACS was originally con-ceptualized and developed.

A TIMMS provides the ICT based infrastructure necessary forsurgical/interventional workflow management of the modern dig-ital operation room (DOR). The concept and design of a TIMMS isbased on the assumption that significant improvement in the qualityof patient care, as well as ergonomic and health economic progressin the OR can only be achieved by means of an ICT infrastructure(based for example on a suitable DICOM extension) for data, image,information, model and tool communication. A proper design of aTIMMS, taking into account modern software engineering princi-ples, such as service oriented architecture, will clarify the right posi-tion of interfaces and relevant standards for a surgical assist system(SAS) in general and their components specifically.

31.2 TIMMS AND ITS INTERFACES

Engineering of ICT systems for the assistance of surgical interven-tional activities implies the specification, design, implementationand testing of computer assisted surgery (CAS) or IGT systems.



Components of a Surgical Assist System

Modelling Simulation

Kernel for

WF and K+D

management

Visualisationmanager Intervention Validation

Repo-sitory

Engine Data Exchange

IO Imagingand

Biosensors

Imagesand

signals

Modellingtools

Computingtools

WF andK+Dtools

Presentationtools

Devices/Mechatronics

tools

Validationtools

Fig. 1. Components of a surgical assist system.

A number of components for such systems have been developedin academic and industrial settings and are applied in various sur-gical disciplines. In most cases, however, they are stand alone sys-tems with specific ad hoc propriety or vendor interfaces. They canbe considered as islands of IT engines and repositories with varyingdegrees of modularization and interconnection.

Figure 1 shows an abstraction of seven engines with associatedrepositories, which may form part of an SAS. Ideally they should beintegrated by a suitable TIMMS infrastructure.

Considering software engineering principles, such a systemneeds to be designed to provide a highly modular structure. Mod-ules may be defined on different granulation levels. A first list ofcomponents (e.g. high and low level modules) comprising enginesand repositories of an SAS, which should be integrated by a TIMMS,is currently being compiled in a number of R&D institutions and alsowithin the DICOM “DICOM in surgery.”

Figure 2 shows a concept (meta architecture) of a high levelgeneric modular structure of a surgical assist system. The high levelmodules are abstracted from many specific CAS/IGT systems whichhave been developed in recent years. In general, a combination ofthese can be found in most R&D as well as commercial SAS systems.



Modules of a Surgical Assist System

Modelling

Models(simulated

objects)

Therapy Imaging and Model Management System (TIMMS)ICT infrastructure (based on DICOM-X) fordata, image, modeland tool communication for patient model-guided therapy

Simulation

Kernel forWF and K+DManagement

VisualisationManager Intervention Validation

Repo-sitory

Engine

IO Imagingand

Biosensors

Imagesand

signals

Modellingtools

Computingtools

WF andK+Dtools

Presentationtools

Devices/Mechatr.

tools

Validationtools

WF`s, EBM,cases

Data andinformation

Models andinterventionrecords

incl. a Therapy Imaging and Model Management System (TIMMS) Data Exch.

Control

Fig. 2. Therapy imaging and model management system (TIMMS).

A central position in Fig. 2 is occupied by the “Kernel for workflowand knowledge and decision management.” It provides the strategicintelligence for preoperative planning and intraoperative execution.Often this module (or parts thereof) is integrated into some of theother engines, as the need may have demanded.

Low level modules (LLM’s) responsible for interfacing and com-munication are embedded in each of the engines and repositoriesgiven in Fig. 2. LLM’s should be derived from a single or froma combination of several distinct surgical workflows. In the lat-ter case, these are sometimes referred to as surgical integrationprofiles (SIP’s). An LLM may be a surgical function or related activ-ity using information objects, which ideally, may be part of differ-ent types of interventions. In order to identify LLM’s which satisfythe above requirements, it is of critical importance to select a repre-sentative set of surgical/interventional workflows which cover thedomain of interest for standardization of image and model-guided



interventions. This selection should not only focus on the presentstate of the art of surgery, but also take into account future potentialdevelopments of patient image- and model-guided interventions.

31.3 COMPONENTS OF TIMMS AND FUNCTIONALITIES

31.3.1 Engines and Repositories

The components of TIMMS,3 which are modular, scalable and maybe distributed in location, act synergistically to provide function-ality and utility that exceeds the sum of its individual parts. Thecomponents include:

(1) Seven “engines” which work independently and dependently,and account for all facets of complex medical and surgical pro-cedures. Engine may be defined as a software module which canbe executed on an appropriate computing machine.

The seven engines are:

• Intraoperative imaging and biosensors engine• Modelling engine• Simulation engine• Kernel for workflow and knowledge and decision manage-

ment engine• Visualization engine• Intervention engine• Validation engine.

(2) Associated repositories linked to each of the seven engines arepository may be defined as an integrated hardware and soft-ware structure which stores, and makes available, data and/ordata processing tools.

• Images and signals repository for the intraoperative imagingand biosensors engine.

• Modelling tools repository for the modelling engine.• Computing tools repository for the simulation engine.• Workflow and knowledge and decision tools repository for

the kernel for workflow and knowledge and decision man-agement engine.



• Representation tools repository for the visualization engine.• Devices and mechatronic tools repository for the intervention

engine. Mechatronics is defined as the synergistic combina-tion of mechanical engineering, electronic engineering, andsoftware engineering.

• Validation tools repository for the validation engine and forthe kernel for workflow and knowledge and decision man-agement engine.

(3) Additional repositories which are provided for

• Models (models are defined as simulated objects).• References such as workflow models, evidence-based medical

data, case-based medical data.

The system provides for real time data mining from these repos-itories during the performance of the surgical procedure.

(4) Kernel for workflow and knowledge and decision manage-ment engine. The central computing kernel (or “brain”) of thesystem may use different forms of logic, different databasestructuring, agents and other forms of artificial intelligence,depending on the specific applications of the procedure orprocedures being performed. Agents may be defined as soft-ware modules, containing some form of intelligence, which,with some degree of autonomy and adaptability, carry outfunctions or tasks. Agents may be called by the workflowengine when executing a given activity component/element ofa given workflow. In general, agents are part of the Kernel forworkflow and knowledge and decision management, but theymay also be part of and/or be accessible to the other enginesof TIMMS.

(5) Information and communication technology infrastructureallowing for intercommunication and interactivity between allcomponents of TIMMS. All of the engines, tools, repositories,ICT infrastructure, data sources, including the operative teamare linked, through a distributed network, providing for the fullfunctionality of TIMMS, including planning, guidance, learning,and data mining and processing.



The ICT infrastructure used by TIMMS includes structures, objects,processes and interfaces from well established sources, to ensurecompatibility. This includes, but is not limited to:

• IHE• HIS• RIS• PACS• DICOM• HL7

Interfaces are provided for the input of data and informationfrom the outside world which are then processed and utilized bythe functional components of TIMMS and stored within the repos-itories. A possible realization of interfaces required between majorfunctional groups within and outside TIMMS is shown in Fig. 3.

Fig. 3. Data interfaces of TIMMS.



Interfaces are also provided for the output of various models, inter-vention records, data and information that have been synthesizedwithin the TIMMS structure.

31.3.2 Major Functionalities

31.3.2.1 Patient specific modelling

TIMMS is based on an underlying construct or approach topatient management entitled a model-centric view. Traditionally, theapproach to medical imaging when applied to clinical aspects ofpatient care has been limited to the realm of the images themselves.This has been called the image-centric world view.

However, the approach to medical imaging employed by TIMMSis extended far beyond the realm of the images. In the model-centricworld view a wide variety of information, relating to the patient,can be integrated with the images, providing a more comprehensiveand robust view of the patient. TIMMS employs the model-centricworld view, providing and utilizing all available data for surgicalinterventions.

31.3.2.2 Adaptive workflow engines

The incorporation and utilization of workflow processes, within thekernel for workflow and knowledge and decision management iscentral to the functioning of TIMMS. TIMMS employs an adap-tive workflow engine that is flexible and capable of learning andproviding guidance throughout the procedure. A reference work-flow, which provides the basic framework for a surgical procedure,evolves into an executing workflow, which is patient specific andis based on the model-centric view of the patient that also evolvesthroughout the entire patient encounter. For example, modificationsto the executing workflow may be based on feedback from physi-ologic monitoring of the patient, from the surgeon, from operativerobots, from operative haptic devices, from stored data within repos-itories. Modifications to the executing workflow engine are in syn-chronization with updates to the patient model by the modelling



engine. The selected reference surgical workflows is extracted fromthe appropriate repository during the planning stage of the surgicalprocedure.

31.3.2.3 Validation processes

Data collection is automated for all aspects of the presurgical eval-uation, intraoperative procedures, and postoperative evaluation.Methodology is provided for the application of statistical processesto the accumulated data.

The methodology for error handling and validation is built intothe system so that variations in human performance, as well asmachine performance, and patient response are factored in, andlearned from, at any given step of the surgical procedure.

The system contains the functionality to achieve refinementsin medical and surgical “best practices” and to facilitate qualityimprovement programs. Prospective medical research projects willbe more easily achieved through the automated collection, mon-itoring and measuring of large volumes of data, with numerousvariables.

Key aspects for the validation engine:

• Assess the surgical workflow activities, in particular the imaging,model and representations accuracy of the surgical intervention.

• Assess specific surgical domain data, information, knowledge anddecision presentations, intervention protocols.

• Ascertain that the specific surgical workflow selected fulfils thepurpose for which it is intended and is properly executed.

• Ascertain that selected critical activities, which imply givenaccuracy, precision, real time response, etc. are properlycarried out.

• Ascertain that the appropriate tool sets selected from the reposi-tories will provide the capabilities required.

• Secure that completeness and consistency checks produce the cor-rect results.

• Ascertain that appropriate documentation and reporting for theintervention is carried out.



• Ascertain that the appropriate hardware and software devicesrequired are online and functioning.

31.4 INCORPORATION OF SURGICAL WORKFLOW

Organized activities such as those observed in the operating room,regardless of complexity, may be better understood and character-ized through the process of workflow analysis and diagramming.By analyzing, synthesizing and filtering multicomponent processesinto their fundamental functional components, a workflow diagrammay be generated. To provide consistency and reproducibility thisprocess must utilize a uniform and consistent ontology. The work-flow diagram thus generated may be viewed at different levels ofgranularity or orders. These may be described from the broadest cat-egories (first-order processes) through the finest levels of the surgicalprocedure (n-order process).

The specific workflow diagrams generated through precise andanalytic description of actual surgical procedures may be further dis-tilled into generic, or reference, workflow diagrams for categories ofprocedures. The reference workflow diagrams thus generated pro-vide the underlying roadmap to be followed by TIMMS throughoutan entire operative procedure. This includes each of the three first-order processes: preoperative assessment and planning; operativeprocedure; and postoperative care.

The reference workflow diagram is a dynamic and flexible struc-ture, designed to be transformed into a patient specific workflow, orexecuting workflow, by TIMMS throughout the entire procedure.The workflow Kernel and the various cognitive agents of TIMMSgenerate a patient-specific model from all of the available sources ofdata, such as imaging, physiological monitoring, EMR, data reposi-tories, generated simulations, input and feedback from mechatronicdevices. Furthermore, on the basis of changes in the patient modelthroughout the entire procedure, the executing workflow may bemodified and updated as necessary. This provides the necessaryflexibility required for a surgical procedure in which both minor andmajor variations are the norm. As variations or deviations from the



active executing workflow are encountered, the patient model andthe executing workflow are updated as required. It should be notedthat the patient specific model may be influenced by any and allfactors directly impacting the procedure. These include factors thatare both intrinsic and extrinsic to the patient, including the func-tions and status of surgical tools and devices, and activities of theoperating surgeon and assistants.

As a surgical procedure progresses through the executing work-flow, active links between the workflow engine and the TIMMSagents are activated in sequence in order to accomplish the tasksrequired for TIMMS to help facilitate the surgical process.

A reference workflow diagram for an hepatic tumor radiofre-quency ablation procedure, which will be used to demonstrate theactive links between workflow and TIMMS in Sec. 31.5, is presentedin Figs. 4(A)–4(E).

31.5 EXAMPLE OF A TIMMS PROJECT

31.5.1 Active Links Between Surgical Workflow and TIMMS

A TIMMS project is designed to function throughout a surgicalworkflow at all levels of granularity of each of the three first-orderprocesses: preoperative assessment and planning; operative proce-dure; and post-operative care, as described in Sec. 31.4. The initi-ation of a TIMMS project, in a clinical setting, may be consideredto take place at the time a request for a procedure is received bythe surgeon, and concludes when all post-operative care issues andpost-operative quality assurance and archiving activities have beenaddressed.

An example of a TIMMS project (treatment for a solitaryliver metastasis from a previous colon cancer with radiofrequencyablation) is presented in this section. The relationship between theworkflow steps and accompanying TIMMS functions and actionsare outlined in detail. Schematically, this will be represented as con-nections between the workflow steps and the TIMMS agents andengines which are accessed through the TIMMS network [Fig. 5(A)].



No

Priorto Day of

Precedure

Radiologist Logs onPACS Workstation and

RIS/HIS

Does TumorMeet Criteria

for RF Ablationwith CT Guidance

Referfor Surgical

Ablation

Clinical Evaluation:Review patient's Historyand Physical, CompleteBlood count, CoagulationStudies, Liver FunctionTests, and Biopsy Results.

Continueto Fig. 4(B)for Day ofProcedure

Laboratory DataReviewed byRadiologist

Previous ImagingReviewed byRadiologist

Localization:Confirm tumor location

andcharacteristics

Targeting:Plan and Determine

Access and trajectoryof electrodes

Hepatic Tumor Radiofrequency AblationPreoprative Assessment and Planning

Yes

Fig. 4(A). The workflow steps involved in preoperative assessment and plan-ning (the initial first-order process) for an hepatic tumor radiofrequency ablationprocedure.



Continuedfrom Fig. 4(A)

Day ofProcedure

Patient EntersCT Suite

Radiologist EntersCT Suite

Radiologist Scrubs

Input datafrom

pre-procedure CT

Place fine needlefor local anesthesia and

to confirm trajectory

Re-Image with CT

Is NeedlePositioned

Adequately?

Prep Site

RepositionNeedle

Continueto Fig. 4(C)

No

Yes

Image Guidance:Final Plans and measurements

for electrode placement

Navigation:1. Real-time CT-Fluoro

2. Electro-magnetic Targeting System3. Laser Goniometer

Previous ImagingReviewed byRadiologist

Patient Placed onCT Table

Tech Opens Orderin RIS

Tech EntersPatient ID inCT Console

Patient Monitoring,Intubation, and

General Anesthesia

LocalizingCT Scan of Upper

Adbomen

RadiologistPrepares to Place

Fine Needlefor Local Anesthesia

Skin is markedusing CT laser light

Select Entrance Site

Hepatic Tumor Radiofrequency AblationOperative Procedure: Preparation for Electrode Placement

Fig. 4(B). The initial workflow steps involved in the operative procedure (thesecond first-order process) for an hepatic tumor radiofrequency ablation procedure.This portion of the workflow covers the preparatory steps required for placementof the radiofrequency electrode.



Hepatic Tumor Radiofrequency AblationOperative Procedure: Ablation of the Tumor

Continuefrom Fig. 4(B)

Insert Electrode withTandem Technique

With CT-Fluoro

Re-Image with CT

Re-Image with CT

RepositionElectrode

RepositionElectrode

Has all ofthe tumor been

ablated?

Re-Image with CTwith IV contrast

Have CriteriaBeen Met for

CompleteAblation

Continueto

Fig. 4(D)

Yes

Yes

No

No

No

Monitoring

Is ElectrodePositioned

Adequately?

Ablate Tumor as perProtocol for Ablation

Device

Yes

Fig. 4(C). The next workflow steps involved in the operative procedure for anhepatic tumor radiofrequency ablation procedure. These workflow steps are relatedto the ablation process.



Hepatic Tumor Radiofrequency AblationOperative Procedure: Completion

Continuefrom

Fig. 4(C)

Continueto

Fig. 4(E)

Rescan with CT toR/O Complications

Remove Electrode(While Heating Tract)

Fig. 4(D). The workflow steps involved in the completion of the ablation process.

31.5.2 Preoperative Assessment

31.5.2.1 Initiation of a new TIMMS project

When a request for an operative procedure is received, the surgeonmay launch the TIMMS software to initialize a new project at aTIMMS medical workstation [Fig. 5(B)]. The TIMMS engines andrepositories will start up and undergo an automated system check,and all of the engine activities which operate in the backgroundwill commence. At this time, the validation engine will check that allTIMMS software components are online and functioning properly.The default settings of all connected hardware and software deviceswill be initialized and their proper function will also be confirmedby the validation engine. At this time, the surgeon may modify thespecific connections through the TIMMS computer interface.

The surgeon will then establish a new “TIMMS PROJECT” whichwill have its own unique TIMMS Project ID Number and will enterpatient’s name and medical record number. In order for TIMMS to



Hepatic Tumor Radiofrequency AblationPost-Operative Care

Continuefrom

Fig. 4(D)

Images Sent to PACSby Technologist

Case Dictated byRadiologist

Case Finalized in RISby Tech

Radiologist Signs OffDictation

Apply Dressing

Extubation andPost-Procedure Careby Anesthesiologist

Patient Transferred toRecovery Room

Patient Dishcharged

Fig. 4(E). The workflow steps involved in post-operative care (the third first-orderprocess) for a hepatic tumor radiofrequency ablation procedure.

begin to select the appropriate reference workflow from the workflowrepository and to perform the data mining from electronic medicalrecords and data repositories, the surgeon will enter identifying fea-tures of the surgical procedure to be performed, such as a procedureclass (radiofrequency ablation) and code (solitary liver metastasisfrom colon cancer). The reference workflow would be selected by acognitive agent of the kernel for workflow and knowledge and decisionmanagement (workflow kernel). The data mining functions are medi-ated by the electronic medical record (EMR) agent of the workflow kernel.Patient information and images would be retrieved from sourcesincluding the radiology information system (RIS), hospital informa-tion system (HIS), picture archiving and communications system(PACS) and from TIMMS data repositories.



I&BE+R

ModE+R

SimE+R

ModelsRep

TIMMS ICT Infrastructure

KerE+R

VisE+R

IntE+R

ValE+R

LAN / WAN

EMR

RIS

HIS

MechatronicsWF+EBMRep

Fig. 5(A). TIMMS components and the LAN/WAN through which it connects toexternal elements such as data sources and mechatronic devices.

Priorto Day of

Procedure ValidationEngine

1. Receive request for new procedure.2. Initialize new case.3. TIMMS engines and repositories, and connections will start up and undergo system check.

Fig. 5(B). The initial links between the surgical workflow and TIMMS when arequest for a procedure is first received and the TIMMS software is started up.

31.5.2.2 Collection of patient information and images

A TIMMS cognitive agent, the EMR Agent, performs retrieval ofdata from the electronic medical record and the data repositories.This includes all relevant patient information, such as history andphysical, past medical history, laboratory data, pathology reports,consultations, etc. The imaging agent of the TIMMS Imaging andBiosensors Engine will also retrieve and download pertinent medi-cal imaging studies [Fig. 5(C)].



Table 1. First-Order Process: Preoperative Assessment and Second-OrderProcess: Initiative of New TIMMS Project

WorkflowStep

Related TIMMSAction/Function

Agent/Device/Description

TIMMS Engine orRepository

Receiveconsult forRF ablationprocedure.

1. Launch TIMMSSYSTEMhardware andsoftware.

1. All TIMMS enginesand repositorieswill start up and acognitive agent ofthe validationengine will conducta system check.

2. All engine activitieswhich operate inthe background willcommence.

1. Kernel for workflowand knowledge anddecisionmanagement.

2. Validation engine.

1. Check that allhardware andsoftware devicesare connectedand functioning.

1. Connection checker, acognitive agent ofthe validation engine,will check that allexternal hardwareand softwaredevices areconnected andfunctioningthrough the TIMMSinfrastructure.


Initializenew case.

1. Establish a new“TIMMSPROJECT” whichwill have its ownunique TIMMSProject ID # .

2. Enter patientname, medicalrecord number,procedure class(RFA) and code(liver metastasis).



EMR AgentLocalization:

Confirm tumor locationand

characteristics

Targeting:Plan and determine

access and trajectoryof electrodes

Previous imagingreviewed byradiologist

Radiologist Logs onPACS Workstation and

RIS/HIS

Laboratory datareviewed byradiologist

Clinical Evaluation:Review patient's historyand physical, completeblood count, coagulationstudies, liver functiontests, and biopsy results.

Imaging Agent

Fig. 5(C). The links between the surgical workflow and TIMMS for collecting therequired clinical information and images from the electronic medical record (whichmay include PACS, HIS, and RIS systems, as well as TIMMS data repositories). Thisincludes patient information and imaging data. The Patient-Model Integrator createsand updates the patent-model which is used by TIMMS throughout the operativeprocedure.

Table 2. First-Order Process: Preoperative Assessment, Second-OrderProcess: Collection of Patient Information and Images

Workflow Related TIMMS Agent/Device/ TIMMS Engine orStep Action/Function Description Repository

Collect allknownpatient data.

1. Access theelectronicmedical record(EMR) todownload allpatientinformation;history andphysical; labdata; pathologyresults.

1. EMR agentaccesses HIS, andRIS systems, aswell as TIMMSdata repositories.


2. Data repository.

Collect allknownpatientimages.

1. Access the EMRto download allpertinent patientimaging studies.

1. Imaging agentaccesses PACSand TIMMS datarepositories.

1. Imaging andbiosensors engine.

2. Data repository.



31.5.2.3 Development of the patient model and treatment plan

Once the required patient information and images are retrieved andmade available, the next step is determining whether or not themetastatic tumor in the liver from a previous colon cancer, in thispatient, meets criteria for treatment with radiofrequency ablation[Fig. 5(D)].

One of the core functions throughout the TIMMS project is thecreation and maintenance of the patient-model [Fig. 5(D)]. A cogni-tive agent of the TIMMS modeling engine, the Patient-model integra-tor, which creates and updates the patient-model, will be activated.The information compiled in the patient-model is used to deter-mine whether the patient is a suitable candidate for undergoingradiofrequency ablation and if the features of the lesion are favor-able for radiofrequency ablation. Examples of parameters collectedand analyzed would include the features of the tumor (histologi-cal characteristics; stage, grade, size/volume; shape; proximity tosurface, diaphragm, vessels; portal vein patency and flow; prox-imity of diaphragm, gallbladder, colon, stomach); imaging features(CT, ultrasound, MRI, PET characteristics); and, previous treatment(systemic chemotherapy, surgery, chemoembolization). Informationobtained from the previously retrieved images [Fig. 5(B)] wouldbe used to determine feasibility of treatment based on location of

No

Yes

Patient-Model Integrator

Does TumorMeet Criteria

for RF Ablationwith CT Guidance

Visualization Manager

Surgical Modeler

Scheduling Agent

Outcomes Predictor

Treatment Assessment Simulator

Adaptive Workflow Agent

Fig. 5(D). TIMMS components that are called into play in order to create thepatient model, to determine the feasibility of the proposed radiofrequency ablationtreatment, and to specify the treatment plan.



the tumor, and, the access and trajectory of the electrodes to beemployed.

Another of the core functions of a TIMMS project is the selec-tion of the reference workflow and its modification into an execut-ing workflow which is updated as changes in the patient-model areencountered. The adaptive workflow agent and the treatment assessmentsimulator of the workflow kernel and the simulation engine, respectively,would be instrumental in determining the suitability of radiofre-quency ablation and in selecting the appropriate reference workflow,in this example, all available workflows for radiofrequency ablationof solitary hepatic metastasis from colon cancer would be consid-ered [Fig. 5(D)]. A group of possible reference workflows would beselected, simulations would be conduct, and the “best-fit” referenceworkflow would be selected. The reference workflow would thenbe transformed into the executing workflow based on the specificfeatures delineated in the patient-model. This executing workflowforms the basis for the treatment plan.

Cognitive agents of the workflow kernel and validation engines,such as the outcomes predictor, then perform data mining and out-comes predictions [Fig. 5(D)]. The patient-model, the executingworkflow, and data mined from data and peer-to-peer repositories, areanalyzed by the surgeon, assisted by the workflow kernel, to providea prospective quantitative and qualitative assessment of the likeli-hood of technical success. If the outcomes prediction is favorable,the following are potential recommendations that might be made:

(1) Literature does not support additional chemotherapy at thistime; Proceed with radiofrequency ablation.

(2) Available, connected equipment does not support cryotherapy.(3) Adjacent areas do not require protection with saline.(4) Suggest multi-prong electrode at x location/angle/depth for

portion x of the tumor, as displayed.

When the surgeon determines that the patient is a suitable candi-date, the scheduling agent of the workflow Kernel will proceed to sched-ule the procedure [Fig. 5(D)], and the TIMMS will continue to updatethe patient-model and executing workflow, in the background as



Table 3. First-Order Process: Preoperative Assessment, Second-Order Process:Development of Patient Model and Treatment Plan

Workflow Related TIMMS Agent/Device/ TIMMS EngineStep Action/Function Description or Repository

Evaluation ofwhether the livermetastasis in thispatient, meetscriteria fortreatment.

1. A comprehensiveworking patientmodel isconstructed andmaintainedthroughout theprocedure.

1. Patient-modelintegratorcreates andupdatespatent-model.

1. Modelingengine.

Select treatmentplan.

1. A group ofpossible referenceworkflows areselected.

2. Conductsimulations.

3. Select “best-fit”“executingworkflow.”

1. Adaptiveworkflowagent.

2. Treatmentassessmentsimulator.

1. Kernel forworkflow andknowledge anddecisionmanagement.

2. Simulationengine.

Assess feasibility;outcomeprobabilities;determinepotential pitfalls.

1. Outcomeprediction isperformed.

2. The patientspecific model, theexecutingworkflow, datamined from dataand peer-to-peerrepositories, areanalyzed.

1. Outcomespredictorperform datamining andoutcomespredictions.


2. Validationengine.

3. Data andpeer-to-peerrepositories.

Revise workflowbased onoutcomesassessment.

1. The adaptiveworkflow agentwill “suggest”additional changesto the executingworkflow.

1. Adaptiveworkflow agent.


The treatmentplan will befinalized.

1. Final simulationwill be run andanalyzed.

1. Treatmentassessmentsimulator.


2. The final executingworkflow isaccepted.

(Continued)



Table 3. (Continued)


The procedure isscheduled.

1. The procedure isscheduled throughthe interface withRIS/HIS.

1. The schedulingagent.


Patientundergoespresurgicallaboratorytesting andanesthesiaassessment.

1. The patient modelis updated withresults frompresurgicaltesting.

2. Simulationreassessedautomatically.

3. Any changes arehighlighted.

1. EMR agentretrieves patientdata.

2. Patient-modelintegratorupdatespatent-model.

3. Treatmentassessmentsimulatorperformssimulations.


2. Modelingengine.


The executingworkflow isrevised ifindicated.

1. The adaptiveworkflow agentwill “suggest”changes to theexecutingworkflow basedon current data.

1. Adaptiveworkflow agentsuggestsrevisions toexecutingworkflow.


3D illustrationsand models maybe constructed.

1. Preoperative 3Dillustrations ormodels may becreated to facilitatesurgery.

1. Cognitiveagents of themodeling andvisualizationmanager enginesare used tocreate requireddiagrams andillustrations.

1. Visualizationmanagerengine.

2. Modelingengine.

any additional information, such as laboratory data collected dur-ing presurgical testing, is accumulated.

If needed, 3D illustrations or 3D models to facilitate surgeryare created by the visualization manager and surgical modeling engines[Fig. 5(D)]. Through its infrastructure, TIMMS is capable of remotely



initiating the design and building of surgical 3D models by devicesthat are networked to TIMMS.

31.5.3 Operative Procedure

31.5.3.1 Initiation of operation and patient assessment

On the day of the operative procedure, after the TIMMS is startedup and its functions and connections are checked and the physi-ological monitoring has been initiated, the patient model integratorupdates patient-model from real time physiologic data. Revisionsto workflow are suggested by the adaptive workflow agent of the work-flow engine as the patient model integrator updates the patient-model[Fig. 5(E)].

Prior to the onset of the administration of anesthesia and theonset of the surgical procedure, preanesthesia assessment is requiredto ensure patient safety. Patient data is acquired by the efforts ofthe imaging and biosensors engine and the patient safety agent of thevalidation engine, and/or entered by operating room personnel. Theprocedure can only commence when the preanesthesia assessmentis complete [Fig. 5(E)].

At the onset of the procedure the cognitive agents of the workflowkernel monitor, the procedure in parallel with the evolving execut-ing workflow, recording the actual executing workflow ultimatelyused.

Validation Engine

Radiologist entersCT suite

Patient entersCT suite

Tech opens orderin RIS

Tech enterspatient ID inCT console

Patient monitoring,intubation, and

general anesthesia

Patient placed onCT table

Previous imagingreviewed byradiologist

Radiologist scrubs

Patient Safety Agent

Imaging & Biosensors Engine

Patient-Model Integrator


Fig. 5(E). TIMMS components utilized during the initiation of the operative pro-cedure and during preanesthesia patient assessment.



Table 4. First-Order Process: Operative Procedure, Second-Order Process:Initiation of Operation and Preanesthesia Assessment


Launch TIMMSSYSTEMhardware andsoftware.

1. All TIMMSengines andrepositories willstart up andundergo systemcheck.

Validationengine.

2. All engineactivities whichoperate in thebackground willcommence.

1. Check that allTIMMS hardwareand softwaredevices areconnected andfunctioning.

2. Use “defaultsettings” ormodify the specificconnections.

3. Re-enter theunique TIMMSproject ID #.

1. Connection checkerconfirms that allTIMMS hardwareand softwaredevices areconnected andfunctioning.

Validationengine.

1. Connections andlinks betweenimaging devices,monitoringdevices, displaysand monitors,input devices, etc.are checked.

1. Cognitive agentscheck that alloperativehardware andsoftware imagingdevices;mechatronicdevices; displays;and biosensordevices areconnected andfunctioningproperly.

1. Imaging andbiosensorsengine.


(Continued)





Workflowmonitoringbegins.

1. Initiate workflow. 1. Adaptive workflowagent monitors theprogress of theexecutingworkflow; anychanges to theexecutingworkflow arerecorded.


Patientmonitoringbegins.

1. The patient modelis updated fromreal timephysiologic data.

1. Patient modelintegrator updatepatient-model asthe operationprogresses.


2. Modelingengine.

Preanesthesiaassessmsnt.

1. To ensure patientsafety, an indicatorwill be displayedand the procedurecan continue, onlywhen thepreanesthesiaassessment iscomplete.

1. Patient safety agentensures thatpreanesthesiaassessment iscomplete andwithin acceptablelimits.


2. Workflowkernel.

Theexecutingworkflow isrevised ifindicated.

1. Revisions toworkflow aresuggested by theadaptive workflowagent of theworkflow engine asthe patient modelintegrator updatesthe patient-model.


2. Modelingengine.


31.5.3.2 Planning of electrode placement

As the procedure progresses, the flow of images and data throughthe TIMMS Infrastructure is maintained between the imaging equip-ment (e.g. the CT scanner and/or ultrasound) and the registrationand navigation agents of the intervention engine. Mechatronic and



LocalizingCT scan of upper

abdomen

Radiologistprepares to place

fine needlefor local anesthesia

Image Guidance:Final plans and measurements

for electrode placement Input datafrom

pre-procedure CT


Validation Engine


Visualization Manager

Navigation Agent

Registration Agent

TIMMS ICT Infractructure

Navigation:1. Real-time CT-Fluoro

2. Electro-magnetic targeting system3. Laser goniometer

Fig. 5(F). TIMMS components in preparing for electrode placement.

navigation devices are brought online. All available imaging andphysiologic data is fed through the TIMMS infrastructure to themechatronic and navigation devices for maximum operative pre-cision. This data is also assimilated by the adaptive workflow agentinto the executing workflow [Fig. 5(F)].

Any additional visualization devices, such as stereoscopic over-lay, are brought online, with all available data and images input fromthe visualization manager engine.

Once all available data has been processed by TIMMS, the adap-tive workflow agent makes final revisions to the executing workflow,and the efficacy of the proposed treatment is confirmed through thevalidation engine [Fig. 5(F)].

31.5.3.3 Placement of fine needle

When the final, specified coordinates, angles, and depths of elec-trodes are calculated and displayed, the surgeon then proceeds withskin preparation and administration of anesthesia. A fine needle,long enough to extend from the skin to the tumor to be ablated, isdeployed for two purposes. This needle will be used to adminis-ter local anesthesia and to verify adequacy of the planned electrodetrajectory. This fine needle will also serve as a guiding needle for“tandem” placement of the radiofrequency (RF) electrode along sidethe fine needle.

Feedback from Navigational devices will dictate modificationsto the executing workflow by the adaptive workflow agent [Fig. 5(G)].



Table 5. First-Order Process: Operative Procedure, Second-Order Process:Planning of Electrode Placement


Planning ismade forelectrodeplacement.

1. Imagingregistrationbegins.

2. Technologist/physician beginimaging thepatient.

3. Feedback from theimagingmodalities that areused, such as CTand/orultrasound, are fedinto TIMMS.

1. The flow of imagesand data through theTIMMS Infrastruc-ture is maintainedbetween the imagingequipment (e.g. theCT scanner and/orultrasound) and theregistration andnavigation agents.

2. This data is thenassimilated by theadaptive workflowagent into theexecuting workflow.

1. Interventionengine.


3. Workflowengine.

Mechatronicset up(robotics,naviga-tion).

1. Mechatronic andNavigationdevices arebrought online.

2. Feedback from themechatronic andnavigation devicesare transmitted toTIMMS.

1. Flow of images anddata is maintainedbetween TIMMS andmechatronic andnavigation devices.

2. This data is thenassimilated by theadaptive workflowagent into theexecutingworkflow.


2. Workflowengine.

Visualization(such asstereo-scopicoverlay)set-up.

1. Additionalvisualizationdevices arebrought online.

2. Data/images areinput fromTIMMS.

1. Flow of images anddata is maintainedbetween TIMMS andvisualizationhardware andsoftware.

1. Visualizationmanagerengine.

Finaladjust-mentsmade toexecutingworkflow.

1. Modification andacceptance ofexecutingworkflow.

1. Adaptive workflowagent makes finalrevisions toexecuting workflow.

2. Final coordinates,angles, depths ofelectrodes aredisplayed.






Validation Engine

Skin is markedusing CT laser light

Select entrance site Prep site

Repositionneedle

Place fine needlefor local anesthesia and

to confirm trajectory

Re-Image with CT

Is needlepositioned

adequately?

Yes

No

Fig. 5(G). TIMMS components for fine needle placement for local anesthesia.Ade-quacy of the trajectory of the fine needle will be confirmed, so that the radiofre-quency electrode will be placed appropriately, in tandem, along side the fine needle.

Table 6. First-Order Process: Operative Procedure, Second-Order Process: FineNeedle Placement

Workflow Step Related TIMMS Agent/Device/ TIMMS Engine orAction/Function Description Repository

Placement of fineneedle for localanesthesia.

1. Adequacy ofthe trajectoryof the fineneedle will beconfirmed, sothat theradiofre-quencyelectrode willbe placedappropriately,in tandem,along side thefine needle.

1. Feedback fromnavigationaldevices willdictatemodificationsby the adaptiveworkflow agentto theexecutingworkflow.



31.5.3.4 Placement of radiofrequency electrode and ablation of tumor

The radiofrequency (RF) electrode is deployed with the biosensorand imaging, intervention, and validation engines enabling coordinated,synchronized function of real time imaging devices (such as CT-fluoro or ultrasound), registration and navigation agents, androbotic devices. Once proper placement of the RF electrode is



Insert electrode withtamdem technique

with CT-fluoro

Re-Image with CTImaging & Biosensors Engine

Validation Engine

Intervention Engine

Navigation Agent

Registration Agent

Is electrodepositioned

adequately?

Ablate tumor as perprotocol for ablation

device

Yes

NoRepositionelectrode

Fig. 5(H). TIMMS components for radiofrequency electrode placement and forablation of the tumor.

confirmed, the RF generator may be activated and the tumor ablatedvia the workflow kernel and intervention engines [Fig. 5(H)].

31.5.3.5 Assessment of initial ablation of tumor and completionof operation

After the initial treatment, the results of the ablation are evaluated.The post-ablation images and data from biosensors will be analyzedand the patient model will be updated through a synchronized effort

Table 7. First-Order Process: Operative Procedure, Second-Order Process:Electrode Placement and Ablation

Workflow Related TIMMS Agent/Device/ TIMMS Engine orStep Action/Function Description Repository

RF electrodeinsertion andfeedbackbasedadjustments.

1. The RF generatoris brought online.

2. The RF electrodeis deployed.

1. Registration andnavigation agentsprovidecoordinated,synchronizedfunction with realtime imaging.





(Continued)





Ablation withphysiologic,and imagingfeedback.

1. Tumor ablation isperformed.

1. Workflowkernel.



Evaluateresults ofinitialablation.

1. The post-ablationimages and datafrom biosensorswill be analyzedand the patientmodel will beupdated.

2. The adaptiveworkflow agentwill suggestchanges to theexecutingworkflow basedon current data.

3. Need to repeatablation will beindicated.

4. The planningsteps will berepeated withdevelopment ofnew executingworkflow.


2. Patient modelintegrator.


2. Modelingengine.

3. Workflowkernel.


Additionalablationsperformedwithphysiologicand imagingfeedback ifnecessary.

1. Reposition RFelectrode asindicated.

2. Perform additionaltumor ablation.

1. Registration andnavigation agentsprovidecoordinated,synchronizedfunction with realtime imaging.







Remove electrode(while heating tract)

Re-Image with CTwith IV contrast

Monitoring


Validation Engine

Workflow Kernel

Modeling Engine

Has all ofthe tumor been

ablated?

Have criteriabeen met for

completeablation

Repositionelectrode

No

Yes

YesNo

Re-Image with CT

Rescan with CT toR/O complications

Fig. 5(I). TIMMS components for evaluating adequacy of tumor ablation, and forthe completion of the operative procedure.

of the imaging and biosensors engine, the modeling engine, the workflowkernel, and the validation engine. If necessary, the adaptive workflowagent will suggest changes to the executing workflow based on cur-rent data, for a second ablation [Fig. 5(I)].

If all parameters indicate a successful ablation treatment, theelectrode will be removed and a completion scan will be performedto rule out complications.

31.5.4 Post-Operative Care

31.5.4.1 Completion of operation and patient assessment

After the ablation procedure is completed, physiological monitoringcontinues, with the imaging and biosensors engine updating the patientmodel. The EMR Agent will update the patient’s medical recordswith a report of the procedure and its outcome [Fig. 5(J)].

The validation engine will perform a variety of validation func-tions, including outcomes analysis, statistical evaluation, compli-cation recording, etc. This data is sent to repositories and the EMR,and will be available for additional evaluation and research pur-poses. All required quality assurance procedures and documenta-tion will be completed. When post-operative assessment indicates



Table 8. First-Order Process: Operative Procedure, Second-Order Process:Assessment of Initial Ablation and Completion of Operation


Evaluateresults ofinitialablation.

1. The post-ablationimages and datafrom biosensorswill be analyzedand the patientmodel will beupdated.

2. The adaptiveworkflow agentwill suggestchanges to theexecutingworkflow basedon current data.

3. Need to repeatablation will beindicated.

4. The planningsteps will berepeated withdevelopment ofnew executingworkflow.


2. Patient modelintegrator.


2. Modeling engine.3. Workflow kernel.4. Validation engine.

Additionalablationsperformedwithphysiologicandimagingfeedback ifnecessary.

1. Reposition RFelectrode asindicated.

2. Performadditional tumorablation.

1. Registration andnavigation agentsprovidecoordinated,synchronizedfunction withreal timeimaging.





that patient is stable and ready for transfer, and when validationprocedures have been completed, the patient safety agent will indi-cate that the patient is ready for transfer to the recovery room[Fig. 5(J)].



Extubation andpost-procedure careby anesthesiologist

Apply dressing

Patient transferred torecovery room

Radiologist signs offdictation

Images sent to PACSby technologist

Case dictated byradiologist


Patient Safety Agent

Validation Engine

EMR Agent

Case finalized in RISby tech

Patient discharged

Fig. 5(J). TIMMS components involved during post-operative care.

31.6 MODELLING TOOLS OF TIMMS AND STEPSTOWARDS STANDARDS

Standards relating to medical imaging and communication fornon-real time diagnostic and related activities are well defined byDICOM and are an integral part of TIMMS. Most of the image andpresentation states IOD’s, which are defined in DICOM, are alsorelevant to surgery.

Models and their associated management have not been con-sidered in DICOM intensively, except through some work done inDICOM WG 07, WG 17 and WG 22. Modelling and simulation insurgery however, are key functions for SAS’s pre- and intraopera-tively. Interfacing of tools which support these functions comprisesa relatively new scope for DICOM.

To define model and simulation, a definition by O Balci4 may beused “A model is a representation or abstraction of something suchas an entity, a system or an idea. Simulation is the act of experiment-ing with or exercising a model or a number of models under diverseobjectives including acquisition, analysis and training.”As indicatedin Fig. 2, both modelling and simulation are critical components ofan SAS, particularly for planning and intervention activities.

It will be a significant extension of current DICOM efforts tocomplement the image centric view with a model centric view fordeveloping DICOM objects and services. Some IOD’s which makeuse of the concept of a model are listed in DICOM PS 3.3 as part



Table 9. First-Order Process: Post-Procedure Care

WorkflowStep

Related TIMMSAction/Function

Agent/Device/Description

TIMMS Engineor Repository

Physiologicand post-anesthesiamonitoring.

1. Post-operativeassessment isperformed.

2. The patient modelis updated fromreal timephysiologic data.

3. Orders forpost-operativecare are written.

1. Patient modelIntegrator andEMR agent updatepatent model andpatient record.


2. Modelingengine.

3. Kernel forworkflow andknowledgeand decisionmanagement.

Procedurevalidations.

1. Validation processes,including outcomesanalysis, statisticalevaluation,complicationrecording, etc. areperformed.

2. Data sent torepositories andEMR.

Cognitive agentsperform qualityassuranceprocedures andEMR agent addsdocumentation torepositories andEMR.



Dischargepatient torecoveryroom.

1. When post-operativeassessment indicatesthat patient is stableand ready fortransfer, and whenvalidationprocedures havebeen initiated, anindicator willlight up.

1. Patient safety agent. 1. Validationengine.

of annex C 8.8. “radiotherapy modules.” Currently, approximately40 modules have been specified for radiation therapy. They implya limited spectrum of data types and data structures with differ-ent degrees of complexity, e.g. simple lists or tree structures. Inthe context of a TIMMS, a more comprehensive view on mod-elling than for example in radiation therapy, will be necessary.



Not only as regards the modelling tools for generating differenttypes of data structures, but also with respect to the modellingengine which carries out the modelling task. This engine willoccupy a central position in the design of a SAS and the TIMMSinfrastructure.

By default, the broader the spectrum of different types ofinterventional/surgical workflows which have to be considered forstandard interfacing support, the more effort has to be given fordesigning appropriate IOD modules and services. The followinglist contains some examples of modelling tools and aspects, derivedfrom different types of surgical workflows, which may have to beconsidered for future standard activities such as DICOM:

• Geometric modelling including volume and surfacerepresentations

• Properties of cells and tissue• Segmentation and reconstruction• Biomechanics and damage• Tissue growth• Tissue shift• Prosthesis modelling• Fabrication model for custom prosthesis• Properties of biomaterials• Atlas-based anatomic modelling• Template modelling• FEM of medical devices and anatomic tissue• Collision response strategies for constraint deformable objects• Variety of virtual human models• Lifelike physiology and anatomy• Modelling of the biologic continuum• Animated models• Multiscale modelling• Fusion/integration of data/images• Registration between different models including patient, equip-

ment and OR• Modelling of workflows



Real time aspects identified for imaging during intervention areequally applicable for the generation and management of these mod-els. In addition to defining mechanisms to enable real time communi-cation, it will also be one of the first tasks of standardization to agreeon a list of relevant models to be considered for DICOM IOD’s.

31.7 GENERAL MOTIVATION FOR STANDARDSIN SURGERY

31.7.1 Meetings

Anumber of special workshops and seminars which have addressedthe medical, technical, economic and related problems of the ORhave taken place in recent years in Europe and in the USA. The mostnotable recent meetings with a focus on IGT, surgical workflow andstandards in the OR were:

(1) UCLA Seminars on Imaging and Informatics, September 22–25,2003, Lake Arrowhead, CA, USA.5

(2) Leipzig University Forum, ICCAS, October 2003.6

(3) Workshop on “OR2020: Operating Room of the Future”, March18–20, 2004, Ellicott City, MA, USA.7

(4) CARS/SPIE “Joint meeting on Surgical Workflow, PACS and theOR of the Future,” June 26, 2004, Chicago, IL, USA.8

(5) UCLA Seminars on Imaging and Informatics, October 4–6, 2004,Lake Arrowhead, CA, USA.9

(6) NCIGT and NA-MIC Workshop on Image Guided Therapy,Rockville, MD, October 19–20, 2006.10

Standards and interoperability of devices were a common theme ofalmost all of these meetings. Exemplary for this effort are the insightand results given by two working groups established for the OR2020 Workshop.7 It is worth noting that approximately one third ofthe participants of the OR 2020 Workshop were MD’s, R&D PhD’sand representatives from the industry and government institutionsrespectively. The problems which were identified before and thenelaborated during the workshop by the two working groups, aresummarized as follows:



31.7.1.1 Working group 1: Operational efficiency and workflow

This group focused on examining requirements for achievingincreased efficiencies in the OR. These requirements focused onneeded mechanisms for accessing and obtaining correct and cur-rent patient-related information and scheduling, and accessing useof correct surgical tools. The group also discussed developing sur-gical practice standards that define day-to-day, step-by-step sur-gical workflows. Four of the most critical technical needs whichwere identified for improving OR efficiencies and workflow are asfollows:

(1) creating accessible “patient-centric” medical records,(2) developing readable equipment locator/tracking mechanisms,(3) resolving OR teamwork/personnel issues; and(4) developing and following technical standards in the OR.

31.7.1.2 Working group 2: Systems integrationand technical standards

This group focused on the need for interoperability among a broadrange of devices that are used in the OR. To achieve seamless integra-tion among devices, it was recommended, that a standard interfacefor interoperability among these technologies should be developedusing a plug and play platform. This group also discussed the needfor device standards that will enable configurability and easy use ofthese tools in the OR.

31.7.2 Recommendations

Many details have been listed by the two working groups as poten-tial solutions to the above problems, here included as a summaryrecommendation7:

(1) Standards, standards, standards. If there was an overarchingtheme of the workshop, this was it. Standards are needed in allareas, and must be developed through a concerted effort involv-ing companies, government agencies, academic institutions, and



perhaps standards organizations. Research studies of surgicalworkflow and efficiencies are required to develop practice stan-dardization and thus realize improvements.

(2) Progress on the first recommendation will also enable progresson device interoperability. It is recommended that research bedevoted to developing common user interfaces among medicaldevices, and that the device industry take the lead in perform-ing this research with input for academic institutions and gov-ernment agencies. A “plug and play” architecture for medicaldevices is also needed.

Of particular interest is here the statement that standards areneeded in all areas and must be developed through a concerted effortinvolving companies, government agencies, academic institutions,and perhaps standards organizations. Motivating these players towork in a concerted effort towards standards can only be achieved,of course, if it is in their business interest. One of the critical questionswhich needs to be addressed is:

“Is the OR of the Future (ORF) a viable economic reality?”11

31.8 SURGICAL WORKFLOWS (WF) FOR MEDICALIMAGING (MI) IN SURGERY

Standards for creating and integrating information about patients,equipment, and procedures are vitally needed at the outset in plan-ning for an efficient ORF. To determine these standards, research isneeded to define day-to-day, step-by-step surgical workflow prac-tices and create surgery workflow models per procedures or pervariable cases.

An example that might be used to better understand (and even-tually improve on) or workflows and efficiencies is the recent workcarried out by the integrating the healthcare enterprise (IHE) initia-tive and its definitions of work profiles and efficiencies in healthcareoutside of the surgical room. This body of experts develops rec-ommendations for the healthcare industry on how to implement



standards. (Note: IHE’s members do not develop the standardsthemselves.)

Furthermore, the IHE initiative has developed “integration pro-files” that enable consistent access to images and reports for cer-tain medical specialties (such as radiology). Surgical profiles havenot been developed yet, but they are needed (a widespread opin-ion expressed at the OR 2020 Workshop), as is a “surgical DICOM.”Today’s DICOM standard is not suitable for many imaging typesand working modes that are needed in the or (e.g. it does not coverreal time, and 3D and higher dimensional issues, nor does it addressinteractivity).

31.8.1 Recording of Workflows

With these objectives in focus, a detailed workflow analysis12 hasbeen carried out by the Technical University Berlin (TUB) andthe Innovation Center for Computer Assisted Surgery (ICCAS) inLeipzig. The aim is to model and visualize surgical procedures inorder

– to allow a correlation between workflows of different types ofsurgical procedures, e.g. to obtain a measure of similarity betweenworkflows,

– to assist in identifying (e.g. through simulation, see Fig. 6), thoseparts of the same and between different workflows (Surgical Inte-gration Profiles — SIP’s) where a process redesign with automatedactivities may prove to be of a clinical and economic advantage,

– to provide concepts and data to assist in the specification, design,implementation and in vivo usage of new information and com-munication technology and mechatronic systems.

An important aspect when recording workflows is their mod-elling and representation technology. Amongst many possibilitiesand derived from the above work, the workflow management coali-tion standard is being recommended for workflow recording withinWG24. Figure 7 shows an example of a surgical workflow in ortho-pedic surgery.



Fig. 6. Simulation of surgical workflow.

31.8.2 Dynamics of Workflows and the Model of the Patient

It is important to consider workflows to be dynamic entities. ForWG24, they serve as reference (not best practiced!) workflows andare updated at regular intervals to detect within the workflows pos-sible changes in imaging and patient modelling requirements. Forexample, it can be expected, that molecular imaging modalities willimpact workflow for oncologic patients substantially.13 Radiationresistant parts of a tumor may be defined with molecular imagingto a higher precision giving rise to include surgical/interventionalablation procedures combined with radiation therapy as a possibleregimen.

A well defined workflow and a high fidelity patient model willbe the base of activities for both, radiation therapy and surgery. Con-sidering the present and future requirements for surgical planningand intervention, such a patient model must be n-dimensional, weren may include the spatial and temporal dimensions as well as a



Fig. 7. A workflow example in orthopedic surgery.



number of functional variables. 2D imaging and 2/2 D or 3D recon-structions are, by definition as subset of an n-dimensional patientmodel and its representation in the electronic medical record (EMR).

As the boundaries between radiation therapy, surgery and inter-ventional radiology are becoming less well defined,14 precise patientmodels will become the greatest common denominator for all ther-apeutic disciplines.

31.9 CONCLUSION

In summary, TIMMS provides a process and system for a com-prehensive surgical assist system, which combines and integratesall of the necessary information and communication technology;workflow analysis, data processing and data synthesis; interactiveinterfaces between surgeon and mechatronic devices; and agents; toprovide comprehensive assistance and guidance throughout com-plex medical and surgical therapies, such as image guided surgery.The components of TIMMS, which are modular, scalable and maybe distributed in location, act synergistically to provide functionalityand utility that exceeds the sum of its individual parts.

References

1. Pentacost M, Review of the operating room of the future, OR-2020,UCLA Seminar on Imaging and Informatics, October 4–6, 2004.

2. Lemke HU, Vannier MW, The operating room and the need for anIT infrastructue and standards, International Journal of CARS 1(3):Springer, November, 2006.

3. Lemke HU, Berliner L, Specification and design of a therapy imagingand model management system (TIMMS), SPIE Proceedings 2007.

4. Balci O, Verification, validation and certification of modeling and simu-lation applications, Proceedings of the 2003 Winter Simulation Conference,2003.

5. http://www.radnet.ucla.edu/Arrowhead2004/Seminar2003.html.6. Leipzig University Forum 2003 (Technical Report).7. Cleary K, Kinsella A, OR 2020: The operating room of the future,

J Laproendosc Adv Surg Tech A 15(5): 497–573, 2005.



8. Lemke HU, Trantakis C, Kochy K et al., Workflow analysis for mecha-tronic and imaging assistance in head surgery, in Lemke HU, VannierMW, Inamura K et al. (eds.), Computer Assisted Radiology and Surgery,pp. 830–835, Elsevier, Chicago, IL, 2004.

9. http://www.radnet.ucla.edu/Arrowhead2004/.10. Image Guided Therapy Workshop, Rockville, MD, October 19–20,

2006, Technical Report (to be published).11. Dritz R, Is the operating room of the future a viable economic reality?

Anesthesiology 104(6), 2006.12. Lemke HU, Surgical Workflow and Surgical PACS, UCLA Seminar on

Imaging and Informatics, October 4–6, 2004.13. Niederlag W, Lemke HU et al., Molecular Imaging, Health Academy,

2006.14. Onik G, Prostate imaging goals shift as therapeutic options expand,

Diagnostic imaging 55–64, November, 2005.


CHAPTER 32

Future Trends in Medical andMolecular Imaging

Atam P Dhawan, HK Huang and Dae-Shik Kim

Recent advances in computerized medical imaging and associated areasof basic and applied sciences, engineering, medicine and computing tech-nologies have created a synergy among researchers and scientists toexplore complex issues related to the onset of critical diseases for betterunderstanding of physiological processes from molecular to organ andbehavioral levels. Future trends are expected to continue to develop morecomplex and sophisticated tools in the investigation of biological func-tional and pathologies associated with the onset of critical diseases forearly diagnosis, treatment, evaluation and interventional protocols. Thischapter points out some areas and challenges of future technology devel-opment with potential applications.

32.1 FUTURE TRENDS WITH SYNERGY IN MEDICALIMAGING APPLICATIONS

In recent years, clinical medicine and healthcare have been rev-olutionarized through multidisciplinary technological advances.Critical health care technologies including diagnostic radiology,surgery and rehabilitation extensively use computerized systemsto continuously improve diagnosis, treatment and prognosis. Thesetechnological advances have emerged from a synergy of manyspecialized areas including engineering, computer science, math-ematics, and other basic, applied and social sciences. Today, we cancritically measure neurological signals of the brain with under a

829



millimeter spatial resolution over a fraction of second to diagnoseand characterize neurological disorders and diseases. As the techno-logical contributions are impacting medicine and clinical practice,higher goals and standards are being established to achieve betterdiagnosis, treatment and healthcare. The synergy of advanced tech-nologies such as computerized medical imaging, high volume datastorage and database architecture, picture archiving and commu-nications systems, wireless networking, and display technology isleading to better patient care with more computer processing, mod-eling and analysis, leaving less room for guesswork.

Medical imaging technologies provide complimentary informa-tion from molecular to organ levels. Current and future trends infMR, diffusion-MR, positron emission tomography (PET), ultra-sound, and optical imaging are targeted towards obtaining molecu-lar information from the cellular structure of tissue.

Advanced imaging techniques are expected to explore biolog-ical investigations to develop signatures and models for under-standing physiological processes associated with “presymptomatic”conditions leading to specific diseases and pathologies. Futuretechnological developments in multimodal molecular and cellularimaging should allow early detection of cellular/neurological devia-tions in critical diseases such asAlzheimer’s disease, autism, or mul-tiple sclerosis, before the first symptomatic sign(s). This is of greatimportance given the fact that sometimes many neurological dis-eases (such as Alzheimer’s disease, where several decades can passbetween the initial neurological deviations and observed behav-ioral changes) have exceedingly long incubation periods. Currentimaging paradigms rely on the expression of the first symptomaticsign(s), scientists then try to correlate, on an ad hoc basis, the observedsigns with cellular and/or neurological deviations. The problem isthat by the time the symptoms are expressed, the disease is prob-ably already in a relatively advanced stage (e.g. shrinkage of thebrain in the case of Alzheimer’s disease). Therefore, it is importantto improve the patient care that the future imaging methods and pro-tocols must be able to detect critical diseases in a presymptomaticstage for better preventive treatment, and at the same time provide


Future Trends in Medical and Molecular Imaging 831

robust, efficient and reliable support for early diagnosis, treatment,evaluation and intervention protocols.

32.1.1 Trends in Targeted Imaging and Image Fusion

Targeted imaging provides a systematic investigation into a physio-logical process for the assessment of the nature and extent of pathol-ogy through multilevel analysis of information from molecular toorgan levels. Recent discoveries in molecular science and medi-cal imaging contrast agents are setting up future trends in design-ing specific contrast agents for multidimensional medical imagingmodalities such as ultrasound, PET, fMR and optical fluorescenceimaging to study molecular interactions defining abnormal physio-logical processes linked with the onset of a disease. Targeted contrastagents provide an opportunity to image physiology or pathologythat might be otherwise difficult to distinguish from the surroundingtissue without targeted contrast enhancement. For example, encap-sulated microbubbles in ultrasound imaging can provide informa-tion about activated neutrophil, a cell involved in the inflammatoryresponse (www.ImaRx.com).

The technology of using specific contrast agent for targetedimaging can also be used for better drug delivery in criti-cal therapeutic protocols. It is expected that future diagnostic,treatment-evaluation and therapeutic-intervention protocols willuse specific multimodality targeted imaging with computerizedanalyses through models using molecular signatures of physiolog-ical processes. For example, tumor-induced angiogenesis is a com-plex process involving tumor cells, blood, and the stroma of thehost tissue. Studies related to angiogenic growth factors linked withendothelial cells has shown that vascular integrin alpha v beta 3may be a useful therapeutic target for diseases characterized byneovascularization.1 Thus, αvβ3 is a prime candidate for molec-ular targeting and can be monitored through advanced imagingmethods.

Furthermore, nanoparticle conjugated novel MRI contrastagents can be used in order to directly observe gene expression,



metabolism and neurotransmission. The great advantage of thesenovel contrast agents is their ability to provide such information ina noninvasive fashion. This enables important cellular and metabolicprocesses to be observed for the first time from whole animals overrepeated time periods.2

One of the major challenges in using several advanced technolo-gies such as EKG, EEG, CT, MRI, fMRI, PET, etc. is how to integrateinformation from several complimentary technology instruments.The process of information fusion requires computer processing oflarge data files with a common standard and coordinate system sothat information from different instruments can be easily read andintegrated to target the specific region. Though efforts have beenmade in establishing common formats for images and data fromdifferent instruments, the files are usually transported to a commoncomputing environment off-line after the images and measurementsare acquired from corresponding instruments. Such systems providelarge datasets to handle in the real-time on-demand environment. Itis still a challenge for acquisition, storage, analysis, and communica-tion of integrated information resulting from multimodality imagefusion.

32.1.2 Image Fusion for Surgical Intervention

Real-time information fusion and analysis is critical for interactivesurgical interventional protocols. Today, a surgeon studies everypiece of available radiological and pathological information includ-ing 3-D images from X-ray CT, MRI, Nuclear medicine and ultra-sound images, and pathology reports before planning a surgery.Computers are used to plan surgical or radiation procedure before apatient is brought into the operating room in piecemeal fashion.Computers are used to integrate all information from radiologi-cal imaging and diagnostic protocols but without a user friendlygraphic user interface display mechanism. Computer-based mod-eling and simulations can also be used to predict the outcome tostudy and compare alternatives. Though this is a grand step for-ward in using technologies in surgery, this is just a beginning in



learning how a spectrum of technologies can be used to optimizethe diagnosis, interactive surgery, treatment, and patient care.

However, the current operating rooms and instrumentationtechnologies do not allow an interactive surgical procedure in theoperating room where the intermediate steps can be evaluated inreal-time to ensure the success of the planned surgical procedure.The new goal in operating room of the future is to integrate tech-nologies to facilitate interactive surgery with intervention proce-dures for minimally invasive surgery with completely/maximallypossible successful outcome.

There are some leading clinical and research facilities wherethe operating rooms of the future are being designed. At MGH,Dr Warren Sandberg is designing an operating room of thefuture (ORF). Information is found on the internet (http://www.cimit.org/orfuture.html). Another ORF is being built toinclude MRI in the operating room at the University ofMaryland Medical Center (http://www.umm.edu/news/releases/or_future_opening.html) (see Chapter 31 for additional details).These efforts are starting to realize the potential of interactiveintegrated information to perform minimally invasive surgery.However, defining specific sources of information needed for dataacquisition and information fusion to be used during the surgery toevaluate surgical plan enroute with dynamics of the tissue responsewould be a key factor that would guide future design of ORs. Theother issue is whether there should be one general ORF or theyshould be designed on the basis of specific categorical needs defin-ing specific instrumentations for data fusion and analysis for thesurgeon’s use. The design of future operating room has to addresschallenges in the following three categories:

(1) Architecture: If the patient has to be examined through differ-ent complimentary diagnostic and radiological systems suchas EKG, ECG, EMG, DF, CT, ultrasound, MRI and/or nuclearmedicine before, and sometimes during the operation, thepatient should remain on a stable operating table without muchmovement. Therefore, every instrument has to come to the



patient bed rather than having the patient moved into differentrooms. Therefore, the first challenge is to have instrumentationand architecture of the operating room synchronized in such away that any instrument needed for measurement or imagingcan be easily slipped around the patient bed without obscuringthe surgeon’s access to the patient.

(2) Networking and Information Fusion: If there are severalinstruments that are being used to acquire complimentary infor-mation necessary to evaluate the surgical procedure and tissueresponse in real-time, all data files from different instrumentsmust follow common format and standards for networkingand communications to a central computer. Thus, data output,communication and networking among all equipment must beeffective and without any noise interference. Such type of dataacquisition and wireless communication environment requirea very fast and high volume data throughput with commonstandards.

(3) Human-Machine Interface: The overall goal of bringing alltechnologies together is to help a surgeon in the continuousevaluation of the ongoing surgical procedure for any neces-sary modification to successfully perform minimally invasivesurgery. This requires an enormous task of information fusion ofhigh-volume data from several instruments, and analyses to fil-ter out only the useful and necessary information that a surgeonneeds to know to evaluate and revise the surgical plan.All of thishas to be done in real-time with a proper and effective human-machine interface to the surgeon and in some cases, guiding thesurgical instruments as well such as needed in robotic assistedsurgery.

Even with the above conceptual description of the future operat-ing room, it is quite clear that technologies and expertise from somany disciplines including architecture, engineering, computer sci-ence, basic and applied sciences, psychology and human perceptionhave to work together to create a synergy to successfully develop anefficient operating room of the future.



32.2 TRENDS IN LARGE-SCALE MEDICAL IMAGE DATASTORAGE AND ANALYSIS

With the large amount of image data accumulated daily from med-ical imaging modalities, picture archiving and communication sys-tems (PACS) in hospitals, we can take advantage of these resourcesto investigate the concept of imaging informatics. PACS-based med-ical imaging informatics is to use existing PACS resources includingimages and related data for systematic large-scale horizontal andlongitudinal clinical service, education, and research applicationsthat could not have been performed before because of insufficientdata and unavailable tools.

32.2.1 PACS-Based Medical Imaging Informatics

Medical imaging informatics infrastructure (MIII) is the vehicleto facilitate the utilization of PACS in addition to its daily clini-cal service. Figure 1 illustrates MIII components and their logicalrelationship.3 The PACS, Data Grid, Grid Computing, and CAD-PACS integration discussed in Chapter 21 are components in theMIII infrastructure. The integration of CAD-PACS, Data Grid andGrid Computing is an example of large-scale MIII components inte-gration. Another example is the use of Data Grid and ComputingGrid for image-based clinical trials to be discussed in the following

Customized Software

ResearchApplication Middleware

Clinical ServiceApplication Middleware

Education Application Middleware

MII Database & Knowledge Base Management, Simulation and Modeling, Data Mining

Image ProcessingAnalysis, CAD, Grid

Computing, StatisticsTools

Visualizationand Graphics

Tools

Graphical UserInterface

Tools

DataSecurity

CommunicationNetworks

Medical Images, PACS, Data Grid, and Related Database

Fig. 1. MIII components and their logical relationship. The second layer frombottom are common tools in MIII; the third layer is general database, knowledgebase, simulation and modeling, and data mining software packages; the fourthlayer is application specific software; and the top layer is customized applicationsoftware. CAD, PACS, data grid and grid computing facilities are components ineach layer.



subsection. The use of MIII concept for other applications in large-scale medical image data storage and analysis will be continuouslyexplored and solidified by researchers in medical imaging.

32.2.2 Data and Computing Grids for Image-BasedClinical Trials

Clinical trials play a crucial role in testing new drugs or devices inmodern medicine. Medical imaging has also become an importanttool in clinical trials because images provide a unique and fast diag-nosis with visual observation and quantitative assessment. Atypicalimaging-based clinical trial consists of: (1) A well-defined rigorousclinical trial protocol; (2) a medical image core that has a qualitycontrol mechanism, image analysis, a biostatistics component, anda server for storing and distributing data and analysis results; and(3) many field sites that generate and send image studies to the medi-cal imaging core.As the number of clinical trials increases, it becomesnecessary for the core which services multiple trials to have a serverrobust enough to administrate and quickly distribute informationto worldwide participants. The Data Grid can satisfy the aforemen-tioned requirements of image-based clinical trials.4 A general orga-nization of an image-based clinical trial with responsibilities of eachcomponent is depicted in Fig. 2. An example of a fault-tolerant DataGrid with computational services testbed for image-based trials withtwo storage nodes (University of Southern California and Univer-sity of California, Los Angeles), and a third storage with computa-tion services (Image Processing and Informatics Laboratory (IPI)) isshown in Fig. 3. In this testbed, the DICOM GAP provides DICOMworkstations at field sites access to the Data Grid for trial imagestorage and analysis, and results retrieval.

32.3 MEDICAL IMAGING TO BRIDGE THE GAP BETWEENDIAGNOSIS AND TREATMENT

Most medical images have been used for diagnostic purpose. Inorder to communicate images from modalities to workstations for



Fig. 2. Ageneric image-based clinical trial organization layout and responsibilitiesof each component.

Fig. 3. An image-based clinical trial testbed with two storage nodes (USC, UCLA)and a third storage node with computational services (IPI, USC). The DICOM GAPprovides access for DICOM workstations at filed sites to store, perform image anal-ysis and retrieve results from the DICOM Data Grid.



different applications, DICOM standard was developed in 1992.Radiation therapy (RT) was the first to use medical images for treat-ment planning and dose calculation. Since RT uses many other medi-cal image data unique to its own application, DICOM standard wasnot sufficient to cover its utilization for RT. As a result, DICOMRT was ratified with seven RT-specific DICOM objects in 1999.5

Both DICOM and RT DICOM are evolving standards, new DCIOMobjects and specifications are being developed and rectified continu-ously. During the past ten years, image-assisted and guided surgeryhas become popular. In order to perform image-assisted surgerysuccessfully, it requires image communication and display as wellas surgical workflow profiles. The results are the need of an extendedDICOM standard for image-guided surgery. The new DICOM Work-ing Group (WG24) with experts in both medical imaging andsurgery has been formed to develop an image-assisted and guidedsurgery specification.6 We use minimally invasive spinal surgery(MISS) as an example to describe the concepts of image-guided andassisted surgery and the role of medical imaging in bridging the gapbetween diagnosis and treatment, as well as the informatics aspectof MISS.

32.3.1 Minimally Invasive Spinal Surgery (MISS) — Background

Back and neck pain is the price human beings pay for poor pos-ture, prolonged sitting, lifting, repeated bending, obesity, and injuryfrom accidents. It is providing the USA with a massive economicheadache. Approximately 85 percent of inhabitants of the west-ern world are afflicted with some degree of back or neck pain atsome point in their lives. About 25 percent of our population hasbeen incapacitated for two weeks or more due to back pain andan estimated eight to ten million people have a permanent disabil-ity from it. The economic impact is obvious. In most cases, simpletreatments such as bed rest, exercise, physiotherapy, and pain med-ication bring relief. Many sufferers are not so fortunate. If one ormore of their vertebral discs ruptures and presses on nerve roots,



Fig. 4. Examples of MISS on lumbar, cervical, and thoracic spines. Arrows showthe areas where the disc protrudes the spine. Upper row: preoperation, Lower row:postMISS operation (courtesy of Dr John Chiu).

the pain radiating from the back or neck and down the limbs canbe incapacitating and severe (see top row, Fig. 4). Until recently,the only treatment was surgical removal of part of the ruptureddisc, a major operation that required general anesthesia, the dis-section of muscle, removal of bone, manipulation of nerve roots,and, at times, bone fusion. In an effort to overcome the disadvan-tages of traditional surgical techniques, the scientific medical com-munity began exploring the use of endoscope (arthroscopy) for MISSoperation.

An endoscope provides clear visualization and magnification ofdeep structures. With the advancement of scientific technology andminiaturization, including fiber optics, video imaging technology,laser treatment and experience gained through minimally invasivespinal surgery, there is a less traumatic discectomy procedure forsome patients with disc problems. In the recent years, the devel-opment of image-guided surgery has improved the precision andreduced surgical tissue trauma.7



32.3.2 The MISS Procedure

The MISS procedure is performed in a digital endoscopic operat-ing room (OR, Fig. 5) with an array of various types of presurgicaldiagnostic images including digital fluorography (DF), CT, MR, andultrasound; and real-time vital sign waveforms (right bottom andleft upper, Fig. 5), surgical images like DF and digital endoscopicimages (Top left and right upper, Fig. 5).8 Depending on the type ofspinal surgery, the MISS procedure is done with the patient undereither a local anesthesia or in some situations, a brief general anes-thesia. Using the minimal exposure Digital Fluoroscopy (DF) andthe endoscope digital video image for guidance, a small hollow

Digital Endoscopic MISS OR facility

MD’s

StaffRN, Tech

EMG Monitoring

C-Arm Fluoroscopy

C-Arm Images

Image manager reportVideo endoscopy

EEG monitoring

Left side of OR

Image view boxes

Teleconferencing - telesurgery

Laser generator

MRI image — PACS

Fig. 5. A digital endoscopic OR for MISS operation of today with various imagesand waveform data scattering in the suite. With the design and implementationof the ePR, future MISS OR will have the benefit of streamlined patient pre andduring surgical operation data to improve the efficiency and effectiveness of MISS(courtesy of Dr John Chiu).9



tube (∼ 6 mm in diameter) is inserted into the disc space. A vari-ety of surgical instruments can be used through the hollow tubeincluding miniforceps, curettes, trephines, rasps, burrs, cutters, andother types of probes for disc decompression. Lasers are also used toshrink and tighten the disc and to remove portions of the protrudeddisc. The procedure takes about 15 minutes per disc on the average.The discectome, a hollow probe, is used to cut, suction and removesmall pieces of disc material. Enough disc material is removed for

C-ARM Gateway ePRServer &

Web PortalWeb 1000PACS Server

EndoscopicVideo Image

Surgical Video

Waveform Data fromCIS Clinical Database

(A)

Patient WorklistJohn Doe

PACS ImagesMRCTUS

Real Time Surgical ImagesC-ARM FluoroscopyEndoscopic ImagesSurgical VideoWave form Signal

EMGEEG

Jane Doe

(B)

Fig. 6. (A) The infrastructure of a MISS ePR system showing the connections ofvarious presurgical and during operation MISS images, waveforms and relatedclinical data (see also Fig. 5). (B) The patient worklist graphic user interface designof the ePR.



decompression of the nerve root. A laser is used to shrink and totighten the disc. The supporting structure of the disc is not affected.Upon completion, sutures and a small band-aid are applied to theincision. This endoscopic procedure is also used currently for bonydecompression in spinal stenosis. Overall, endoscopic spine surgeryhas a patient satisfaction score of 91 percent, and a 94 percent suc-cess rate (for a single level of disc problem). The complication rateis much less than 1 percent and mortality rate directly from spinaldisc surgery is zero.9 Figure 4 shows a cervical, thoracic and lumbarspine before and after MISS operations.

32.3.3 The Informatics Aspect of MISS

Image informatics technologies can been used to facilitate MISS10 byfirst integrating all images, vital sign waveforms, and other relateddata to streamline surgical workflow; and implementing an ePR(electronic patient record) system for data management and out-come analysis. Figure 6 shows the work-in-progress of a MISS ePRsystem design; Fig. 6(A) depicts the workflow of integrating imageand related data, and Fig. 6(B) is the graphic user interface of theMISS patient worklist. The deployment of such a MISS ePR willfacilitate the improvement of effectiveness and efficiency of MISS ofthe future.11

32.4 ACKNOWLEDGMENT

Liu B and Zhou Z, at IPI, USC contributed substantially to theprojects described in this chapter, which have been partially sup-ported by the National Institutes of Health, USA: NIH R01 EB 00298,NIH R01 LM 07606, T32 EB00438; and the MII Corp. Dr John Chiu ofthe California Spine Institute provided materials used in the MISSoperation.



References

1. Brooks PC, Clark RA, Cheresh DA, Requirement of vascular integrinalpha v beta 3 for angiogenesis, Science 264(5158): 569–571, 1994.

2. Atanasijevic T, Shusteff M, Fam P, Jasanoff A, Calcium-sensitive MRIcontrast agents based on superparamagnetic iron oxide nanoparticlesand calmodulin, Proc Natl Acad Sci USA 103: 14707–14712, 2006.

3. Huang HK, PACS and Imaging Informatics: Principles and ApplicationsJohn Wiley & Sons, Hoboken, New Jersey, 2004.

4. Zheng Zhou, MarcoAGutierrez, Jorge Documet, Lawrence Chan, et al.,The role of a data grid in worldwide imaging-based clinical trials, J HighSpeed Networks 1–13, 2006.

5. Law MYY, A model of DICOM-based electronic patient record in radi-ation therapy, J Comp Med Imag Graphics 29(2–3): 125–136, 2006.

6. Lemke H, Summary of the white paper of DICOM WG24, Proc SPIEMedical Imaging, pp. 6516–6522, February, 2007.

7. Chiu J, Savitz MH, Use of laser in minimally invasive spinal surgeryand pain management, in Kambin P (ed.), Arthroscopic and EndoscopicSpinal Surgery — Text and Atlas, 2nd ed., Humana Press, New Jersey,Vol. 13, pp. 259–269, 2005.

8. Chiu J, Technological developments for computer assisted endo-scopic minimally invasive spinal surgery (MISS), Proceedings ComputerAssisted Radiology and Surgery, 20th International Congress, Osaka,Japan, June 28–July 1, 2006.

9. Chiu J, Savitz M, Multicenter study of percutaneous endoscopic dis-cectomy, in Savitz M, Chiu J, Rauschning W, Yeung A (eds.), The Prac-tice of Minimally Invasive Spinal Technique: 2005 Edition, AAMISS Press,New City, New York, pp. 622–626, 2005.

10. Huang HK, PACS, Informatics, and the neurosurgery command mod-ule, J Mini Invasive Spinal Technique 1: 62–67, 2001.

11. Chiu JC, Savitz MH, Operating room of the future for spinal pro-cedures, in Savitz MH, Chiu JC, Rauschning W, Yeung AT (eds.),The Practice of Minimally Invasive Spinal Technique, AAMISS Press, NY,pp. 645–648, 2005.

January 22, 2008 12:3 WSPC/SPI-B540:Principles and Recent Advances Index FA

Index

18F-fluorodeoxyglucose (FDG), 673D (RT3D) echocardiography, 3483D ultrasound, 3484D medical imaging, 7754D radiation therapy (4DRT), 599, 7464D treatment delivery, 7764D treatment planning, 776

accuracy, 26activation likelihood estimation

(ALE), 665adaptive arithmetic mean filter, 183adaptive radiation therapy (ART), 746adaptive workflow engines, 791advantage of 4DRT, 774ALE meta-analysis, 665ALE statistic, 666algebraic reconstruction method, 48algebraic reconstruction techniques

(ART), 163, 375analysis of meta-analysis networks,

671Anger gamma camera, 73angular momentum, 100annihilation radiations, 64anisotropic diffusion, 291architecture, 833atlas construction, 498atlas-based segmentation, 483attenuation, 136

B-Spline transformation, 493back-projection method, 48, 159basic principles of MERT, 613Bloch equations, 106blood oxygenation level dependent

(BOLD), 267body immobilization, 765BOLD signal, 272boundary tracking, 200BPF algorithm for image

reconstruction, 365brachytherapy, 746brain atrophy, 687brownian motion, 290

cancer of the breast, 618cancer of the orbit, 607cancer of the parotid gland, 623cancer of the scalp, 608cardiac imaging, 337, 726cardiac MRI, 344cine XCT, 53classification, 455clinical applications of IMRT+e, 607clinical applications of MERT, 618clustering, 229, 231computational atlases, 481computational services in the data

grid, 538computed radiography, 35

845


846 Index

computed tomography, 361computer aided interpretation, 728cone beam CT (CBCT), 650constrained least square filtering, 188construction of a statistical atlas, 681content-based image retrieval (CBIR),

737convex hull, 462CT simulation, 751

data acquisition for DWI and DTI, 292data and computing grids, 836data back up and disaster recovery,

525data grid, 521data mining, 559decision-support tools, 549deformable segmentation, 734dense field representation, 488DICOM, 521DICOM-RT data model, 552diffuse optical tomography, 322diffusion magnetic resonance

imaging, 289diffusion tensor imaging (DTI), 290,

703diffusion tensors, 294diffusion weighted imaging (DWI),

703diffusion weighted magnetic

resonance imaging (DWI), 289diffusivity analysis, 709digital fluorography, 30digital mammography, 40digital operation room (DOR), 785digital radiography, 43digital signature embedding, 573dimensionality reduction, 244distance-dependent resolution, 385dose planning and delivery, 767dose-volume histogram, 638dual-photon imaging, 86DWI for tissue structural

characterization, 303DWI/DTI space, 710

dynamic patterns of brain atrophy, 689dynamics of workflows, 824

echo planar imaging (EPI), 126echocardiography, 725edge detection, 198, 732edge enhancement, 183edge-based image segmentation, 198electronic patient record, 548embed or extract, 590, 591embedding, 577, 581endocardium motion tracking, 344endogenous fluorophores, 314energy minimization and

optimization, 732engines and repositories, 788estimation methods, 166exogenous fluorophores, 315exponential radon transform, 383external beam radiation therapy, 746extracting, 582extracting and verifying, 580

false negative fraction, 25false positive fraction, 25feature extraction, 197, 214, 447feature extraction through wavelet

transform, 450feature-based, 418feature-based approach, 424finite elements (FE) models, 494flipping operation, 596fluorescence contrast agent, 313fluorescence imaging, 313, 397fluorescence macroscopic imaging, 320fluorescence microscopic imaging, 317fluorescence molecular imaging, 311fluorescence molecular tomography,

322fluoroscopic imaging system, 647forward modeling, 401four-dimensional (4D) imaging, 599four-dimensional (4D) XCT, 57four-dimensional radiation therapy

(4DRT), 772


Index 847

fourier projection theorem, 46free induction decay, 111frequency domain filtering, 184frequency of imaging, 656function-location meta-analysis, 664functional brain mapping, 663functional magnetic resonance

imaging (fMRI), 267, 663fusion imaging in nuclear medicine, 93future trends, 829fuzzy k-means clustering, 241fuzzy c-means clustering, 207fuzzy clustering, 238fuzzy membership function, 466

genetic algorithms, 247globus, 521GM diffusivity study, 716, 717goodness-of-alignment metrics, 420gradient echo, 122grid computing, 518, 520grid methods, 517

helical cone beam CT, 364helical MV CT, 650hemodynamic basis, 269hierarchical clustering, 234high-pass filtering, 192Hilbert space, 439histogram equalization, 177histogram modification, 177histogram representation, 175Hough transform, 222human brain mapping, 677human-machine interface, 834hypo-fractioned, ablative RT, 764

image archiving, 517image averaging, 179image formation, 22image fusion, 416, 423, 482, 831, 832image guidance in radiation therapy,

635image management and analysis, 530image management systems, 521

image processing in spatial domain,175

image processing methods, 173image processing using wavelet

transform, 444image reconstruction, 46, 151, 361, 413image reconstruction in positron

emission tomography, 369image reconstruction in single photon

emission computed tomography,380

image registration, 481, 753image security, 574image segmentation, 197image segmentation using neural

networks, 213image sharpening, 183image-guided radiation therapy

(IGRT), 600, 746image-guided radiotherapy, 639imaging for organ motion, 642imaging for radiation therapy

planning, 641imaging for treatment delivery, 646imaging informatics, 546imaging plate technology, 33impedance, power and reflection, 132individual diagnosis, 692information and communication

technology (ICT), 783integrated CT/linear accelerator, 648intensity modulated radiation therapy

(IMRT), 545, 599, 746intensity modulated radiation

therapy+electrons, 601intensity-based approach, 417, 426interfaces, 785inverse filtering, 186inverse radon transform, 158inverse treatment planning, 545ionizing radiations, 64iso-value curves, 341

k-means clustering, 206, 237k-nearest neighbors, 738


848 Index

k-space, 121knowledge base development, 556knowledge-based treatment planning,

545

Larmor frequency, 14, 104laser-stimulated luminescence, 33limitations of DTI techniques, 302linear attenuation coefficient, 65low-pass filtering, 189

magnetic resonance imaging, 13, 99,289

maximum likelihood discriminantanalysis, 458

maximum likelihood discriminantfunctions, 456

mechatronic (MT) systems, 783median filter, 182medical image analysis, 437medical image integrity, 573medical image security, 574medical imaging, 1, 9medical imaging (MI) in surgery, 822medical imaging informatics, 545medical imaging modalities, 3, 10meta-analyses of human cognition

and perception, 667meta-analysis, 663meta-analysis of stroop interference

studies, 668metadata database, 527minimally invasive spinal surgery

(MISS), 838model-based image reconstruction,

374model-based methods, 733modelling tools, 817modulated electron radiation therapy,

613molecular diffusion, 290molecular imaging, 829moments, 217motion analysis, 740MRI (DWI), 289

multichannel fusion, 715multiclass classification, 456multidimensional medical imaging,

831multimodality image reconstruction,

413multiresolution decomposition, 443multiresolution signal representation,

442multislice XCT, 54mutual information, 418, 429

nearest neighbored classifier, 243neighborhood operations, 180networking and information fusion,

834neural network for classification, 253neuro-fuzzy classifiers, 460neurodegenerative diseases, 705neurological disorders, 703neurooncology, 708neurosurgical planning, 708NMR spectrum, 111non-conventional fMRI, 277non-parametric classifiers, 253non-rigid transformations, 417nonuniform attenuation, 385nuclear magnetic moment, 102nuclear medicine imaging, 63

operative procedure, 807optical coherence tomography, 398optical flow, 337optical flow (OF) tracking, 339optical imaging, 393optical imaging methods, 395optical spectroscope, 398optical tomography, 399optical transillumination imaging, 395

PACS-based medical imaginginformatics, 835

partitional clustering, 237patient immobilization, 750patient model, 824


Index 849

patient positioning, 749patient specific modelling, 791pattern classification, 229permutation tests, 666PET/XCT fusion, 58picture archiving and communication

system (PACS), 517, 519, 783, 835pixel classification through clustering,

205pixel-based direct classification

methods, 202planar fluorescence imaging, 320positron emission tomography (PET),

18, 86post-operative care, 815preoperative assessment, 798pretreatment quality assurance (QA),

759principal component analysis, 245principles of IMRT+e, 601prior transformation models, 488probabilistic atlases, 501process of radiation therapy, 749proton and heavy-ion radiation

therapy, 767

radial basis functions, 257radiation oncology, 745radiation therapy (RT), 545, 599, 635,

745radiation therapy planning, 546radiopharmaceuticals in nuclear

medicine, 69radon transform, 153receiver operating characteristics, 24reconstruction with fourier transform,

156recording of workflows, 823region of interest imaging, 386region-based segmentation, 209region-growing, 210region-splitting, 211regularization, 495respiratory control, 765RF pulses, 110rigid registration, 482

rigid transformation, 417rotating frame of reference, 108

scaling function, 439sensitivity, 26series expansion, 440shape features, 216shape-based reconstruction, 393shape-based reconstruction algorithm,

404sign or verify, 590, 591signing and embedding, 579similarity metrics, 485single photon emission computed

tomography, 16single-photon tomography, 83skin cancer, 394spatial encoding, 118spatial normalization, 681specificity, 26SPGR space, 709spin echo, 116spiral (helical) XCT, 49statistical atlas, 501, 678, 687, 692statistical pixel-level image features,

215stereotactic body radiation therapy

(SBRT), 762support vector machine, 470surface imaging, 396surgical intervention, 832surgical workflow, 793, 822synergy in medical imaging, 829

T1 relaxation, 114T2 relaxation, 115target delineation, 755targeted imaging, 831texture features, 219therapy imaging and model

management system (TIMMS), 783,785, 817

time performance, 588tissue classification, 711tissue scattering, 135tomography, 83


850 Index

tomographic imaging, 81tracking the endocardium in real-time

3D ultrasound, 348treatment delivery, 761treatment planning, 636, 758true negative fraction, 25true positive fraction, 25tumor-targeted molecular probes, 326

ultrasonic imaging, 144ultrasonic instrumentation, 141ultrasound imaging, 19, 129, 726ultrasound-guided radiation therapy,

653uniform attenuation, 385

validation processes, 792visualization tool, 559, 563

wave equation, 130wavelet transform, 437, 439web-based GUI, 564white matter tractography, 297wiener filtering, 186workflow model development, 550

X-ray computed tomography (XCT),49

X-ray CT, 46X-ray imaging, 11

medical imaging modality, 29

principles and advanced_methods_in_medical_imaging_and_image_analysis_(wsp,_2008)

Education

eld of medical imaging

medical sciences

medical research

com puterized medical

text book

recent advances fm preface

diagnostic radiology

usa office