c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 182–192

journa l homepage: www. int l .e lsev ierhea l th .com/ journa ls /cmpb

Feature-based registration of histopathology images withdifferent stains: An application for computerized follicularlymphoma prognosis

Lee Coopera,∗, Olcay Sertela, Jun Konga, Gerard Lozanskib, Kun Huanga, Metin Gurcana

a Department of Biomedical Informatics, The Ohio State University, 3190 Graves Hall, 333 W. 10th Avenue,Columbus, OH 43210, United Statesb Department of Pathology, The Ohio State University Medical Center, 129 Hamilton Hall, 1645 Neil Avenue,Columbus, OH 43210, United States

a r t i c l e i n f o

Article history:

Received 26 September 2008

Received in revised form

28 February 2009

Accepted 22 April 2009

Keywords:

Image registration

Image analysis

Feature extraction

a b s t r a c t

Follicular lymphoma (FL) is the second most common type of non-Hodgkin’s lymphoma.

Manual histological grading of FL is subject to remarkable inter- and intra-reader varia-

tions. A promising approach to grading is the development of a computer-assisted system

that improves consistency and precision. Correlating information from adjacent slides with

different stain types requires establishing spatial correspondences between the digitized

section pair through a precise non-rigid image registration. However, the dissimilar appear-

ances of the different stain types challenges existing registration methods.

This study proposes a method for the automatic non-rigid registration of histological sec-

tion images with different stain types. This method is based on matching high level features

that are representative of small anatomical structures. This choice of feature provides a rich

Follicular lymphoma grading

Digital pathology

matching environment, but also results in a high mismatch probability. Matching confidence

is increased by establishing local groups of coherent features through geometric reasoning.

The proposed method is validated on a set of FL images representing different disease stages.

Statistical analysis demonstrates that given a proper feature set the accuracy of automatic

registration is comparable to manual registration.

ticular cancer type, follicular lymphoma (FL) [3,4].

1. Introduction

Histopathological examination is a crucial step in cancer prog-nosis. Pathological analysis of biopsy samples is necessary tocharacterize the tumor for treatment planning. Cancer prog-nosis that relies on this qualitative visual examination mayhave significant inter- and intra-reader variability due to sev-

eral factors, such as experience or fatigue at the time ofexamination [1,2]. Poor reproducibility of histological gradingmay lead to inappropriate clinical decisions on the timing and

∗ Corresponding author. Tel.: +1 614 313 7468.E-mail addresses: cooperl@ece.osu.edu (L. Cooper), osertel@bmi.osu

gerard.lozanski@osumc.edu (G. Lozanski), khuang@bmi.osu.edu (K. Hu0169-2607/$ – see front matter © 2009 Elsevier Ireland Ltd. All rights resdoi:10.1016/j.cmpb.2009.04.012

© 2009 Elsevier Ireland Ltd. All rights reserved.

type of therapy, and may result in under- or over-treatmentof patients with serious clinical consequences. A computersystem capable of extracting quantitative, and thereby moreprecise and objective prognostic clues, may provide moreaccurate and consistent evaluations. For this reason we aredeveloping a computer-assisted grading system for one par-

.edu (O. Sertel), kongj@ece.osu.edu (J. Kong),ang), gurcan@bmi.osu.edu (M. Gurcan).

FL is the second most common type of non-Hodgkin’slymphoma that consists of a group of cancers developingfrom the lymphatic system. The term “follicular” is derived

erved.

i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 182–192 183

fwtft(Wfr0bosr

mct(agaaiib

astvtecmi2apsmph

tIcaaoli

safdRmt

Fig. 1 – Sample image regions from CD3 and H&E stainedFL slides captured at 2× magnification. (a) and (b)correspond to adjacent sections from the same specimenand demonstrate local and global deformations and thedifficulty of identifying follicles from H&E-stained slides.Sample regions corresponding to the same follicle arehighlighted in red. (For interpretation of the references tocolor in this figure legend, the reader is referred to the web

and warping [22,23,14]. Like most optimization processes, agood initialization is critical for a global optimum outcome.In many cases, a good rigid registration serves as an idealinitialization for non-rigid registration [13]. For large images

c o m p u t e r m e t h o d s a n d p r o g r a m s

rom round-shaped biological structures, namely “follicles”,hich are visible under microscope. In current clinical prac-

ice, the risk stratification and subsequent choice of therapyor FL mainly depends on the histological grading processhat involves computing the average number of centroblastsCBs), i.e., malignant follicle center cells, as recommended by

orld Health Organization [5–7]. Due to the large number ofollicles usually exhibited in biopsy samples, only 10 follicleegions equivalent to a microscopic high power field (HPF) of.159 mm2are randomly sampled to make this process feasi-le in practice. Performing CB count over a limited numberf follicles can introduce a considerable sampling bias as theelected follicles may not be representative of other sampleegions, especially in heterogeneous tumors [2].

With sampling regions identified, centroblasts are thenanually counted in HPFs of the selected follicle regions. FL

ases are classified into three histological grades based onhe centroblast average count: grade I (0–5 CB/HPF), grade II6–15 CB/HPF) and grade III (> 15CB/HPF) [5]. Grade I is usu-lly associated with indolent disease and not treated, whilerade III is associated with aggressive disease and treatedggressively. A multi-site study reported only 61–73% gradinggreement across expert pathologists [1]. In addition to thisnter-observer variation, the manual counting of centroblastss very time-consuming, especially when a large number ofiological samples need to be examined.

In the current follicle grading processes, pathologists usu-lly resort to using pairs of adjacent slides dyed with differenttains to enhance visual contrasts. For example, immunohis-ochemical (IHC) stains, e.g. CD3 and CD20, provide a clearisual contrast for the follicle structures at low magnifica-ions, e.g. 2×, 4× and 8×. By comparison, hematoxylin andosin (H&E) stain enhance the contrast of the cytologicalomponents, and provide better cellular-level detail at higheragnifications, e.g. 20× and 40×. Two representative sample

mage regions from IHC and H&E stained images captured at× magnification are shown in Fig. 1, where follicle boundariesre clearly visible in the IHC (CD3 stain in this specific exam-le) stained image, but are not clearly discernible in the H&Etained counterpart. The proposed computer-assisted systemimics the manual grading procedure, working jointly with

airs of images with IHC and H&E stains. The flowchart of thisybrid FL grading system is presented in Fig. 2.

One of the key steps in this system is to map the spa-ial coordinates of the detected follicle positions from theHC stained image to the H&E counterpart image where theentroblast detection will occur. In order for the IHC imagenalysis to be able to interact with the H&E analysis process,n image registration algorithm is required that allows theutput of IHC follicle detection to be fed into the H&E centrob-

ast detection stage. In this paper, such a methodology and itsmplementation on clinical cases are reported.

Image registration for biological applications has beentudied extensively [8–19]. Registration can be considered asn optimization problem, posed as finding the optimal trans-ormation T between two images I and I to maximize a

1 2

efined similarity measure such as mutual information [20].egistration may also be formulated as a problem of featureatching: finding correspondence between sets of represen-

ative features using descriptors and spatial relations [21].

version of the article.)

The space of transformations includes rigid, that deals withonly rotation and translation, and non-rigid, that compen-sates for deformations such as bending, stretching, shearing

Fig. 2 – Flowchart of the computer-aided FL grading system.

s i n

184 c o m p u t e r m e t h o d s a n d p r o g r a m

with conspicuous deformations, hierarchical multi-resolutionregistration methods have also been widely used in medicalimaging applications [24,25].

The key challenge for the registration of sectionedhistopathological images is to compensate for distortion intro-duced by slide preparation. The input slide pairs are cut witha 5 �m thickness from adjacent locations so that the mor-phological structures vary minimally between image pairs.However, there are discernible global and local deformationsbetween these neighboring tissue sections due to the slidepreparation procedure (i.e., sectioning, fixation, embedding,and staining). The preparation process can introduce a vari-ety of non-rigid deformations including bending, shearing,stretching, and tearing. At micron resolutions, even minordeformations become conspicuous and may prove problem-atic when accuracy is critical to the end application. In orderto compensate for such deformations, a non-rigid registrationis essential and success depends on establishing a large num-ber of precise spatial correspondences throughout the extentof the image.

An additional challenge for the registration of histopatho-logical images exists when the images to be registered arestained with different stain types, and consequently have dis-similar appearances. An approach based on intensity valuesrequires the ability to resolve similarity between intensitysignals using a measure such as mutual information. Suchsimilarity is not necessarily guaranteed for combinations ofstain pairs, since for some stain combinations only com-plex high-order perceptual qualities will be consistent. Ifthe images do exhibit a significant visual similarity, then anapproach exists that uses correlation sharpness as a meansfor classifying local similarity between intensity information[26]. However, in the case of follicular lymphoma images withH&E and IHC staining, content at local scales appears as a uni-form texture of cellular components, certainly not an idealcondition for intensity comparison between distinct sections.Another approach exists that uses a segmentation of tissuetypes as input to a registration process [13]. The registra-tion reconciles differences in the segmentation by calculatinga displacement field that is used for non-rigid registration.Again, this approach is not reasonable in the case of follicularlymphoma, where the content is textural and segmentationis the original problem that a registration is intended toaid.

To address these challenges, this paper proposes a reg-istration approach based on the matching of small salientanatomical features. Small features such as blood vesselsappear universally in most tissues and have a commonappearance in many stains, making their extraction andmatching feasible. These features are used to establish spatialcorrespondences and register the images in two stages: firstrigidly, to roughly align the images, then non-rigidly, to cor-rect for elastic distortions introduced by preparation. The firststage uses a previously established mismatch-tolerant votingprocedure [17]. With the rough alignment of the images calcu-lated, the second stage establishes coherent local networks of

matched features between the images to enhance the confi-dence of matching and reduce the probability of mismatch andprovide a set of spatial correspondences that is satisfactory fornon-rigid registration.

b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 182–192

The outline of the remaining paper is organized as follows.Section 2 describes the proposed algorithm for registeringmulti-stained consecutive histopathological FL images. Twocomponents, including the feature extraction and the actualtransformation, are presented. In Section 3, extensive exper-imental results and the validation processes are presented.Conclusions are presented in Section 4.

2. Methods

To address the challenges of comparing content from con-secutive slides stained with different stain types, non-rigiddistortion, and feature-rich content, a two stage algorithmis proposed that consists of rigid initialization followed bynon-rigid refinement. Both stages operate by matching highlevel features, image regions that correspond to distinct andanatomically significant features such as blood vessels, otherductal structures, or small voids within the tissue area. Thesematches serve as the control points for calculating spatialtransformations to register the image pair. Rigid initializationestimates the rigid alignment of the image pair from the looseconsensus of correspondences between anatomical features,following the method presented in [17]. The non-rigid stagerefines the initialization, by establishing a more accurate set offeature correspondences at a local scale. Initialization reducesthe search for matching in the refinement stage, resulting in alower likelihood of erroneous matches and less computation.

2.1. Data

The input images of FL tissue slides are digitized using a ScopeXT digitizer (Aperio, San Diego, CA) at 40× magnification. Tis-sue slides are collected from the Department of Pathology,The Ohio State University in accordance with an InstitutionalReview Board (IRB) approved protocol. Slides are prepared byslicing the biopsy specimen in 5 �m sections. Adjacent sec-tions are stained pairwise, one of each pair with CD3 and theother with H&E. In this study five pairs of whole-slide biopsyspecimens associated with multiple FL patients having differ-ent grades of the disease were used.

2.2. Measure for evaluating image registration

For images with the same stain type, an ideal registrationwould be expected to match the areas of corresponding fol-licles with perfect overlap, natural morphological differencesaside. However, this expectation does not apply to the sce-nario of images with different stain types, as the differencein appearance of corresponding follicles in each stain typeresults in significantly different follicle boundaries. In gen-eral, the follicles in CD3-stained images appear smaller thantheir H&E counterparts due to the preparation process (thetissue is boiled or microwaved), and so when correctly regis-tered the CD3 follicles only cover the interior “kernel” regionsof those follicle regions in the H&E images. As illustrated in

Fig. 3, this fact implies a possible ambiguity in evaluating reg-istration accuracy from a ground truth perspective in that adecision cannot be made on which result is more optimal.However, since the aim is to identify regions of interest in

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 182–192 185

Fig. 3 – Overlap ratio score. The corresponding boundaries of a follicle from the CD3 image (a) and it H&E counterpart (b). Asshown in (c), different registration results can produce a perfect overlap ratio score due to the differences in follicleappearance between the CD3 and H&E stains. In (c) the red line indicates the H&E follicle boundary, and the green and bluelines indicate different manual registrations of the CD3 follicle boc ersi

teirf

r

wai

f

2

EmtFbMf

FEcm

olor in this figure legend, the reader is referred to the web v

he H&E image, this ambiguity will not compromise accuracyvaluation from the perspective of follicular lymphoma grad-ng. Therefore, we propose the performance measure as theatio between the overlap area of the registered CD3 and H&Eollicles and the area of the CD3 follicle as follows:

= Area(T(SCD3) ∩ SH&E)Area(SCD3)

, (1)

here SCD3 and SH&E are follicle regions detected in the CD3nd H&E images and T is the transformation between the twomages.

This quantity is measured for multiple manually markedollicles in each image as described in Section 3.

.3. Feature extraction

xtraction of high level features is a simple process as forost types of stains these features correspond to large con-

iguous regions of pixels with a common color characteristic.

or each stain type, a particular color segmentation followedy morphological operations for cleanup usually suffices.orphological opening is performed to reduce small noisy

eatures resulting from the color segmentation, and morpho-

ig. 4 – Feature extraction. This figure contains high-level featurextracted features, shown in a binary image(right), represent regombination of color segmentation and morphological operationajor-axis orientation are calculated for each feature.

undary to the H&E. (For interpretation of the references toon of the article.)

logical closing follows to fill in small gaps. The computationalcost of these operations can be significantly reduced by per-forming the extraction on down-sampled versions of theoriginal images without compromising the quality of the finalnon-rigid result. Fig. 4 demonstrates sample input and outputof the extraction process.

Given the base image B, and float image F, we extracttheir respective feature sets B = {bi} and F = {fj} accordingto the process described above. Each feature has associatedwith it a set of descriptors used for the matching processes,bi = (�xb

i , sbi, eb

i, �b

i) and fj = (�xf

j, s

f

j, e

f

j, �

f

j), where �x = (x, y) is the

feature centroid, s the feature area in pixels, e the featureeccentricity, and � the feature semimajor axis orientation.

2.4. Feature matching

Both the initialization and refinement stages use featurematching schemes to establish correspondences betweenthe base and float images. The following describes the con-ventions used for feature matching in both stages. Matches

between individual features are referred to as match candidatesif their size and eccentricity descriptors are consistent. That is,given the feature sets B,F, a match candidate (bi, fj) is estab-

lished if the descriptors of size sbi, s

f

jand eccentricity eb

i, e

f

jare

extraction results from a typical H&E image (left).ions such as blood vessels recognized by the use of as. Descriptions of centroid location, size, eccentricity, and

s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 182–192

Fig. 5 – Rigid feature matching. Features are matchedbetween the base and float images based on size andeccentricity to form m atch candidates (bi, fj), (bk, fl).Intra-image distance between pairs of match candidatesare compared to identify c andidate pairs. A model rigid

186 c o m p u t e r m e t h o d s a n d p r o g r a m

consistent within given percent difference thresholds �s, �e:

(bi, fj) ⇔

⎧⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎩

|sbi

− sf

j|

min(sbi, s

f

j)

≤ �s

|ebi

− ef

j|

min(ebi, e

f

j)

≤ �e

. (2)

If the base and float images are already roughly alignedthen �-consistency may also be enforced in the identificationof match candidates.

Both stages also use feature matches to generate modelrigid transformations (�̃, T̃x, T̃y) as part of their matchingschemes. Generating a model rigid transformation requires, atminimum, a pair of match candidates. To identify models orig-inating from coherent pairs of match candidates, geometricconsistency criteria are used to ensure consistent intra-imagedistances between feature centroids and also consistent fea-ture orientations. For a pair of match candidates to form ac andidate pair, {(bi, fj), (bk, fl)}, the intra-image centroid-to-centroid distances between features bi, bk and fj, fl are requiredto be consistent within the percent difference threshold ��x.Additionally, for the initialization stage, the orientations ofthe feature semimajor axes must be consistent with the modeltransformation angle �̃:

{(bi, fj), (bk, fl)} ⇔

⎧⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎩

|‖�xbi

− �xbk‖2 − ‖�xf

j− �xf

l‖2|

min(‖�xbi

− �xbk‖2, ‖�xf

j− �xf

l‖2)

≤ ��x

|�bi

− �f

j− �̃| < ��

|�bk

− �f

l− �̃| < ��

. (3)

The model transformation (�̃, T̃x, T̃y) for the candidate pair{(bi, fj), (bk, fl)} is calculated by first solving for the angle �̃ =tan−1((yf

i− y

f

k)/(xf

i− x

f

k)) − tan−1((yb

j− yb

l)/(xb

j− xb

l)), corrected

to the interval [−�, �]. The translation components T̃x, T̃y arecalculated using �̃ and least squares.

The match candidate and candidate pair concepts are illus-trated in Fig. 5.

2.5. Rigid initialization

Determining an estimate for rigid registration from a set offeature matches requires a method that is robust to erro-neous matchings. This is especially true in microscope images

Table 1 – Summary of parameter values used in the tests and v

Rigid

Parameter description Value

Size similarity (�s) 0.1Eccentricity tolerance (�e) 0.1Distance tolerance (��x) 0.1Orientation tolerance (��) 5◦

Voting interval for � (ω� ) 0.5◦

Voting interval for T (ωT ) 30

transformation, (�̃, T̃x, T̃y), is defined for candidate pairswith consistent distances.

where many features are indistinguishable, and a substantialamount of mismatches are inevitable. The fundamental ideaof the method presented in [17] is the recognition that anycandidate pair {(bi, fj), (bk, fl)} defines a model rigid transfor-mation (�̃, T̃x, T̃y), and for carefully chosen candidate matchesand candidate pairs, a large portion of the concomitant modeltransformations will concentrate around the desired parame-ters in the Euclidean transformation space. Careful choice ofmatches and match pairs is achieved with a set of consistencycriteria enforced at two levels: between feature descriptors formatches between individual base and float features, and geo-metrically between pairs of such matches. With a set of modeltransformations identified from consistent candidate pairs, ahistogram voting scheme is used to estimate the initializationparameters (�, Tx, Ty).

The details of rigid initialization are previously published in[17]. Sample voting results from a follicular lymphoma imagepair are presented in Fig. 6. The associated parameter valuesare presented in Table 1.

2.6. Non-rigid refinement

The challenge in non-rigid registration is the sensitivityof computed non-rigid transformations to errors in match-

ing, a consequence of the freedom of such transformationsto accommodate distortion. In computing a relatively con-strained transformation such as a rigid transformation, theeffect of mismatches can be mitigated through the constraints

alidation.

Non-rigid

Parameter description Value

Base neighborhood (Rb) 1000Float neighborhood (Rf ) 1100Search neighborhood (S) 250Match neighborhood (ı) 30�̃ angle tolerance (�) 5◦

Pattern match minimum (�) 4

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b

Fig. 6 – Sample histogram voting result for rigidinitialization of follicular lymphoma image pair. Manualparameter results are shown in red and automatic resultsin green. (For interpretation of the references to color in thisfigure legend, the reader is referred to the web version oft

onfni

F

M

p

fı

r

he article.)

f the transformation and least squares. For most common

on-rigid transformation types the effect of a mismatched

eature is certainly strong locally, and depending upon theumber of matches used may also affect the registration qual-

ty globally.

ig. 7 – Non-rigid feature matching. (a) Locations of feature bi (re

atch candidate fj (red) and surrounding features in the Rf

j-neigh

airings that generate a model local rigid transformation. (c) The

eatures (blue dots). In this case, the number of base features wit-neighborhood (green circle) is three. (For interpretation of the referred to the web version of the article.)

i o m e d i c i n e 9 6 ( 2 0 0 9 ) 182–192 187

For this reason the standard for establishing matches tocompute a non-rigid transformation must be strict to achievea low probability of mismatch. In the rigid stage, feature com-parisons are made globally to accommodate the possibly grossmisalignment of the image pair. The rigid transformation isinferred from the modes of the collection of model transfor-mations resulting from the set of all possible candidate pairs(which inevitably includes a large proportion of mismatches).Due to the presence of mismatches from model transforma-tions surrounding these modes these candidate pairs are notappropriate input for computing the transformation of thenon-rigid stage. However, the rigid initialization provides astarting point that can reduce the search area for a stricterfeature matching procedure that can reduce the likelihood ofmismatching and also computation.

Given the rigid initialization, the problem of matching indi-vidual features with high confidence can be formulated asa pattern matching problem. Instead of comparing individ-ual features solely via their descriptors, the spatial patternsformed by the collection of features within their neighborhoodcan be compared to increase the matching confidence. Fea-tures that match with a high degree of confidence will havesimilar spatial patterns of neighboring features with consis-tent descriptors. Since these neighborhood comparisons aremade at a local scale non-rigid distortion is usually mild andlocal rigidity can be assumed.

Procedurally, the non-rigid matching scheme is as follows:given feature sets B and F, for each base feature bi, thesurrounding features in the Rb-neighborhood are identified.Match candidates for bi are located in the float image withinthe S-neighborhood centered at �xb

i, and are matched to bi

based on size sf

j, eccentricity e

f

j, and orientation �

f

j(orienta-

tion can be used as criteria now that the images are rigidlyaligned). For each match candidate fj, the surrounding fea-

tures are identified within the Rf -neighborhood of �xf , and

j

match candidates other than (bi, fj) are identified. From theseother match candidates, candidate pairs are formed with(bi, fj), and pairs with model rotation angle |�̃| > � are elimi-nated. The model for each of the remaining candidate pairs

d) and surrounding features in Rbi-neighborhood (blue). (b)

borhood (blue). Green lines in (a) and (b) indicate the

float features of Rf

j(red x’s) are transformed onto Rb

i

h a consistent transformed float feature within itseferences to color in this figure legend, the reader is

s i n

188 c o m p u t e r m e t h o d s a n d p r o g r a m

is used to transform the two neighborhoods, and the numberof base features in Rb that fall within ı of an s, e, �-consistentfloat feature are counted. A match (bi, fj) is established if themaximum count exceeds the pattern match threshold � and|Rb

i|/2. This process is illustrated in Fig. 7 and summarized in

Algorithm 1.

Parameters for the non-rigid matching procedure have

clear interpretations and can be selected by examining thefeatures for a particular dataset. Neighborhood size Rb is cho-

b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 182–192

sen to capture small local networks of features, and dependson the density of features and scan magnification. The matchcandidate search neighborhood, S, is selected to account forerror in the rigid alignment. The match neighborhood size,ı, is chosen to account for physical distortion and noise dueto feature extraction including natural morphological differ-ences. Parameter values for the dataset used in this paper arepresented in Section 3.

Algorithm 1. Non-rigid feature matching

i n b

2

Trmwpe[

ctruwppppclcmtspbirta

amwoims

pd{

es

2

TraSA1pt

c o m p u t e r m e t h o d s a n d p r o g r a m s

.7. The polynomial transformation

he collection of point correspondences generated by non-igid matching provides the information needed to form a

apping that transforms the float image into conformationith the base. A variety of non-rigid mappings are used inractice, differing in computational burden, robustness torroneous correspondences, and existence of inverse form22,23,14].

The desired transformation qualities include not only theapability to correct non-rigid distortions, but also robustnesso match errors, closed inverse form, and computationallyeasonable calculation and application. Of the commonlysed non-rigid mapping types such as thin-plate spline, localeighted mean, affine, polynomial, and piece-wise variations,olynomial offers a good compromise between warp com-lexity and the aforementioned qualities. Thin plate splinerovides a minimum energy solution which is appealing forroblems involving physical deformation, however, perfectonformity at correspondence locations can potentially causearge distortion in other areas and excess error if an erroneousorrespondence exists. The lack of an explicit inverse formeans the transformed image is calculated in a forward direc-

ion, likely leaving holes in the transformed result. Methodsuch as gradient search can be used to overcome the inverseroblem, but at the cost of added computation, which canecome astronomical when applied to each pixel in a gigapixel

mage. Kernel-based methods such as local weighted meanequire a uniform distribution of correspondences. Givenhe heterogeneity of tissue features this distribution cannotlways be guaranteed.

Polynomial warping admits an inverse form, is fast inpplication, and is capable of satisfactorily correcting theild distortion encountered in sectioned images. Polynomialarping parameters can be calculated using least squaresr its variants which can mitigate the effect of match-

ng errors. Affine mapping offers similar benefits but isore limited in the complexity of the warping it can repre-

ent.Second degree polynomials are used for the results in this

aper. Specifically, for a point (x, y) in the base image, the coor-inate (x′, y′) of its correspondence in the float image is

x′ = a1x2 + b1xy + c1y2 + d1x + e1y + f1,

y′ = a2x2 + b2xy + c2y2 + d2x + e2y + f2,(4)

Since each pair of matched correspondences provide twoquations, we need at least six pairs of correspondences toolve for the coefficients in (4).

.8. Experimental procedures

o demonstrate the effectiveness of the automatic non-rigidegistration method, the feature extraction and registrationlgorithms were applied to the five image pairs described inection 2.1. Magnification was reduced from 40× to 4× using

perio’s ImageScope software, resulting in images roughly0, 000 × 7500 pixels in size. For feature extraction, the samearameters for color segmentation and morphological opera-ions were used for all image pairs. The automatic registration

i o m e d i c i n e 9 6 ( 2 0 0 9 ) 182–192 189

parameters, presented in Table 1, were also identical for allimage pairs. For comparison, manual rigid and manual non-rigid registrations were also performed to the five image pairs,using eight manually selected control point pairs per imagepair. A simple Euclidean transformation was used for the rigidregistrations. A second degree polynomial transformation wasused for the non-rigid registrations.

All computations were carried out on a dual core 2.6 GHzAMD Opteron system with 8 Gigabytes of RAM. Software wasdeveloped using a combination of Matlab, and Matlab’s C/C++interface MEX. With the RGB images loaded into memory, theentire process executes in 2 min for a single image pair. Lessthan 1 s of that is devoted to the non-rigid matching proce-dure.

Visual inspection of the feature extraction results revealedthat features in two of the five image pairs are not uniformlydistributed, being concentrated almost entirely in one half ofthe tissue area in each case. In regions where features aresparse, non-rigid refinement matches are hard to establishsince it is difficult to identify coherent networks of features at alocal scale. This can result in spatially clustered control points,and depending on the severity of distortion between the slides,a transformation that is significantly biased to the feature-richareas of the tissue. The validation analysis that follows is car-ried out separately on these challenging image pairs and thefeature regular image pairs, to illustrate the importance of fea-ture input and the expected outcome if a sufficient feature setcan be identified.

2.9. Validation

The procedure for registration validation was motivated bythe application of automated FL grading. The goal in thisapplication is to correctly register follicle regions so that folli-cle segmentations from the CD3 image can be used to directgrading analysis in the counterpart H&E image. To evaluateregistration performance in the context of this application, theoverlap of manually identified follicle regions was comparedfor different registration methods.

For each H&E/CD3 image pair, five corresponding test fol-licle pairs were selected. The boundaries of each of these testfollicle pairs were then marked by five different observers.The same test follicle pairs were marked by each observer,generating a total of 25 follicle pair markings per observer.The overlap ratio demonstrated in Fig. 3 was then computedfor every follicle test pair marking using the manual rigid,manual non-rigid, and automatic non-rigid registrations foreach image pair. We denote these overlap ratios for observer i,image pair j, and follicle test pair k as Rigidi(j, k), Manuali(j, k),and Autoi(j, k), respectively. The feature regular image pairsare the set j ∈ {1, 2, 3} and the challenge image pairs are the setj ∈ {4, 5}.

This validation aims to illustrate two points: (1) that non-rigid registration is beneficial in terms of follicle overlap and(2) that the automatic non-rigid registration is comparable

to a reasonable manual non-rigid registration. We addressthese points with three statistical analyses: the boxplot graph-ical analysis, significance testing by paired t-test, and theBland–Altman graphical analysis.

190 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 182–192

Fig. 8 – Boxplots of overlap ratios for observer-methods sets from feature regular image pairs. Outlier overlap ratios frompoorly registered follicles are indicated by red cross-markers. Mean performance is comparable between manual non-rigid

the r

presented in Table 2. Comparing manual rigid and manualnon-rigid registrations, the non-rigid registration improvesthe mean overlap ratio for all markings except those ofobserver two, demonstrating the benefit of correcting non-

Table 2 – Mean overlap ratios and standard deviationsfor observer-method sets of feature regular image pairs.

Observer i Rigidi Manuali Autoi

Mean ±S.D. Mean ±S.D. Mean ± S.D.

1 0.8943 ± 0.0930 0.9373 ± 0.0889 0.9306 ± 0.1152

and automatic non-rigid registrations. (For interpretation ofreferred to the web version of the article.)

The boxplot is a graphical analysis that presents thedistributions of the overlap ratios for feature image pairs,separated by both registration method and observer. Themedian, inner-quartile range, and outliers are plotted foreach observer-method set, {Methodi(j, k)}, ∀(j, k) ∈ {1, 2, 3} ×{1, . . . , 5}, for some i.

To demonstrate the similarities of manual non-rigidregistrations, significance testing was performed onthese observer-method sets using the paired t-test.For each observer i, the overlap ratios were paired bymethod for all follicles in the feature regular image pairs,{(Manuali(j, k), Autoi(j, k))}∀(j, k) ∈ {1, 2, 3} × {1, . . . , 5}. The t-statistic was calculated for these method-pair sets:

ti = D̄i

√15

2i

, (5)

where D̄i and 2i

are the mean and variance:

D̄i =3∑

j=1

5∑k=1

(Autoi(j, k) − Manuali(j, k)),

2i

= 115 − 1

3∑j=1

5∑k=1

(Autoi(j, k) − Manuali(j, k))2.

The t-statistic ti was compared against the Student’s t dis-tribution to compute the p-value pi.

To further illustrate the similarities between automaticand manual non-rigid registrations, a Bland–Altman graph-ical analysis was performed. The Bland–Altman analysis iscommonly used in biostatistics to examine the extent of agree-ment between to distinct measurement methods (Fig. 9). It isincluded here because it illustrates the performance of theautomatic and manual methods well. We note, however, thatcomparing the overall performance of two registration meth-ods is fundamentally different from the assessment of theagreement of measurement methods. In the case of measure-

ment assessment, agreement between individual samples iscritical, since the measurements intended to provide the sameinformation about some underlying physical state. In regis-tration, follicle overlaps may disagree individually between

eferences to color in this figure legend, the reader is

methods, but the collection of overlaps may still indicate com-parable performance.

For each observer i, the difference dj,k and mean j,k werecomputed:

j,k = Autoi(j, k) + Manuali(j, k)2

(6)

dj,k = Autoi(j, k) − Manuali(j, k), (7)

and the mean and difference tuples (j,k, dj,k) were plotted forall follicles in the feature regular image pairs. Along with themean and difference tuples, the average-difference and 95%confidence intervals are plotted to provide information on themean performance of the methods and their range of agree-ment.

Finally, a simple analysis is performed to demonstrate thespatial variation of registration quality in the challenge imagepairs. For each follicle k, the overlap ratios Autoi(j, k), j ∈ {4, 5}are averaged over observer i.

3. Results

The boxplot is presented in Fig. 8. The corresponding meansand standard deviations of the observer-method sets are

2 0.9223 ± 0.0667 0.9190 ± 0.0950 0.9213 ± 0.07183 0.9428 ± 0.0838 0.9520 ± 0.0617 0.9562 ± 0.04774 0.9167 ± 0.0850 0.9278 ± 0.0969 0.9316 ± 0.07275 0.9247 ± 0.0732 0.9384 ± 0.0691 0.9351 ± 0.0614

c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 182–192 191

Fig. 9 – Bland–Altman analysis of manual and automatic non-rigid registrations. Average difference is indicated in red. The95% confidence limits are indicated in green. (For interpretation of the references to color in this figure legend, the reader isr

rri

pim

pmoofhcppa

trfciTtqf

Table 4 – Challenge image pair overlap ratios Autoi(j, k),separated by follicle k, and averaged over observers i.

Follicle k 1 2 3 4 5

Image Pair j = 4 0.8482 0.2601 0.0505 0.8377 0.9366

eferred to the web version of the article.)

igid distortion. Mean overlap ratios for automatic non-rigidegistration are comparable to manual non-rigid, with slightmprovements noted for the markings of three observers.

The p-values for the t-statistics of the method-pair sets areresented in Table 3. These p-values range from 0.79 to 0.93

ndicating no statistically significant difference between theanual and automatic methods.The Bland–Altman plot is presented in Fig. 9. Tuples

lotted above zero indicate better performance for the auto-atic method. The average-difference is nearly zero for all

bservers. Most tuples are clustered tightly in the center rightf their plot, indicating a high average overlap and small dif-erence for the manual and automatic methods. Each observeras at least one outlier tuple with a difference beyond the 95%onfidence limits. For each outlier tuple indicating superiorerformance for the manual registration, there is a com-lementary tuple indicating superior performance for theutomatic method.

The overlap results from the challenge image pairs illus-rate the impact of feature input to the automatic non-rigidegistration. Where the test follicle pairs were chosen uni-ormly throughout the extent of the tissue, the features in thehallenge image pairs were not uniformly distributed, result-ng in a transformation that is biased to feature-rich areas.

he overlap ratios of Table 4 demonstrate this point, where

est follicles located in feature rich regions show comparableuality and others apparently suffer from a lack of proximaleature matches.

Table 3 – Significance values of paired t-tests formethod-pair sets {(Manuali(j, k), Autoi(j, k))} from featureregular images. The p-values indicate no statisticallysignificant difference between the overlaps for manualand automatic non-rigid registration methods.

Observer i pi

1 0.79812 0.93013 0.81944 0.89015 0.8905

Image Pair j = 5 0.9189 0.4540 0.9187 0.8862 0.9886

4. Conclusion and discussion

This paper presents a method for the non-rigid registra-tion of distinctly stained follicular lymphoma section images.As a key step for fusing the information extracted fromimages of two different stains, i.e., IHC and H&E, comput-erized registration serves as a bridge that allows for thecombination of valuable information otherwise unique ineach resource in a meaningful way. In this particular study,the registration step makes it possible to recognize salientfeatures from both stained images and map the follicle bound-aries detected in IHC images to appropriate locations in H&Eimages. As a consequence, further grading analysis can pro-ceed with H&E counterparts where cellular-level analysis isfavorable. In the end, by providing accurate follicle bound-aries on the H&E images, the registration contributes to moreprecise CB count, the essential step in the FL grading pro-cess.

The automatic matching method presented in this paperoffers a solution for applications such as microscopy imag-ing, where a large number of nondescript features are to bematched with high-fidelity. Matching such features individ-ually is a high probability-of-error endeavor, and matchingerrors can result in poor conformation between the registeredimage pair due to the freedom of non-rigid transformations.Here, confidence in matches between individual features isenhanced by verifying the existence of coherent networks offeatures in the surrounding areas.

In terms of registration accuracy, the quality of transfor-mations derived from automatic matching depends on the

ability to extract features throughout the extent of the tis-sue area. When excluding the image pairs where extractedfeatures are sparse and highly spatially clustered, the registra-

s i n

r

high resolution microscopic images with different

192 c o m p u t e r m e t h o d s a n d p r o g r a m

tions based on automatic matching are indistinguishable fromthose based on the manual non-rigid method. This suggeststhat the registration framework could benefit from a moresophisticated feature extraction process. However, in practice,poorly registered follicles located in feature sparse areas couldpossibly be avoided by analyzing the spatial distribution offeature matches and their proximities to each follicle.

Conflict of interest

None declared.

Acknowledgements

The project described was supported by Award NumberR01CA134451 from the National Cancer Institute. The contentis solely the responsibility of the authors and does not nec-essarily represent the official views of the National CancerInstitute or the national Institutes of Health.

e f e r e n c e s

[1] F. Dick, S. Vanlier, P. Banks, Use of the working formulationfor non-Hodgkin’s lymphoma in epidemiological studies:agreement between reported diagnoses and a panel ofexperienced pathologists, Journal of National CancerInstitute 78 (1987) 1137–1144.

[2] G.E. Metter, B.N. Nathwani, J.S. Burke, C.C. Winberg, R.B.Mann, M. Barcos, C.R. Kjeldsberg, C.C. Whitcomb, D.O.Dixon, T.P. Miller, S.E. Jones, Morphological subclassificationof follicular lymphoma: variability of diagnoses amonghematopathologists, a collaborative study between therepository center and pathology panel for lymphoma clinicalstudies, Journal of Clinical Oncology 3 (1) (1985) 25–38.

[3] O. Sertel, J. Kong, G. Lozanski, U. Catalyurek, J.H. Saltz, M.Gurcan, Computerized microscopic image analysis offollicular lymphoma, in: Proceedings of the SPIE MedicalImaging, 2008.

[4] O. Sertel, J. Kong, U.V. Catalyurek, G. Lozanski, J.H. Saltz, M.Gurcan, Histopathological image analysis usingmodel-based intermediate representation and color texture:follicular lymphoma grading, Journal of Signal ProcessingSystems for Signal, Image, and Video Technology 55 (2009)169–183.

[5] E.S. Jaffe, N.L. Harris, H. Stain, J.W. Vardiman, Tumors ofHaematopoietic and Lymphoid Tissues, IARC, Lyon, France,2001.

[6] W. Hiddemann, C. Buske, M. Dreyling, O. Weigert, G. Lenz, R.Forstpointner, C. Nickenig, M. Unerhalt, Treatmentstrategies in follicular lymphomas: current status and futureperspectives, Journal of Clinical Oncology 23 (26) (2005)6394–6399.

[7] R.B. Mann, C.W. Berard, Criteria for the cytologicclassification of follicular lymphomas: a proposedalternative method, Hematology Oncology 1 (1983) 187–192.

[8] C. Levinthal, R. Ware, Three-dimensional reconstructionfrom serial sections, Nature 236 (1972) 207–210.

[9] J. Capowski, Computer-aided reconstruction of neuron trees

from several sections, Computational Biomedical Research10 (6) (1977) 617–629.

[10] V. Moss, The computation of 3-dimensional morphologyfrom serial sections, European Journal of Cell Biology 48(1989) 61–64.

b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 182–192

[11] R. Brandt, T. Rohlfing, J. Rybak, S. Krofczik, A. Maye, M.Westerhoff, H.-C. Hege, R. Menzel, A three-dimensionalaverage-shape atlas of the honeybee brain and itsapplications, The Journal of Comparative Neurology 492 (1)(2005) 1–19.

[12] J. Hajnal, H. Derek, D. Hawkes, Medical Image Registration,CRC Press, Boca Raton, FL, 2001.

[13] U. Braumann, J. Kuska, J. Einenkel, L. Horn, M. Luffler, M.Huckel, Three-dimensional reconstruction andquantification of cervical carcinoma invasion fronts fromhistological serial sections, IEEE Transactions on MedicalImaging 24 (10) (2005) 1286–1307.

[14] W. Crum, T. Hartkens, D. Hill, Non-rigid image registration:theory and practice, The British Journal of Radiology 77 (2)(2004) S140–S153.

[15] P. Wenzel, L. Wu, R. Sharp, A. de Bruin, J. Chong, W. Chen, G.Dureska, E. Sites, T. Pan, A. Sharma, K. Huang, R. Ridgway, K.Mosaliganti, R. Machuraju, J. Saltz, H. Yamamoto, J. Cross, M.Robinson, G. Leone, Rb is critical in a mammalian tissuestem cell population, Genes & Development 21 (1) (2007)85–97.

[16] L. Cooper, K. Huang, A. Sharma, K. Mosaliganti, T. Pan,Registration vs. reconstruction: building 3-d models from2-d microscopy images, in: Proceedings of the Workshop onMultiscale Biological Imaging, Data Mining and Informatics,Santa Barbara, CA, 2006, pp. 57–58.

[17] K. Huang, L. Cooper, A. Sharma, T. Pan, Fast automaticregistration algorithm for large microscopy images, in:Proceedings of the IEEENLM Life Science Systems &Applications Workshop, Bethesda, MD, 2006, pp. 1–2.

[18] J. Prescott, M. Clary, G. Wiet, T. Pan, K. Huang, Automaticregistration of large set of microscopic images usinghigh-level features, in: Proceedings of the IEEE InternationalSymposium on Medical Imaging, Arlington, VA, 2006, pp.1284–1287.

[19] K. Mosaliganti, T. Pan, R. Sharp, R. Ridgway, S. Iyengar,A. Gulacy, P. Wenzel, A. de Bruin, R. Machiraju, K. Huang,G. Leon, J. Saltz, Registration and 3d visualization oflarge microscopy images, in: Proceedings of the SPIEMedical Imaging Meeting, vol. 6144, San Diego, CA, 2006, pp.923–934.

[20] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, P.Suetens, Multimodality image registration by maximizationof mutual information, IEEE Transactions on MedicalImaging 16 (2) (1997) 187–198.

[21] T. Kim, Y.-J. Im, Automatic satellite image registration bycombination of matching and random sample consensus,IEEE Transactions on Geoscience and Remote Sensing 41 (5)(2003) 1111–1117.

[22] F. Bookstein, Principal warps: thin-plate splines and thedecomposition of deformations, IEEE Transactions onPattern Analysis and Machine Intelligence 11 (6) (1989)567–585.

[23] K. Rohr, Landmark-Based Image Analysis: Using Geometricand Intensity Models, Springer, New York, NY, 2007.

[24] R. Bajcsy, S. Kovacic, Multiresolution elastic matching,Computer Vision, Graphics, and Image Processing 46 (1989)1–21.

[25] M. Lefebure, L. Cohen, A multiresolution algorithm for signaland image registration, in: Proceedings of the IEEEInternational Conference on Image Processing, vol. 3,Washington, DC, 1997, pp. 252–255.

[26] L. Cooper, S. Naidu, G. Leone, J. Saltz, K. Huang, Registering

histochemical stainings—a tool for mapping geneexpression with cellular structures, in: Proceedings of theWorkshop on Microscopic Image Analysis with Applicationsin Biology, Piscataway, NY, 2007.