[0933 1752010 Bioinformatics-btq219tex] Page i57 i57ndashi63

BIOINFORMATICS Vol 26 ISMB 2010 pages i57ndashi63doi101093bioinformaticsbtq219

As-rigid-as-possible mosaicking and serial section registration oflarge ssTEM datasetsStephan Saalfeld1 Albert Cardona2 Volker Hartenstein3 and Pavel Tomancaacutek1lowast1Max Planck Institute of Molecular Cell Biology and Genetics Dresden Germany 2Institute of NeuroinformaticsUniversity and ETH Zuumlrich Switzerland and 3Department of Molecular Cell and Developmental Biology University ofCalifornia Los Angeles CA 90095-1606 USA

ABSTRACT

Motivation Tiled serial section Transmission Electron Microscopy(ssTEM) is increasingly used to describe high-resolution anatomyof large biological specimens In particular in neurobiology TEMis indispensable for analysis of synaptic connectivity in the brainRegistration of ssTEM image mosaics has to recover the 3Dcontinuity and geometrical properties of the specimen in presenceof various distortions that are applied to the tissue during sectioningstaining and imaging These include staining artifacts mechanicaldeformation missing sections and the fact that structures mayappear dissimilar in consecutive sectionsResults We developed a fully automatic non-rigid but as-rigid-as-possible registration method for large tiled serial section microscopystacks We use the Scale Invariant Feature Transform (SIFT) toidentify corresponding landmarks within and across sections andglobally optimize the pose of all tiles in terms of least squaredisplacement of these landmark correspondences We evaluate theprecision of the approach using an artificially generated datasetdesigned to mimic the properties of TEM data We demonstratethe performance of our method by registering an ssTEM dataset ofthe first instar larval brain of Drosophila melanogaster consisting of6885 imagesAvailability This method is implemented as part of the open sourcesoftware TrakEM2 (httpwwwiniuzhchsimacardonatrakem2html)and distributed through the Fiji project (httppacificmpi-cbgde)Contact tomancakmpi-cbgde

1 INTRODUCTIONTransmission electron microscopy (TEM) offers currently thehighest available resolution to investigate the structure of biologicalsamples The superior resolution of TEM is classically used to revealsubcellular structures such as organelles and some of the largestmacromolecular assemblies inside cells For anatomical studies onthe tissue level TEM finds most applications in neurobiology toreveal neuronal connectivity maps that underlie the function ofthe nervous system The reconstruction of the complete neuronalconnectivity map in Caenorhabditis elegans remains the pinnacle ofthese studies (White et al 1986) On the nanometer scales brains aregigantic structures and many overlapping images must be assembledas a mosaic to cover a substantial portion of the tissue Moreoverserial sectioning is required to capture the volume of the specimen(Fig 1andashd)

Digital imaging and high-precision beam or specimen shiftingequipment (Suloway et al 2005) simplifies the reconstruction of

lowastTo whom correspondence should be addressed

mosaics within one section by providing a good initial estimateof the tile configuration (Fig 1b) However due to the minutedistances involved the precision of the instruments is insufficientfor seamless stitching of the tiles (Fig 1c) Adjacent sections arerotated and translated arbitrarily with respect to each other (Fig 1d)Physical sectioning the subsequent manipulation of the sections bycontrast gaining staining procedures and imaging with the electronbeam introduce significant distortions and artifacts into the capturedimages (Fig 1endashh) The goal in reconstruction of serial sectionTEM (ssTEM) data is to identify the configuration of all tiles thatbest preserves both the continuity and geometry of 3D structures inthe specimen making it amenable for subsequent segmentation andanalysis Particularly in neurobiology it is important to preservethe continuity of axons across large distances and the ability todistinguish and localize synaptic connections between neurons

Anderson et al (2009) propose a registration pipeline for largessTEM datasets based on correlation of image intensities withinand across sections They are able to identify overlapping tilesin unordered sets automatically mosaic them by propagatingpairwise translation and refine this initial layout by warping eachtile to its preceding partners The resulting section mosaics areregistered relative to each other by searching a rigid transformationinitially and subsequently warping one section to the other Thissolution guarantees continuity of 3D structures by maximizingthe overlap of similar image content in adjacent sections On theother hand it cannot preserve geometrical properties because theunavoidable registration error is accumulated with each consecutiveregistration step

Instead of using all image intensities it is possible to registertwo or more images using corresponding salient points as a statisticabout the image content as proposed by Brown and Lowe (2003)for panorama stitching Reducing the problem from full resolutionimage content to a relatively small number of corresponding pointssimplifies the estimation of the globally optimal configuration for alarge number of tiles

State of the art techniques (Bay et al 2008 Lowe 2004 Mataset al 2002 Mikolajczyk and Schmid 2004 Tuytelaars and VanGool 2004) allow both automatic detection of interest points andextraction of affine invariant local descriptors for these pointsAffine invariant matching becomes nearest neighbor search in a localfeature descriptor space It was shown by Mikolajczyk and Schmid(2003) that the local descriptor as used in the Scale Invariant FeatureTransform (SIFT Lowe 2004) outperforms competing techniquesin both distinctiveness and robustness with respect to significantimage deformation and other disturbing artifacts

In this article we present a fully automatic registration methodfor large ssTEM image mosaics that globally minimizes the

copy The Author(s) 2010 Published by Oxford University PressThis is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (httpcreativecommonsorglicensesby-nc25) which permits unrestricted non-commercial use distribution and reproduction in any medium provided the original work is properly cited

by on June 7 2010 httpbioinform

aticsoxfordjournalsorgD

ownloaded from

[0933 1752010 Bioinformatics-btq219tex] Page i58 i57ndashi63

SSaalfeld et al

Fig 1 Properties of ssTEM image mosaics (a) Low magnification overview of a single section mosaic consisting of 9times9 registered tiles (b) Each sectionis imaged as a sequence of overlapping tiles that is assumed to be a regular grid (c) Each tilersquos true pose in the local section coordinate frame is affectedby odometry errors of the microscope that are propagated over consecutive tiles (d) The relative pose of two consecutive sections is arbitrarily shifted androtated Clean sections are relatively rare (e) typically sections contain artifacts such as dirt (f) staining precipitate (g) and folds or cracks (h)

registration error We identify overlapping images by correspondingimage content within and across sections using SIFT features anda geometric consistency constraint We approximate the non-lineartransformation between adjacent sections by an independent rigid-body transformation (rotation and translation) for each image tile ofthe section mosaics In a TEM section series none of the sectionscan be considered non-deformed and thus no section can serve asa template for registration Therefore we define the registrationproblem to be template free and instead explicitly minimize thenon-rigid deformation applied to all sections This is achieved byglobally minimizing the square displacement of all correspondingSIFT landmarks within and across sections By this means theresulting tile configuration is non-rigid but as-rigid-as-possible Inorder to evaluate the performance of this approach we developeda synthetic ground truth dataset that is designed to mimic theproperties of ssTEM data We were able to register 6885 imagesfrom 85 serial sections through the Drosophila first instar larvalbrain each section consisting of 9times9 tiles of 2048times2048 px Theregistered dataset serves as a starting point for characterizing thefine architecture of this large brain at unprecedented resolution

2 METHODS

21 Definition of the taskSpecimens prepared for serial sectioning are typically embedded in a blockof rigid medium such as resin plastic or ice Using a microtome ultrathinconsecutive slices are cut from the rigidly mounted block The slices float onthe surface of a liquid in the lsquoknife boatrsquo and are manually picked up onto aTEM grid While the section thickness may vary slightly within and acrosssections in this work we consider sections to be planar and of constant

thickness In reality the cross-section variation can be minimized by carefuloperation of the microtome and as both the specimen itself and the knifeof the microtome are rigidly mounted intrasection variation is compensatedthroughout the section series by complementary variation in other sections

Each section is imaged as a set of overlapping tiles For each tile tthe microscope registers a 2D translation vector tt= (uv)T in a relativesection reference frame R

2s whereas the section index s is given by the

human operator These coordinates are inaccurate due to measurement errorsθt= (uv)T (Fig 1b and c) A sectionrsquos pose relative to its adjacent sectionsis an unknown rigid transformation Rs (Fig 1d) Moreover sectioningpreparation after sectioning and even the imaging process itself inducenoticeable deformations of both sections as a whole (Ds) as well as on thescale of single tiles (Dt) (Fig 1a and endashh)

Therefore the task is to reconstruct a 3D image volume from serialsections that have to be reconstructed from single tiles while compensatingfor noticeable deformation within and across sections Particularly thedeformationndashcompensation poses the question how to invert the largelyunknown transforming process that mapped a 3D coordinate (xyz)T inscene space R

3S into a 2D coordinate (uv)T in tile space R

2t Assuming

planar sections of constant thickness as mentioned above we attach sceneand section spaces by a permutation Pzs between the scene dimension z andsection index s Now this mapping shows as

(uv

)=RsDtDsPzs

⎛⎝x

yz

⎞⎠+tt+θt (1)

with RsDtDs and (xyz)T being unknown Without precise knowledgeabout the deforming process DtDs the appropriate solution is that withminimal non-rigid deformation for all sections a model being as-rigid-as-possible This coincides with the researcherrsquos request not to introduceunmotivated artificial deformation to the image data that would adulterate

i58

by on June 7 2010 httpbioinform

aticsoxfordjournalsorgD

ownloaded from

[0933 1752010 Bioinformatics-btq219tex] Page i59 i57ndashi63

ssTEM registration

Fig 2 Similarity of serial sections depends on the scale of observation Two consecutive 60 nm sections from the registered Drosophila first instar larvalbrain dataset showing a part of the cortex and the neuropile The same region is shown with a resolution of 326 nmpx (ab) and 5216 nmpx (cd) It isclearly visible that the latter having approximately isotropic resolution shows significant similarity across sections while it is hard to identify correspondingpixels in the higher resolution example

his observations Declaring a tile being the unit of rigidity we approximateRsDtDs+tt by a rigid-per-tile transformation Rt such that

(uv

)=RtPzs

⎛⎝x

yz

⎞⎠+ωt+θt (2)

with ωt being the transformation error introduced by the approximationMaximal consistency relative to the scene is guaranteed by minimizing theremaining error εt=ωt+θt

Without knowing the scene it is impossible to estimate εt directlyHowever consistency with respect to the scene implies consistency relativeto overlapping tiles Let T be the set of all tiles tisinT and R be the set of allrigid transformations Rt isinR then the best configuration R is that minimizingthe sum of all relative transfer errors of pairwise overlapping tiles

argminR

sumtisinT

⎛⎝ sum

oisinTtεto

⎞⎠ (3)

Using landmark correspondences as a statistic about corresponding imagecontent εto is the sum of all square correspondence point displacements LetCto be a set of landmark correspondences (xy) between two tiles tisinT andoisinT twith xisin t and yisino then the best configuration R is that minimizingthe sum of all square correspondence displacements

argminR

sumtisinTf

⎛⎝ sum

oisinTt

⎛⎝ sum

(xy)isinCto

RtxminusRoy2⎞⎠

⎞⎠ (4)

with f isinT being a fixed tile that defines the scene reference frame All tilesmust represent one single interconnected graph being pairwise connected bysets of landmark correspondences

This optimization task requires a reliable landmark correspondencedetector Within a section the appropriate interest point detector needs tobe covariant to translation and robust against small amounts of non-rigiddeformation as the overlapping parts of two consecutive tiles show theprojection of the same tissue slightly deformed by heat during imagingFor cross-section matching the appropriate interest point detector needscovariance to rotation as well as an automatism to detect only structuresthat appear similar since two adjacent tiles show different portions of thetissue and are related by a rigid transformation and some plastic deformationIn ssTEM data depending on section thickness corresponding locationscan be identified by local appearance of large enough structures sectionedperpendicularly Such structures appear on different scales (for instance

mitochondria 500 nm nuclei 2microm and neuropile compartments 40microm) andthus detecting structures with large (xy)T-scale raises the chance to detectthe same structures in adjacent sections (Fig 2)

For its scale invariant nature and robustness we decided to useSIFT (Lowe 2004) for both intra- and intersection landmark correspondencedetection

22 SIFTSIFT (Lowe 2004) detects blobs of arbitrary size as interest points usingthe Difference of Gaussian detector estimates one or more dominantorientations for each detection and extracts a scale and orientation invariantlocal descriptor for each Such blob-like structures are common in mosttextured images including TEM data Typically we detect several hundredsto thousands of interest points in a TEM micrograph of 2000times2000 pximage size

The detectionrsquos size orientation and 2D location define a local coordinateframe for extraction of an invariant descriptor We found empirically thata larger descriptor built from 8times8 local histograms with eight bins each(512 dimensions) provides substantially better matching results for our datathan the originally proposed 4times4times8 descriptor The smaller descriptorswere proposed for natural images where viewpoint change and occlusionplay a significant role In TEM data where two regions are related onlyby a rigid transformation with some deformation a larger descriptor regionnecessarily increases the distinctiveness (Fig 3)

Interest point correspondence candidates are identified by nearestneighbor matching in the local descriptor space As suggested by Lowe(2004) good candidates are those that are significantly better than thenext nearest neighbor Significantly better means that the ratio between theEuclidean distances to the nearest and the next nearest neighbor is smallerthan a given threshold Empirically we identified a threshold of 092 forTEM images to yield a reasonable number of correspondence candidatesSince real-time performance is not required we identify the exact nearestneighbor by exhaustive search

Local descriptor matching results in a significant number of falsecorrespondences We use the Random Sample Consensus (RANSAC)(Fischler and Bolles 1981) to separate true correspondences that behaveconsistently with respect to a rigid transformation up to a maximalcorrespondence displacement εmax (see Algorithm 1) For estimation ofthe optimal rigid transformation by means of least square correspondencedisplacement we use the closed form solution described by Schaeferet al (2006) Significant deformation requires εmax to be relativelytolerant thus accepting some false correspondences We filter these witha robust regression scheme that iteratively removes correspondences with

i59

by on June 7 2010 httpbioinform

aticsoxfordjournalsorgD

ownloaded from

[0933 1752010 Bioinformatics-btq219tex] Page i60 i57ndashi63

SSaalfeld et al

Fig 3 SIFT-feature correspondences in two overlapping tiles from adjacentsections 1003 feature candidates were extracted in tile (a) 971 in tile (b)41 correspondence pairs were identified by local feature descriptor matching19 of them are true matches consistent to common transformation modelFalse matches are displayed in white true matches in black The size of thecircles is proportional to the features scale the filled part visualizes a featurersquosorientation Two true correspondence pairs are selected as an example forthe local SIFT-descriptor The values of the local histogram bins are shownas combs on top of the local region the descriptor is extracted from

Algorithm 1 RANSAC

Input Set of data points CModel MMaximal number of iterations kMaximal allowed error εmaxMinimal required number of inliers nmin

Output Set of inlier data points C+Model M best fitting to C+

1 C+larrempty Initialize best set of inliers2 for k iterations do3 Cminlarr choose random minimal subset of C to solve M4 Mlarr fit to Cmin5 C+larrempty Initialize set of inliers6 for all cisinC do7 if M(c)ltεmax then8 C+larrC+cupc9 end if

10 end for11 if (|C+|genmin)and(|C+|gt |C+|) then12 C+larrC+13 end if14 end for15 if |C+|genmin then16 Mlarr fit to C+ Refine M with respect to all inliers17 end if

Algorithm 2 Iterative square displacement minimization

Input Set of tiles TSet of fixed tiles Tf subTSet of landmark correspondences C

Output Optimal configuration R=R1R2R|T |

1 εRlarrinfin Initialize the average displacement2 repeat3 for all tisinT Tf do4 Rlarr fit rigid transformation to Ct5 for all (xy)isinCt do6 xlarrRx Update landmarks in Ct 7 end for8 RtlarrRRt9 end for

10 ε2Rlarr 1

|C|sum

(xy)isinC yminusx211 until εR converges

a displacement larger than 3times the median displacement For details seeSaalfeld and Tomancaacutek (2008)

23 Global optimization of the tile configurationHaving the set of true landmark correspondences identified successfullythe optimal configuration R of all tiles has to be estimated As shown inEquation (4) this is the configuration with minimal square correspondencedisplacements

We minimize this term using an iterative optimization scheme (seeAlgorithm 2) In case that prior knowledge about the gross configuration ofall tiles is available the minimization is initialized with this configurationOtherwise all tiles start from Rt=I In each iteration i the optimal Rt for eachsingle tile tisinT f relative to the current configuration is estimated usingthe closed form least square solution by Schaefer et al (2006) and applied toall landmark coordinates in this tile The scheme terminates on convergenceof the average correspondence displacement εRi that is estimated after eachiteration As convergence criteria we require εRi to have a low absoluteslope |nablah εRi| and an absolute value below some threshold given by the userIt is crucial to prevent being trapped in local minima or at long plateaus Weaddress this effectively by requiring |nablah εRi| to be very low for all h from a setof decreasing lengths The maximal length h of a plateau or local valley themaximal total number of iterations and the maximal accepted remaining errorare user-defined parameters with empirically estimated defaults suggested

24 Notes on implementationThe described registration algorithm including SIFT RANSAC closed formleast square error solutions for a variety of transformation models the robustregression scheme and the iterative optimizer are implemented using the programming language and provided as an Open Source library through theImageJ distribution Fiji (Schindelin 2008) The registration procedure isincluded in the software toolkit TrakEM2 (Cardona 2006) for managementregistration and analysis of massive serial section microscopy datasets andby that in daily use in a large number of laboratories allover the world

3 RESULTS

31 Overview of the registration algorithmPrincipally the registration procedure consists of extractinglandmarks from all tiles identifying landmark correspondencesbetween tile pairs and estimating the tile configuration thatminimizes the sum of all square correspondence displacements asshown in Algorithm 3

i60

by on June 7 2010 httpbioinform

aticsoxfordjournalsorgD

ownloaded from

[0933 1752010 Bioinformatics-btq219tex] Page i61 i57ndashi63

ssTEM registration

Fig 4 Artificially generated evaluation dataset The dataset simulates thin transilluminated volumes of 20 px thickness with membrane- and blob-likestructures at various scales Structures are defined by volumetric density functions Locations with higher density scatter more lsquolightrsquo than those with lowerdensity and appear darker in the shadow projection Two adjacent sections are shown to visualize the cross-section change of visible structures In (ab) thefull dataset (4096times4096 px) is shown at low magnification an area of 512times512 px is marked and shown in (cd) at high magnification

Algorithm 3 Register TEM-dataset principal procedure

Input Set of tiles TOutput Tile configuration R

Extract interest points from all tiles1 for all tisinT do2 Ftlarrextract interest points3 end for

Identify interest point correspondences for pairs of tiles4 C empty set of correspondences5 for all tisinT do6 Ct empty set of correspondences7 for all oisinT t do8 Ctolarr identify correspondence pairs from Ft and Fo9 CtlarrCtcupCto

10 end for11 ClarrCcupCt12 end for

Estimate the optimal tile configuration R13 RlarrargminεR

Practically since adjacent sections are related by a rigidtransformation Rs plus some non-rigid deformation Ds that isto be compensated we would like to use the transformation Rsas initialization for the sought after configuration R and as ahint which tiles to exclude from pairwise feature comparisonUnfortunately single sections do not always represent a singlelandmark-interconnected graph of tiles while the whole volumeeventually does This is a typical scenario when imaging tree-likestructures and the operator following only the branches ignoringthe rest of the volume That is instead of the section as a wholeall graphs of a section have to be tested against all graphs presentin the previous section Two graphs that can be registered to eachother are then joined into a single one by matching and filteringthe features of overlapping tiles Eventually this will result inone single graph representing the whole volume After joiningtwo graphs the configuration of the resulting graph is optimized

thus sequentially building up the optimal global configuration fromoptimal intermediate configurations

We applied this registration approach to test sets of TEM data ofthe Drosophila first instar larval brain sectioned into 60 nm sectionsimaged at 326 nmpx resolution We developed a visualizationof the global optimization progress where for each iteration thecorresponding landmarks and the residual transfer error betweenthem are highlighted by green dots and red lines respectivelyMovie 1 available as supplementary on-line material shows theprogress of the optimization for a single section dataset consistingof 6times6 tiles1 It highlights the ability of the method to correctlyplace even tiles that contain mostly background (lower left cornertile) and gain very little SIFT feature detections as long as these areattached to the graph If a tile lacks SIFT correspondences altogetherit represents an independent graph on its own The user can choose todrop or hide all but the largest graph from the dataset Movie 2 showsthe progress of the optimization on three serial sections imaged as4times4 tile mosaics demonstrating that the procedure for multiplesection is essentially the same as for a single section2

32 Quantitative assessment of registration results onsynthetic evaluation data

We created an artificial dataset with the ray-tracing programPOV-ray (Cason et al 2007) that allows to define the interior ofa volume as a density pattern based on an arbitrary volumetricfunction where high density scatters more light than low density(Fig 4)3 This simulates the imaging process in the TransmissionElectron Microscope where structures with a higher density ofheavy-metal atoms scatter more electrons than structures with lowerdensity and appear darker at the screen The interior of the evaluationdataset contains filamentous and lsquoblobrsquo-like structures both overthree scale octaves with multiscale turbulence In this way it mimicstypical structures in the stained tissue like membranes nucleoli andmitochondria

To emulate the ssTEM data we generated 16 projectionsof 4096times4096 px each through consecutive volumes of 20 px

1Movie 1 httpflympi-cbgdesaalfeld36avi2Movie 2 httpflympi-cbgdesaalfeld48avi3Evaluation-dataset httpflympi-cbgdesaalfeldtem-evaluation

i61

by on June 7 2010 httpbioinform

aticsoxfordjournalsorgD

ownloaded from

[0933 1752010 Bioinformatics-btq219tex] Page i62 i57ndashi63

SSaalfeld et al

Fig 5 Drosophila first instar larval brain ssTEM dataset Two consecutiveregistered sections from the dataset as red-cyan color merge The diameterof the brain is sim60microm (a) The section mosaics as a whole with theareas zoomed in (b) marked (c) A single perpendicular section through theentire registered volume Several sections were lost during sectioning andcollecting onto the electron microscopy grid and shown here as black rows

thickness which is roughly equivalent to the ratio of xy versusz resolution in typical TEM data We split the synthetic groundtruth data into 1024times1024 px tiles with 102 px overlap resultingin 25 images per section We ran our registration program on theresulting 400 images as if there was nothing known about the tilesposes other than the section index The registration results in a stackthat is anchored to a randomly selected tile in the world referenceframe Thus missing comparable world coordinates we evaluate thequality of the registration by estimating the coordinate translationthat gives the minimal average displacement when applied to theregistered stack This identifies the world reference frame of theregistered stack and gives an average registration error of a pixelrelative to the original scene

We select a random sample of 1000 locations in each tile andtransfer all locations into the world reference frame using the trueand the registered transformation model of the tile The translationbetween registration and ground truth domain is that between thecentroids of both point clouds The average residual transfer errorserves as a quantitative measure of the registration process Weestimated an average displacement ε = 414 px SD σε = 363 px andmaximal displacement εmax= 1571 px for the registration resultThese results demonstrate that with the proposed method we canboth identify overlap in an unknown configuration of images andregister them

33 Registration of the Drosophila first instar larvalbrain ssTEM dataset

As an example of real biological data we registered the Drosophilafirst instar larval brain ssTEM dataset consisting of 85 sections of60 nm thickness covering lateral neuronal layers and part of theneuropile of the left hemisphere Each section was imaged withTEM using a moving stage operated by the Leginon software as9times9 tiles overlapping bysim6 Tiles have a size of 2048times2048 pxand a resolution of 468 nmpx All 6885 tiles were registered fully

automatically Intrasection configurations were initialized with theodometry data of the microscope Correspondence estimation usingSIFT descriptors and robust geometric consensus filters performedvery well even in presence of significant changes in illuminationand sharpness dirty sections and significant gaps of up to sixsections in the stack Visual inspection of the registered data showsreliable continuity of biologically relevant structures such as axonbundles within and across sections both at low and high scales(Fig 5a and b)4 Moreover when the registered dataset is cutperpendicularly the resulting image whose lateral resolution islimited by section thickness resembles electron microscopy datawithout major discontinuities suggesting that the registration wassuccessful (Fig 5c)5 For visualization and collaborative annotationwe present the registered dataset on-line through the CATMAID webinterface (Saalfeld et al 2009)6

4 DISCUSSIONWe presented a fully automated method for registration oflarge ssTEM image mosaics The method identifies the optimalconfiguration of image tiles regardless whether or not an initial guessof the configuration is available It is capable of correctly placingtiles with only minimal image content provided that this content isconnected to the rest of the tile graph Disconnected graphs of tilesare registered independently Thus the approach is ideally suitedfor alignment of ssTEM data generated either manually or using arobotic setup

We use SIFT to identify corresponding landmarks and use themas a statistical sample for similar image content in the TEMimages The scale invariance of SIFT is well suited to capture thechanging appearance of small structures at low scale levels and thepresence of meaningful similar features at multiple scales Largermore distinctive local descriptors together with the requirement thatcorrespondence candidates must be geometrically consistent yieldsreliably large sets of true matches

The reduction of the registration task to a compact set ofcorresponding landmarks enables global minimization of squarecorrespondence displacements for very large tile systems Regis-tering the 6885 tiles Drosophila larval brain dataset took sim24 hon an Intelreg Xeonreg computer with two 266 GHz dual-core-CPUsand 8 GB of RAM with the most time-consuming operation in theprocedure being the exhaustive nearest neighbor search in the 512 Dlocal descriptor space Principally there is no upper limit to the sizeof the dataset other than the available storage space

The described methods for robust outlier removal and globaloptimization are applicable to a wide variety of microscopyimage mosaicking tasks We used it successfully to register lightmicroscopy image mosaics including sets of overlapping 3Dconfocal image stacks (Preibisch et al 2009b) and applied it tobead-based registration of Selective Plane Illumination Microscopy(SPIM Huisken et al 2004) datasets consisting of 3D stacks of thesame specimen taken from different angles (Preibisch et al 2009a)In Preibisch et al (2009b) we identify the pairwise 3D translationbetween overlapping confocal image stacks by normalized cross-correlation and globally minimize the translational offset of large

4Section series as a movie httpflympi-cbgdesaalfeldseriesavi5Resliced series as a movie httpflympi-cbgdesaalfeldreslicedavi6Registered dataset on-line httpflympi-cbgdefirst-instar-brain

i62

by on June 7 2010 httpbioinform

aticsoxfordjournalsorgD

ownloaded from

[0933 1752010 Bioinformatics-btq219tex] Page i63 i57ndashi63

ssTEM registration

sets of overlapping stacks In Preibisch et al (2009a) we proposedthe usage of fluorescent beads embedded in the rigid mountingmedium as fiduciary markers We use the local constellationsof neighboring beads to automatically identify correspondencesinvariantly to 3D rotation and scale The model for the geometricconsistency constraint and global minimization of the square transfererrors is a 3D affine transformation with one of the 3D stacks servingas a template

We generated a unique evaluation dataset designed to resemblethe appearance of biological tissue in ssTEM data for quantitativecomparison of the registration results The evaluation framework canbe used to assess the performance of any registration scheme thatrecords its transformation model and could become the standardfor objectively measuring the performance of various registrationapproaches In future work we will induce artificial deformationand variance in section thickness to the dataset in order to examinethe limits of our registration method As an alternative more realisticground truth evaluation dataset we propose to use Serial Block-FaceScanning Electron Microscopy data (Denk and Horstmann 2004)where consecutive sections are aligned per definition

The presented registration method is non-rigid but as-rigid-as-possible with a tile being the unit of rigidity for intrasectionalignment This delivers an ssTEM image dataset amenable toquantitative analysis such as double dissector estimation of synapticdensity (Geinisman et al 1996) which require volumes to be asreliable as possible On the other hand rigid-per-tile registrationcannot compensate large-scale deformation by tile displacementwithout introducing noticeable discontinuities at the tile bordersThe solution to this problem lies in extending the approach toinclude arbitrary non-rigid deformation at scales below the tile levelpreserving the regularization in terms of local rigidity That meansdeforming all images minimally We are currently exploring variousideas of how to express this regularization for a completely non-rigid global registration The evaluation dataset will be instrumentalin assessing the performance of such approaches with respect toreflecting faithfully the ground truth data

The globally optimal reconstruction of entire brains on TEM levelwill enable registration of 3D light microscopy data onto electronmicroscopy volumes By that it will be possible to establish theconnection between brain macro (neuronal lineages) and micro(synaptic connectivity) circuitry (Cardona et al 2009)

ACKNOWLEDGEMENTSWe want to thank the HHMI Janelia Farm Research Campus forhosting the Janelia Hackathons 2007 and 2008 Mitya Chklovskiifor hosting us and sharing ideas and Johannes Schindelin formaintaining the Fiji platform

Funding DIGS-BB PhD stipend (to SS)

Conflict of Interest none declared

REFERENCESAndersonJR et al (2009) A computational framework for ultrastructural mapping of

neural circuitry PLoS Biol 7 e1000074BayH et al (2008) SURF Speeded up robust features Comput Vis Image

Understand 110 346ndash359BrownM and LoweDG (2003) Recognising panoramas In ICCV rsquo03 Proceedings

of the Ninth IEEE International Conference on Computer Vision IEEE ComputerSociety Washington DC pp 1218ndash1225

CardonaA (2006) TrakEM2 an ImageJ-based program for morphological data miningand 3D modeling In Proceedings of the 1st ImageJ User and Developer ConferenceLuxembourg May 18ndash19

CardonaA et al (2009) Drosophila brain development Closing the gap betweena macroarchitectural and microarchitectural approach Cold Spring Harb SympQuant Biol [Epub ahead of print doi101101sqb200974037]

CasonC et al (2003ndash2007) POV-ray ndash the persistance of vision raytracer [releaseversion 361] Availabe at httpwwwpovrayorg (last accessed date August 82007)

DenkW and HorstmannH (2004) Serial block-face scanning electron microscopy toreconstruct three-dimensional tissue nanostructure PLoS Biol 2 1900ndash1909

FischlerMA and BollesRC (1981) Random sample consensus a paradigm for modelfitting with applications to image analysis and automated cartography CommunACM 24 381ndash395

GeinismanY et al (1996) Unbiased stereological estimation of the total number ofsynapses in a brain region J Neurocytol 25 805ndash819

HuiskenJ et al (2004) Optical sectioning deep inside live embryos by Selective PlaneIllumination Microscopy Science 305 1007ndash1009

LoweDG (2004) Distinctive image features from scale-invariant keypoints Int JComput Vis 60 91ndash110

MatasJ et al (2002) Robust wide baseline stereo from maximally stable extremalregions In RosinPL and MarshallD (ed) Proceedings of the British MachineVision Conference Vol 1 BMVA London pp 384ndash393

MikolajczykK and SchmidC (2003) A performance evaluation of local descriptorsIn International Conference on Computer Vision and Pattern Recognition Vol 2pp 257ndash263

MikolajczykK and SchmidC (2004) Scale and affine invariant interest point detectorsInt J Comput Vis 60 63ndash86

PreibischS et al (2009a) Bead-based mosaicing of single plane illuminationmicroscopy images using geometric local descriptor matching In PluimJPWand DawantBM (ed) Proceedings of The International Society for OpticalEngineering Vol 7259

PreibischS et al (2009b) Globally optimal stitching of tiled 3D microscopic imageacquisitions Bioinformatics 25 1463ndash1465

RasbandW (1997ndash2009) ImageJ Image processing and analysis in Java [version 143l]Available at httprsbinfonihgovij (last accessed date November 24 2009)

SaalfeldS and TomancaacutekP (2008) Automatic landmark correspondence detectionfor ImageJ In Proceedings of the ImageJ User and Developer ConferenceLuxembourg pp 128ndash133

SaalfeldS et al (2009) CATMAID Collaborative annotation toolkit for massiveamounts of image data Bioinformatics 25 1984ndash1986

SchaeferS et al (2006) Image deformation using moving least squares ACM Transon Graph 25 533ndash540

SchindelinJE (2008) Fiji is just ImageJmdashbatteries included In Proceedings of theImageJ User and Developer Conference Luxembourg pp 99ndash104

SulowayC et al (2005) Automated molecular microscopy the new Leginon systemJ Struct Biol 151 41ndash60

TuytelaarsT and Van GoolL (2004) Matching widely separated views based on affineinvariant regions Int J Comput Vis 59 61ndash85

WhiteJG et al (1986) The structure of the nervous system of the nematodeCaenorhaditis elegans Phil Trans R Soc Lond Ser B Biol Sci 314 1ndash340

i63

by on June 7 2010 httpbioinform

aticsoxfordjournalsorgD

ownloaded from