A FAST AND AUTOMATED FRAMEWORK FOR EXTRACTION OF NUCLEI FROMCLUTTERED 3D IMAGES IN FLUORESCENCE MICROSCOPY

Sorin Pop1,3, Alexandre Dufour1,3, Jean-Francois Le Garrec2,4, Chiara V. Ragni2,4,Margaret E. Buckingham2,4, Sigolene M. Meilhac2,4 and Jean-Christophe Olivo-Marin1,3

1 Institut Pasteur, Quantitative Image Analysis Unit, F-75015 Paris, France2 Institut Pasteur, Molecular Genetics of Development Unit, F-75015 Paris, France

3 CNRS URA 2582, F-75015 Paris, France4 CNRS URA 2578, F-75015 Paris, France

ABSTRACTConfocal fluorescence microscopy has become a standard tool to im-age thick 3D tissue samples, permitting the observation of cell be-haviour, such as cell division within developing organs. However,robust and automatic extraction of nuclear shape and mitotic ori-entation may be hindered by a highly cluttered environment as forexample in mammalian tissues. We propose a fast and automatedframework for the segmentation of nuclei from cluttered 3D images,allowing robust quantification of various parameters such as numberof cells, number of mitoses and mitotic orientation. We have appliedthis framework to scans of the developing mouse heart, and manualvalidation on three independent experiments indicates a detectionrate of 93% in all cases. Moreover, the proposed tool permits fast,real-time 3D rendering of the data set during the analysis, and can beeasily adapted to other applications related to dense tissue analysis.

Index Terms— organ development, cell division analysis, 3Dsegmentation, active meshes.

1. INTRODUCTION & RELATED EFFORTS

The study of organ morphogenesis has benefited from recent ad-vances in 3D fluorescence microscopy techniques, giving novel in-sight into numerous processes including aspects of cell behavioursuch as cell division, cell shape changes, cell death and cell polarity[1]. In vertebrates, since cell behaviour is not stereotyped, quantita-tive and statistical analyses are required. Moreover, analysis of cellpolarity or division orientation relies on good-quality segmentationof the nuclei. Reliable quantification requires robust and automatedmeasurements applied onto a large number of experiments. In thisrespect, state-of-the-art confocal microscopy systems have allowedimaging of tissue samples in 3D and over time with multiple fluores-cent markers in a very reproducible manner, leading to an exponen-tial increase in the amount of imaging data produced. Analyzing thisquantity of data can no longer be done manually, calling for robustand automated analysis and visualization tools to handle the data inthe simplest possible way.

Yet, automated image analysis in developmental biology is avery challenging task; highly cluttered and dense cell environmentlimit the penetration depth of state-of-the-art confocal microscopes,

�Correspondence:{adufour,spop,jcolivo}@pasteur.frThis work was funded by Institut Pasteur (PTR 335), CNRS, INSERMJFLG is financed with a grant from the E.U. Project ’CardioCell’

(LT2009-223372).

beyond which light scattering fades out the fluorescent signal de-tection. Moreover, phototoxicity and photobleaching phenomenaboth impact on the viability of the samples, imposing low exposuretimes and therefore low signal to noise ratio, further complicatingthe image analysis process, especially in a time-lapse imaging con-text [2, 3, 4].

Recent efforts have been devoted to improving the imaging pro-cess. In [3], transgenic quail lines were generated, where only en-dothelial cells express a fluorescent marker, thus simplifying thequantification of multi-cellular movements in an embryo which isamenable to live-imaging. Nevertheless, images still suffer frompoor spatial resolution, hindering precise segmentation at the nuclearlevel.

Another solution lies in more subtle imaging setups such as sin-gle plane illumination microscopy or SPIM [5], in which a thin sam-ple plane is illuminated simultaneously, while the emitted light iscollected in the normal direction, thereby reducing bleaching andtoxicity. The light sheet (or the sample) can then be displaced toacquire the full 3D sample from different positions or angles. Un-fortunately such systems are currently limited to small-sized samples(typically zebrafish embryos or drosophila larvae) and are not com-mercially available, hence their use is limited to a small number oflaboratories.

On the automated image analysis side, the most recent effortsto obtain precise quantification of 3D morphogenetic information atthe single cell level are presented in [4]. Here the authors propose aworkflow for nuclear and cellular segmentation in the upper sectionof a developing zebrafish embryo observed by time-lapse confocalmicroscopy. They describe an automatic image processing work-flow that estimates the location of the nuclei using data filteringtechniques and a 3D Hough transform, in order to initialize levelset-based active contours to achieve segmentation of the nuclear andcellular membranes. Validation is presented against manual mem-brane delineation. However, validation of the nuclei segmentation isnot described, and the proportion of correctly detected cells in theimaged tissue is not mentioned.

In the work presented here we focus upon morphogenesis of themouse heart, which differs from the zebrafish model in three mainaspects: a) the mouse embryo is significantly more difficult to cul-ture, so we have used fixed samples; b) the overall cost induced byperforming experiments on mice is such that the amount of infor-mation extracted from each sample should be maximized; c) the ac-quired images suffer from a much lower signal to noise ratio, mostlydue to the fact that the tissue is not transparent as in the zebrafish

2113978-1-4244-4128-0/11/$25.00 ©2011 IEEE ISBI 2011

case, yielding more light scattering and hence a more cluttered envi-ronment.

In this paper we propose a fully integrated framework for the au-tomated segmentation of nuclei from dense and cluttered 3D tissue,enabling the systematic quantification of parameters of organ devel-opment such as the number of nuclei, mitosis detection and measure-ment of the orientation of cell division. We also focus on provid-ing an efficient framework that is able to analyze the data strikinglyfaster than current approaches using a standard workstation, whileallowing live 3D manipulation of the analyzed data set through theuse of the active mesh framework [6]. In the following we presenta global overview of the image processing workflow, and refer thereader to appropriate references for further details at each step.

2. PROPOSED FRAMEWORK

We describe here the main steps of the image processing workflow,summarized in Fig. 1 and illustrated in Fig. 2. We start by describingthe pre-filtering approaches used to enhance the input signals, thendetail the pre-detection algorithm that is used to initialize the finalsegmentation step achieved using active meshes.

Red channel (membranes)

Blue channel(nuclei)

Green channel(mitotic bridges)

filtering

filtering & subtraction

pre-detection

segmentation

validation

filtering (section 2.1)

(section 2.2)

(section 2.3)

(section 2.4)

(section 3.2)

Fig. 1. Overview of the image processing workflow for automated segmen-tation of nuclei (refer to indicated sections in text for further details). Notethat the final validation step (dashed arrows) is done manually to assess theperformance of the framework.

2.1. Nuclear signal filtering

Our images suffer from the traditional set of disturbances inducedby digital light microscopy systems: blur effect induced by the pointspread function of the optical setup, Poisson noise depicting the pho-ton counting process, and additive gaussian shot noise. Additionally,the nuclear signal is not homogeneous due to pecularities of the la-beling protocol, yielding random bright spots within the nuclei (cf.Fig. 2a left). To solve these issues, we first perform a 3× 3× 3 me-dian filter on the data, to remove the bright nuclear spots, followedby a classical PDE-based diffusion filter [7] to smooth the remainingnoise and enhance the boundaries of nuclei (cf. Fig. 2b left).

2.2. Membrane signal filtering & subtraction

Once the signal from the nucleus has been filtered, some close nu-clei may now appear to touch and therefore cannot be distinguished,even visually. Here we exploit the membrane signal to separate thesevisually touching nuclei. We start by filtering the membrane signalusing an anisotropic diffusion filter we developed for seismic data in

[8], which will enhance this signal while smoothing out surroundingnoise. We then subtract the filtered membrane data from the filterednuclear signal and thereby recover the intensity cusp between neigh-boring nuclei (cf. Fig. 2c left).

2.3. Pre-detection of nuclei

In order to initialize the active mesh framework described subse-quently, we apply a fast pre-detection method that will automaticallydetect the approximate shape and location of the nuclei in the tissue.Since the filtered data (including background subtraction) is still notsuitable for basic thresholding techniques, we apply a more evolvedpre-detection method, that we had proposed in [9], which performsa multi-class K-Means thresholding of the intensity distribution ofthe data, followed by size constrained object extraction in ascendingorder of intensity class (cf. Fig. 2c right). This method is able toseparate nuclei clusters efficiently, while it requires only 2 parame-ters (given by prior knowledge of the manipulated sample) which arethe minimum and maximum size of a typical nucleus. This methodhas proven to be particularly powerful in conjunction to active con-tours methods for automated segmentation of cells in 2D high con-tent screening applications (see [10, 11] for further details).

a) pre-detection on raw data

b) pre-detection after data filtering

c) pre-detection after membrane subtraction

Fig. 2. Impact of the pre-processing steps (cf. sections 2.1 and 2.2) on thepre-detection algorithm (cf. section 2.3). Note that data filtering (b) removesimaging artifacts but also blurs nuclear membranes. After membrane sub-traction (c) more nuclei have been detected, while some erroneous detectionshave been corrected (blue stars). The final result (bottom right) is used toinitialize the active meshes described in section 2.4

2.4. Segmentation of nuclei using active meshes

Once each nucleus has been individually detected during the previ-ous pre-detection step, we perform a fast and robust segmentation ofthe nuclei using the active mesh framework [6], which is a fast dis-crete implementation of our multiple coupled active surfaces model[12]. The principle is to initialize a triangular mesh close to eachobject of interest (which we do here automatically by triangulat-ing each pre-detected nucleus into a mesh using the fast MarchingTetrahedra algorithm [13]), and deform all meshes simultaneously

2114

by minimizing an energy functional combining several sets of termsrelated either to the image or to geometrical characteristics of the de-forming meshes. The final energy functional for an image I wheren contours C1..n evolve simultaneously reads:

E(R0..n, C1..n) = λ0

ZR0

|I − c0|2dΩ +

nXi=1

"λ1

ZRi

|I − ci|2dΩ + μ

ICi

gdC + γ

nXj=i+1

ZRi∩Rj

dΩ

#

where c0 is the average intensity of the background region R0, ci

is the average intensity inside the segmented region Ri bounded byCi, g is the result of an edge detector function applied to the originalimage, dC is the N − 1-dimensional boundary element, dΩ is theelementary space and λ0, λ1, μ, γ, η are empirical weights. The firsttwo terms (weighted by λ{0,1}) correspond to the data attachment,and are a multi-phase extension of the popular Chan-Vese-Mumfor-Shah functional [14]. The third term (weighted by μ) regularizes thisill-posed inverse problem and expresses the geodesic length of thecontour. The final term (weighted by γ) couples the evolution of allmeshes to prevent them from overlapping during their deformation,allowing us to handle touching nuclei. Further details on the methodcan be found in [6].

3. EXPERIMENTS AND RESULTS

3.1. Biological and imaging protocol

The heart has a precise geometry which is essential to orchestratethe circulation of the blood [15]. To analyze the cell behaviourwhich underlies the shaping of organs it is necessary to preservethis geometry. Therefore, mouse hearts were dissected out fromembryos at 8.5 days after fertilisation (E8.5). They were fixed in0.5% paraformaldehyde and labeled as a whole mount. With fluo-rescent imaging, it is possible to combine multiple markers to fol-low several cell characteristics at once. The transgenic line R26-mT/mG [16] provided a red-fluorescent marker of cell membranes,which was amplified by immunostaining with an anti-DsRed anti-body. Labeling of the nuclei in blue was performed with Hoechststaining. In principle, dividing cells can be detected with this stain-ing at anaphase. However, the percentage of cells in anaphase perscan of the mouse heart was too low (less than 1%), precluding sta-tistical analysis. In contrast, immunostaining with an anti-Aurkbantibody, which recognizes the AuroraB kinase, led to the detectionin green of the cytoplasmic bridges which connect recently dividedsister cells. This led to the identification of a cell population referredto as telophase nuclei, representing more than 10% of the total num-ber of cells. Whole mount hearts were mounted between coverslips.Multi-channel 16-bit images were acquired sequentially on a LeicaSP5 inverted confocal microscope with a 40× 1.25 oil immersionobjective. Z-stacks of 30 to 40 μm were scanned every 1 μm. Anexample of an image is shown in Fig. 3.

3.2. Results & Validation

We present here segmentation results on three independent scans ofthe heart. Due to the high number of cells in the observed tissue,manual validation on the entire data set is not really feasible. Sincevalidation on simulated data may not reflect exactly the entire char-acteristics of the imaging process, we have chosen to validate seg-mentation on a fraction of the entire set that is relevant to our analy-sis, namely on nuclei in telophase, which give valuable information

Fig. 3. 3D view of a mouse heart scan (#170709, cf. Table 1). Nuclei are inblue, membranes in red and cytoplasmic bridges in green. Data size: 1024 x1024 x 40 voxels. Resolution: 0.379 x 0.379 x 1 μm.

Scan IDNuclei Nuclei in Correctly

extracted telophase segmented

170709 1626 194 (12%) 181 (93.3%)

250310 985 166 (17%) 155 (93.4%)

260310 857 104 (12%) 97 (93.3%)

Table 1. Validation of the proposed approach on three independent scansof E8.5 mouse hearts. Manual identification was performed on nuclei intelophase, based on the presence of a cytoplasmic bridge. Segmentation per-formance is given by the ratio of telophase cells correctly identified.

about the orientation of cell division. The advantage of choosing thissub-population of nuclei is that manual validation is greatly facili-tated by the presence of a bright cytoplasmic bridge between pairsof sister cells within the green image channel (cf. Fig. 3). Hence,segmentation performance was assessed by visually identifying thetwo nuclei in telophase for each bridge, and checking whether thesewere correctly segmented by the method. Table 1 summarizes theperformance of the proposed framework on the three observed scans.The number of nuclei in telophase (third column) which have beenmanually identified via the cytoplasmic bridges, cover from 12% to17% of the entire population of nuclei, which we believe representsa good assessment of the performance of the framework. The finalcolumn gives the number of nuclei in telophase which have beencorrectly segmented by the proposed approach. The proportion ofcorrect segmentation is 93% in all cases. The remaining 7% includeunsegmented nuclei in the deeper z sections (where the signal tonoise ratio decreases dramatically) and touching nuclei segmentedas a unique nucleus. Overall this segmentation performance is satis-factory for extracting biological information.

2115

3.3. Performance & implementation

The entire framework has been developed in Java and computationswere done on a quad core 2.4 GHz CPU with 6 GigaBytes of RAM.The total processing time per scan ranged from 10 to 30 minutes,depending on the total number of nuclei in the scan. Although pub-lications of other frameworks with the same purpose do not provideanalysis time, we believe that this is an important parameter. Unpub-lished information indicates that our framework performs strikinglyfaster than current tools, thanks to the fast pre-detection step and tothe active mesh segmentation. Furthermore, the active mesh frame-work offers live 3D visualization and manipulation of the data dur-ing its analysis, providing 3D views of the evolving scene (cf. Fig.4) and also permitting adjustment of the algorithm parameters in aninteractive and intuitive manner.

4. CONCLUSION AND PERSPECTIVES

We have presented an integrated framework which can automati-cally extract nuclei from dense and highly cluttered environmentsobserved in 3D confocal microscopy. The proposed framework com-bines a number of robust PDE-derived approaches at every step ofthe analysis process, providing a fast and efficient toolset for robustquantification of morphogenesis on a nuclear scale. The approachwas successfully applied to several tissue samples from developingmouse hearts, with a segmentation performance of 93% on a val-idated data set. These encouraging results lead us to pursue thedevelopment of the proposed approach in several directions. Ourfirst goal is to perform semi-automated validation of the extractionprocess, by automatically extracting cytoplasmic bridges and iden-tifying the corresponding pair of nuclei. In a second step, we willconcentrate on extracting the cell shapes via a similar strategy onthe membrane channel by exploiting the extracted nuclear informa-tion. This will allow a more in-depth analysis of cell characteristicsand thereby provide a complete toolset for morphogenetic analysisof developing tissues.

5. REFERENCES

[1] M. Mavrakis, O. Pourquie, and T. Lecuit, “Lighting up developmentalmechanisms: how fluorescence imaging heralded a new era,” Devel-opment, vol. 137, pp. 373–387, 2010.

[2] S. Megason, “In toto imaging of embryogenesis with confocal time-lapse microscopy,” Methods in Molecular Biology, vol. 546, pp. 317–332, 2010.

[3] Y. Sato, G. Poynter, D. Huss, M.B. Filla, A. Czirok, B.J. Rongish, C.D.Little, S.E. Fraser, and R. Lansford, “Dynamic analysis of vascularmorphogenesis using transgenic quail embryos,” PLoS One, vol. 5, no.9, pp. e12674, Sept. 2010.

[4] C. Zanella, M. Campana, C. Melani, G. Sanguinetti, P. Bourgine,K. Mikula, N. Peyrieras, and A. Sarti, “Cells segmentation from 3-D confocal images of early zebrafish embryogenesis,” IEEE Trans. onImage Processing, vol. 19, no. 3, pp. 770–781, Mar. 2010.

[5] J. Huisken, J. Swoger, F. del Bene, J. Wittbrodt, and E.H.K. Stelzer,“Optical sectioning deep inside live embryos by selective plane illumi-nation microscopy,” Science, vol. 305, pp. 1007–1009, 2004.

[6] A. Dufour, R. Thibeaux, E. Labryuere, N. Guillen, and J.-C. Olivo-Marin, “3d active meshes: fast discrete deformable models for celltracking in 3d time-lapse microscopy,” IEEE Trans. on Image Pro-cessing, 2011, in press.

[7] F. Catte, P.-L. Lions, J.-M. Morel, and T. Coll, “Image selectivesmoothing and edge detection by nonlinear diffusion,” SIAM J. Nu-merical Analysis, vol. 29, no. 1, pp. 182–193, 1992.

Fig. 4. Segmentation result on the scan illustrated in Fig. 3. Each meshsegments an individual nucleus, and is colored randomly for visualizationpurposes. Total processing time: 30 minutes on a Quad core 2.4 GHz CPU.

[8] O. Lavialle, S. Pop, C. Germain, M. Donias, Sebastien Guillon, Naa-men Keskes, and Yannick Berthoumieu, “Seismic fault preserving dif-fusion,” J. Applied Geophysics, vol. 61(2), pp. 132–141, 2007.

[9] A. Dufour, V. Meas-Yedid, and J.-C. Olivo-Marin, “Automated quan-tification of cell endocytosis using active contours and wavelets,” inProc. ICPR’2008, Tempa, FL, USA, Dec. 2008.

[10] V. Meas-Yedid, A. Dufour, B. Zhang, A. Grassart, N. Sauvonnet, andJ.-C. Olivo-Marin, “Automated quantification of cellular processeswith applications to high-content screening,” in Handbook on PatternRecognition and Computer Vision, C.H. Chen, Ed. 2009, pp. 1–13,World Scientific.

[11] A. Grassart, V. Meas-Yedid, A. Dufour, J.-C. Olivo-Marin, A. Dautry-Varsat, and N. Sauvonnet, “Pak1 phosphorylation enhances cortactinN-WASP interaction in clathrin-caveolin independent endocytosis,”Traffic, vol. 11, no. 8, pp. 1079–1091, Aug. 2010.

[12] A. Dufour, V. Shinin, S. Tajbaksh, N. Guillen, J.-C. Olivo-Marin, andC. Zimmer, “Segmenting and tracking fluorescent cells in dynamic3d microscopy with coupled active surfaces,” IEEE Trans. on ImageProcessing, vol. 14, no. 9, pp. 1396–1410, Sept. 2005.

[13] G.M. Treece, R.W. Prager, and A.H. Gee, “Regularised marching tetra-hedra: improved iso-surface extraction,” Tech. Rep., Cambridge Uni-versity Engineering Department, 1998.

[14] T. Chan and L.A. Vese, “Active contours without edges,” IEEE Trans.on Image Processing, vol. 10, no. 2, pp. 266–277, Feb. 2001.

[15] S. Meilhac and M. Buckingham, “The behaviour of cells that formthe myocardial compartments of the vertebrate heart,” in Heart De-velopment and Regeneration, N. Rosenthal and R. Harvey, Eds. 2010,Elsevier.

[16] M.D. Muzumdar, B. Tasic, K. Miyamichi, L. Li, and L. Luo, “A globaldouble-fluorescent Cre reporter mouse,” Genesis, vol. 45, pp. 593–605,2007.

2116