AN OPEN-SOURCE VAA3D PLUGIN FOR REAL-TIME 3D VISUALIZATION OFTERABYTE-SIZED VOLUMETRIC IMAGES

Alessandro Bria1,2 Giulio Iannello1 Hanchuan Peng3,4

1Integrated Research Centre, University Campus Bio-Medico of Rome, 00128 Rome, Italy2Department of Electrical and Information Engineering, University of Cassino and L.M., 03043 Cassino, Italy

3Janelia Farm Research Campus, Howard Hughes Medical Institute, Ashburn, Virginia 20147, USA4Allen Institute for Brain Science, 551 North 34th Street, Suite 200, Seattle, Washington 98103, USA

ABSTRACTModern high-throughput bioimaging techniques pose the un-precedented challenge of exploring and analyzing the pro-duced Terabyte-scale volumetric images directly in their 3Dspace. Without expensive virtual reality devices and/or par-allel computing infrastructures, this becomes even more de-manding and calls for new, more scalable tools that help ex-ploring these very large 3D data also on common laptops andgraphic hardware. To this end, we developed a plugin for theopen-source, cross-platform Vaa3D system to extend its pow-erful 3D visualization and analysis capabilities to images ofpotentially unlimited size. When used with large volumet-ric images up to 2.5 Terabyte in size, Vaa3D-TeraFly exhib-ited real-time (subsecond) performance that scaled constantlyon image size. The tool has been implemented in C++ withQt and OpenGL and it is freely and publicly available bothas open-source and as binary package along with the mainVaa3D distribution.

Index Terms— Terabyte, visualization, Vaa3D

1. INTRODUCTION

Modern high-throughput bioimaging techniques enable rapidthree-dimensional imaging of large specimens at submicro-metric resolution [1, 2, 3]. This led to the generation of im-ages that easily reach or exceed the Terabyte size and, asconsequence, to the subsequent effort of biomage informati-cians to develop more scalable techniques for typical bioim-age informatics tasks such as image stitching and registration[4, 5, 6], visualization [5, 7, 8] and automated analysis [9, 10].In addition to scalability, another bottleneck is the inabilityto efficiently explore the complicated 3D image content andthus to input user-specified information of the observed im-age patterns directly in the 3D space. Indeed, a number ofbioimage-oriented software tools for Terabyte-scale [7, 8, 11]and Gigabyte-scale [12, 13] images are based on 2D cross-sectional views or a combined display with an arbitrary-anglecutting plane. On the other hand, the few tools supporting3D visualization and analysis either do not scale well to the

Terabyte-size [14, 15], or are limited owing to high expense oflicenses (e.g. Amira(VSG), Imaris (Bitplane)) and infrastruc-tures [16]. This presents an obstacle for the unbiased, high-throughput and quantitative analysis of bioimage data andcreates tremendous need for the development of new tech-niques that help explore very large 3D data directly and effi-ciently without expensive virtual reality devices and/or paral-lel computing infrastructures.

The open-source Vaa3D system [17] was specifically de-signed for bridging such gaps, particularly for the tasks ofvisualization, interaction, analysis and management of large(Gigabyte-size), multidimensional (including 3D, 4D and 5Dand so on) microscopic images [18]. The underlying idea is tomap users’ inputs in the 2D plane of a computer screen to the3D locations of biological entities (for example, cells, neu-rons or microtubules) in the volumetric space of a 3D imagestack. This is achieved through a family of 3D-WYSIWYG(what you see in 2D is what you get in 3D) computer algo-rithms named Virtual Finger [19].

We showed in [19] how a single finger operation is effi-ciently used by our Vaa3D plugin TeraFly to generate a 3Dregion of interest (ROI) for subsequent fetching and render-ing of image data at a higher resolution, thus enabling fast(subsecond) visualization of Terabyte-size images. In this ar-ticle, we focus on the technological aspects of the tool anddemonstrate its scalability to images of potentially unlimitedsize. We also propose a method to generate a 3D ROI formouse scroll input, which provides a user’s experience sim-ilar to how Google Earth allows viewing satellite imageries.Finally, we show that Vaa3D-TeraFly exhibits real-time per-formance regardless on image size when used on both high-and medium-end laptops.

2. APPROACH

Vaa3D-TeraFly is designed around three basic functionali-ties enabling the instant 3D Google Earth-like visualizationof Terabytes, and virtually infinite amount, of high-resolutionimage data: (i) volume import, i.e. the generation of an ad

978-1-4799-2374-8/15/$31.00 ©2015 IEEE 520

X

Y

Z

Layer 0 𝐼 0

[…]

[…] row (3,0)

column 0

column 3

row (3,3)

block (3,3,0) block (3,3,1) block (3,3,2)

Layer 1 𝐼 1

Layer 2 𝐼 2

TMITREE

Fig. 1. An example of TMITREE composed by three lay-ers. Each layer is composed by a number of nonoverlappingblocks of image data that are stored into separate files. Thesefiles are organized according to a two-level hierarchy of fold-ers where the first level subdivides the volume into columns(or rows) and the second one into rows (or columns).

hoc multiresolution tree-structured representation of the vol-umetric image; (ii) zoom-in to a ROI, i.e. the generation of aROI when the user zooms-in into the displayed image with themouse scroll; and (iii) fetch and display of image data, i.e. theloading of the ROI’s content from the storage and subsequent3D display. These functionalities are detailed in the followingsubsections.

2.1. Volume import

Vaa3D-TeraFly is built on the top of an image tiled multires-olution representation, hereafter referred to as Tiled Multires-olution Image Tree (TMITREE), which is generated as fol-lows. Let be I the original, highest-resolution, very large-sized image to be imported. Starting from I , we produce an-other k images {I(1), I(2), . . . , I(k)} obtained by iterativelydownsampling by two I(j) from I(j−1) ∀j = 1, . . . , k, whereI(0) = I . This process is iterated until the lowest-resolutionimage I(k) fits within the 3D viewer box of size Bx(x) ×By(y) × Bz(z). These define the maximum size in pixels ofdisplayable image data in the 3D renderer and can be set bythe user according to the graphic hardware capabilities (de-fault is 256(x)×256(y)×256(z) pixels). At all resolutions,image data are stored according to a tile-wise organization ofnonoverlapping three-dimensional blocks (see Fig. 1).

A converter tool, named TeraConverter, comes alongwith Vaa3D-TeraFly for converting a given bunch of imagefiles into the TMITREE representation. With TeraConverter,the user can select the number of layers to be produced, theblock dimensions and the format of the image files. Once theTMITREE has been produced, it can be opened with Vaa3D-

prev: Explorer

volume: Volume

next: Explorer

set ROI

load ROI

show

rend: Renderer

rend: Renderer

get interpo-lated ROI

hide

new renderer

loading ended

update image

new explorer

an object instantiated in the GPU thread

an object instantiated in the IO thread

total time

IO time saved

(a)

(b)

(i) (ii) (iii)

(ii)

(iii)

(i)

(c)

(i) (ii) (iii)

Fig. 2. ROI’s loading and display. (a) UML Sequence Dia-gram. (b) The effect of actions (i), (ii) and (iii) of (a) on theviewport when zooming-in with the mouse scroll. The actionstaken by the tool to preview/display and to load the data areperformed in parallel on two different threads, named GPUand IO, respectively. While the preview is calculated and dis-played by the GPU thread, image data are loaded by the IOthread. In this way, the first part of the IO time is transparentto the user, i.e., it is not perceived as a delay.

TeraFly by selecting any of its layers I(j), j = 0, . . . , k.The 3D exploration then automatically starts by loading anddisplaying the entire image content of the highest layer inthe TMITREE, i.e., the lowest resolution image I(k). Thisusually takes < 5 seconds as I(k) is conveniently stored inone binary uncompressed file for fast volume opening.

2.2. Zoom-in to a ROI

The underlying idea is that zoom-in with the mouse wheelis usually performed in the nonempty regions of an imageand, thus, without losing in generality, we can assume thepresence of one or more biological entities displayed on theforeground of the viewport. Since the mouse-wheel zoom-inin the Vaa3D renderer is viewport-centered, these entities arelikely to occur around the center of the viewport and, thus,

521

cache

Fig. 3. 3D exploration though different layers of the TMITREE with mouse-wheel scrolls up (zoom-in) and down (zoom-out)and caching of ROIs for quick navigation

we can focus on this region to calculate the ROI’s center. Tothis end we define a 2D point in the center of the viewportand apply the point-pinpointing algorithm (PPA) of VirtualFinger [19] to determine the correspondent 3D location in thevolumetric image. PPA repeatedly applies the mean-shift al-gorithm to each channel intensity along the shooting ray or-thogonal to the screen until convergence and finally selectsthe best point among the points found for all the channels.The ROI is finally calculated as the Bx(x)×By(y)×Bz(z)sized box centered on the 3D point.

2.3. Fetch and display of image data

The proposed mouse scroll zoom-in to a ROI requires theimage content of the requested ROI to be retrieved and dis-played as quickly as possible for fast and smooth explorationof the image. This is accomplished by performing, in paral-lel, two different operations: (i) reusing part of the currentlyviewed content to display instantly a preview of the ROI; and(ii) loading from the TMITREE only the blocks that containthe required image content. More formally, let be ROI` theROI currently viewed that belongs to the TMITREE layer` (ROI` ≡ I(k) when the exploration starts) and ROIL

the ROI to be retrieved and displayed from the higher res-olution layer L, with L ≤ ` (usually L = ` − 1). Then,Vaa3D-TeraFly interpolates to the resolution of L the por-tion of ROI` that intersects with ROIL and displays theresult. This gives the clue that the system has correctly andinstantly responded to the request, and also makes more ac-ceptable to wait for the full image to be loaded. Meanwhile,

Vaa3D-TeraFly loads the image content from the blocks ofthe TMITREE layer L that intersect with ROIL and updatesthe viewed ROIL with the higher resolution data. The de-tailed sequence of actions taken by the tool is shown in Fig. 2.Throughout 3D exploration, the ROIs that have been alreadyviewed are cached in the graphic card memory (if possi-ble) or in the system memory following a web-browser-likestrategy in which the user can instantly go back and forward(see Fig. 3). This requires storing in the graphic card mem-ory (k + 1) × Bx × By × Bz × c bytes at most, where cis the number of channels and we assumed 8 bits per pixelfor display. Based on the criterion to choose k described inSection 2.1, and after simple algebraic manipulations, themaximum memory requirement for a cubic image of N pix-els is

⌈(log(N1/3)− log(B)) log2(e)) + 1

⌉× B3 × c bytes

where we chose Bx = By = Bz = B. Remarkably, thiscorresponds to only 336 Megabytes for 1 Teravoxel three-channels image (i.e. 10, 0003 voxels) and to 480 Megabytesfor 1 Petavoxel three-channels image (i.e. 100, 0003 voxels).

3. RESULTS

We tested visualization of 3D image volumes of rat neuronsand whole mouse brains with sizes ranging from 22.8 Gb to2.5 Tb. The statistics for each test case were produced by onat least 20 trials of randomly selected target ROIs in arbitrarilydetermined scales of the respective images. The total time forgeneration, loading and displaying of a 3D ROI is reported inFig. 4 for three different laptops with different CPUs, GPUs

522

0

500

1000

1500

2000

2500

10 100 1.000

3D-v

iew

er t

ota

l gen

erat

ion

tim

e fo

r a

RO

I (m

s)

Image size (Gigabytes)

Asus M70vn + external hard drive 2.5'' WD 1Tb USB 2.0CPU Intel Core 2 Duo P8600, RAM 4 Gb, GPU NVIDIA GeForce 9650M GT 1Gb

Sony Vaio VPCF21Z1E + external hard drive 2.5'' WD 1Tb USB 3.0CPU Intel Core 2 i7-2630QM, RAM 8 Gb, GPU NVIDIA GeForce GT 540M 1Gb

MacBook Pro Retina + external hard drive Seagate 4Tb USB 3.0CPU Intel Core i7, RAM 16 Gb, NVIDIA GeForce GT 650M 1Gb

2.520

Fig. 4. Bench-test of ROI-generation time (mean ± s.d.) fordifferent large images on different laptops. Each point in thegraph corresponds to one test case run on one dataset of theindicated size.

and external storages. Remarkably, the time scaled constantlyon image size regardless of the hardware employed. In par-ticular, subsecond performance was achieved with a high-endlaptop, whereas on medium-level laptops it ranged from 1.3to 2.2 seconds, which is still acceptable for effective 3D vi-sualization. RAM usage was also monitored in all the experi-ments and resulted below 1 Gb.

4. CONCLUSIONS

3D visualization and analysis of very large bioimages are instrong need of powerful and more scalable 3D explorationtechniques for high-throughput and quantitative analysis ofbioimage data. We have discussed here a software tool thatextends the Vaa3D powerful 3D visualization and analysis ca-pabilities to images of potentially unlimited size. The resultsprovided demonstrate that the tool can visualize in real-timeTerabytes of 3D images and can scale well to even larger im-ages without the need of expensive hardware or parallel in-frastructures.

5. REFERENCES

[1] R Tomer, L Ye, B Hsueh, and K Deisseroth, “Advanced clarityfor rapid and high-resolution imaging of intact tissues,” Nat.Protocols, vol. 7, no. 9, pp. 1682–1697, 2014.

[2] K Chung and K Deisseroth, “CLARITY for mapping the ner-vous system,” Nat. Methods, vol. 10, no. 6, pp. 508–513, 2013.

[3] L Silvestri, A Bria, L Sacconi, G Iannello, and FS Pavone,“Confocal light sheet microscopy: micron-scale neuroanatomyof the entire mouse brain,” Optic Express, vol. 20, no. 18, pp.20582–20598, 2012.

[4] A Bria and G Iannello, “Terastitcher - a tool for fast au-tomatic 3D-stitching of teravoxel-sized microscopy images,”BMC Bioinformatics, vol. 13, no. 1, pp. 316, 2012.

[5] AG Arivis, “Vision 4D,” http://vision.arivis.com/, 2014, [accessed 23-October-2014].

[6] L Cooper, K Huang, and M Ujaldon, “Parallel Automatic Reg-istration of Large Scale Microscopic Images on Multiproces-sor CPUs and GPUs,” in Parallel and Distributed Process-ing Workshops and Phd Forum (IPDPSW), 2011 IEEE Inter-national Symposium on, May 2011, pp. 1367–1376.

[7] T Pietzsch, “Bigdataviewer,” http://fiji.sc/BigDataViewer, 2014, [accessed 23-October-2014].

[8] W-K Jeong, J Beyer, M Hadwiger, R Blue, C Law, A Vazquez-Reina, RC Reid, J Lichtman, and H Pfister, “SSECRETand neurotrace: Interactive visualization and analysis tools forlarge-scale neuroscience data sets,” IEEE Comput. Graph.Appl., vol. 30, no. 3, pp. 58–70, May 2010.

[9] P Frasconi, L Silvestri, P Soda, R Cortini, FS Pavone, andG Iannello, “Large-scale automated identification of mousebrain cells in confocal light sheet microscopy images,” Bioin-formatics, vol. 30, no. 17, pp. i587–i593, 2014.

[10] J Stegmaier, JC Otte, A Kobitski, A Bartschat, A Garcia,G Nienhaus, U Strhle, and R Mikut, “Fast segmentation ofstained nuclei in terabyte-scale, time resolved 3D microscopyimage stacks,” PLoS ONE, vol. 9, no. 2, pp. e90036, 02 2014.

[11] Z Husz, N Burton, B Hill, N Milyaev, and R Baldock, “Webtools for large-scale 3d biological images and atlases,” BMCBioinformatics, vol. 13, no. 1, pp. 122, 2012.

[12] MD Abrmoff, PJ Magalhes, and SJ Ram, “Image Processingwith ImageJ,” Biophotonics International, vol. 11, no. 7, pp.36–42, July 2004.

[13] F Long, J Zhou, and H Peng, “Visualization and analysis of3d microscopic images,” PLoS Comput Biol, vol. 8, no. 6, pp.e1002519, 2012.

[14] T Pietzsch, S Preibisch, C Rueden, S Saalfeld, B Schmid, J-Y Tinevez, DJ White, V Hartenstein, K Eliceiri, P Tomancak,and A Cardona, “Fiji: an open-source platform for biological-image analysis,” Nat. Methods, vol. 9, no. 7, pp. 676–682,2012.

[15] JL Clendenon, CL Phillips, RM Sandoval, S Fang, andKW Dunn, “Voxx: a pc-based, near real-time volume render-ing system for biological microscopy,” American Journal ofPhysiology - Cell Physiology, vol. 282, no. 1, pp. C213–C218,2002.

[16] D N Williams et al., “Ultrascale visualization of climate data,”Computer, vol. 46, no. 9, pp. 68–76, 2013.

[17] H Peng, Z Ruan, F Long, JH Simpson, and EW Myers, “V3denables real-time 3d visualization and quantitative analysis oflarge-scale biological image data sets,” Nat Biotech, vol. 8, no.4, pp. 246–249, 2010.

[18] H Peng, A Bria, Z Zhou, G Iannello, and F Long, “Extensiblevisualization and analysis for multidimensional images usingVaa3D,” Nat. Protocols, vol. 9, no. 1, pp. 193–208, 2014.

[19] H Peng et al., “Virtual finger boosts three-dimensional imagingand microsurgery as well as terabyte volume image visualiza-tion and analysis,” Nat. Communications, vol. 5, 2014.

523