Image and Vision Computing 29 (2011) 434–441

Contents lists available at ScienceDirect

Image and Vision Computing

j ourna l homepage: www.e lsev ie r.com/ locate / imav is

Editor’s Choice Article

Video-based, real-time multi-view stereo☆,☆☆

George Vogiatzis a,⁎, Carlos Hernández b

a Computer Science, Aston University, Birmingham, B4 7ET, UKb Google, Seattle, WA 98103, USA

☆ Editor's Choice Articles are invited and handled byEditorial Board committee. This paper has been recoPhilippos Mordohai.☆☆ This work was carried out while both authorsGroup, Toshiba Cambridge Research Laboratory, CambriThe research was fully funded by Toshiba Research Eur⁎ Corresponding author. Tel.: +44 7900023260.

E-mail addresses: g.vogiatzis@aston.ac.uk (G. Vogiatcarloshernandez@google.com (C. Hernández).

0262-8856/$ – see front matter © 2011 Elsevier B.V. Adoi:10.1016/j.imavis.2011.01.006

a b s t r a c t

a r t i c l e i n f o

Article history:Received 17 November 2010Accepted 28 January 2011

Keywords:Real-timeMulti-view stereo3d reconstructionShape-from-X

We investigate the problem of obtaining a dense reconstruction in real-time, from a live video stream. In recentyears, multi-view stereo (MVS) has received considerable attention and a number of methods have beenproposed. However,mostmethods operate under the assumption of a relatively sparse set of still images as inputand unlimited computation time. Video based MVS has received less attention despite the fact that videosequences offer significant benefits in terms of usability of MVS systems. In this paper we propose a novel videobasedMVSalgorithm that is suitable for real-time, interactive3dmodelingwith a hand-held camera. Thekey ideais a per-pixel, probabilistic depth estimation scheme that updates posterior depth distributions with every newframe. The current implementation is capable of updating 15 million distributions/s. We evaluate the proposedmethod against the state-of-the-art real-time MVS method and show improvement in terms of accuracy.

a select rotating 12 membermmended for acceptance by

were at the Computer Visiondge Science Park, CB4 0GZ, UK.ope Ltd.

zis),

ll rights reserved.

© 2011 Elsevier B.V. All rights reserved.

1. Introduction

In the last few years, binocular and multi-view stereo (MVS)research has reached a certain level of maturity where high qualityreconstruction results can readily be obtained for a variety of scenes[23,26]. However, the possibility of applying MVS methods to videostreams has received less attention. There are several reasons why avideo based, real-time MVS system is an interesting alternative to stillimage systems. For small scale reconstructions, video acquisition canpotentially be faster andmore user-friendly than acquiring still picturese.g. rapid prototyping or industrial modeling. Similarly, for very largescene reconstructions like city-wide 3d models, where large quantitiesof data are required, video can offer an affordable way of capturing amassive amount of data in a user-friendly manner. Since large scalereconstruction algorithms are very data hungry, it is no surprise that thetwo main paradigms used to feed them are either video [6,22] orcommunity photo collections [1,21,24]. Photo collections have the bigadvantage of being constantly updated. The main disadvantage is that,for themoment, only touristic places such as the Colosseum in Rome orthe city of Dubrovnik have enough photographs to achieve high qualityreconstructions [1]. Video on the other hand can be used to reconstructany scene of interest on demand.

From an algorithmic point of view, video has several characteristicsthat distinguish it from still image capture. Firstly, the quality of data istypically lower than that of still images in terms of image resolution,motion blur or compression artifacts. Secondly, the quantity of data isorders ofmagnitude bigger than a still image sequence. Most of the top-performing MVS techniques would not cope in terms of memory andcomputation time with a relatively small video sequence. The largeamount of data however is also an advantage because it resolves manyof the ambiguities inherent in MVS that arise from repeated texture,texture-less regions or occlusion. This is in contrast to the traditionalapproach of addressing ambiguities in MVS through computationallyexpensive regularization. Finally, the baseline between successiveframes is small which means that image flow is limited to a few pixelsbetween successive frames. The search for correspondences is thereforeeasier because there are less image locations to search.

In this paperwepresent an algorithm that exploits the advantages ofvideo input while also being resilient to the challenges it poses. Thesystem we describe maintains a large set of candidate 3d features(~250,000 of them) at any given point in time. It has an estimate of their3d position that is improved with every incoming frame. Whenconfidence in this estimate is sufficiently high, the candidate 3d featureis consolidated into a3dpoint and leaves the candidate poolwhile a newone is introduced in its place.

The key requirements of this strategy are:

1. a sequential update scheme to obtain intermediate 3d estimates asvideo frames are acquired,

2. a precision measure to assert the accuracy of the 3d estimates,3. an outlier rejection scheme to reliably reject outliers that arise in

MVS due to occlusion, lack of texture, etc.,4. efficiency both in terms of memory and computation time.

Fig. 1. Searching for a match along an optic ray. For a given pixel p we wish to find thedepth Z along the optic ray through p such that the 3d point x(Z) projects to similar imageregions in images I and I ′ .We canmeasure this similarity by computing amatching scorebetween the two image patches W and W′ .

435G. Vogiatzis, C. Hernández / Image and Vision Computing 29 (2011) 434–441

An elegant framework that satisfies the first three requirements isprobabilistic inference [16]. In this paper we propose a novel,parametric, Bayesian approximation to the MVS problem that complieswith all of the above requirements, including being extremely fast tocompute and having a very lowmemory footprint (each 3d feature uses9 floats to model the position plus 25 B for the reference image patch).The main contributions of this paper are:

1. a probabilistic treatment of occlusion robust photo-consistency [13],2. a parametric approximation to the full probabilistic inference

problem that makes the real-time BayesianMVS problem tractable,3. a video based MVS system that is shown to process video input in

real-time (60 Hz) while providing intermediate reconstructionresults as user feedback.

2. Previous work

This paper is primarily related to MVS literature but also to real-timepose and scene reconstruction from video. We start by referring to theMVS evaluation by Ref. [23]. Looking at the top performers in thatevaluation, we can distinguish twomain trends: region growingmethods[8,11,12,20] and occlusion-robust photo-consistency methods[3,5,10,13,18,27]. The best performing region growing method [8] uses acombination of photo-consistency based patch fitting, growing andfiltering in order to reconstruct the scene of interest. This approach issuccessful sinceplane-basedphoto consistencyperformsverywell on lowtextured regions or sparse data sets. However it is not obvious how thepatch growing step could be transferred to a video setting. This is becauseby the time a patch has been optimized and new patches must begenerated in its vicinity the camera will have already moved away fromthat region. The algorithmwould therefore have to keep previous imagesin memory which is not feasible for long sequences at 30–60 fps.Alternatively the systemwould have to wait until the camera revisits thepatches whichmakes it difficult to use in camera drive-through scenarios(e.g. a car-mounted MVS system reconstructing parts of a city).

Occlusion-robust photo-consistency methods provide a very simplepipeline using off-the-shelf algorithms such as dense stereo and 3dsegmentation algorithms. However, they rely on a much simplerwindow-based photo-consistency, less robust to sparse images andlack of texture. The top performer in this group [13] estimates depth byhistogram voting of local maxima of a photo-consistency measure. In areal-time video setting however, depth estimation using histogramspresents the following difficulties: (a) As new frames are acquired andthe histogram is updated with new local maxima, it is not clear how tomeasure confidence in the current depth estimate. (b) Estimationaccuracy depends on bin size. (c) Histograms tend to be memoryintensive. Our method is inspired by this second group of methods andproposes a probabilistic interpretation of occlusion-robust photo-consistency. We derive a parametric approximation to the posteriordepth distribution which overcomes the difficulties of the histogramapproach: (a) The probabilistic framework offers confidence measuresfor the estimates computed while (b) estimates are not quantized.Finally (c) our representation of the depth posterior has a lowmemoryfootprint.

Our work is also related to real-time urban reconstruction methods[6,22]. While Ref. [6] assumes a very simple shape model for thebuildings, the method of Ref. [22] could be used to reconstruct general3d scenes. The main differences with their approach are twofold:robustness to camera motion and improved accuracy. The core of theiralgorithm is based on producing a dense stereo depth-map every 0.5 susing the frames captured during that time. The depth-map is thenfused in 3d with the previously generated structure resolving anyinconsistencies that may arise. This system works very well for car-mounted cameras where the motion of the camera is smooth and withslow-varying speed. If the baseline of the cameras is too small (the carstops) or too big (the car is too fast), the system just drops those frames.

This makes their algorithm less suitable for hand-held interactive MVS,our goal, where the cameramotion is generally not smooth. In contrast,our formulation is independent of the type of cameramotion. Eachbit ofthe geometry is only generatedwhenever a certain degree of confidenceand 3d accuracy is reached. Thismeans that a given part of the geometrywith good focus and enough baseline could be generated in just a fewtenths of a second while other parts, with occlusions or with shakycamera motion may take longer.

This paper also shares similarities with visual SLAM [7] in the waywe sequentially update a pool of 3d features. The main difference isthat we aim to maximize the production of 3d features and weassume the camera pose estimation is given so we can exploitepipolar geometry. Ref. [29] is a recent MVS method based on real-time visual SLAM. A small number of video frames are automaticallyselected and an optic-flow based dense stereo algorithm is applied tothose frames, leading to a dense, smooth surface representation ofthe scene within seconds. The results obtained are compelling,however their approach of using a small image set and heavyregularization contrasts with our approach of using hundreds ofvideo frames and no regularization. It is also not clear how well [29]would cope with closed surfaces.

Finally, a probabilistic approach toMVShas alreadybeenproposed ina number of papers [2,4,9,14,25]. However, while these methods modelocclusion explicitly, our approach assumes probabilistic independenceof the depth of different pixels and occlusion is implicitly modeled asanother source of noise. The independence assumption is key in order tomake the real-time MVS problem tractable and we believe that ourresults justify it in practice.

3. Probabilistic depth sensor

Let p be a pixel of image I that has been calibrated for pose andinternal camera parameters. For a particular depth value Z one canobtain the corresponding 3d point x(Z) that is located Z units away fromI , along the optic ray of p (see Fig. 1). Let I ′ be another calibrated imageacquired from a nearby viewpoint while W and W denote two squarepatches centered on the projection of 3d point x(Z) onto I and I ′respectively. We can evaluate the photo-consistency [19] at 3d locationx(Z) using Normalized Cross Correlation (NCC). Fig. 2 shows a plot ofNCC scores for a pixel across depth as well as a histogram of the localmaxima of these curves for 60 neighboring images.

Zmin Zmax−1

−0.5

0

0.5

1

mat

chin

g sc

ore

(a)

Zmin Zmax0

10

20

30

40

50

60

(b)

Fig. 2. Depth estimation with NCC maxima. (a) NCC score across depth along optic ray. The black dots correspond to local maxima. (b) Histogram of local maxima for 60 neighboringimages. Local maxima are either generated in the vicinity of the true depth or are uniformly generated across the depth range.

436 G. Vogiatzis, C. Hernández / Image and Vision Computing 29 (2011) 434–441

We observe that the histogram is concentrated on a single modecorresponding to the true depth with a uniform component thatcorresponds to occlusion, image warping, repetitive texture etc. Thispicture suggests a probability distribution that is a mixture between agood measurement and a bad measurement model. The goodmeasurement model places nearly all its probability mass around thecorrect depth location while the bad measurement model uniformlydistributes its mass in all possible depth locations. The next sectionexplains this in more detail.

We view the local maxima x1,…,xN as a set of noisy measurementscoming from a depth sensor. Wemodel the sensor as a distribution thatmixes a good measurement model with a bad one as is common inrobust sensor fusion problems (e.g. chapter 21 of Ref. [16]). Our sensorcan produce two types of measurement with a probability π and 1−πrespectively: (1) a good measurement that is normally distributedaround the correct depth Z or (2) an outlier measurement that isuniformly selected from the interval [Zmin,Zmax]. The limits Zmin and Zmax

can be determined by someprior knowledge of the scene geometry. Theobject of interest is guaranteed to be entirely contained between Zmin

and Zmax. The following Gaussian+Uniform mixture model describesthe probability distribution of the n-th measurement given the correctdepth location Z and the inlier probability π

p xn jZ;πð Þ = πN xn jZ;τ2n� �

+ 1−πð ÞU xn jZ min;Z maxð Þ: ð1Þ

The variance of a good measurement τn2 can be obtained from therelative position of the cameras at frame I and I ′ that produced themeasurement. This is becausewe assume that themeasurement xnhas afixed variance of one pixel when projected in I ′. We then back-projectthis variance in 3d space to compute the variance of themeasurement indistance units.

3.1. Bayesian inference

The likelihood introduced inEq. (1) is a typicalmixturemodel and, assuch, its parameters could be estimated from the data x1,…,xN in amaximum likelihood framework using Expectation Maximization.However, it is crucial to have a measure of confidence in our depthestimate as it can be used to inform the system when enoughmeasurements have been collected as well as to detect when theestimation has failed. This is not offered by a maximum likelihoodapproach. Also, in our experiments EMwas trapped in a local optimumfor a significant percentage of cases. We therefore opt for a Bayesianapproach where we define a prior over depth and inlier ratio and thencalculate the posterior distribution given all measurements. The

estimated depth is then the maximum of this posterior distributionwhile its shape (through 2nd order moments) determines theconfidence in our estimation.

Assuming all the measurements x1,…,xN are independent, theposterior has the form

p Z;π jx1…xNð Þ∝p Z;πð Þ∏np xn jZ;πð Þ ð2Þ

where p(Z,π) is our prior on depth and inlier ratio. Fig. 3 (top row)shows some snapshots from the evolution of p(Z,π|x1…xn) asmeasurements are collected. The prior is assumed to be uniformand the distribution is modeled using a dense 2d histogram. Theposterior converges to the correct values for Z and π. In an experimentdescribed in Section 5 and summarized in Table 1 we show how thisprobabilistic formulation outperforms the histogram voting approachused in Refs. [13,27].

However modeling the posterior with a full 2d histogram for eachpixel is impractical due to memory and computation limitations. Oursystem maintains 250,000 seeds at any given time. A reasonable 2dhistogram should be quantized with at least 500 values for depth and100 values for the inlier ratio. To keep these histograms in memory wewould need to store 12.5 billion floats (which is non-trivial). Further-morewe found that evenwith aGPU implementation itwasnot possibleto perform updates of the full 2d histograms in real-time. Our approachis to use a parametric approximation to the posterior as outlined next.

3.2. Parametric approximation to the posterior

When the seed corresponds to awell-textured unoccluded pixel, thehistogram of depth observations has a single mode (see Fig. 5a). Thismotivates the use of a uni-modal parametric posterior. A goodapproximation to the depth posterior (4) is the product of a Gaussianfor the depth with a Beta distribution for the inlier ratio. In thesupplementary material we provide a variational argument for thisform. In particular we show how it has the smallest Kullback Leiblerdivergence from the true posterior out of a wide set of possibleapproximation distributions that share a weak factorization property.We therefore define our approximation to the posterior of Eq. (2) as

q Z;π jan; bn; μn;σnð Þ : = Beta π jan; bnð ÞN Z jμn;σ2n

� �: ð3Þ

InEq. (3), an andbn canbe thoughtof asprobabilistic counters of howmany inlier andoutliermeasurements haveoccurredduring the lifetimeof the seed. Theother two, μn andσn

2, represent themeanand variance of

Fig. 3. Non-parametric vs. parametric modeling of posterior distribution. The first row shows the posterior distribution that is modeled non-parametrically as a 2d histogram. Thefour columns represent four time instances (after 5, 10, 20 and 100 updates). The second row shows the evolution of our parametric Gaussian×Beta approximation. Even though ourmodel cannot capture the multi-modal nature of the true posterior, after a few iterations it converges to the same point estimate. The third row shows the histogram ofmeasurements that have been seen by the system in each time instance. The last three rows show one of the few cases where the parametric model cannot follow the non-parametricone.When this happens it can be detected because the parametric posterior predicts a very low inlier ratio. We can therefore safely discard it. The x axis denotes depth along optic rayfor all images.

437G. Vogiatzis, C. Hernández / Image and Vision Computing 29 (2011) 434–441

ourGaussiandepth estimate. Now, if q(Z,π|an−1,bn−1,μn−1,σn−12 )was

the true posterior after n−1 measurements, the new posterior afterobserving xn would have the form

C × p xn jZ;πð Þq Z;π jan−1; bn−1; μn−1;σ2n−1

� �ð4Þ

for some constant C. This distribution is no longer of the formGaussian×Beta but we can approximate it using moment matching.We therefore define the new parameters an,bn,μn,σn

2 such that theproduct in Eq. (4) and our approximation to the true posterior q(Z,π|an,bn,μn,σn

2) share the same first and second order moments for Z and π.This update is straight forward to calculate analytically but we refer thereader to the supplementary material for the actual formulae. Thesecond and fifth row of Fig. 3 show the parametric approximation to theposterior as it evolves through the Bayesian updates. Even though ourapproximation is uni-modal while the true posterior is not, it is nearlyalways able to converge to the same values of Z and π. In the few caseswhere it is not able to converge (fifth rowof Fig. 3), thedistributiongiveshigh probability to a very low inlier ratio.When this happens,we can beconfident that the estimationhas failed andwe candisregard the results.

Fig. 4 shows a typical evolution of this Bayesian update with theparametric approximation. The estimates of Z and π (Fig. 4a and b)converge to the correct values as can be seen by superimposing themeasurement histogramwith themarginalizedmeasurement posteriorp(x|x1,…,xn) (Fig. 4c).

It is important to note that the success or failure of the estimationproblem also depends upon the quality of the reference patch W thatstays fixed throughout the evolution of the sensor's depth posterior. Ifthat patch is well textured and visible in the subsequent frames theestimation is typically successful. If on the other hand the pixel is either

Table 1Accuracy/completeness comparison between histogram voting (first row) and ourprobabilistic sensor model on the Middlebury ground truth data [23] (second row). Theprobabilistic model is outperforming the histogram approach across all completenesslevels.

Completeness 50% 80% 85% 90% 95% 100%

Histograms (mm) 0.36 0.80 0.99 1.34 2.10 4.00Gauss+Uniform (mm) 0.33 0.70 0.84 1.05 1.57 3.42

untextured or becomes occluded in the subsequent frames then theestimation fails. Crucially, these failure cases can bedetected because theestimated inlier ratio is very low. Such cases are shown in Fig. 5b and c.

4. System details

One of the aims of this paper is to evaluate the feasibility andusability of a video basedmulti-view stereo algorithm. In this sectionwedescribe a real-time implementation of a reconstruction system basedon the ideas described previously. The system is a CUDA based, mixedCPU/GPU implementation. It can update 250,000 seeds, 60 times/s,running on an Intel XEON 2.33 GHz with an NVIDIA GTX 260. We alsohave a portable implementation on a dual-core laptop with an NVIDIAGTX 280M chipset. The portable version updates 250,000 seeds at30 times/s.

4.1. Camera pose estimation

Recently, there have been major advances in visual SLAM techni-ques. References to recent work can be found in Refs. [7,17]. Thesemethods can track the 6DOF motion of a camera from point and linefeatures in a video sequence of a rigid scene. In this paper however ouraim is to evaluate the dense3d structure estimation independently fromany inaccuracies in a SLAM based camera tracker. To achieve this wechose a simple but very accurate template based camera trackingmethod using Ref. [28]. This technique which includes a per framebundle-adjustment stage obtains reprojection errors of less than0.05 pixels.

4.2. Evolution of seeds

The system performs sequential inference of the depth of variouspixels in the video sequence. A seed corresponds to a particular pixel pwhose depth we aim to estimate. Due to memory and computationlimitations we can maintain a fixed number of seeds throughout theprocess. Each seed is associated with a set of parameter values (an,bn,μn,σn

2,W). The first four parameters that evolve during the lifetime of theseed, describe our posterior given the first n observations according to Eq.(3). With each seed we also store a reference image patchW around thepixel location of the seed on the reference image, I . This patch remainsconstant and is used to compare against target images and obtain depth

438 G. Vogiatzis, C. Hernández / Image and Vision Computing 29 (2011) 434–441

measurements. When a seed is created we set a0=10 and b0=10. Thiscorresponds to a prior for the inlier ratio centered on 0.5 and with astandard deviation of approximately 0.1. The depth parameters μn and σn

2

are set such that 99% of the prior probabilitymass is between some presetZmin and Zmax. These limits define a bounding volume which is known tocontain the object of interest.

During the lifetime of a seed we obtain depth measurements byevaluating NCC between the stored patch W and patches W on theepipolar lineon thecurrent frame I ′ (see Fig. 1). Ideallywewould like tosearch the entire epipolar line for local maxima in NCC score but this isnot feasible computationally with ordinary hardware. Instead, weexploit the small inter-frame motion by only searching within a radiusof w pixels away from the projection of the prior mean μn. This violates

50 100 150−10

−5

0

5

10

(a)

50 100 150

0

0.2

0.4

0.6

0.8

1

(b)

−5 0 50

5

10

15

20

25

(c)

Fig. 4. Parametric update evolution. The first two plots show the evolution of the depthestimate Z and the inlier probability π. We show themean±two standard deviations. Thethird plot shows the measurement histogram superimposed with the measurementposterior p(x|x1,…,xn). Both the mean and the outlier level have been correctly captured.

Fig. 5. Three types of pixel sensors. Thesefigures show themeasurementhistogramsand thesuperimposedmeasurement posterior p(x|x1,…,xm) for three types of pixel sensor. In (a) thepixel is a well-textured point on the object. In (b) the pixel corresponds to a completelyuntextured white point on the ground. In (c) the pixel corresponds to a point that will getoccluded within the next few frames. The estimated inlier ratio is shown in the three cases.The two pathological cases, (b) and (c), can be identified from their low inlier ratio.

the independence assumption of Eq. (2) because previous measure-ments will now dictate the search region for new measurements. Inspite of this the approximationworkswell in practice. In caseswhen thetrue depth falls outside this searchwindow of the epipolar line the seedwill be producing erroneous depthmeasurements.We rely on the inlierratio estimation to detect that themeasurements coming from this seedare outliers. The seed will subsequently be discarded as outlined in thenext section. In the experiments shown in this paperw is set to 3 pixelsfor our 2 million pixel camera. In the case when no local maximum isdetected, we penalize the seed by setting bn+1:=bn+1. This has thesame effect as observing a depth measurement which was known withcertainty to be an outlier.

Algorithm 1. The video based MVS algorithm.

M := themaximumnumber of seeds that canbemaintained inmemory.S := the current number of seeds in the system.For each new frame I

1. If SbM generate M−S new seeds at I .2. For each seed

(a) Project optic ray of seed on I .(b) Detect largest local maximum xn+1 of NCC score within

search window (Section 4.2).(c) Update posterior parameters with new depth measure-

ment xn+1(Section 3.2).

0 0.2 0.4 0.6 0.8 10

10

20

30

40

50

60

70

80

90

100P

reci

sion

(%

)

Error (mm)

this paper (600 frames)pollefeys08 (600 frames)this paper (30 frames)

(a)

0 0.2 0.4 0.6 0.8 10

10

20

30

40

50

60

70

80

90

100

Com

plet

enes

s (%

)

Error (mm)

this paper (600 frames)pollefeys08 (600 frames)this paper (30 frames)

(b)

Fig. 7. Accuracy and completeness curves for ground truth experiment. Panel (a) showsaccuracy results for our algorithm running in 30 and 600 frames of a video sequence of ahouse. The graph shows for a given distance d, howmuch of the reconstructedmodel fallswithin d of the ground truth. Panel (b) measures completeness. I.e. for a distance d howmuch of the ground truth falls within d of the reconstructed model. Our results are moreaccurate but somewhat less complete. This is because our method performs noregularization and returns an unmeshed point-cloud.

Fig. 6. Comparison against ground-truth. Our algorithmwas comparedwith Ref. [22] on a 600 frame video sequence of a toy house. (a) Ground truthmodel of the house. (b) The result ofRef. [22]. (c)Our result on theentire 600 frames. (d) Result of ourmethod running on every 20 frames (total of 30 input images). Our results appearmoredetailed compared to Ref. [22] butespecially in the case of the30 frame result, less complete. This is due to the lackof any spatial regularization inourmethodaswell as the fact thatRef. [22]produces ameshwhile our resultsare 3d point-clouds. Full completeness-precision curves for these results can be found in Fig. 7.

439G. Vogiatzis, C. Hernández / Image and Vision Computing 29 (2011) 434–441

3. Remove all seeds with inlier ratio less than ηoutlier.4. Convert into 3d points (and remove from seed list) all seeds

with inlier ratio higher than ηinlier and σnb .

4.3. Pruning of seeds

After the seed evolution step described in the previous sectionthere are three possible outcomes:

• The seed has converged to a good estimate and therefore it is removedfrom the seed list and a 3d point is generated at the current posteriormean μn.

• The seed has failed to converge due to too many outliers present.The seed is then removed from the list.

• The seed has not been left to converge long enough and therefore itsurvives into the next evolution step.

To decide on the appropriate outcome we use the variance of thedepthposteriorσn

2 and the estimated inlier probabilityπ.We employ thefollowing criteria:

1. If according to our current posterior distribution q(Z,π|an,bn,μn,σn2)

the inlier ratio π is less than ηoutlier with a probability of 99% then wecan conclude that thedepth estimation has failed. This is typically thecase when the seed is initialized on an image patch that was out offocus, or there was not enough texture to match in subsequentimages (Fig. 5b,c).

2. If the mean inlier ratio of our posterior is more than ηinlier and thedepth variance σn is less than then we assume that the depthestimation has succeeded (Fig. 5a).

3. In all other cases we let the seed evolve further.

Throughout all our experiments the threshold parameterswere keptfixed at ηoutlier=0.05, ηinlier=0.1. The variance threshold was set at1/10,000th of the bounding volume size Zmax−Zmin. The generated 3dpoints are collected into an octree structure that is graphically renderedwith z-buffer shading in real-time. Algorithm 1 provides a summary ofour method.

5. Evaluation

Here we present the results of two evaluations of our methodagainst ground truth data. In the first experiment we compare thehistogram voting approach of Ref. [13] with our probabilisticformulation. We focus on depth estimation performance, isolatingeffects such as surface regularization or meshing. To that end, wegenerated depth estimates for 1.5 million pixels randomly selectedfrom the 312 images of the ‘fullTemple’ sequence in the Middleburyevaluation [23]. For each pixel, we estimated its depth using ourprobabilistic formulation as well as the histogram voting approach.We then ran the standard completeness/precision tests on the twopoint-clouds. The results are summarized in Table 1. The probabilistic

formulation is outperforming the histogram approach across allcompleteness levels. This confirms that our model provides betterdepth estimation for the same data while offering the benefits of aprobabilistic approach.

The second experiment involves comparing against [22], which isone of the few MVS methods that offer real-time performance. Thesubject is a small toy house which we reconstructed to a very highaccuracy using a sequence of 36, 8-megapixel images and the publicly

Fig. 8. Various 3d models acquired with our system. Top row: Left: raw point cloud. Right: a 3d mesh extracted using a graph-cut method similar to Ref. [15]. Bottom row: Left a welltextured elephant figurine. Right: a textureless toy car. The model is incomplete but contains few spurious points.

440 G. Vogiatzis, C. Hernández / Image and Vision Computing 29 (2011) 434–441

available PMVS [8] software. This reconstruction is treated as groundtruth for the purposeof this experiment.We then captured a 600 framevideo sequence of the sameobject and ran ourmethodon the full videosequence as well a sub-sampled sequence of only 30 frames (skipping19 out of every 20 frames) to evaluate how ourmethod degrades withless data. We also asked the authors of Ref. [22] to run their algorithmon the same video sequence. The results are shown in Figs. 6 and 7. Insummary, compared to Ref. [22] our results are more precise (89.6% ofour reconstruction falls within 5 mm of the ground truth compared to77.0% for Ref. [22]). However, because of our lack of regularization andthe fact that Ref. [22] is providing a 3d surface our results suffer incompleteness (We cover 76.4% of the ground truth surface comparedto 82.7% for Ref. [22] for a distance threshold of 0.5 mm). From theexperiment on the subsampled sequence we note that our methoddegrades gracefully with less data. The reconstructed points we returnare still accurate, however the algorithmmanages to convert less seedsinto 3d points, which leads to lower completeness figures.

Finally, Fig. 8 shows several challenging objects that were recon-structed by a user operating our system. The time required (includinguser time and computation time) was between 1 and 2 min per model.In the supplementary material we provide additional videos of oursystem in action.

6. Conclusion

This paper presented a video MVS method based on independentper-pixel depth estimation. We look for local maxima of correlationscore along the epipolar line and fuse these candidate 3d locationswithin a probabilistic framework. Our implementation of this methodcan process 2-megapixel video at 60 Hz, producing accurate reconstruc-tionsof small objectswhile providing anonline feedback to theuser. Thevalidity of our approach was evaluated against ground-truth and wasfound to produce accurate reconstructions, degrading gracefully as thequantity of input data decreases.

One of the aims of this paper was to demonstrate the usability ofvideo-based real-time MVS systems that provide an online feedbackthrough intermediate results. To that end we showed how our methodcan be used to obtain 3d models of a variety of objects within a fewseconds. Another aimwas to evaluate howwell does video data resolve

ambiguities inMVSwithout any type of regularization.Our results showimprovement in terms of accuracy compared to regularizedmethods, inexchange for lower completeness.

We believe that video-based MVS systems have great potential forreconstructing large-scale models when acquisition time is at apremium. This is because they provide a denser coverage of the objectthan still photographs while the online feedback helps avoid costlyreturn visits to the scene. In future work we intend to verify this bydeploying our method outdoors and applying it to the reconstruction oflarge scale scenes.

Appendix A. Supplementary data

Supplementarydata to this article canbe foundonline at doi:10.1016/j.imavis.2011.01.006.

References

[1] S. Agarwal, N. Snavely, I. Simon, S.M. Seitz, R. Szeliski, Building rome in a day, Proc.12th Intl. Conf. on Computer Vision (ICCV), 2009.

[2] R. Bhotika, D. Fleet, K. Kutulakos, A probabilistic theory of occupancy andemptiness, Proc. 7th Europ. Conf. on Computer Vision (ECCV), 2002, pp. 112–130.

[3] D. Bradley, T. Boubekeur, W. Heidrich, Accurate multi-view reconstruction usingrobust binocular stereo and surface meshing, Proc. IEEE Conf. on Computer Visionand Pattern Recognition (CVPR), 2008.

[4] A. Broadhurst, T. Drummond, R. Cipolla, A probabilistic framework for spacecarving, Proc.8th Intl. Conf. on Computer Vision (ICCV), volume 1, 2001,pp. 338–393.

[5] N. Campbell, G. Vogiatzis, C. Hernández, R. Cipolla, Using multiple hypotheses toimprove depth-maps for multi-view stereo, Proc.10th Europ. Conf. on ComputerVision (ECCV), 2008.

[6] N. Cornelis, B. Leibe, K. Cornelis, L. Gool, 3d urban scene modeling integratingrecognition and reconstruction, Intl. J. Comput. Vis. 2–3 (78) (July 2008) 121–141.

[7] A. Davison, Real-time simultaneous localisation and mapping with single camera,Proc. 9th Intl. Conf. on Computer Vision (ICCV), 2003.

[8] Y. Furukawa, J. Ponce, Accurate, dense, and robust multi-view stereopsis, Proc.IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2007.

[9] P. Gargallo, P. Sturm, Bayesian 3d modeling from images using multiple depthmaps, Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),volume 2, Jun. 2005, pp. 885–891.

[10] M. Goesele, B. Curless, S. Seitz, Multi-view stereo revisited, Proc. IEEE Conf. onComputer Vision and Pattern Recognition (CVPR), volume 2, 2006,pp. 2402–2409.

[11] M. Goesele, N. Snavely, B. Curless, H. Hoppe, S.M. Seitz, Multi-view stereo forcommunity photo collections, Proceedings of ICCV 2007, October 2007.

441G. Vogiatzis, C. Hernández / Image and Vision Computing 29 (2011) 434–441

[12] M. Habbecke, L. Kobbelt, A surface-growing approach to multi-view stereoreconstruction, Proc. IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2007.

[13] C. Hernández, F. Schmitt, Silhouette and stereo fusion for 3d object modeling,Comput. Vis. Image Underst. 96 (3) (December 2004) 367–392.

[14] C. Hernández, G. Vogiatzis, R. Cipolla, Probabilistic visibility for multi-view stereo,Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2007.

[15] A. Hornung, L. Kobbelt, Robust reconstruction of watertight 3d models from non-uniformly sampled point clouds without normal information, SGP'06, 2006, pp. 41–50.

[16] E. Jaynes, Probability Theory, The Logic of Science, Cambridge University Press,2003.

[17] G. Kleinand, D. Murray, Improving the agility of keyframe-based slam, Proc.10thEurop. Conf. on Computer Vision (ECCV), 2008.

[18] K. Kolev, D. Cremers, Integration of multiview stereo and silhouettes via convexfunctionals on convex domains, Proc. 10th Europ. Conf. on Computer Vision(ECCV), 2008.

[19] K.N. Kutulakos, S.M. Seitz, A theory of shape by space carving, Intl. J. Comput. Vis.38 (3) (2000) 199–218.

[20] M. Lhuillier, L. Quan, A quasi-dense approach to surface reconstruction fromuncalibrated images, IEEE Trans. Pattern Anal.Mach. Intell. 27 (3) (2005) 418–433.

[21] X. Li, C. Wu, C. Zach, S. Lazebnik, J.-M. Frahm, Modeling and recognition oflandmark image collections using iconic scene graphs, ECCV'08: Proceedings ofthe 10th European Conference on Computer Vision, 2008, pp. 427–440.

[22] M. Pollefeys, et al., Detailed real-time urban 3d reconstruction from video, Intl. J.Comput. Vis. 78 (2–3) (2008) 143–167.

[23] S. Seitz, B. Curless, J. Diebel, D. Scharstein, R. Szeliski, A comparison and evaluationof multi-view stereo reconstruction algorithms, Proc. IEEE Conf. on ComputerVision and Pattern Recognition (CVPR), volume 1, 2006, pp. 519–528.

[24] N. Snavely, S.M. Seitz, R. Szeliski, Photo tourism: exploring image collections in 3d,ACM Transactions on Graphics (Proceedings of SIGGRAPH 2006), 2006.

[25] C. Strecha, R. Fransens, L.V. Gool, Wide-baseline stereo from multiple views: aprobabilistic account, Computer Vision and Pattern Recognition, IEEE ComputerSociety Conference on, 1, 2004, pp. 552–559.

[26] C. Strecha, W. von Hansen, L.V. Gool, P. Fua, U. Thoennessen, On benchmarkingcamera calibration and multi-view stereo for high resolution imagery, Proc. IEEEConf. on Computer Vision and Pattern Recognition (CVPR), 2008.

[27] G. Vogiatzis, C. Hernández, P. Torr, R. Cipolla, Multi-view stereo via volumetricgraph-cuts and occlusion robust photo-consistency, IEEE Trans. Pattern Anal.Mach. Intell. 29 (12) (December 2007) 2241–2246.

[28] Z. Zhang, A flexible new technique for camera calibration, IEEE Trans. PatternAnal. Mach. Intell. 22 (11) (2000) 1330–1334.

[29] A. Richard, Newcombe, A.J. Davison, Live Dense Reconstruction with a SingleMoving Camera, Proc. IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2010.