Efficient Temporal Consistency for Streaming Video Scene Analysis

Ondrej Miksik Daniel Munoz J. Andrew Bagnell Martial Hebert

Abstract— We address the problem of image-based sceneanalysis from streaming video, as would be seen from amoving platform, in order to efficiently generate spatially andtemporally consistent predictions of semantic categories overtime. In contrast to previous techniques which typically addressthis problem in batch and/or through graphical models, wedemonstrate that by learning visual similarities between pixelsacross frames, a simple filtering algorithm is able to achievehigh performance predictions in an efficient and online/causalmanner. Our technique is a meta-algorithm that can be ef-ficiently wrapped around any scene analysis technique thatproduces a per-pixel semantic label distribution. We validateour approach over three different scene analysis techniques onthree different datasets that contain different semantic objectcategories. Our experiments demonstrate our approach is veryefficient in practice and substantially improves the quality ofpredictions over time.

I. INTRODUCTION

A semantic understanding of the environment from im-ages, as illustrated in Fig. 1, plays an important role fora variety of robotic tasks, such as autonomous navigation.Many works have investigated this problem using differenttechniques such as graphical models [1], [2], deep learning[3], [4], exemplar-based [5], [6] and iterative decoding tech-niques [7], [8]. Typically, they address the problem of sceneanalysis from a single image. Extending these techniquesto temporal sequences of images, as would be seen from amobile platform, is very challenging. Simply applying thescene analysis algorithm to each image independently is notsufficient because it does not properly enforce consistentlabels over time. In practice, the temporally inconsistentpredictions result in “flickering” classifications. This effectis not just due to the motion of the camera through the3D scene: we often observe this behavior even on imagesof a static scene due to subtle illumination changes. Theseinconsistencies in predictions can have a major impact onrobotic tasks in practice, e.g., predicted obstacles may sud-denly appear in front of the robot in one frame and thenvanish in the next. The situation is further complicated by theneed for online, causal algorithms in robotics applications,in which the system does not have access to future frames,unlike video interpretation systems which can proceed inbatch mode by using all the available frames [9], [10].

O. M. is with the Center for Machine Perception, Czech TechnicalUniversity in Prague. ondra.miksik@gmail.com

D. M., J. A. B. and M. H. are with The Robotics Institute, CarnegieMellon University. {dmunoz, dbagnell, hebert}@ri.cmu.edu

This work was conducted through collaborative participation in theRobotics Consortium sponsored by the U.S. Army Research Laboratoryunder the Collaborative Technology Alliance Program, Cooperative Agree-ment W911NF-10-2-0016.

road

sidewalk

treecar

building

tree

tree

sky

building

sidewalkpedestrian

Fig. 1: Predicted semantic categories from our approach.

In this work, inspired by standard linear causal filters,we consider the most natural technique for maintainingtemporally consistent predictions: a temporal filter based onexponential smoothing over past predictions. Our approachis a meta-algorithm in the sense that it is agnostic to thespecific way in which predictions are generated, so that iscan be used with any per-frame scene analysis technique.Our only requirement is that the per-frame scene analysistechnique predicts a per-pixel label probability distributioninstead of a single label.

Our algorithm is illustrated in Fig. 2: At the currentframe It, each pixel i is associated with a label probabilitydistribution y

(t)i , which is produced by a scene analysis

algorithm. Our goal is to ensure that the final label dis-tribution that we return for pixel i is consistent with thetemporal prediction y

(t−1)j from its corresponding pixel j in

the previous frame It−1, which we do not know. Instead, weuse optical flow [11] to estimate a neighborhood of candidatecorrespondences in the previous frame. Giving all neighborsequal weight and defining the smoothed prediction basedon the average of the neighborhood’s predictions is unwisebecause the neighborhood could include pixels of completelydifferent objects. Therefore, between pixels i ∈ It andj ∈ It−1, we propose to learn a data-driven, visual similarityfunction to assign a high weight wij between pixels thatcorrespond to each other (and low weight for those that donot) in order to select correct correspondences and accuratelypropagate predictions over time. In summary, our algorithmconsists of two main components:

1) Using optical flow to initialize a local neighborhood.

i

jwij

(a) (b) (c) (d) (e)

Fig. 2: Overview of our approach. The distribution of labels at a pixel in frame It (b) is combined with a weighted averageof the distributions of labels in a neighborhood of pixels in the previous frame It−1 (a). This neighborhood is estimatedusing optical flow techniques (c). We propagate predictions across time by learning a similarity function between pixelsi ∈ It (e) and j ∈ It−1 (d). This similarity assigns high values wij between visually similar pixels (green cells) and lowvalues over visually different pixels (red cells).

2) Selectively combining the past predictions y(t−1) inthe neighborhood with the current prediction y

(t)i using

learned weights.

In Section III, we present an efficient algorithm to learnthis similarity function, and then in Section IV, we showhow to transfer predictions to obtain a temporally smoothedprediction y

(t)i . We then validate our temporal smoothing

over three distinct semantic labeling algorithms on threedifferent datasets in Section V. Our experiments confirm thatthis natural approach yields drastically improved predictionsover time, with the additional important benefits of beingvery efficient and simple to implement.

II. RELATED WORK

One popular way to incorporate temporal information isto compute Structure from Motion (SfM) between consec-utive frames in order to compute geometric/motion-basedfeatures [12]–[14]. One drawback of this approach is thataccurate SfM computation may be slow and/or may requirea large buffer of previous frames to process. Alternatively,and/or in addition, a large graphical model can be definedamong multiple frames, where edges between frames propa-gate predictions over time [15]–[19]. Performing bounded,approximate inference over such large models remains achallenging problem. Furthermore, in order to efficientlycompute approximate solutions, only an estimate of the MAPdistribution is returned, i.e., there is no uncertainty in thelabeling or marginal distributions. To further improve effi-ciency in practice, techniques make further approximationsat the cost of loss of guarantees on the solution quality.By returning label probabilities, our approach may be moreuseful as input for subsequent robotic algorithms, such asreasoning about multiple interpretation hypotheses. Anothertechnique for maintaining temporal consistency, which issimilar to defining a spatio-temporal graphical model, is toanalyze volumes from a spatio-temporal segmentation [10].This batch approach is omniscient in the sense that it requiresprocessing the entire video sequence, which is typically notsuitable for most robotic applications.

III. LEARNING SIMILARITY

A. Metric Learning

In order to selectively propagate predictions from theprevious frame, we assign high weight between pixels thatare visually similar. One standard way to measure similaritywij between two pixels is through a radial basis functionkernel

wij = exp

(−d(fi, fj)

σ2

), (1)

where fi ∈ Rd is the vector of visual features of pixeli, σ = 0.4 controls the bandwidth of the kernel, andd : Rd × Rd → R+ is a distance function. Directly usingthe pixels’ RGB values for the feature representation is notdiscriminative enough to correctly match correspondences.We instead augment the color with local texture and localbinary pattern features, resulting in fi ∈ R141. Because of theincrease in dimensionality, the standard squared Euclideandistance

d(fi, fj) = (fi − fj)T (fi − fj) = Tr(∆ij), (2)

where ∆ij = (fi − fj)(fi − fj)T , is typically large between

most feature vectors, as illustrated in Fig. 3-b. Alternatively,we can learn a distance that has the desirable properties forour application. That is, for a pair of pixels (i, j) from the setof pairs of pixels that truly correspond to each other, Ep, wewant d(fi, fj) to be small, and for a pair of pixels (i, k) fromthe set that do not correspond, En, we want d(fi, fk) to belarge. Learning a distance, or metric, also remains an activearea of research [20]–[23], and a variety of techniques couldbe used to learn d. As we are concerned with efficiency inthe predictions, we use a simple Mahalanobis distance

dM(fi, fj) = (fi − fj)TM(fi − fj) = Tr(MT∆ij), (3)

and propose an efficient method to learn the parameters Moffline from training data.

We follow the max-margin learning strategy and learn ametric such that the distances between pixels that do notcorrespond (i, k) ∈ En are larger by a margin than thedistances between pixels that do correspond (i, j) ∈ Ep. This

can be formulated as the convex, semidefinite program

minM,ξ,ζ

‖M‖2F + α∑(i,j)

ξij + β∑(i,k)

ζik

s.t. dM(fi, fj) ≤ 1 + ξij , ∀(i, j) ∈ EpdM(fi, fk) ≥ 2 + ζik, ∀(i, k) ∈ EnM ∈M, (4)

where M = {M|M � 0,M = MT } is the convex cone ofsymmetric positive-semidefinite matrices, and α and β penal-ize violating the margins. This program can be equivalentlyrewritten as the unconstrained, convex minimization

minM∈M

Tr(MTM) + α∑(i,j)

max(0,Tr(MT∆ij)− 1)

+ β∑(i,j)

max(0, 2− Tr(MT∆ik)), (5)

and can be efficiently optimized using the projected subgra-dient method [24]. We define Υij and Ψij to be subgradientsof the respective summands:

Υij =

{∆ij , Tr(MT4ij)− 1) > 00, otherwise

,

Ψij =

{−∆ik, 2− Tr(MT4ij)) > 00, otherwise.

(6)

The subgradient update, with small step-size ηt, is then

Mt+1 ← PM

Mt − ηt(Mt + α∑ij

Υij + β∑ij

Ψik)

,

(7)where PM projects to the closest matrix on the convexconeM. Since Ψij and Υij are symmetric by construction,the closest projection, with respect to the Frobenius norm,is computed by reconstructing the matrix with its negativeeigenvalues set to 0 [25]. To further improve run-timeefficiency, we also constrain M to be a diagonal matrix.

As illustrated in Fig. 3-c, our learned metric greatlyimproves at discriminating pixels by visual similarity thanusing Euclidean distance and is often small among pixelswith the same semantic category. Although the computeddistances may not be optimal across the entire image, weobserve correct behavior over a local area. Thus, we donot have to be globally correct and can use optical flow toinitialize the area in which to compute distances.

B. Obtaining Training Data

As we propagate predicted probabilities between frames,one immediate idea is to sample, from a labeled training set,pairs of pixels that belong to the same semantic categoryto create Ep and use pairs between the different categoriesto create En. The result will be a general metric betweencategories rather than correspondences between pixels and isa much harder problem. Furthermore, as we expect motionin the data sequence, objects will be observed under differentviewpoints, and we want our similarity metric to handle this.

We instead generate our pairs of training data using pixelcorrespondences across multiple frames and viewpoints.

++

(a)

(b)

(c)

Fig. 3: Comparing similarity metrics. (a) A scene with twopixels of interest selected. (b) The inverted heatmaps ofEuclidean distances of the respective pixel of interest to everyother pixel in the image (bright means small distance). (c)The inverted heatmaps from our learned Mahalanobis metric.

These correspondences can be easily obtained through avariety of different keypoint-based algorithms. Specifically,we use the publicly available SfM package, Bundler [26].Given a collection of images, Bundler will produce a 3-Dreconstruction of the scene (which we ignore) as well as thecorresponding pixels across multiple frames. As illustrated inFig. 4, we use pairs of pixels from the same correspondenceto construct Ep and use pairs that do not correspond to eachother to construct En when we learn our metric offline. Actualcorrespondences between 5 pairs of pixels across frames areshown in Fig. 5.

IV. TEMPORAL CONSISTENCY

With the ability to measure similarities between twopixels, we now describe how we combine predictions overtime.

A. Optical Flow

We are interested in general scene analysis without mak-ing strong assumptions on the movement/frame-rate of thecamera and/or the type of object motions in the scene.Furthermore, we require the procedure to be as efficient as

Fig. 4: Generating pairs of training data. The blue and thegreen pixels are correspondences. The positive examples arepairs of blue and green pixels. Negative examples are pairsbetween the blue and red pixels. The orange areas representthe same object under multiple viewpoints.

possible so it can potentially be used onboard of a mobilerobot. To generate initial hypothesis between frames, weuse the efficient dense Anisotropic Huber-L1 optical flowalgorithm from Werlberger et al. [11].

1) Data Warping: Given an image It at time t, insteadof finding each pixels neighborhood in the previous frameIt−1, we warp the previous frame into the current frameusing the estimated optical flow field. Let (xt−1, yt−1) bea pixel coordinate in It−1 and Vx and Vy be the estimatedoptical flow fields. The coordinates of each pixel with knownvelocity vector are projected from It−1 to It via[

xtyt

]=

[xt−1yt−1

]+

[Vx[xt−1, yt−1]Vy[xt−1, yt−1]

]. (8)

Although we use a state-of-the-art optical flow algorithm,it is not perfect and some velocity vectors may not matchexactly and/or some correspondences between frames mightbe missing. To help recover from missing flow information,we warp the image data and previous predictions yt−1with small patches, instead of one pixel, in the followingway. For each pixel (xt−1, yt−1) in frame It−1, we useVx[xt−1, yt−1] and Vy[xt−1, yt−1] to project all pixels ina small 5 × 5 pixel patch centered at (xt−1, yt−1) into awarped image It. For each warped pixel, we accumulateits RGB value and previous temporal prediction y

(t−1)i into

its projected coordinate. After all pixels from the previousframe have been warped, the RGB values and predictionsare appropriately averaged by the number of projections thatfell into each coordinate, resulting in a warped RGB imageIt and warped temporal predictions y(t)

2) Forward-Backward Error: The most typical situationswhen the optical flow estimation fails are occlusions orincorrect estimations. If such a situation occurs, a reasonablebehavior is to stop smoothing and to use only the currentmeasurement. Otherwise, we propagate pixels and predic-tions that might belong to different parts of an image.

A common method to detect optical flow failures is theforward-backward error. This technique has demonstratedto be effective for object [27] and point [28] tracking. Wedetect failures by ensuring that the optical flow forward intime from It−1 → It is consistent with the flow backwardsin time from It → It−1. Letting

−→Vx,−→Vy denote the forward

flow fields and←−Vx,←−Vy denote the backward flow fields, we

+ +

+++

+ +

++

+ +

+++

++++ +

+ +++

+ ++++

Fig. 5: Example correspondences. Same colored correspon-dences are used in Ep and mixed colored correspondencesare used in En.

consider pixel coordinates (xt, yt) in It to be invalid if∥∥∥∥∥[ −→Vx[xt−1, yt−1]−→Vy[xt−1, yt−1]

]+

[ ←−Vx[xt, yt]←−Vy[xt, yt]

]∥∥∥∥∥2

≥ κ, (9)

where κ = 13 is related to the size of the averagingneighborhood, since this neighborhood is designed to allowsome inaccuracies. Invalid pixels do not contribute to theexponential smoothing algorithm described in the followingsubsection.

B. Temporal Smoothing

Now we have all the necessary components for recursivetemporal smoothing: a metric for measuring visual similaritywij between two pixels and a method to warp pixels betweenframes. At the recursive step of our procedure, we have thecurrent image It with its per-pixel probabilities yt fromthe scene analysis algorithm, and the warped image ofthe previous frame It with its warped temporal predictionsy(t). For each pixel i ∈ It and j ∈ It, we compute thepixel features (texture gradients, local binary patterns, color)fi, fj that were used to learn our metric. Using exponentialsmoothing, we can define a pixel’s new temporally smoothedpredictions using the update rule

y(t)i =

1

Zi

∑j∈N(t)

i

wijy(t)j + cy

(t)i , (10)

where N (t)i is a 5× 5 spatial neighborhood in It centered at

pixel i excluding invalid pixels as defined by Eq. 9, c = 0.25is our prior belief on the independent prediction from thescene analysis algorithm, and Zi =

∑j∈Ni

wij + c ensuresthat y

(t)i sums to one. The procedure then recurses.

V. EXPERIMENTAL ANALYSIS

We evaluate our approach over three sophisticated sceneanalysis algorithms over three different datasets. All resultswere obtained using the same smoothing parameters acrossthe different algorithms/datasets.

A. Algorithms and Datasets

1) CamVid: This dataset [12] consists of over 10 minutesof 30 Hz video captured in an urban environment duringdaylight and dusk containing 11 semantic categories. Thesequences are sparsely annotated in time at 1 Hz. For thisdataset, we use the hierarchical inference machine algorithm

TABLE I: F1 scores and overall accuracies on CamVid

05VD 16E5Class Independent Smoothed Independent Smoothedsky .303 .682 .237 .639tree .352 .563 .336 .518road .197 .546 .150 .529sidewalk .277 .512 .188 .357building .165 .232 .275 .395car .127 .456 .304 .597pole .201 .386 .252 .285person .138 .218 .324 .193bicycle .323 .165 .348 .043fence .325 .712 .335 .668sign .367 .029 .378 .039Accuracy .261 .591 .286 .530

from [8]. Without using any temporal information, we obtainstate-of-the-art overall pixel and average per-class accuraciesof 84.2% and 59.5%, respectively, and is comparable [13] orexceeds [12] other techniques which use temporal features.For evaluating temporal consistency, we trained two separatemodels. The first is trained on the standard training fold from[12], and we evaluate the test sequence 05VD. The secondmodel is evaluated on the 16E5 sequence and trained on theremaining images not from this sequence.

2) NYUScenes: This dataset consists of 74 annotatedframes and is captured by a person walking on an urbanstreet with a hand-held camera. In addition to the objectsmoving, the video has a lot of camera motion due to theperson walking. For this dataset, we used the outputs fromthe deep learning architecture [4], provided by the authors.This model was trained on the SIFTFlow dataset [6], whichcontains 33 semantic categories.

3) MPI-VehicleScenes: This dataset [29] consists of156 annotated frames captured from a dashboard-mountedcamera while driving in a city and consists of only 5 semanticclasses. For this dataset, we used the outputs from the per-pixel, boosted classifier [15], provided by the authors, whichis based on the JointBoost algorithm [30].

B. Efficiency

There are two main components of our approach: 1)forward and backward, dense optical flow computation and2) temporal smoothing (which includes warping, pixel fea-tures, and the weighted averaging). Between two frames,the average timing of the first component is 0.02 s using aGPU implementation on a GeForce GTX 590 graphics card,and the average timing of the second component is 0.75 susing a CPU implementation on an Intel X5670 processor.We observe that dense optical flow computation time can bebrought down from the order of seconds with using a CPU to10s of milliseconds with using a GPU. As our approach relieson many, simple numeric computations, we would expect asimilar efficiency improvement with a GPU implementation.

C. Analysis

Videos comparing per-frame and the temporally smoothedclassifications for each sequence are available at [31]–[34],and we show qualitative examples for each sequence in Figs.

TABLE II: F1 scores and overall accuracies on NYU

Class Independent Smoothedbuilding .231 .547car .207 .630door .046 .000person .094 .080pole .169 .139road .152 .575sidewalk .373 .274sign .127 .000sky .002 .019tree .353 .630window .102 .101Accuracy .209 .500

TABLE III: F1 scores and overall accuracies on MPI

Class Independent Smoothedbackground .422 .730road .497 .340lane-marking .763 .330vehicle .076 .280sky .209 .560Accuracy .405 .537

6, 7, and 8. The videos demonstrate the substantial benefitof using temporal smoothing, especially on the CamVidsequences which are captured at a much higher frame rate.

It is important to remember that our approach relies onthe output of the inner scene analysis algorithm and cannotfix misclassifications due to the biases of the base algorithm.Hence, for quantitative evaluation we first only consider pix-els from which the prediction obtained by the scene analysisalgorithm differs with our temporal smoothing. We computea confusion matrix over these differing pixels and report theper-class F1 scores in Tables I, II, III, as well as the per-pixelaccuracy on the differing pixels; improvements greater than0.03 are bolded. This evaluation measures whether we areworse or better off with using temporal smoothing.

The behavior across datasets is consistent: categorieswhich occupy large areas of the image (e.g., sky, trees, cars,buildings) are significantly improved upon and predictionson some of the smaller objects (e.g., signs, people, lane-markings) are sometimes oversmoothed. There are variousreasons as to why performance may decrease. 1) Optical flowestimation on small objects may fail due to large displace-ments and/or excessive blurring, resulting in neighborhoodsthat are not adequately initialized. 2) As these objects occupya small number of pixels, over-smoothing from spatiallyadjacent objects will cause a large drop in performance.Similarly, it is challenging to accurately annotated theseintricate objects, and imperfections in the ground truth canfurther skew evaluation. 3) These small object categoriesare typically challenging to classify. When the predictedlabel distributions from the scene analysis algorithm are lessconfident, i.e., have high entropy, the resulting weightedcombination may be incorrect. Nonetheless, the overallimprovement in accuracy shows a clear benefit of usingtemporal smoothing rather than per-frame classification.

The comparisons of overall per-pixel accuracy for each

TABLE IV: Overall pixel accuracies (%)

Dataset Independent SmoothedCamVid-05VD 84.60 86.85CamVid-16E5 87.37 88.84NYUScenes 71.11 75.31

MPI-VehicleScenes 93.27 93.76

sequence are shown in Table IV. Due to the sparseness of theCamVid annotations, the quantitative improvements are notas drastic as we would expect, however, there is a noticeablegain. We also observe a large quantitative improvement in theNYUScenes sequence, even in the presence of large cameramotion. The improvement in the MPI-VehicleScenesdataset is modest, however, this can be attributed to a smalllabel set of 5 categories (vs. 11 and 33 from the other two)which often have little confusion. Furthermore, we note thepredictions are qualitatively much smoother in appearance,even from using a per-pixel classifier.

VI. CONCLUSION

We propose an efficient meta-algorithm for the problem ofspatio-temporal consistent 2-D scene analysis from streamingvideo. Our approach is based on recursive weighted filteringin a small neighborhood, where large displacements arecaptured by dense optical flow and we propose an efficient al-gorithm to learn image-based similarities between pixels. Aswe do not require information about future frames, our causalalgorithm can handle streaming images in a very efficientmanner. Furthermore, we demonstrate that our approach canbe wrapped around various structured prediction algorithmsto improve predictions without a difficult redefinition of aninference process.

ACKNOWLEDGMENTS

We thank A. Wendel for helping with the optical flowcomputation, C. Fabaret for providing the NYUScenesdataset and his classifications, C. Wojek for providing hisclassifications on the MPI-VechicleScenes dataset, andJ. Tighe for discussions about the CamVid dataset.

REFERENCES

[1] L. Ladicky, P. Sturgess, K. Alahari, C. Russell, and P. H. S. Torr,“What, where & how many? combining object detectors and crfs,” inECCV, 2010.

[2] D. Munoz, J. A. Bagnell, N. Vandapel, and M. Hebert, “Contextualclassification with functional max-margin markov networks,” in CVPR,2009.

[3] R. Socher, C. C.-Y. Lin, A. Y. Ng, and C. D. Manning, “Parsing naturalscenes and natural language with recursive neural networks,” in ICML,2011.

[4] C. Farabet, C. Couprie, L. Najman, and Y. LeCun, “Scene parsingwith multiscale feature learning, purity trees and optimal covers,” inICML, 2012.

[5] J. Tighe and S. Lazebnik, “Superparsing: Scalable nonparametricimage parsing with superpixels,” in ECCV, 2010.

[6] C. Liu, J. Yuen, and A. Torralba, “SIFT flow: Nonparametric sceneparsing via label transfer,” IEEE T-PAMI, vol. 33, no. 12, 2011.

[7] Z. Tu and X. Bai, “Auto-context and its application to high-level visiontasks and 3d brain image segmentation,” IEEE T-PAMI, vol. 32, no. 10,2010.

[8] D. Munoz, J. A. Bagnell, and M. Hebert, “Stacked hierarchicallabeling,” in ECCV, 2010.

[9] W. Brendel and S. Todorovic, “Learning spatiotemporal graphs ofhuman activities,” in ICCV, 2011.

[10] M. Grundmann, V. Kwatra, M. Han, and I. Essa, “Efficient hierarchicalgraph-based video segmentation,” in CVPR, 2010.

[11] M. Werlberger, W. Trobin, T. Pock, A. Wedel, D. Cremers, andH. Bischof, “Anisotropic huber-L1 optical flow,” in BMVC, 2009.

[12] G. J. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla, “Segmentationand recognition using structure from motion point clouds,” in ECCV,2008.

[13] P. Sturgess, K. Alahari, L. Ladicky, and P. H. S. Torr, “Combiningappearance and structure from motion features for road scene under-standing,” in BMVC, 2009.

[14] B. Micusik, J. Kosecka, and G. Singh, “Semantic parsing of streetscenes from video,” IJRR, vol. 31, no. 4, 2012.

[15] C. Wojek and B. Schiele, “A dynamic conditional random field modelfor joint labeling of object and scene classes,” in ECCV, 2008.

[16] A. Ess, T. Muller, H. Grabner, and L. Van Gool, “Segmentation-basedurban traffic scene understanding,” in BMVC, 2009.

[17] J. Xiao and L. Quan, “Multiple view semantic segmentation for streetview images,” in ICCV, 2009.

[18] R. de Nijs, J. S. Ramos, G. Roig, X. Boix, L. V. Gool, andK. Kuhnlenz., “On-line semantic perception using uncertainty,” inIROS, 2012.

[19] A. Y. C. Chen and J. J. Corso, “Temporally consistent multi-classvideo-object segmentation with the video graph-shifts algorithm,” inWACV, 2011.

[20] E. Xing, A. Y. Ng, M. Jordan, and S. Russell, “Distance metriclearning, with application to clustering with side-information,” inNIPS, 2003.

[21] J. Davis, B. Kulis, P. Jain, S. Sra, and I. Dhillon, “Information-theoreticmetric learning,” in ICML, 2007.

[22] J. Goldberger, S. Roweis, G. Hinton, and R. Salakhutdinov, “Neigh-bourhood components analysis,” in NIPS, 2004.

[23] R. Hadsell, S. Chopra, and Y. Lecun, “Dimensionality reduction bylearning an invariant mapping,” in CVPR, 2006.

[24] N. Ratliff, J. Bagnell, and M. Zinkevich, “Online subgradient methodsfor structured prediction,” in AISTATS, 2007.

[25] G. Golub and C. F. Van Loan, Matrix Computations. John HopkinsUniversity Press, 1996.

[26] N. Snavely, S. M. Seitz, and R. Szeliski, “Photo tourism: Exploringphoto collections in 3d,” ACM SIGGRAPH, 2006.

[27] Z. Kalal, K. Mikolajczyk, and J. Matas, “Forward-Backward Error :Automatic Detection of Tracking Failures,” in ICPR, 2010.

[28] N. Sundaram, T. Brox, and K. Krutzer, “Dense point trajectories bygpu-accelerated large displacement optical flow,” in ECCV, 2010.

[29] C. Wojek, S. Roth, K. Schindler, and B. Schiele, “Monocular 3d scenemodeling and inference: Understanding multi-object traffic scenes,” inECCV, 2010.

[30] A. Torralba, K. P. Murphy, and W. T. Freeman, “Sharing visual featuresfor multiclass and multiview object detection,” IEEE T-PAMI, vol. 29,no. 5, 2007.

[31] D. Munoz, “Temporal consistency on CamVid 05VD,”http://www.youtube.com/watch?v=q071f75nWNk.

[32] ——, “Temporal consistency on CamVid 16E5,”http://www.youtube.com/watch?v=iEgljSK8Oxc.

[33] ——, “Temporal consistency on NYU Scenes,”http://www.youtube.com/watch?v=4pVzNaUd2yw.

[34] ——, “Temporal consistency on MPI-StreetScenes,”http://www.youtube.com/watch?v=7MeZLVFxRC4.

(a) frame 00055 (b) frame 00056 (c) frame 00057 (d) frame 00058 (e) frame 00059

— sky — tree — road — sidewalk — building — car — column pole — pedestrian — bicycle — fence — sign symbol

Fig. 6: CamVid classifications. Top: per-frame. Bottom: temporally smoothed. Inconsistent predictions are highlighted inwhite boxes. The full videos are available at [31], [32].

(a) frame 00055 (b) frame 00056 (c) frame 00057 (d) frame 00058 (e) frame 00059— unknown — awning — balcony — building — car — door — person — road — sidewalk — sign— sky — streetlight — sun — tree — window

Fig. 7: NYUScenes classifications. Top: per-frame. Bottom: temporally smoothed. Inconsistent predictions are highlightedin white boxes. The full video is available at [33].

(a) frame 00265 (b) frame 00266 (c) frame 00267 (d) frame 00268 (e) frame 00269

— void — road — lane-marking — vehicle — sky

Fig. 8: MPI-VehicleScenes classifications. Top: per-frame. Bottom: temporally smoothed. The misclassificationsbetween objects are not as drastic due to the small set of labels, however, there is a lot of noise in the predictions that arecorrected. The full video is available at [34].