Dense Semantic Correspondence where Every Pixel is a Classifier

Hilton Bristow,1 Jack Valmadre1 and Simon Lucey2

1Queensland University of Technology, Australia2Carnegie Mellon University, USA

Abstract

Determining dense semantic correspondences acrossobjects and scenes is a difficult problem that underpinsmany higher-level computer vision algorithms. Unlikecanonical dense correspondence problems which con-sider images that are spatially or temporally adjacent,semantic correspondence is characterized by imagesthat share similar high-level structures whose exact ap-pearance and geometry may differ.

Motivated by object recognition literature and re-cent work on rapidly estimating linear classifiers, wetreat semantic correspondence as a constrained detec-tion problem, where an exemplar LDA classifier islearned for each pixel. LDA classifiers have two dis-tinct benefits: (i) they exhibit higher average precisionthan similarity metrics typically used in correspondenceproblems, and (ii) unlike exemplar SVM, can outputglobally interpretable posterior probabilities without cal-ibration, whilst also being significantly faster to train.

We pose the correspondence problem as a graphi-cal model, where the unary potentials are computed viaconvolution with the set of exemplar classifiers, andthe joint potentials enforce smoothly varying correspon-dence assignment.

1. Introduction

Unlike canonical dense correspondence problemswhich consider images that are spatially (stereo) ortemporally (optical flow) adjacent, semantic correspon-dence is characterized by images that stem from thesame visual class (e.g . elephants, lammergeiers, car-lined streets) whilst exhibiting individual appearanceand geometric properties.

For example, given two images of elephants (see Fig-ure 1), we would like to predict where each pixel on thefirst elephant corresponds to on the second. This isparticularly challenging because the space of elephantsexhibits significant intra-class appearance and geomet-ric variation. A related problem is that of pose es-timation [17, 24], which considers a smaller fixed setof landmarks stemming from a labelled dataset of a

Figure 1. Dense semantic correspondence estimates howpoints are related between images that stem from the samevisual class. Here, we wish to predict where each pixel onthe first elephant corresponds to on the second, whilst be-ing robust to appearance, pose and background variation.The points labelled are representative of the dense corre-spondence field estimated by our method.

known object class. From this dataset, one can learn(i) the geometric dependency between landmarks, and(ii) local detectors that discriminate the appearance ofeach landmark from the background. When presentedwith a new image, one can then estimate the landmarklocations by solving a graphical inference problem.

1

Liu et al .’s seminal work of SIFT Flow [12] estab-lished that a similar strategy could be applied for es-timating dense semantic correspondence between twoimages stemming from the same semantic class. Thereare three complicating factors however: (i) learning ge-ometric dependencies between landmarks is impossiblefrom only a single example, (ii) learning local detec-tors is problematic due to the lack of positive trainingsamples, and (iii) computational complexity is a ma-jor concern as we are treating each pixel coordinatewithin the image as a landmark. Liu et al . proposedto circumvent these problems by assuming the densegeometric dependencies in an image can be adequatelygoverned by a variational regularizer, and that accu-rate local detections between semantically similar im-ages can be attained through the L1 distance betweenSIFT descriptors. Since there is no learning required,this can be performed in a computationally tractablemanner.

In this paper, we explore the possibility of actu-ally learning a discriminative detector at every pixelcoordinate in an image. Motivated by object detec-tion literature, we learn a linear classifier per pixel inthe reference image and apply it in a sliding-windowmanner to the target image to produce a match like-lihood estimate. Learning a multitude of linear detec-tors such as exemplar support vector machines (SVMs)has typically had two issues: (i) each detector mustparse the negative set, often with hard-negative min-ing techniques, leading to long training times, whichmakes training a classifier for every pixel in an imageintractable, and (ii) since the scale of the outputs de-pends on the margin, the output confidences of twodifferent SVMs are not directly comparable.

We leverage recent work on learning detectorsquickly with linear discriminant analysis (LDA), bycollecting negative statistics across a large number ofimages in a pre-training phase. Learning a new exem-plar detector then involves a single matrix-vector mul-tiplication. Since LDA uses a generative model of theclass distributions, the posterior probabilities providea quantity that is comparable between detectors. Thisallows us to estimate both the likelihood of matchesfor each pixel individually, and also a global belief ofmatch quality.

2. Prior Art

Canonical correspondence problems such as stereoand optical flow typically rely on simple (dis-)similaritymetrics to describe the likelihood of two pixels match-ing. In the original work of Horn and Schunck [7], thiswas Euclidean distance on raw pixel intensities, whichmanifested a brightness constancy assumption.

Since then, significant literature has focused on de-termining robust metrics under increasingly adverseconditions - from non-rigid deformations and occlu-sions, to non-global intensity, constrast and colorimet-ric changes [1, 14, 19, 20]. Importantly, however, allof these works assume the images being observed stemfrom the same underlying scene.

SIFT Flow first introduced the idea of semantic cor-respondence across scenes [12]. While the method usesa simple L1 metric, the images are represented in denseSIFT space typically associated with sparse keypointmatching.1 This sacrifices some localization accuracyfor improved geometric invariance. We maintain, how-ever, that similarity metrics are insufficient for estimat-ing the likelihood of pixels matching between differentscenes. Instead, we advocate the use of classifiers, asper deformable face fitting and pose estimation litera-ture, except where a classifier is trained per pixel.

We leverage recent work on rapid estimation of LDAclassifiers [4, 22] to achieve this goal, though fast corre-lation filter estimation [5] is potentially equally appli-cable. The method we present is largely agnostic to theobjective used to learn the linear detectors (e.g . SVM,LDA, correlation filters), however LDA classifiers areattractive in producing globally interpretable outputsacross pixels, and requiring only a single matrix-vectormultiplication to train, which is critical to learning> 10, 000 classifiers per image.

A number of dense correspondence methods havemade use of discriminative pre-training [11, 18, 20],with the recent work of [10] being particularly rele-vant to our discussion. In this work, a classifier of theform f(Φ(x1)−Φ(x2)) is trained to predict a (binary)likelihood of two pixels matching. Intuitively, the clas-sifier learns the modes and scale of variation in the un-derlying feature space Φ that are important and thosethat are distractors. Training is fully supervised fromgroundtruth optical flow data.

Like SIFT Flow, [10] formulate the correspondenceobjective as a graphical model ([8, 9] respectively).This has the distinct advantage over variational meth-ods of permitting very large displacements and arbi-trarily complex data terms, at the expense of requiringsimple regularizers to keep inference tractable. Morerecently, a number of variational methods have usedsparse descriptor matching to anchor larger displace-ments [2, 23]. While both methods use robust SIFTdescriptors for keypoint matching, in a semantic cor-respondence setting the best match is infrequently thetrue correspondence, leading to poor initialization ofthe densification stage.

1Feature representation and similarity metric are intrinsicallyrelated, since f(φ(x1), φ(x2)) = g(x1,x2).

Mobahi et al . [15] present a parametric approach tothe general alignment problem, in a similar vein to con-gealing [3] or RASL [16]. Parametric methods generallyhave limited degrees of freedom, which enables strongcoarse localization, at the expense of fine-grained lo-calization accuracy.

Finally, the recent work of FlowWeb [25], is par-ticularly complementary to the ideas presented in thispaper, since they are agnostic to the method of appear-ance matching, and we are agnostic to the method ofregularization or global matching consistency. We usethe variational regularizer of SIFT Flow in our exper-iments, though the ideas we present are equally appli-cable to FlowWeb.

3. Dense Semantic Correspondence

Given two images, IA ∈ RMN and IB ∈ RPQ, anda discrete set of points x = [x1, x2, . . . , xMN ], dense se-mantic correspondence involves minimizing the inversefitting problem,

x∗ = argmin

x

MN∑

i=1

fi(xi) + λg(x) (1)

where f is the unary function that evaluates the like-lihood of a particular assignment for each xi based onthe image content, and g is a regularizer which enforcesconstraints on the joint configuration of the points. Insemantic correspondence, the unary function must bea good indicator of semantic similarity, and so mustbe robust to significant intra-class variation. In theframework we adopt, there are no constraints on itscomplexity or properties.

SIFT Flow [12] adopts a unary of the form,

fi(xi) = h(i,xi) = ||ΦA(i)− ΦB(xi)||1 (2)

where ΦA(xi) = Φ(xi; IA) is a feature representationof the image IA evaluated at the point xi. Φ subsumesthe exact detail of the feature extraction, which mayproduce vector-valued output. For our LDA classifiers,we extract features from a window of pixels around xi

and return a multi-channel patch centered at xi.In [10], the L1 norm on the difference between fea-

tures is replaced with a more general learned represen-tation,

h(i,xi) = H(ΦA(i)− ΦB(xi)) (3)

In both formulations, however, the unary function is astationary kernel. This implies a feature space capableof producing similar outputs for semantically similarinputs. Finding such a feature embedding is a difficult

task in general, and as a result significant object de-tection literature has focussed on learning classifiers todistinguish classes instead.

The use of classifiers has two distinct advantagesover stationary kernels for describing match likelihood.First, linear classifiers define half-spaces in which sam-ples are either classified as positive or negative. Thustwo points with dissimilar appearances can still be af-forded a high match likelihood. Second, the impor-tance of different dimensions in the feature space canbe learned from data.

In this paper, we advocate a unary function of theform,

fi(xi) = h(i,xi) = wA(i)TΦB(xi) (4)

where wA(i) is a linear classifier trained to predict cor-respondences to pixel i in IA, with ideal response,

wA(i)TΦB(xi) =

{

1 xi = x∗i

−1 otherwise(5)

This is traditional binary classification, where thepositive class contains the reference pixel, and its truecorrespondence in the target image, and the negativeclass contains all other pixels. Since the correspon-dence in the target image is not known a priori how-ever, we rely on the classifier wA(i) to generalize froma single training example: ΦA(i). This is known asexemplar-based classification [13].

The challenge is how to rapidly estimate thousandsof exemplar classifiers per image in reasonable time.The remainder of this section focuses on addressingthat challenge, and a number of interesting propertiesthat arise from our approach.

3.1. Learning Detectors Rapidly using StructuredCovariance Matrices

Linear classifiers have a rich history in computer vi-sion, not least because of their interpretation and effi-cient implementation as a convolution operation. Sup-port vector machines have proven particularly popular,due to their elegant theoretical interpretation, and im-pressive real-world performance, especially on objectand part detection tasks. A challenge for any objectdetection problem is how to treat the potentially in-finite negative set (comprising all incorrect correspon-dences in our case). Object detection methods usingsupport vector machines employ hard negative miningstrategies to search the negative set for difficult exam-ples, which can be represented parametrically in termsof the decision hyperplane. This feature is also theirlimitation for rapid estimation of many classifiers, sinceeach classifier must reparse the negative set looking for

hard examples – knowing one classifier does not helpin estimating another.2

Linear Discriminant Analysis (LDA), on the otherhand, summarizes the negative set into its mean andcovariance. The parameters w of the decision hyper-plane wT

x = c are learned by solving the system ofequations,

Sw = b (6)

where S is the joint covariance of both classes andb = xpos − xneg is the difference between class means.[4] made two key observations about LDA: (i) if thenumber of positive examples is small compared to thenumber of negative examples, the joint covariance S

can be approximated by the covariance of the negativedistribution alone, and reused for all positive classes,and (ii) gathering and storing the covariance can beperformed efficiently if the negative class is shift in-variant (i.e. a translated negative example is still anegative example).

This second fact implies stationarity of the nega-tive distribution, where the covariance of two pixels isdefined entirely by their relative displacement. Impor-tantly, both [4] and [6] showed that the performance oflinear detectors learned by exploiting the stationarityof the negative set is comparable to SVM training withhard negative mining.

The covariance S can be constructed from a relativedisplacement tensor, according to,

S(u,v,p),(i,j,q) = g[i− u, j − v, p, q] (7)

where i, j, u, v index spatial co-ordinates, and p, q in-dex channels. We call the maximum displacement ob-served abs(i−u), abs(j−v) the bandwidth of the tensor.Also note that stationarity only exists spatially – cross-channel correlations are stored explicitly. The storageof g thus scales quadratically in both bandwidth andchannels, though since the detectors we consider aretypically small-support, we can entertain feature repre-sentations with large numbers of channels (i.e. SIFT).

In order to compute S, and thus g, we gather statis-tics across a random subset of 50, 000 images fromImageNet. We precompute the covariance matrix ofthe chosen detector size (typically 5 × 5) and factorit with either a Cholesky decomposition, or its ex-plicit inverse. The constructed covariance S is a sam-ple covariance matrix estimated from missing data, sowhilst it is symmetric by construction, it is not strictly

2This is not strictly true. Warm starting an SVM from a pre-vious solution, especially in exemplar SVMs where only a sin-gle positive example changes, can induce a significant empiricalspeedup, however is unlikely to change the O() complexity of thealgorithm.

positive-semidefinite. To ensure positive-definiteness,we subtract the minimum of zero and the minimumeigenvalue from the diagonal, i.e. (S − min(0, λmin) ·I)−1.

For each pixel in the reference image, we compute,

wA(i) = S−1(xpos − xneg) (8)

which involves a single vector substraction and matrix-vector multiplication, where,

xpos = ΦA(xi) (9)

The likelihood estimate for the i-th reference pointacross the target image can be performed via convolu-tion over the discretize pixel grid,

fi(x) = wA(i) ∗ ΦB(x) (10)

Since storing the full unary is quadratic in the num-ber of image pixels (quartic in the dimension), we per-form coarse-to-fine or windowed search as per SIFTFlow [12].

3.2. Posterior Probability Estimation

Linear Discriminant Analysis (LDA) has the attrac-tive property of generatively modelling classes as Gaus-sian distributions with equal (co-)variance. This per-mits direct computation of posterior probabilities viaapplication of Bayes’ Rule:

p(Cpos|x) =p(x|Cpos) p(Cpos)∑

n∈{pos,neg}

p(x|Cn) p(Cn)(11)

where,

p(x|Cn) =1

(2π)|S|1

2

e−1

2(x−xn)

TS−1(x−xn) (12)

With some manipulation, the posterior of Equation(11) can be expressed as,

p(Cpos|x) =1

1 + e−y(13)

y = xTS−1(xpos − xneg) (14)

+ 12 x

TposS

−1xpos −

12 x

TnegS

−1xneg (15)

+ ln

(

p(Cpos)

p(Cneg)

)

(16)

Equation (13) takes the form of a logistic function,which maps the domain (−∞,∞) to the range (0 . . . 1).

The logistic function is typically used to convertSVM outputs to probabilistic estimates, however a“calibration” phase is required to learn the bias and

Figure 2. From left to right: (a) reference image with reference point labelled in red, and posterior estimates for (b) LDA and(c) L1 norm. We present a range of points, from distinctive to indistinctive or background. LDA and L1 norm have similarlikelihood quality for distinctive points, but LDA consistently offers better rejection of incorrect matches and backgroundcontent.

variance of each SVM in the ensemble so their outputsare comparable. With LDA, these parameters are de-rived directly from the underlying distributions.

Equation (14) is the canonical response to the LDAclassifier, Equation (15) represents the bias of the dis-tributions, and Equation (16) is the ratio of prior prob-abilities of the classes. This must be determined bycross-validation (once, not for each classifier), basedon the desired sensitivity to true versus false positives.

By completing the squares in Equation (15), we yieldthe final expression for computing the posterior prob-ability,

y = (x− 12 (xpos + xneg))

TS−1(xpos − xneg) + µ

= (x− 12 (xpos + xneg))

Twi + µ (17)

The implication of Equation (17) is that it is nomore expensive to compute probability estimates thanto just evaluate the classifier – the computation isstill dominated by the single matrix-vector product re-quired to learn the classifier.

Figure 2 illustrates a representative set of likelihoodestimates output by our method and SIFT Flow re-spectively. LDA typically has tighter responses aroundthe true correspondence, and better suppression of falsepositives, especially on background content that has noclear correspondence.

4. Evaluation

In order to evaluate the efficacy of our method, wefirst wanted to understand how well human annota-tors perform at semantic labelling tasks. Since we areprimarily interested in estimating correspondences forreconstruction-type objectives, we gathered 20 pairsof images from visual object categories which exhibitanatomical correspondence, including an assortmentof animals, trucks, faces and people. Given a set ofsparsely selected keypoints in the first image of eachpair, 8 human annotators were tasked with labellingthe corresponding points in the second image. A rep-resentative subset of the data is shown in Figure 3.

Figure 3. A representative subset of the groundtruth dataset. From top to bottom: (a) the source images, (b) the targetimages, and (c) the distribution of points selected by the human annotators on the target images. The structure of theobject is often clearly discernible from the annotations alone.

A similar experiment was performed in [12], howeverthey focussed on correspondences across scenes, whichoften have no clear correspondence, even for humanannotators. In contrast, the agreement on our datasetis high, with a natural increase in uncertainty fromcorner features, to edges and textureless regions.

In recognizing that not all features are equally dis-tinctive, we measure distance from estimated points xi

to the groundtruth using Mahalanobis distance,

di(xi) =

√

(xi − µi)TS−1i (xi − µi) (18)

where µi and Si are the 2D mean and covariance of thegroundtruth labellings across annotators. [21] motivatea similar procedure for human pose estimation. Thismetric has two advantages over Euclidean distance: (i)it takes into account spatial and directional uncertainty(e.g . correspondences are afforded some slack along anedge, but not perpendicular to it), and (ii) it is reso-lution independent, since distance is measured in stan-dard deviations.

Our dataset and metric therefore sets a higher stan-dard for what is considered a good correspondence,both empirically and qualitatively (since readers canaccurately discriminate good from poor results). Allresults presented in the following section are measuredunder this metric.

4.1. Experiments

In all of our experiments we resize the source (A)and target (B) image so max(M,N) = 150, preservingthe aspect ratio, and extract densely sampled SIFTfeatures.

The stationary distribution (mean and covariance)of SIFT features is estimated from 50, 000 randomlysampled images from ImageNet. Classifiers with spa-tial support 1×1, 3×3, 5×5, 7×7 and 9×9 were eval-uated. The different sizes tradeoff speed, localizationaccuracy and generalization. We found 5× 5 classifiersprovided a good balance between these tradeoffs, andthe results throughout our paper use this support.

While the LDA likelihoods are more computation-ally demanding to compute than L1-norm likelihoods,the construction and application of the classifiers canbe accelerated with BLAS. Estimating 10, 000 5 × 5classifiers and applying them in a sliding window fash-ion to a 80×125 SIFT image (with 128 channels) takesapproximately 6 seconds.

We apply our LDA-based correspondence methodin the same graphical model framework as SIFT Flow.We use a coarse-to-fine scheme to handle inference overlarger images, and grid searched the hyperparametersfor both LDA and L1 based unary functions. Resultsare shown in Figure 4.

We display the cumulative density for increasingnumber of standard deviations from groundtruth (i.e.fraction of points falling within an increasing radius

from groundtruth). As a baseline, we simply setxi = i,3 which acts as a proxy to the global alignmentbias of the dataset (small flow assumption). In addi-tion to SIFT Flow, we also compare our method to aleading optical flow method, Deep Flow [23].

We truncate the CDF due to the long tails for allmethods compared. This is an artefact of the non-global regularization schemes, which allow some pointsto be arbitrarily far from groundtruth without affectingothers. Finally, in Figure 5 we illustrate a number ofexemplar correspondences to show the visual quality ofmatches produced by our method.

5. Conclusion

In this paper we motivated the application of densesemantic correspondence for a range of computer vi-sion problems which currently rely on synthetic dataor specialized imaging devices. In contrast to existingcorrespondence methods, which typically use similar-ity kernels, we proposed using exemplar classifiers fordescribing the likelihood of two points matching. Weshowed that LDA classifiers exhibit 3 desirable proper-ties: (i) higher average precision than simple measuresof image similarity such as the L1 norm, (ii) signifi-cantly faster training than exemplar SVMs, and (iii)estimates of match confidence that are directly compa-rable across pixels.

We presented a small semantic correspondencedataset and metric in a bid to measure the performanceof different methods in a quantifiable manner, andshowed that under this metric our classifier-based ap-proach offered improvements over the L1 norm, withinthe same SIFT Flow optimization framework. Thequalitative results illustrate our method’s ability to es-timate high-quality dense semantic correspondences.

References

[1] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. Highaccuracy optical flow estimation based on a theory for warp-ing. European Conference on Computer Vision (ECCV),2004. 2

[2] T. Brox, J. Malik, and C. Bregler. Large displacement opti-cal flow. International Conference of Computer Vision andPattern Recognition (CVPR), 2009. 2

[3] M. Cox, S. Sridharan, and S. Lucey. Least-squares congeal-ing for large numbers of images. International Conferenceon Computer Vision (ICCV), 2009. 3

[4] B. Hariharan, J. Malik, and D. Ramanan. Discriminativedecorrelation for clustering and classification. EuropeanConference on Computer Vision (ECCV), 2012. 2, 4

[5] J. a. F. Henriques, R. Caseiro, P. Martins, and J. Batista.High-Speed Tracking with Kernelized Correlation Filters.Pattern Analysis and Machine Intelligence (PAMI), 2014.2

3For images of different sizes, we set xi = W(i) where W isa function that maps the span of IA to IB .

[6] J. F. Henriques, J. Carreira, R. Caseiro, and J. Batista. Be-yond Hard Negative Mining: Efficient Detector Learning viaBlock-Circulant Decomposition. International Conferenceon Computer Vision (ICCV), 2013. 4

[7] B. K. Horn and B. G. Schunck. Determining optical flow.Artificial Intelligence, 1981. 2

[8] V. Kolmogorov. Convergent tree-reweighted message pass-ing for energy minimization. Pattern Analysis and MachineIntelligence (PAMI), 2006. 2

[9] P. Krahenbuhl and V. Koltun. Efficient inference in fullyconnected crfs with gaussian edge potentials. In Advancesin Neural Information Processing Systems (NIPS), pages1–9, 2011. 2

[10] L. Ladicky, C. Hane, and M. Pollefeys. Learning the Match-ing Function. arXiv preprint, 2015. 2, 3

[11] Y. Li and D. Huttenlocher. Learning for optical flow usingstochastic optimization. European Conference on ComputerVision (ECCV), 2008. 2

[12] C. Liu, J. Yuen, and A. Torralba. Nonparametric sceneparsing via label transfer. Pattern Analysis and MachineIntelligence (PAMI), 2011. 2, 3, 4, 6, 8

[13] T. Malisiewicz, A. Gupta, and A. a. Efros. Ensemble ofexemplar-svms for object detection and beyond. Interna-tional Conference on Computer Vision (ICCV), 2011. 3

[14] Y. Mileva, A. Bruhn, and J. Weickert. Illumination-RobustVariational Optical Flow with Photometric Invariants. Pat-tern Recognition, 2007. 2

[15] H. Mobahi and W. T. Freeman. A Compositional Model forLow-Dimensional Image Set Representation. InternationalConference of Computer Vision and Pattern Recognition(CVPR), 2014. 3

[16] Y. Peng, A. Ganesh, J. Wright, W. Xu, and Y. Ma. RASL:robust alignment by sparse and low-rank decomposition forlinearly correlated images. Pattern Analysis and MachineIntelligence (PAMI), 2012. 3

[17] V. Ramakrishna, D. Munoz, M. Hebert, J. A. Bagnell, andY. Sheikh. Pose Machines: Articulated Pose Estimationvia Inference Machines. European Conference on ComputerVision (ECCV), 2014. 1

[18] S. Roth and M. Black. On the spatial statistics of op-tical flow. International Conference on Computer Vision(ICCV), 2005. 2

[19] S. M. Seitz and S. Baker. Filter flow. International Con-ference on Computer Vision (ICCV), 2009. 2

[20] D. Sun, S. Roth, J. P. Lewis, and M. J. Black. Learn-ing optical flow. European Conference on Computer Vision(ECCV), 2008. 2

[21] J. Tompson, R. Goroshin, A. Jain, Y. Lecun, and C. Bre-gler. Efficient Object Localization Using Convolutional Net-works. arXiv preprint, 2015. 6

[22] J. Valmadre, S. Sridharan, and S. Lucey. Learning detec-tors quickly using structured covariance matrices. In AsianConference on Computer Vision (ACCV), 2014. 2

[23] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid.DeepFlow: Large Displacement Optical Flow with DeepMatching. International Conference on Computer Vision(ICCV), 2013. 2, 7, 8

[24] Y. Yang and D. Ramanan. Articulated pose estimation withflexible mixtures-of-parts. Computer Vision and PatternRecognition (CVPR), 2011. 1

[25] T. Zhou, Y. J. Lee, S. X. Yu, U. C. B. Icsi, and A. A. Efros.FlowWeb: Joint Image Set Alignment by Weaving Con-sistent, Pixel-wise Correspondences. International Confer-ence of Computer Vision and Pattern Recognition (CVPR),2015. 3

0 2 4 6 8 10

Mahalanobis Distance (σ)

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ula

tive D

ensi

ty

LDA (ours)SIFT FlowDeep FlowArgmaxBaseline

0 2 4 6 8 10

Mahalanobis Distance (σ)

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ula

tive D

ensi

ty

LDA (ours)SIFT FlowDeep FlowArgmaxBaseline

0 2 4 6 8 10

Mahalanobis Distance (σ)

0.0

0.2

0.4

0.6

0.8

1.0

Cum

ula

tive D

ensi

ty

LDA (ours)SIFT FlowDeep FlowArgmaxBaseline

Figure 4. Comparison of sparse keypoint localization for our method, SIFT Flow [12] and Deep Flow [23]. The baselinemeasures the global alignment bias of the dataset (how well one would perform by simply assuming no flow). The argmaxconsiders taking the single best match without regularization. The graphs measure the fraction of correspondences whichfall within an increasing distance from groundtruth. 3 standard deviations is inperceptible from human annotator accu-racy. From left to right: (a) aggregate results across all images, (b) the truck pair which our method localizes well, and(c) the biking pair for which our method fails to produce any meaningful correspondences.

Figure 5. Example correspondences discovered by our method, across a broad range of image pairs from our dataset. Thetruck pair produces good localization of points (see Figure 4b), whilst the biking pair shows a failure to produce anythingmeaningful (see Figure 4c).