Estimating Uncertainty in SSD-BasedFeatureTracking

Kevin Nickelsa� � , SethHutchinsonb

aDept.of Engineering Science, Trinity University, SanAntonioTX78212,USAbDept.of Electrical andComputer Engineering andTheBeckmanInstitute, University of

Illino is at Urbana-Champaign, Urbana IL 61801,USA

Abstract

SSD-basedfeaturetrackershaveenjoyedgrowing popularity in recentyears,particularlyin the field of visual servocontrol of robotic manipulators. Thesetrackers usesum-of-squared-differencescorrelation measuresto locate target featuresin sequencesof images.Theresults canthenbeusedto estimatethemotionof objectsin thescene, to infer the3Dstructureof thescene,or to control robotmotions.

Thereliability of theinformationprovidedby thesetrackerscanbedegradedby avarietyof factors, including changes in illumination, poor imagecontrast,occlusionof features,or unmodeled changes in objects. This has led other researchersto develop confidencemeasuresthatareusedto either accept or rejectindividual featuresthatarelocatedby thetracker. In this paper, we derive quantitative measures for the spatial uncertainty of theresults providedby SSD-basedfeaturetrackers.Unlike previousconfidencemeasuresthathavebeenusedonly to acceptor reject hypotheses, ournew measureallowstheuncertaintyassociatedwith a feature to beusedto weight its influenceon theoverall tracking process.Specifically, wescaletheSSDcorrelationsurface,fit aGaussiandistribution to thissurface,andusethisdistribution to estimatevaluesfor acovariancematrix.Weillustratetheefficacyof these measuresby showingtheperformanceof anexampleobject tracking systemwithandwithout themeasures.

Key words: FeatureTracking, Sumof SquaredDifferences,Uncertainty Estimation

�Corresponding author.Email addresses:knickels@trinity.edu (Kevin Nickels),seth@uiuc.edu

(SethHutchinson).

Article publishedin ImageandVision Computing 20 (2002) 47–68

1 Intr oduction

A growingnumberof applicationsin roboticsandcomputervision rely onreal-timetrackingof featuresin imagesequences.Theseapplicationsincludeestimatingthemotion of autonomousmobile vehicles,inferring the 3D structureof an environ-mentfrom 2D imagedata,andcontrollingroboticmanipulators. Thereliability offeaturetrackerscanbedegradedby many factors,e.g.,changesin illuminationorpoorimagecontrast.Nevertheless,many featuretrackerstreatall featuresin auni-form manner, without consideringin any way the uncertaintyassociatedwith thefeaturetrackingprocess.In thispaper, wepresenta formalismby which theuncer-taintyassociatedwith individualfeaturescanbeassessedandsubsequentlyusedinthetrackingprocess.Thisallowsfeaturesto beaccordedalevel of confidencecom-mensuratewith the uncertaintyin their measurement,which enablesapplicationsto weightfeaturesappropriately, dependingon theassociatedconfidencelevel andon theapplication’sneeds.

In this paper, we consideronly the caseof featuretrackers that usethe sum-of-squared-differences(SSD) correlationmethod[1] [2] [3]. In SSD-basedfeaturetracking,a feature templateis comparedto portions of animageto locatethatfea-ture.A similarity metricis usedto ratethesimilarity of thetemplateandtheimagepatch.The imageregion found to be the mostsimilar to the templateis typicallytaken to be the locationof the feature.In our work, we considerfeaturesthat aredefinedasgivengreyscalepatternsin animage.

We arespecificallyconcernedherewith theuncertainty in thecomputation of the(2D) locationof featuresin animage.Thesefeatureuncertaintiescanthenbeusedby higherlevel applicationsoftware.For example,in relatedwork, we have usedthemethodspresentedin this paperto drive anuncertaintyestimation processforan objecttrackingsystemthatalsotakesinto accountkinematic andimagingun-certaintiesin trackingcomplex articulatedobjects[4].

Therehave beenseveralotherattemptsto incorporatefeatureuncertaintyinto thetrackingprocess.In thework describedby Gennery[5] andLowe[6] objecttrack-ing is performedby trackingsalientfeatureson the imageof an object,andthisinformationis usedto computetheobjectmotionthatgaveriseto theobservedfea-ture motion. Gennerytracksestimatederror in the position, orientation,velocity,andangularvelocity of the rigid objectto aid in predictionof objectmotion. Heusesnaturallyoccurringedgeson the polyhedralobjectasfeatures.Lowe distin-guishesbetweentwo different typesof errors,matchingerrorsandmeasurementerrors,andattemptsto utilize separatemechanismsto dealwith each.Matchinger-rors,mismatchesbetweenfeaturepointson theobjectmodelandfeaturepointsinthe image,aredealtwith by removing outliers.Measurementerrorsaredealtwithby usingthevariancein theobjectlocation,orientation,andconfigurationto com-putetheexpectedvariancein measurements.In [3], estimatesfrom regionsof high

48

confidenceareusedto improveestimatesin regionsof low confidence.To facilitatethis,a metricfor theconfidencein a motionestimateis developed.Finally, featuretrackingconfidencemeasureshavebeenusedin visualservo controlto increasetherobustnessof thecontrol [2]. In eachof thesecases,theuncertaintycharacteriza-tionsareusedonly to rejector acceptfeatures.In contrast,theapproachpresentedhererefersto the certaintyof the locationof the featureratherthanthe certaintyin themeasurementof thatfeature.It gives a quantitativeevaluation of uncertaintythatcanbeusedto weightfeaturemeasurementsin proportionto their reliability invariousdirectionsin theimage.

In our research,we assumethat the shapeand appearanceof the object beingtrackedareknown. This is a particularversionof themodel-basedtrackingprob-lem,which is of currentinterestin boththeroboticsandcomputervision commu-nities(see,e.g.[7], [8], [9]). By exploiting theinformationcontainedin shapeandappearancemodels, weareableto generatetemplatesfor thepredictedappearanceof featuresof interestin a given configuration.An alternative to this model-basedapproachis to usefeaturesgatheredon-line[2], whichmayhelpto ensurethequal-ity of the featurestracked,sincefeaturetemplatesexactly matchprevious featureappearance.However, this methodneglectsany a priori information aboutthege-ometricrelationshipof theindividualfeaturesto theobject.

The remainderof the paperis organizedas follows. We begin with a review oftheSSD-basedfeaturetrackingmethod.Following this,wedescribeourgoalswithrespectto characterizingthe spatialdiscrimination of features.We illustratehowthesegoalsallow a systemto accountfor bothocclusionof featuresandsubopti-mal featureperformanceduring objecttracking.Thenwe presenta Gaussianap-proximation anddescribehow sufficient statistics canbeusedto characterizethisapproximation. Finally, we presentsomefeaturetracking resultstaken from ourimplementedtrackingsystem,illustratingthe informationgainedfrom theseerrorestimatesandshow a casestudyillustrating the degradationin our systemwhenuncertaintyestimation informationis ignored.

2 Corr elation and feature templates

The following sectionsdiscussthe threeareasthataremostcrucial to the perfor-manceof correlationbasedtracking:thecontentof the featuretemplates,thedef-inition and useof a specificsimilarity metric for tracking,and the definition ofconfidencemeasureson thetrackingresults.

49

2.1 Feature templategeneration

Thecontentof thetemplateis an importantchoicein featuretracking.If the tem-platefaithfully reproducestheactualappearanceof thefeaturein theimage,track-ing will work well. However, if a templateis oversimplified or doesnot matchtheappearanceof afeaturein theimagedueto unmodeledeffects,featuretrackingwillalmostcertainlyperformpoorly.

A templatecouldbegeneratedfrom a canonicalview of thefeature,andtemplatematchingdonein a searchwindow centeredaboutthe predictedposition of theimage.Brunelli andPoggiogive a goodreview of this techniquein thecontext offacial featuretracking[10]. Themainproblemwith this straightforwardapproachis that thesimple templateis a 2D entity, andthe imagepatchmayundergo trans-formationsthat thetemplatecannotmodel,suchasrotation,shear, andchangesinillumination[11].

A morecomplex algorithmthatalsoworksin certainsituations is to useanimagepatchfrom theprevious image,taken from theareaaroundthe lastcomputedpo-sition of thefeaturein that image,for thetemplate.Hager[12] usesthis approachfor visual servoing. HagerandBelhumeur[11] have alsousedprevious trackinginformationto warp this imagepatchbeforeuseasa featuretemplateto accountfor illuminationandgeometricchanges,which increasestheflexibili ty of this ap-proach.Themaindifficulty with this approachis feature drift. Featuredrift occurswhenthecomputedpositionof thefeatureis slightly incorrect.Thiscausesthenexttemplateto beslightly offsetfrom thetruefeaturelocation,andthenext computedposition of the featureto be slightly more incorrect.Slowly, the computedposi-tion migratesoff thefeatureof interest,andthe template containsimagestructuredivergentfrom thefeatureof interest.

If objectandscenemodeling arepart of the trackingframework, it is possible tocreatetemplatessoley from this information,neglecting imagecontentfrom previ-ousimages.Lopezet al. [9] have a 3D registeredtextureof a faceaspartof theirobjectmodel.Computergraphics(CG) techniquesareusedto texture-mapthetex-ture onto a wire-framemodelof the faceto estimatethe appearanceof a featurein the image.This imagepatchis thenusedasa template in the featuretrackingportionof thesystem.

Our work uses3D modelsfor complex articulatedobjects,also in a CG basedframework (specifically, OpenGL[13]), to generatefeaturetemplates.Imagingandobjectmodelsareusedto produceaCGimageof thescenein theestimatedconfig-uration.SincetheCGsceneis completely known the2D imagelocationsof salientfeaturesontheCGobjectareknown.Portionsof thisimagearethenusedasfeaturetemplatesto compareagainsttheinput image.More informationonourmethodforfeaturetemplategenerationis givenin [14].

50

2.2 TheSSDsimilarity metric

In correlation-basedtracking,a similarity metric is usedto comparethe featuretemplatedescribedabove to areasof theimageto locatethefeaturein theimage.

Thestandardsum-of-squared-differences(SSD)metricfor grayscaleimagesis de-finedas:

SSD�u� v��� ∑

m� n N

T�m� n��� I

�u m� v n��� 2 � (1)

whereT is the templateimageand I is the input image.The location�u � v� rep-

resentssomelocationin the input imagewhosecontentis beingcomparedto thecontentof the template.Papanikolopoulos [2] usesthe SSDmeasureto generatetrackingresultsthat arethenusedfor roboticvisual servoing experiments. Anan-dan[3] andSinghandAllen [15] usethisSSDmetricfor thecomputationof imageflow. Alternativedissimilarity measurescanbefoundin [16], [17], [18] and[19].

2.3 Windowing

Often, the SSDmeasureis not computedfor the entire input image,but only forsomesearch windowin theinput image.Primarily for computationalreasons,thisrestrictionalso serves as a focus of attentionfor the featuretrackingalgorithm.SinghandAllen [20] [15] definea fixed sizesquaresearchwindow surroundingthe previous locationof the feature.KosakaandKak [21] considerat lengththeshapeand locationof the searchwindow. They model the sceneandcomputeaspatialprobability densityfunctionfor thelocationof eachfeature,thensearchtheimageareacorrespondingto 85%of theprobability mass.

Our work usesa constant-velocity model for an articulatedobject to predict3Dpositionsfor relevantpointsontheobject.Imagingmodelsarethenusedto projecttheselocationstopointsontheimageplane.A fixedsizerectangularsearchwindowcenteredat theselocationsis establishedin the input image.See[22] for moredetailson themodelsusedin thiswork.

3 Confidencemeasuresand spatial uncertainty

It hasbeennoted[3] thatpopularsimilarity measuresoftenleadto someunreliablematches,particularlyin imageregionswith little textural information. For this rea-son,it is often helpful to computea confidenceon the matchfound,aswell asalocation.This confidencemeasuretypically gives information regardingthe relia-bility of thematchscore.Thisscalarscoreoftenis usedto estimatethereliability of

51

thefeature,e.g.,for usein latertrackingoperationsor to propagateimageflow in-formationfrom oneportionof animageto another[15]. Below, wewill describeamatrix-valuedmeasurethatcontainsinformationbothabouttheoverallconfidencein a featuremeasurementandinformation abouthow accuratethemeasurementisin all imagedirections.

Anandan[3] usedtheSSDmatchingscoresof a templatewith a5 � 5 imageregionto developa matchconfidencemeasurebasedon the variationof the SSDvaluesover thesetof candidatematches.Anandanarguedthatif thevariationof theSSDmeasurealonga particularline in the searchareasurrounding the bestmatchissmall,thenthecomponentof thedisplacementalongthedirectionof thatline can-not be uniquely determined.Conversely, if there is significantvariation along agivenline in thesearcharea,thedisplacementalongthis line is morelikely correct.

SinghandAllen definea responsedistribution basedon theSSDmetric(1) as�Dc

�u � v��� exp

� � kSSD�u � v����� (2)

wherek is usedasanormalization factor. Thenormalizationfactork waschosenin[15] sothatthemaximum responsewas0 � 95.SinghandAllen thenarguethateachpoint in thesearchareais acandidatefor the“true match.” However, apointwith asmallresponseis lesslikely to bethetruematchthanapointwith ahigh response.Thus, the responsedistribution could be interpretedasa probability distributionon the true matchlocation – the responseat a point depictingthe likelihood ofthecorrespondingmatchbeingthetruematch.This interpretationof theresponsedistribution allows theuseof estimation-theoretictechniques.

Undertheassumption of additive zeromeanindependenterrors,a covariancema-trix Pm is associatedwith eachlocationestimate.This matrix is constructedfrominformationaboutthe shapeof (2) as u and v (the horizontaland vertical pixellocations)change.

Pm ������������

∑u � v N

�D�u � v� � u � um� 2

∑u � v N

�D�u � v� ∑

u � v N

�D�u � v� � u � um� � v � vm �∑

u � v N

�D�u � v�

∑u � v N

�D�u � v� � u � um� � v � vm�∑

u � v N

�D�u � v� ∑

u � v N

�D�u � v� � v � vm � 2

∑u � v N

�D�u � v�

������������ (3)

whereum andvm aretheestimatedlocations,in theu andv directions,of thefeatureandN is the neighborhood of the pixel. Expressionsfor the normalizedvariancein the horizontalandvertical directionsappearalongthe diagonalof (3), andanexpressionfor thenormalizedcovarianceappearsin bothoff-diagonalelementsofPm. See[15] for morein-depthdiscussion of this formulation.The reciprocalsoftheeigenvaluesof thecovariancematrixareusedasconfidencemeasuresassociated

52

with theestimate,alongthedirectionsgivenby thecorrespondingeigenvectors.Toourknowledge,SinghandAllen arethefirst researchersto treatthelocationof thebestmatchasa randomvectorandto usethe (normalized)SSDsurface(SSDS)to computethe spatialcertaintyof the estimateof this vector. Theseconfidencemeasuresare thenusedin the propagation of high confidencemeasurementsforlocal imageflow to regionswith lower confidencemeasurements,suchas thosecausedby largehomogeneous regions.

As theSSDmeasureis usedto comparethetemplateto areasof theimageneartheareageneratingtheminimum SSDscore,somemeasureof thespatialdiscrimina-tion powerof thetemplatecanbegenerated[3]. Spatialdiscrimination is definedastheability to detectfeaturemotionalonga givendirectionin theimage.This con-ceptis quitesimilar to theconfidencemeasuresdiscussedabove thatestimatethereliability of the locationestimate.However, we interpretthe confidencesasspa-tial uncertaintiesin thereturnedlocation.Papanikolopoulosexplainstheconceptofspatialdiscriminationin detail,giving examplesof cornerfeatures,edgefeatures,andhomogeneousfeatures,alongwith their respective autocorrelationSSDSs[2].We illustratetheeffectonseveralfeaturesin Section5.

While conclusions aboutthe efficacy of a given templatefor featurelocalizationcanbedrawn from thefully computedSSDS,it is bothcomputationally expensiveandmemoryintensiveto maintainthecompletesurfacefor thispurpose.In thenextsection,wederiveanapproximationfor

�D thatis morepracticalto maintainand

propagate.

4 A practical approximation for�

D

In orderto maintainanduserelevant information abouttheshapeof theresponsedistribution, we introducea mathematicalapproximation to thedistribution givenin (2). By suppressingthe off peakresponseof the featuretracking result, thisresponsedistribution functionconvertstheSSDSinto anapproximatelyGaussiandistribution thatcontainsthefeaturetrackinginformationwewish to maintain.

4.1 Uncertainfeaturemeasurements

Themeasurementvectorzk is interpretedasanuncertainlocationin the�u � v� plane,

andmodeledasa 2D Gaussianrandomvector. It is illustrative to analyzethebe-havior of thedensityfunctionfor thisvectorwith respectto thespatialcertaintyofthefeaturetrackingresultasRk, thecovariancematrix for thevector, changes.Forexample,if Rk � σ2I , whereσ2 is thevarianceof thevector, thelocationis equallycertainin eachdirection.The ellipsesof equalprobability on the densitysurface

53

arecircles.If σu �� σv, whereσ2u andσ2

v arethevariancesin theu andv directions,thelocationis morecertainin onedirection(givenby theminoraxisof theellipsesof equalprobability) thanin the otherdirection(given by themajor axis).As thelengthof themajoraxisapproachesinfinity, completeuncertaintyon the locationalong this dimension is asserted.It is well known that the meanandcovariancearesufficient statisticsfor a Gaussianrandomvariable.Therefore,if this Gaussiandensitysurfaceis sufficient to modelthetrackingbehavior, it is nosurprisethatthemeanandcovariancesuffice to maintainthis information. In the next sectionweexplainhow weestimatethesequantitiesfrom

�D.

4.2 Parameterestimation fromtheSSDS

This sectiondescribesa processfor analyzingtheSSDSto arrive at estimatesforthe meanandvarianceof a Gaussianrandomvector. The densityfunction of thisvectoractsasan approximation to the responsedistribution

�D (see(2)) for the

purposeof trackingfeatures.

Our work developsa differentnormalizationprocedurefor�

D that is useful fortheevaluationof isolatedfeaturemeasurementsfrom template images.Thisproce-durediffersfrom thatdescribedabove in severalways.First,SinghandAllen werecomparingquitesimilar imagepatches,leadingto averypeakedresponsedistribu-tion.Ourwork comparesCGgeneratedtemplateimagesto actualcapturedimages,leadingto a muchlesspeakeddistribution. Second,we emphasizethe importanceof suppressing theoff peakresponse,sothatonly theimageareaswith significantagreementto thetemplateimageaffect theuncertainty.

Our computation of thenormalizationfactork in (2) differsfrom thatof SinghandAllen [15]. Wechosek suchthat

∑u � v N

�D�u � v��� 1 � (4)

As canbeseenin Figure1, thishastheeffect of suppressingtheoff peakresponseof thefeaturedetector, whencomparedwith SinghandAllen’s normalization.Webelieve this to beamoreappropriatenormalizationfor oursituation.

Asdescribedin Section4.2,wecomputeonecovariancematrixandonelocationforeachfeature,andusethis information in amodel-basedobjecttrackingframework.Wedonotrejectanytrackinginformationbut insteadweighteachmeasurementonthebasisof thiscovariancematrix,usingasmuchinformationaspossible from thefeaturetracking.

Themode,or mostprobablevalue,of a randomvectoris locatedat thepeakof thedensityfunction.We take the locationof the minimum of the SSDSasour value

54

0

20

400 10 20 30 40

−6

−5

−4

−3

−2

−1

0

x 105

(a) Negativeof SSDSurface

0

20

400 10 20 30 40

0.5

1

1.5

0

20

400 10 20 30 40

0

0.5

1

1.5

x 10−3

(b)�

D from [15] (c) GaussianDensityFunctionapproximating (b)

0

20

400 10 20 30 40

0

0.005

0.01

0.015

0

20

400 10 20 30 40

0

0.002

0.004

0.006

0.008

0.01

0.012

(d)�

D (e) GaussianDensityFunctionapproximating (d)

Fig.1. Approximationof anexampleresponsedistribution by adensity function.Pixelsarealong thex andy axes,andSSDScoreor probability massis alongthez axis.

for themodeof thevector,

zk � argminu � vSSD�u � v��� (5)

Thevarianceof u (σ2u ), thevarianceof v (σ2

v ), andthecovariancebetweenu andv�ρuvσuσv � canbeestimateddirectly from theresponsedistribution usingEquations

(2) and(3), yielding thedesiredcovariancematrix,

Rk � �� �σ2u ρuvσuσv ρuvσuσv �σ2

v

�� � (6)

55

which,asdescribedabove,containscompleteinformationabouttheorientation andshapeof theerrorellipsoids.

By computing thecovarianceaswell asthevariances,we retaininformationabouttheorientationof theellipsoidsof constantprobability, aswell astheir intersectionwith theu andv axes.Therefore,wegaintheability to maintaininformation aboutdirectionsof goodspatialdiscrimination.

Of course,aswe areonly maintaining the meanandvarianceof the randomvec-tor, and not the completeSSDS,this is only an approximation to the completeinformationaboutlocal imagestructuregivenby the SSD.However, it doesgivean indication of both the absolutequality of the matchand,in caseswhereedgefeatures1 exist, thedirectionof theedge.

5 Results

In this section,we review theresultsof featuretrackingwith measurementuncer-taintyestimation. Weexaminesomeillustrativeexamplesgraphicallyandquantita-tively, anddescribetheuseof this information.Finally, wepresentabrief examplefrom our relatedwork in objecttrackingthat usesthis uncertaintyinformationtoaid in trackinganobject.

5.1 Gripper feature

The featureconsideredin this sectionis a portion of the end-effector of a robot.Figures2 and3 show the inputsusedfor thesesearches.Two situations will beconsidered:a reasonablystandardinput imagewith thegrippervisible asexpectedand an input imagein which the gripper is completelyoccludedby an externalobject.Thetrackingresultsandcomputeduncertaintieswill beshown in bothcases.

Figures4 and5 show thetrackingresultsfor thesecases.In eachfigure,thefeaturelocation is shown by a crossin (a), the negative SSDSis shown in (b), and thecomputedprobabilitydensityfunctionof this location(seeSection4 for detailsonthis computation) is shown in (c). In Figure4, both the SSDSandGRV densitysurfacesindicateequalaccuracy of thetrackingresultin all directions.

1 Our referencesto edge featurescanrefer to any featureswith low discrimination powerin onedirection in the current configuration of the object, even if the 3D structurewhichgave rise to the feature hasgoodtexture in multiple directions. SeeSection5.3 for moreexplaination of this case.

56

(b)

(a) (c)

Fig. 2. Inputs to tracking for gripper feature (unoccluded)(a) Full Input Imagewith searchregion marked(b) Template(c) Actual areasearched

(b)

(a) (c)

Fig. 3. Inputs to tracking for gripper feature (occluded) (a) Full Input Imagewith searchregion marked(b) Template(c) Actual areasearched

0

20

400 10 20 30 40

−2.5

−2

−1.5

−1

−0.5

x 106

0

20

400 10 20 30 40

0

1

2

x 10−3

(a) (b) (c)

Fig. 4. Tracking results for unoccluded gripper feature. (a) ResultsImagewith a dottedbox showing theareasearchedanda cross illustrating the feature location determined(b)Negative SSDS(c) GRV density

In Figure5, wepresentanillustration of theusefulnessof theon-lineestimation oftemplateefficacy. By callingthisestimation “on-line”, wedonotmeanto imply thatthismethodis real-time.With nohardwareaccellerationfor theSSDcomputations

57

0

20

400 10 20 30 40

−12

−10

−8

−6

−4

−2

x 105

0

20

400 10 20 30 40

0

0.5

1

1.5

x 10−3

(a) (b) (c)

Fig. 5. Tracking results for occludedgripper feature.(a) ResultsImagewith a dotted boxshowing theareasearchedanda cross illustratingthefeature location determined(b) Neg-ative SSDS(c) GRV density

or theresponsedistribution normalization,thisalgorithmis actuallyquiteslow. Ona Sparc10, eachfeaturetakesapproximately 15 secondsto compute.Instead,wemeanto point out that sincethe templatesusedand input imagesacquiredbothchangeasa functionof timeandconfiguration,theefficacy measuresusedmustbecomputedwithin thetrackingloop.

Thefeatureshown in Figure5 hasthesametemplateasin thepreviouscase.How-ever, a personhassteppedbetweenthecameraandthefeature,occludingthe fea-ture.

A 2D measurement,representedby thecrossin Figure4(a)andFigure5(a),anda2 � 2 covariancemeasurementaretheoutputof thefeaturetracking,andareuseddirectly in theEKF framework describedin [14].

5.2 Edge feature

This caseillustratestheusefulnessof themeasurementuncertaintyestimationfortrackingfeatureswith poorspatialdiscrimination in onedirection.An edgefeaturecanbetrackedwell only in thedirectionorthogonalto theedge.This featurearisesfrom a point on the edgeof the robotic arm.Thus,the orientation of the edgeinthe featuredependson the configurationof the robot.As the configurationof therobotchanges,thedirectionof theedgeprojectedontotheimageplanewill change.Figures6 and7 show theinputsusedfor thissearch.

In Figure8, theedgeis in adiagonalorientation.Thenegativeof theSSDSshown in(b) hasa ridgealongthis direction,indicatinggoodmatchscoresalongtheridge.The locationof the absolute maximimum of the SSDS(i.e. the returnedfeaturelocation), is shown by the crossin (a). After normalization, the densityfunctionshown in (c) exhibits thesameridge,while suppressingtheoff peakmatchscoresonbothsidesof theridge.Similarly, theedgein Figure9 is in averticalorientation,so the ridgesin (b) and(c) are in the vertical direction,andthe returnedfeaturelocationshown by the crossin (a) is known to be accurateonly in the horizontal

58

(b)

(a) (c)

Fig.6. Inputsto tracking for edgefeature(diagonalconfiguration)(a)Full Input Imagewithsearch region marked(b) Template(c) Actual areasearched

(b)

(a) (c)

Fig. 7. Inputs to tracking for edgefeature (vertical configuration)(a) Full Input Imagewithsearch region marked(b) Template(c) Actual areasearched

0

20

400 10 20 30 40

−5

−4

−3

−2

−1

0

x 105

0

20

400 10 20 30 40

0

0.005

0.01

0.015

0.02

(a) (b) (c)

Fig.8.Trackingresults for diagonaledgefeature.(a)ResultsImagewith adotted boxshow-ing theareasearchedanda cross illustrating the feature location determined(b) NegativeSSDS(c) GRV density

directiononly.

By maintainingthis information, the systemcan exploit the featuretracking in-

59

0

20

400 10 20 30 40

−5

−4

−3

−2

−1

0

x 105

0

20

400 10 20 30 40

0

0.005

0.01

0.015

0.02

(a) (b) (c)

Fig. 9.Tracking results for vertical edgefeature.(a)Results Imagewith adottedboxshow-ing theareasearchedanda cross illustrating the feature location determined(b) NegativeSSDS(c) GRV density

(b)

(a) (c)

Fig. 10. Inputs to tracking for point feature (nondegenerate) (a) Full Input Imagewithsearch region marked(b) Template(c) Actual areasearched

formation to its fullest extent. The result is neitherendowed with inappropriateconfidencedue to the goodaccuracy in the directionorthogonal to the edgenorundulydevalueddueto thepooraccuracy in thedirectionalongtheedge.

5.3 Degeneratepoint feature

In this section,we illustrateanotheraspectof theusefulnessof on-lineestimationof template efficacy. Sinceour object-trackingsystemis intendedto work underwidely varyingconfigurationsof theobject,theappearanceof featuresmaychangesignificantly during tracking.A single featureacceptanceor rejectancedecisionwill notsuffice in thiscase.

Figures10 and11 show the inputsusedfor this search.Figures12 and13 showtrackingresultsfor a featurethatundergoessuchachangein appearance,apointofintersectionof ablackline on theedgeof theroboticarmwith therearedgeof thearm.

60

(b)

(a) (c)

Fig. 11. Inputs to tracking for point feature (degenerate— acting asa line feature)(a) FullInput Imagewith search region marked (b) Template(c) Actual areasearched

0

20

400 10 20 30 40

−10

−8

−6

−4

−2

0

x 105

0

20

400 10 20 30 40

0

0.01

0.02

0.03

(a) (b) (c)

Fig. 12. Trackingresults for nondegeneratepoint feature. (a) ResultsImagewith a dottedbox showing theareasearchedanda cross illustrating the feature location determined(b)Negative SSDS(c) GRV density

This featureis a point of high texture in bothdirectionswhenthearmis approxi-matelyparallelto the imageplane,asshown in Figure10(a),andactslike a pointfeature.Thisfeaturelocation,shown in Figure12(a)by thecross,canbefoundwithhighaccuracy in all directionsasshown in theNegativeSSDSshown in (b) andthefinal densityshown in (c).

However, this featureactslike anedgefeaturein otherconfigurations,suchastheconfigurationshown in Figure11, wherethe arm is pointing roughly toward thecamera.In this configuration,thefeatureappearsasa verticaledgefeature,ascanbeseenin the searchimageshown in Figure11(c).The locationof the featureinthis case(shown by thecrossin Figure13(a))canbefoundwith high accuracy inonly thehorizontaldirection.This fact canbeseenin theNegative SSDS(b) andthefinal density(c).

Again, the maintenanceof the covariancematrix insteadof a single confidencemeasuremakesthissuboptimal trackingresultnotonly tolerable,but useful.

61

0

20

400 10 20 30 40

−6

−5

−4

−3

−2

−1

0

x 105

0

20

400 10 20 30 40

0

0.002

0.004

0.006

0.008

0.01

0.012

(a) (b) (c)

Fig. 13.Tracking results for degeneratepoint feature.(a) ResultsImagewith a dottedboxshowing theareasearchedanda cross illustratingthefeature location determined(b) Neg-ative SSDS(c) GRV density

5.4 Objecttrackingusinguncertainty

In eachof theprevious results,wehaveshown how theresultsfrom trackingasin-gle featurecanbeanalyzedto arrive at anefficacy measurefor thelocationof thatfeature.In this section,we briefly describehow this typeof informationfrom sev-eral distinct featurescanbeusedto track themovementsof an articulatedobject.We will alsodemonstratea casein which the ignoranceof this uncertaintyinfor-mationleadsto losingtrackof theobject.As we areusingtheabsoluteminimumof theSSDSfor thefeaturemeasurementin bothexperiments,theaccuracyof thefeaturetrackinginformation is not changed.The independantvariablein the twoexperimentsis insteadtheassociateduncertaintyinformation. KalmanFilter diver-gencedueto incorrectuncertaintyinformation is awell-known phenomenonin theradartrackingliterature[23] [24], andhasreceivedsomeattentionin thecomputervisionandroboticscommunity aswell [25].

In thisexample,wearetrackingatwo degreeof freedomplanararmwith unknownlink lengths.Thus, the systemhasfour total degreesof freedom.We utilize anextendedKalman filter to track this system.For an in-depthlook at this objecttracking system, see[4]. We presenta brief look at the system,from the pointof view of utilizing featuretrackinginformation, below. The systemusedin thisexampleis identicalexcept for the useof measurementuncertaintyasdescribedbelow.

The measurementfunction is the composition of thearm Jacobianandthe imageJacobian(i.e. themappingfrom joint anglesto imagelocationof features).Usingthis framework, themeasurementequationsareimplicitly inverted[26] andfeaturemeasurementsfrom theimagesareusedto updatetheconfigurationof thearm.

Wedefinethestatevectorfor thisextendedKalmanfilter to be

xk �"! q0 q1 q̇0 q̇1 a0 a1 # T � (7)

62

whereqi is theangleof the ith joint, andai is thelengthof the ith link. Weassumeconstantvelocity motion in the joints andfixed unknown link lengths.Measure-mentsfor thissystemarethe

�u � v� locationsin theimageplaneof salientpredefined

featureson thearm,

zk � ! u1 v1 $�$�$ uF vF # T(8)

wherethereare F featuresbeing tracked. Eachfeaturemeasurementhasan un-certaintymatrix Rf associatedwith it. Thesemeasurementsarecombinedinto asystemmeasurementuncertaintymatrix,asshown below.

R fk � �� σ2

ufρuf vf

σufσvf

ρuf vfσuf

σvfσ2

vf

��

Rk ������������R1

k

�1 � 1� R1

k

�1 � 2� $�$�$ 0 0

R1k

�2 � 1� R1

k

�2 � 2� 0 0

......

...

0 0 RFk

�1 � 1� RF

k

�1 � 2�

0 0 $�$�$ RFk

�2 � 1� RF

k

�2 � 2�

������������Figure14 shows the resultsfrom two objecttrackingruns.Both runsutilized thesamesequenceof inputimages.In thefirst case,theuncertaintyestimatesdescribedin thispaperareused.In thesecond,anexperimentallydeterminedconstantexper-imentalvariancewasused.The object tracking portion of the algorithm was in-denticalin bothcases.Thecomputationof thefeaturetrackinguncertaintywastheonly change.Thesolid linesmarkedx0 (with MU) andx1 (with MU) in Figure14representtheangleestimatesfor joints0 and1 respectively, with measurementun-certaintyinformation. The dashedlinesmarked x0 (without MU) andx1 (withoutMU) representtheangleestimatesfor joints0 and1 respectively, replacingtheac-tualmeasurementuncertaintyinformationwith theidentitymatrixmultipliedby anexperimentally determinedscalarvariance.This variancewaschosento optimizeperformancein the system, andworked well in many situations.However, in thecaseshown the initial link lengthswereincorrectby a factorof approximately2,andthe latter systembreaksdown. Sincethesetrackingresultsarefrom an unin-strumentedhumanarm,groundtruth is not available.Qualatitively, the estimatesgivenby theformersystemtracktheobjectwhile theestimatesgivenby thelattersystemdivergefrom theactualangles.

By ignoringtheaugmentedfeaturetrackingmeasurements,thesystemis not ableto identify correctlywhenthesystemis doingwell andwhenit haslost track.Thisfactismoreevidentin Figure15,wheretheestimatesfor thelink lengthsareshown.Again, the solid lines indicatethe systemusingmeasurementuncertaintyandthedashedlinesindicatetheuseof anexperimentallydeterminedscalarvariance.

63

0 50 100 150 200 250 300

−3

−2

−1

0

1

2

3

Frame Number

Join

t Ang

le (

rad)

xo (with MU)

x0 (without MU)

x1 (with MU)

x1 (without MU)

JointAngleEstimates

0 50 100 150 200 250 3000

0.05

0.1

0.15

Frame Number

Join

t Ang

le S

tand

ard

Dev

iatio

n (r

adia

ns)

σxo

(with MU)σ

x0 (without MU)

σx1

(with MU)σ

x1 (without MU)

JointUncertaintyEstimates

Fig. 14.ObjectTracking ResultsandMeasurementUncertainty - JointEstimates

6 Conclusions

ThemethodpresentedusestheSSDS,acommonintermediateresultin correlation-basedfeaturetracking,to computequantitative estimatesfor the spatialaccuracy

64

0 50 100 150 200 250 3000

10

20

30

40

50

60

70

Frame Number

Link

Len

gth

(pix

els)

ao (with MU)

a0 (without MU)

a1 (with MU)

a1 (without MU)

Link LengthEstimates

0 50 100 150 200 250 3000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Frame Number

Link

Len

gth

Sta

ndar

d D

evia

tion

(pix

els)

σao

(with MU)σ

a0 (without MU)

σa1

(with MU)σ

a1 (without MU)

Link LengthUncertaintyEstimates

Fig. 15.Object TrackingResults andMeasurementUncertainty - Link LengthEstimates

of the featuretrackingresult.This estimateconsistsof a covariancematrix for aGaussianrandomvector. Analysisof this matrix yields informationaboutthe di-rections(if any) in which thetemplateis discriminatingthefeaturefrom theimagebackground,andprovidesa quantitativemeasureof confidencein eachdirection.

65

Severalexamplesweregivenillustratingtheintuitivenatureof thisquantitativeun-certaintyestimate.This estimateis particularlyuseful in long sequences,whenagivenfeaturecanperformwell at sometimes, andpoorlyat othertimes.An exam-ple of oneuseof this informationwasalsopresented,in thecontext of articulatedobjecttracking.More examplesof theuseof this matrix in model-basedtrackingof complex articulatedobjectscanbe found in [4], alongwith a more thoroughexplanationof theKalmanfilter basedsystem.

The featuretrackingresults,combinedwith the measurementuncertaintymatrix,yield acompositemeasurethatis usefulwhenanalyzingthetrackingresults.Anal-ysis of the featuretracking resultscan detecttemplatesthat do not discriminateeffectively in any direction.By associatingspatialconfidencemeasureswith fea-ture trackingresults,thoseresultscanbe morefully exploited: the fact thatsomedirectionsmayhavehighconfidencedoesnot leadusto accepttheentiremeasure-ment,andthefact thatsomedirectionsmayhave low confidencedoesnot leadusto disregardusefuldata.

References

[1] J. Shi, C. Tomisito,Goodfeaturesto track, in: ProceedingsIEEE ComputerSocietyConferenceon ComputerVision PatternRecognition, 1994,pp.593–600.

[2] N. P. Papanikolopoulos, Selection of featuresandevaluation of visual measurementsduring robotic visualservoingtasks,Journalof IntelligentandRobotic Systems13(3)(1995) 279–304.

[3] P. Anandan, A computational framework and an algorithm for the measurement ofvisual motion, International Journal of Computer Vision 2 (3) (1989) 283–310.

[4] K. Nickels,S.Hutchinson,Model-based tracking of complex articulated objects,IEEETransactionson RoboticsandAutomation 17 (1) (2001) 28–36.

[5] D. B. Gennery, Visual tracking of known three-dimensional objects, InternationalJournal of Computer Vision 7 (3) (1992) 243–270.

[6] D. Lowe, Robust model-basedmotion tracking through the integration of searchandestimation,International Journal of ComputerVision 8 (2) (1992) 113–122.

[7] S.Lee,B. You,G.Hager, Model-based3-D object tracking usingprojectiveinvariance,in: ProceedingsIEEEInternational ConferenceRoboticsandAutomation,Detroit, MI,1999, pp.1589–1594.

[8] J. E. Lloyd, J. S.Beis,D. K. Pai, D. G. Lowe, Model-basedteleroboticswith vision,in: Proceedings IEEE International Conference Robotics and Automation, Vol. 2,Albuquerque,NM, 1997,pp.1297–1304.

[9] R. Lopez, A. Colmenarez, T. S. Huang, Vision-based head and facial featuretracking, in: Advanced Displays and Interactive Displays Federated Laboratory

66

Consortium, Annual Symposium, Advanced Displays and Interactive DisplaysFederatedLaboratory Consortium, 1997.

[10] R.Brunelli,T. Poggio, Facerecognition:Featuresversustemplates,IEEETransactionson PatternAnalysis andMachine Intelligence15 (10) (1993) 1042–1052.

[11] G.Hager, P. Belhumeur, Efficient region tracking with parametricmodelsof geometryand illumination, IEEE Transactions on PatternAnalysis and Machine Intelligence20 (10) (1998) 1025–1039.

[12] G. Hager, Real-time feature tracking and projective invariance as a basisfor hand-eye coordination, in: ProceedingsIEEE ComputerSocietyConferenceon ComputerVision PatternRecognition, 1994, pp.533–539.

[13] M. Woo, J. Neider, T. Davis, The OpenGLProgramming Guide, Addison-Wesley,Reading, MA, 1996.

[14] K. Nickels, S. Hutchinson, Weighting observations: The use of kinematic modelsin object tracking, in: Proceedings IEEE International Conference Robotics andAutomation, 1998.

[15] A. Singh, P. Allen, Image flow computation: An estimation-theoretic frameworkand a unified perspective, ComputerVision Graphicsand ImageProcessing: ImageUnderstanding 56 (2) (1992) 152–177.

[16] S. Birchfield, C. Tomasi,A pixel dissimilarity measure that is insensitive to imagesampling, IEEE Transactions on PatternAnalysis and Machine Intelligence 20 (4)(1998) 401–406.

[17] P. J. Burt, C. Yen, X. Xu, Local correlation measures for motion analysis: Acomparative study, in: Proceedings IEEE Conference Pattern Recognition ImageProcessing,1982, pp.269–274.

[18] G. Hager, P. Belhumeur, Real-time tracking of image regions with changes ingeometry and illumination, in: Proceedings IEEE Computer SocietyConferenceonComputer Vision PatternRecognition, 1996, pp.403–410.

[19] A. Rosenfeld, A. Kak, Digital Picture Processing, 2nd Edition, Academic Press,Cambridge,MA, 1982.

[20] A. Singh, An estimation-theoretic framework for image-flow computation, in:ProceedingsInternational ConferenceComputer Vision,1990, pp.168–177.

[21] A. Kosaka, A. C. Kak, Fast vision-guided robot navigation using model-basedreasoningandprediction of uncertainties,Computer Vision andImageUnderstanding56 (3) (1992) 271–329.

[22] K. Nickels, S. Hutchinson,Integrated object modelsfor robust visual tracking, in:Proceedingsof WorkshoponRobust Visionfor Vision-basedControlof Motion, 1998.

[23] A. J. Isaksson,F. Gustafsson,Comparison of someKalmanfilter based methods formanoeuvretracking anddetection, in: Proceedingsof theConferenceonDecisionandControl, 1995,pp.1525–1531.

67

[24] A. Zolghadri, An algorithm for real-time failure detection in Kalman filters, IEEETransactionson Automatic Control41 (10) (1996) 1537–1539.

[25] R. Deriche, O. Faugeras,Trackingline segments, ImageandVision Computing8 (4)(1991) 261–270.

[26] W. J.Wilson, C. C. W. Hulls, G. S.Bell, Relative end-effectorcontrol using Cartesianposition basedvisual servoing, IEEETransactionsonRoboticsandAutomation12(5)(1996) 684–696.

68