comparison of local feature descriptorsyang/courses/cs294-6/maji-presentation.pdfvarious feature...

OutlineIntroductionBenchmarks

Experiments and ResultsFuture Work

Comparison of Local Feature Descriptors

Subhransu Maji

Department of EECS,

University of California, Berkeley.

December 13, 2006

Subhransu Maji Comparison of Local Feature Descriptors



1 IntroductionLocal Features

2 BenchmarksMikolajczyk’s DatasetCaltech 101 Dataset

3 Experiments and ResultsEvaluation of Feature DetectorsEvaluation of Feature Descriptors

4 Future Work




Local Features

Applications of Local Features

Multi Camera Scene reconstruction.

Robust to Backgrounds, Occlusions

Compact Representation of Objects for Matching, Recognitionand Tracking.

Lots of uses, Lots of options.

This work tries to address the issue of what features aresuitable for what task, which is currently a black art!!




Local Features

Key properties of a good local feature

Must be highly distinctive, i.e. low probability of a mismatch.

Should be easy to extract.

Invariance, a good local feature should be tolerant to.

Image noiseChanges in illuminationUniform scalingRotationMinor changes in viewing direction

Question: How to construct the local feature to achieve

invariance to the above?




Local Features

Various Feature Detectors

Harris detector find points at a fixed scale.

Harris Laplace detector uses the scale-adapted Harris function to localizepoints in scale-space. It then selects the points for which theLaplacian-of-Gaussian attains a maximum over scale.

Hessian Laplace localizes points in space at the local maxima of theHessian determinant and in scale at the local maxima of theLaplacian-of-Gaussian.

Harris/Hessian Affine detector does an affine adaptation of theHarris/Hessian Laplace using the second moment matrix.

Maximally Stable Exremal Regions detector finds regions such that pixelsinside the MSER have either higher (bright extremal regions) or lower(dark extremal regions) intensity than all the pixels on its outer boundary.

Uniform Detector(unif) - Select 500 points uniformly on the edge mapsby rejection sampling.




Local Features

Various Feature Descriptors

Scale Invariant Feature Transformation A local image is path is dividedinto a grid (typically 4x4) and a orientation histogram is computed foreach of these cells.

Shape Contexts computes the ditance and orientaion histogram of otherpoints relative to the interst point.

Image Moments These compute the descriptors by taking various higherorder image moments.

Jet Decriptors These are essentially higher order derivatives of the imageat the interest point

Gradient Location and Orientaiton Histogram As the name suggests itconstructs a feature out of the image using the Histogram of location andOrientation in of points in a window around the interest point.

Geometric Blur These compute the average of the edge signal responseover small tranformations. Tunable parameters include the blurgradient(β = 1), base blur (α = 0.5) and scale multiplier (s = 9).




Local Features

Example Detections




Mikolajczyk’s DatasetCaltech 101 Dataset

Evaluation Criteria

We want the feature to be repeatable,repeatability = correct−matches

ground−truth−matches

Descriptor Performance:

recall vs 1-precision graphs.recall = #correct matches

#correspondances

correct matches found by neareast neignbour matching in thefeature space.correspondances obtained from ground truth matching.1 − precision = #falsematches

#false matches+#correct matces





Mikolajczyk’s Dataset

8 Datasets, 6 Images per dataset.Ground Truth Homography available for these Images.





Caltech 101 Dataset

101 Categories, man-made objects, motifs, animals and plants.

Foreground Mask is available. Obtain ground truth based on arough alignement of the contours.

Determine the scale, translation which maximizes area overlapof the contours.

Correspondance: Features of the images within a thresholddistance(10 Pixels) under the transformation.

Many clasification techniques use the structure of image forcomputing similarity. For e.g. SC based caracter recognitionusing TSP.

The performance of these algorithms is dependent ondetecting features on the right positions. Ideally we wouldwant the descriptor performance to be better on such a softernotion of matching.





Best 8 and Worst 8

50 100 150 200 250 300

50

100

150

200

250

300100 150 200

50

100

150

200

250

300 350 400

50

100

150

200

250

50 100 150 200 250

40

60

80

100

120

140

160

180

200

220

0 50 100 150 200 250

0

50

100

150

200

250

300

40 60 80 100 120 140 160 180 200 220

80

90

100

110

120

130

50 100 150 200 250 300

0

50

100

150

200

250

30040 60 80 100 120 140 160 180 200 220 240

40

60

80

100

120

140

0.9781 0.9717 0.9642 0.9490 0.9486 0.9483 0.9405 0.9223

0 50 100 150 200 250 300

20

40

60

80

100

0 20 40 60 80 100 120 140 160 180

0

20

40

60

80

100

120

140

160

180

200

100 150 200 250

50

100

150

200

250

50 100 150 200 250

100

120

140

160

180

200

220100 120 140 160 180 200 220

60

80

100

120

140

160

180

200

220

50 100 150 200 250

50

100

150

200

250

50 100 150 200 250 300

20

40

60

80

100

120

140

160

180

20050 100 150 200 250 300

60

80

100

120

0.7097 0.6934 0.6919 0.6658 0.6614 0.6444 0.6426 0.3318





Example Ground Truth Matches

100 200 300 400 500 600 700 800 900 1000

50

100

150

200

250

300

350 100 200 300 400 500 600

50

100

150

Faces car side

100 200 300 400 500 600

50

100

150

200

250

50 100 150 200 250 300 350 400 450 500

50

100

150

stop sign Motorbikes

Figure: Ground Truth matches. We use the harris Affine detector with adistance threshold of 5 pixelsSubhransu Maji Comparison of Local Feature Descriptors



Evaluation of Feature DetectorsEvaluation of Feature Descriptors

Repeatability Results on Benchmarks

Mikolajczyk Dataset: MSER was generally the best followedby Hessian Affine.

Hessian-Affine and Harris-Affine provide more regions than theother detectors, which is useful in matching scenes withocclusion and clutter.

Caltech 101 Dataset: Hessian Affine, Hessian Laplace, MSER,UNIF all perform equally well. Hessian Affine is slightly betterthan others in most cases.

Almost any detector is equally good as the matching is softer.





Desciptor Performance on Mikolajczyk’s Dataset

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1−precision

frac

of c

orre

ctEffect of scale − bikes

gbsiftscspinmomjla

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1−precision

frac

of c

orre

ct

Effect of scale − trees

gbsiftscspinmomjla

(1)bikes (2)trees

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

1−precision

frac

of c

orre

ct

Effect of scale − graf

gbsiftscspinmomjla

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

1−precision

frac

of c

orre

ct

Effect of scale − wall

gbsiftscspinmomjla

(3)graffiti (4)wall





Desciptor Performance on Mikolajczyk’s Dataset

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1−precision

frac

of c

orre

ct

Effect of scale − bark

gbsiftscspinmomjla

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1−precision

frac

of c

orre

ct

Effect of scale − boat

gbsiftscspinmomjla

(5)bark (6)boat

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1−precision

frac

of c

orre

ct

Effect of scale − leuven

gbsiftscspinmomjla

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1−precision

frac

of c

orre

ct

Effect of scale − ubc

gbsiftscspinmomjla

(7)leuven (8)ubc





Desciptor Performance on Caltech 101

0.4 0.5 0.6 0.7 0.8 0.9 10

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

1−precision

frac

of c

orre

ctyin yang

siftscmomglohgbjet

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1−precision

frac

of c

orre

ct

Faces

siftscmomglohgbjet

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.05

0.1

0.15

0.2

0.25

1−precision

frac

of c

orre

ct

Faces Easy

siftscmomglohgbjet

0.95 0.96 0.97 0.98 0.99 10

0.002

0.004

0.006

0.008

0.01

0.012

0.014

0.016

0.018

1−precision

frac

of c

orre

ct

pizza

siftscmomglohgbjet





Desciptor Performance on Caltech 101

0.975 0.98 0.985 0.99 0.995 10

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1−precision

frac

of c

orre

ctbarrel

siftscmomglohgbjet

0.75 0.8 0.85 0.9 0.95 10

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

1−precision

frac

of c

orre

ct

car side

siftscmomglohgbjet

0.4 0.5 0.6 0.7 0.8 0.9 10

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

1−precision

frac

of c

orre

ct

stop sign

siftscmomglohgbjet

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.02

0.04

0.06

0.08

0.1

0.12

0.14

1−precision

frac

of c

orre

ct

Motorbikes

siftscmomglohgbjet





Results on Benchmarks

Mikolajczyk Dataset:1 SIFT and Shape Context do better on wall, bark datasets.2 Geometric Blur(GB) better on bikes, graf datasets3 Both are Comparable on ubc, leuven, boat, trees datasets

Caltech 101 Dataset: GB, Shape Context and SIFT do thebest in all cases.

GLOH which did the best in the Mikolajczyk’s Datasetperforms poorly.

In general the performance in Caltech 101 is much worse thanin Mikolajczyk’s dataset.





Some Observations

The performance difference in significant between SIFT andGB in both 1 and 2.

The performance of SIFT and SC are higly correlated.

The performance of SIFT and GB are higly negativelycorrelated.

Question: Do SIFT, GB carry complimentary information.

When is one more useful than the other?

SIFT does better when there is high texture. High Frequency

Information incorporated better? More experiments required...




Future Work

More flexible notion of Matching, rotations, non-rigidtransformations, etc to incorporate more classes

Extend the analysis to Different Datatsets like PASCAL

A systematic study of the Black Art!




THANK YOU1

1beamer rocks!!Subhransu Maji Comparison of Local Feature Descriptors

comparison of local feature descriptorsyang/courses/cs294-6/maji-presentation.pdfvarious feature...

Documents