Gustavo Carneiro

The Automatic Design of Feature Spaces for Local Image Descriptors using an

Ensemble of Non-linear Feature Extractors

Set of Matching Problems

Visual Object Recognition Visual Class Recognition

Wide baseline Matching

Set of Matching Problems1- Design a feature space that facilitates certain matching problems

SIFT [Lowe,ICCV09]

Shape Context[Belongie et al. PAMI02]

HOG [Dalal & Triggs,CVPR05]

Set of Matching Problems2- Given a matching problem, and a set of feature spaces, combine them in order to minimize probability of error (mismatches) [Varma & Ray, CVPR07]

Target Matching Problem

SIFT

Shape Context HOG

Set of Matching Problems3- Given a matching problem, find the feature space and respective parameters θ that minimizes probability of error (mismatches) [Hua et al.ICCV07]

Target Matching Problem

FeatureTransform 2 (θ*)Feature

Transform 1 (θ)FeatureTransform 2 (θ)

FeatureTransform 1 (θ*)

Set of Matching Problems4- Given future unknown matching problems, find the feature space that minimizes probability of error (mismatches)

Feature TransformMatchingProblem 1

FeatureTransformMatchingProblem 2

Feature TransformMatchingProblem 3

FeatureTransformMatchingProblem 4

FeatureTransformMatchingProblem 5

Target Matching Problem 1

Target Matching Problem 2

The Universal Feature Transform

• Solve random and simple matching problems

• The more matching problems solved, the easier it will be to solve new problems

• Restriction: problems should be in similar feature ranges and similar class statistics

(Linear) Distance Metric Learning[Chopra et al.CVPR05,Goldberger et al.NIPS04, Weinberger & Saul JMLR09]

• Linear transform:

• Image patches:

• Distance in T space:

(Non-Linear) Distance Metric Learning [Sugiyama JMLR07]

• Rewrite S(b) and S(w):

• By taking the following transformation:

• Generalized Eigenvalue Problem:

Dot product replaced by non-linear kernel function

• Feature Transform

Linear vs Non-linear DML

LINEAR

NON-LINEAR

Points from the sameclass collapse and are farfrom each other

Points not belongingto any class collapse at the origin

Intuition

• Train several feature transforms– Random matching problems

• Aggregate distances [Breiman 01]: –

• Threshold-based classifier–

T

IntuitionUnkown target problem

Random training problem 1

Small dist.

Large dist.

ROCAggregated distances

T

IntuitionUnkown target problem

Random training problem 2

Small dist.

Largedist.

ROCAggregated distances

Toy Example

• Combing 100 feature spaces...

Error decreases with number of feature spacesNo matter the error for each space

UFT

Original

NLMSL trained

Experiments• Dataset of for training [Winder & Brown,CVPR07]:

– Backprojecting 3D points to 2D images from scene reconstructions – Variations in scene location, brightness and partial occlusion– Similar pre-processing of [Winder & Brown,CVPR07]

– Train: all patch classes from Trevi & Yosemite dataset– Test: 50K matching and 50K non-matching pairs from Notre Dame dataset

Experiments• Using cross validation– 50 training classes for training each feature space– 50 training feature spaces

Error decreases with number of feature spacesNo matter the error in each spaceUFT (2.28%) SIFT (6.3%) @95% TP

Experiments• Matching database [Mikolajczyk & Schmid,PAMI’05]

Conclusion

• Competitive performance

• Simple ensemble classifier (can be efficiently implemented)

• Adapt to new classification problems (no re-training)

Linear vs Non-linear DML

LINEAR

NON-LINEAR

10 runs, 100 points per classClassifier: threshold matching

Non-linear: low bias, high varianceLinear: High bias, low variance

Combining Feature Spaces

• Breiman’s idea about ensemble classifiers [Breiman 01]: – combine low-bias, high-variance (unstable) classifiers to

produce low-bias, low-variance classifiers.

• Distance