Enhancing Exemplar SVMs usingPart Level Transfer Regularization

1

Problem Definition:

Image Retrieval

2

Problem Definition:

Image Retrieval

query

3

Problem Definition:

Image Retrieval

query

Image Database

Retrieved Images

query Retrieved Images

Retrieving same category in a similar pose

Example: bicycle facing left

4

A Candidate Solution:

Exemplar SVM (E-SVM)

Training a SVM with a single positive and many negative samples

Linear SVMsover

HoG features

[Dalal &Triggs’05], [Felzenszwalb’08] Exemplar SVM

5

[Malisiewicz’11][Shrivastava’11]

A Candidate Solution:

Exemplar SVM (E-SVM)

Linear SVMsover

HoG features

[Dalal &Triggs’05], [Felzenszwalb’08] Exemplar SVM

Training a SVM with a single positive and many negative samples

Retrieval via sliding window search on the image database

Imag

e D

atab

ase

6

A Candidate Solution:

Exemplar SVM (E-SVM)

Retrieval via sliding window search on the image database

Imag

e D

atab

ase

7

Linear SVMsover

HoG features

[Dalal &Triggs’05], [Felzenszwalb’08] Exemplar SVM

Training a SVM with a single positive and many negative samples

Retrieved Images

Framework:

Enhanced Exemplar SVM (EE-SVM)positive sample

negative samples

Train E-SVMover

HoG features

Part-LevelTransfer

Enhanced E-SVM

Exemplar SVM Previously Trained Classifiers

8

Benefit:

Enhanced Exemplar SVM (EE-SVM)Exemplar SVM

Retrieved Subwindows

Enhanced E-SVM

SubwindowRetrieval

SubwindowRetrieval

Image Database

Retrieved Subwindows

9

Que

ry Im

age

Overview

• Transfer Learning in Computer Vision– Classification & Detection

• Enhanced Exemplar SVM

• Feature Augmentation vs Transfer

• Results & Discussion10

Transfer Learning in Computer Vision

• Image Classification– Adaptive SVMs, – Transfer from Multiple Models, – Adaptive Multiple Kernel Learning

• Object Detection– Rigid Transfer– Flexible Transfer

13

[Yang et al. ICDM’07][Tommasi et al. BMVC’09] [Tommasi et al. CVPR’10] [Luo et al. ICCV’11][Duan et al. CVPR’10]

[Stark et al. ICCV’09][Aytar and Zisserman ICCV’11][Gao et al. ECCV’12]

Learning new classes by building upon previously learned classes.

Transfer Learning for Detection

• Rigid Transfer [Aytar and Zisserman ICCV’11]– Transfer between fixed sized templates– Good performance, especially for smaller number of training samples.– Hard to find visually similar detectors with same aspect ratio and size.

• Flexible Transfer – Transfer between different sized templates.– Transferring shape features [Stark et al. ICCV’09]– Deformable Transfer [Aytar and Zisserman ICCV’11] – Transfer via Structured Priors [Gao et al. ECCV’12]

Fixed SizedTransfer

14

FlexibleTransfer

Overview

• Transfer Learning in Computer Vision– Classification & Detection

• Enhanced Exemplar SVM

• Feature Augmentation vs Transfer

• Results & Discussion15

Framework:

Enhanced Exemplar SVM (EE-SVM)

Train E-SVM

Part-LevelTransfer

Enhanced E-SVM Exemplar SVM

Prev

ious

ly T

rain

ed

Cla

ssifi

ers

16

Query

Framework:

Part-Level Transfer Regularization

17Exemplar SVMui

ui

Parameters:

Part-Level Transfer Regularization

18

close to E-SVMclose to construction from ui’s

Framework:

Matching Classifier Patches

19

Previously Learned Classifiers ui

Exemplar SVM

Why is it beneficial?

Part-Level Transfer Regularization• Part level transfer is beneficial because…

– parts can be relocated (deformation), – the possibility of finding a good match for transfer increases when we

look at smaller classifier patches.

• Advantages of transferring parts from well trained classifiers:– Better background suppression and discriminativity due to well trained

source classifiers.– Better handling of local variations since source classifiers are trained

on many positive samples.

• No additional cost on runtime

20

• Unusual Poses

• Composition of Objects [Visual Phrases - Sadeghi CVPR’11]

21

Where is it beneficial?

Part-Level Transfer Regularization

PASCAL 2007:

Results - Left Facing Horse

22

query

E-SVM

Enhanced E-SVM

PASCAL 2007:

Results - Left Facing Bicycle

23

E-SVM

Enhanced E-SVM

query

PASCAL 2007:

Visual Phrase – Riding Horse

24

query

E-SVM

Enhanced E-SVM

ImageNet:

Unusual Pose - Bicycle

25

E-SVM

Enhanced E-SVM

query

Overview

• Transfer Learning in Computer Vision– Classification & Detection

• Enhanced Exemplar SVM

• Feature Augmentation vs Transfer

• Results & Discussion27

. . . .

Implementation:

Transfer vs. Feature Augmentation

29

0.2 0.7 0.1 . . .

is equivalent to learning

Transfer Regularization

“normal” SVM with augmented features.

Implications:

Transfer vs. Feature Augmentation

• This equivalence is not specific to Exemplar SVMs.

• Transfer regularization can be implemented as feature augmentation.

• Transfer regularization can be efficiently solved using standard SVM packages.

30

Overview

• Transfer Learning in Computer Vision– Classification & Detection

• Enhanced Exemplar SVM

• Feature Augmentation vs Transfer

• Results & Discussion31

PASCAL 2007:

Quantitative Results

32

ImageNet:

Quantitative Results

• Three queries are evaluated for each of the five classes.• Precisions at top 5, 10, 50 and 100 are reported.

33

Handling Occlusions

34

Que

ryE-

SVM

EE-S

VM

Que

ryE-

SVM

EE-S

VM

Handling Truncation

35

36

Conclusions

• Boosted the performance of E-SVM which incurs no additional cost on runtime.

• Presented the equivalence between Transfer regularization and feature augmentation.

• Showed the benefit for unusual poses and visual phrases.

• Handling truncation and occlusion.