`  

Adding  Unlabeled  Samples  to  Categories  by  Learned  A6ributes  h6p://umiacs.umd.edu/~jhchoi/addingbya6r  

[1]  Salakhutdinov,  Torralba,  Tenenbaum,  “Learning  to  Share  Visual  Appearance  for  MulBclass  Object  DetecBon”,  CVPR  2011  |  [2]  Rastegari  et  al.,  “ANribute  Discovery  via  Predictable  DiscriminaBve  Binary  Codes”,  ECCV  2012  |  [3]  Malisiewicz,  Gupta,  Efros,  “Ensemble  of  Exemplar-­‐SVMs  for  Object  DetecBon  and  Beyond”,  ICCV  2011  

Be6er  classificaAon  model  

To  obtain  a  be6er  classifier  

More  training  data  

Jonghyun  Choi        Mohammad  Rastegari        Ali  Farhadi          Larry  S.  Davis  

min

Ic2I,wvc ,w

ac

X

c

⇣↵J

vc (Ic, w

vc ) + �J

ac (Ic, w

ac )

⌘+M(I)

subject to

J

vc (Ic, w

vc ) = kwv

ck22 + �v

nX

i=1

⇠c,i

Ic,i · yc,i(wvcxi) � 1� ⇠c,i, 8i 2 {1, . . . , n}

J

ac (Ic, w

ac ) = kwa

c k22 + �a

nX

j=1

⇣c,j �nX

k=l+1

Ic,k

⇣w

ac�(xk)

⌘

Ic,j · yc,j(wac�(xj)) � 1� ⇣c,j , 8j 2 {1, . . . , n}

nX

k=l+1

Ic,k �, Ic,k = 1, 8k 2 {1, . . . , l}

M(I) =

XX

c1 6=c2

Ic1 · Ic2.•  No human required •  No underlying

distribution assumed

Max-­‐margin  Classifier  on  visual  feature  space  

Max-­‐margin  Classifier  on    a6ribute  space  

Top-­‐Lambda  Selector  

Mutual  Exclusion  

Joint  opBmizaBon  for  discovering  discriminaBve  aNributes  and  unlabeled  samples    

Use$all$Except$each$

Score$high$

Given$Training$Samples$

•  AcBve  learning  •  Require  human  in  the  loop  

•  Semi-­‐supervised  learning  •  Unlabeled  samples  are  

assumed  to  follow  the  same  distribuAon  of  labeled  samples  

By learning data driven attributes based on [2]

There  are  many  categories  that  have    a  few  training  samples  

Category Name Init. NN ALC Cat. E+CMashed Potato 45.03 34.02 51.15 61.39 63.92Orange 29.84 16.29 26.97 40.61 41.05Lemon 32.21 27.58 32.43 35.37 34.23Green Onion 25.06 16.50 19.66 38.57 40.20Acorn 13.09 11.05 15.41 19.35 20.10Co↵ee bean 58.29 43.89 56.62 64.65 66.54Golden Retriever 14.54 15.57 12.61 17.54 18.61Yorkshire Terrier 29.62 13.62 27.63 41.41 45.65Greyhound 15.24 15.73 15.64 14.75 15.22Dalmatian 43.84 27.97 37.91 54.42 57.23Miniature Poodle 26.10 12.50 21.16 28.87 30.21Average 30.26 21.34 28.84 37.90 39.36

20 30 40 50 60 70

30

31

32

33

34

35

36

37

38

39

40

Number of Sample Added (!)

Me

an

Av

era

ge

Pre

cis

ion

(%

)

Init. Set+ by C Only+ by E+C

0 5 10 15 20 25 30 35 40 45 50 5520

25

30

35

40

45

50

55

Size of Initial Labeled Set

Me

an

Av

era

ge

Pre

cis

ion

(%

)

Init. Set

+ by C Only

+ by E+C

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Lab

el P

uri

ty

Mashed P

otato

Orange

Lemon

Green O

nion

Acorn

Coffee B

ean

Golden R

etriever

Yorkshire

Terri

er

Greyhound

Dalmatia

n

Min

iatu

re P

oodle

Average

+ by C only

+ by E+C

0

10

20

30

40

50

60

70

Av

era

ge

Pre

cis

ion

(%

)

Mas

hed P

otato

Ora

nge

Lemon

Gre

en O

nion

Acorn

Coffee

Bean

Gold

en R

etrie

ver

Yorksh

ire T

errie

r

Gre

yhound

Dalm

atia

n

Min

iatu

re P

oodle

Avera

ge

Init. SetE!SVMOurs

This research is partly supported by ONR MURI

Grant No. N00014-10-1-0934

Given  labeled  samples  

A6ribute  Mapper  

Number  of  training  samples  in  SUN  09  Dataset[1]  

•  ‘Init.’:  iniAal  labeled  training  set.    •  ‘NN’:  addiAon  by  ‘nearest  neighbor’  in  visual  feature  space  •  ‘ALC’:  addiAon  by  ‘acAve  learning  criteria  (ALC)’  that  finds  the  examples  close  to  the  current  decision  hyper-­‐planes    •  ‘Cat.’:  our  method  of  select  examples  using  categorical  a6ributes  only.    •  ‘E+C’:  addiAon  using  both  categorical  and  exemplar  a6ributes.    

Comparison  with  other  methods  

•  Red:  Using  the  iniAal  labeled  set  (Init.  Set)  •  Green:  The  augmented  set  by  our  method  using  categorical  at-­‐  tributes  only  (+  by  C  only)  •  Blue:  Categorical+Exemplar  a6ributes  respecAvely.  (+  by  E+C)    

Varying  number  of  added  images  

Purity  of  added  examples  Categorical  a6ributes  only  (+  by  C  only)    Categorical+Exemplar  a6ributes  (+  by  E+C).    

Low  Purity  sAll  Improves  Accuracy  

Size  of  IniAal  Labeled  Set  

0

10

20

30

40

50

60

70

80

90

Mea

n A

vera

ge

Pre

cisi

on

(%

)

Mas

hed P

otato

Ora

nge

Lemon

Gre

en O

nion

Acorn

Coffee

Bean

Gold

en R

etrie

ver

Yorksh

ire T

errie

r

Gre

yhound

Dalm

atia

n

Min

iatu

re P

oodle

Avera

ge

Init. Set

+ Similar Only

+ 50 Similar Only

+ 50 by Our Method (C)

+ 50 GND Samples

The  mAP  gain  for  the  smallest  iniAal  labeled  set  (5)  is  the  highest  as  expected.  When  the  number  of  samples  is  larger  than  25,  our  method  (+  by  C  only)  does  not  improve  the  mAP  much,  although  it  sAll  improves  by  1.18  −  2.74%.    

Our  exemplar  a6ribute  discovery  method  (Sec.  4.2)  to  Exemplar  SVM[3]  

Our  method  outperforms  the  exemplar  SVM  in  terms  of  category  recogniAon  accuracy  by  APs  without  the  extra  large  negaAve  example  set  (size  =  50,000).    

Retrieval  rank  change  by  the  absence  of  a  training  sample  

ej(xi) =µ

rg(xi)� ⌫

rj(xi)

More  stable  than  Exemplar-­‐SVM  w/o  large  negaAve  set  •  Non-­‐convex:  I  is  discrete,  w’s  are  conBnuous  

•  Solve  by  block  coordinate  descent  

•  Navy:  using  the  iniAal  labeled  set  (baseline).    •  Blue:  using  only  similar  examples  among  the  selected  50  examples.    •  Green:  using  50  similar  examples  to  compare  with  the  result  of  our  selected  50  examples    

•  Orange:  including  both  similar  and  exact  examples.    •  Red:  using  a  set  of  50  ground  truth  images,  which  is  the  best  achievable  accuracy  (upper  bound).  

Training'Samples'

Build'A2ribute'Space'

Project'

Find'Useful'A2ributes'

Unlabeled'Samples'!!

!!

!!

!!Project'

Choose'Confident'Examples' Selected)by)Categorical)A+ributes)

Selected)by)Exemplar)A+ributes)

Ini=al)Labeled)Training))Examples)

 (Full-­‐set  vs  Leave-­‐one-­‐out  Set)