style-finder fine-grained clothing style detection and retrieval
TRANSCRIPT
Style-Finder: Fine-‐Grained Clothing Style Detec5on and Retrieval
Wei Di*, Catherine Wah+, Anurag Bhardwaj*, Robinson Piramuthu*, Neel Sundaresan*
6/30/13
* eBay Research Labs, Computer Vision Group + Department of Computer Science and Engineering, UCSD
research labs
research labs
Outline
2
q Motivation
q Human annotation of attributes
q Fine-grained classification
q Search by attribute/style
q Text generation
research labs
Motivation - Mobile
3
Vs
Retrieve Clothing with similar Style
Old Navy IVORY LONG PEA COAT WOOL NWT
Women Double-Breasted Long Slim Wool Trench Jacket Coat Outwear
New Navy Blue Double Breasted Long Peacoat Jacket Pea Coat
L.K. Bennett Ami Burgundy Coat
research labs
Coats & Jackets in various Shapes and Forms
Motivation - Style
4
Color Pattern Style
Aeropostale wool peacoat
research labs
Flow: Data Collection, Classification /retrieval /Annotation
Define style elements, Crowd Sourcing
Detected Style:
- Search: Visually similar rank - Annotation: Describe style
Style Modeling, Classification
Local + Global
+
SVM-based style element
detector
5
AMT
…
research labs
Ambiguity (class level)
Challenges in Data
High variability • Shape • How it’s worn/ photographed
Label noise/incomplete
Less detailed descriptions or incorrect labeling
Is this a (a) Puffer jacket (b) Vest (c) Both?
6
research labs
Data collection: Coats/Jackets Dataset
q Harvested images from 18+ online clothing retailers § Used retailer category labels and/or search terms
q Data § Well-aligned, clean images § 2k images, 12 sub-category:
blazer, cape, jean, military, motorcycle, parka, peacoat, poncho, puffer, raincoat, trench, vest.
7
research labs
Round V-shape
Simple Collar
Turtle V-Shirt
Folded Collar
Notched Shawl Peak Round Shirt
Human Annotation of Attributes
Has fur Denim Leather Shiny Wool
Material
Zip Button Open Has belt
Fastener
8
research labs
Human Annotation of Attributes
9
Attribute type Attributes Material Fur, Denim, Leather/is leather-like, Shiny, Wool/is woolen
or felt-like Fastener Zip, Button, Open, Has belt Fastener style Symmetrical (single-breasted), Asymmetrical (single-breasted),
Asymmetrical (double-breasted) Length Short, Medium, Long Cut Fitted, Loose Pocket Chest pocket, Side pocket Collar V-neck collar, Round collar, Turtle neck, V-neck shirt collar,
Round shirt collar, Notched collar, Shawl collar, Peak collar
- 5 annotations per image - Majority agreement on the presence or absence of the
attribute - 27 binary attributes
research labs
Statistics of Crowd Sourcing
10
Average Attributes per Cloth Image
Co-occurrence of Style Attributes
research labs
Fine Grained Classification : Kernel Mapping q Visual features
- SIFT, HOG, GIST, etc - Exclude color features
q Kernelized bag-of-words features
– Kernel trick deal with linear non-separable data – Homogeneous Kernel Mapping – Low dimensional kernel approximations
11
Linear vs non-linear kernels 4
Linear SVM
✔ fast✘ restrictive
Non-linear SVM
✘ much slower✔powerful
>=< xwx ,)(F
∑=
=N
iiiKF
1),()( xxx α
xwxxxxx Tl
kki
Tl
kiif +=+=+= ∑∑
==
βαβαβ11
,)(
∑=
=N
iiiKF
1),()( xxx α
( ))(ˆ),(ˆ),( iiK xxxx ψψ≈
Approximated feature map
)(ˆ,)( xwx ψ=F )(ˆ1
xw ψα∑=
=N
ii
xxxxxxkʹ′+
ʹ′=ʹ′2),(
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=Φ
)log6.0sin(6.0)log6.0cos(6.0
8.0)(
xxxx
Homogeneous kernel map Closed form, simple, small
X2 Kernel Excellent for bag-of-words…
Binary classifier
research labs
Fine Grained Classification : Classifier
Offline: learn binary SVM for each style type
Has style-X Not Has
Style-X
q Visual features (SIFT, HOG, etc)
q Kernelized bag-of-words features
q Binary classifier for each attribute
- Explicit kernel features
- Max-margin Linear SVM
- Fast
12
research labs
Style Recognition/ Retrieval
Input image
At test time:
Has/Is Style-X?
YES
Ranking by attribute similarity
User ask questions: coats with collars like this
q Visual features (SIFT, HOG, etc) q Kernelized bag-of-words features
– Homogeneous Kernel Mapping - Binary classifier - Linear SVM
q Retrieval & Annotation - Recognition - Rank by visual similarity
13
research labs
Result Evaluation q Evaluated by
– Classification accuracy – Precision-recall
14
research labs
Is-shiny
Wool/ wool-like
Query Retrieval Example
Has-fur
15
research labs
Has-button
Has-zip
Has-belt
Query Retrieval Example
16
research labs
Automatic Style Element Attribute Annotation
Image Automatic Generated Descriptions
(a) Shiny, short
(b) Has-fur, leather/is leather-like
(c) Open, loose, round collar
(d) Button, short, chest pocket, v-neck shirt collar
(a) (b) (c) (d)
17
research labs
Automatic Style Element Attribute Annotation
Image Automatic Generated Descriptions
(a) Button, has-belt, asymmetrical and double-breasted, long, slim, v-neck shirt collar, round shirt collar
(b) Wool/is woolen or felt-like, zip, loose, round collar
(c) Has-fur, v-neck-collar
(d) Symmetrical and single-breasted, shawl collar
(a) (b) (c) (d)
18
research labs
Future Work
q Joint learning of Attribute (ex. collar) + Category (ex: coats)
q Multi-attribute query retrieval
q Automatic style element localization
q Personalization based on style
19
research labs 20
Thanks!