second’internaonal’workshop’on’parts’and’a5ributes...
Post on 21-Apr-2018
218 Views
Preview:
TRANSCRIPT
Second Interna,onal Workshop on Parts and A5ributesECCV 2012, Firenze, Italy
October, 2012Discovering a Lexicon of Parts and Attributes
Subhransu MajiResearch Assistant Professor
Toyota Technological Institute at Chicago
Motivation
• Detailed object recognition
• Communication requires a lexicon
Diverse Visual Categories
High%heel(Blue(Shoe(
Berg et al., 10 Farhadi et al, 09
CUB 200 dataset, Visipedia project
Source of part and attribute lexicons
• Field guides provide exhaustive lists when available
• Expert vs. Layman
• Task specific vs. not
Source of part and attribute lexicons
• Captioned images
Limited by sources of such text
Image from Berg et al., ECCV’10
Descriptions are often not visual
What are good attribute lexicons?
• Properties
• It should be easy to communicate
• It should be easy to differentiate instances from one another
images from google, cub 200
Discriminative description task
Describe the (visual) differences between the two
Discriminative description task
Discriminative description task
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
Description
Discriminative description task
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
Description
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
Discriminative Description
Helps elicit a lexicon that enables fine grained discriminationIs task specific by design
Discriminative description task
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
Description
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
Discriminative Description
Helps elicit a lexicon that enables fine grained discriminationIs task specific by design
Non-accidental properties, Biederman 87Pragmatics of Language, Levinson 83
Collecting descriptions on MTurkInterface on AMT
free form text(separated by ‘vs’)
Example annotations
Minimizes instruction biasKeeps the interface simple
Example annotations: airplanespair 82/999; 5 good
facing left facing rightturbofan powered plane propeller powered planelonger tail shorter tailgreen rudder white rudderpassenger door open baggage hold door open"
pair 65/999; 5 good
propeller to the body propeller to the wingone rudder two ruddersthin body fat bodylow wings high wingsfacing towards left side facing slightly towards"
Images from airliners.net
Example annotations: birdspair 10/1600; 5 good
black and white wings spotted wingswhite body spotted bodylarge eyes small eyessmall tail long tailv shaped beak pointed beak"
pair 43/1600; 5 good
yellow black body orange brown bodypointy beak shape beakshort tail long tailblack spot over head brown stripe over headshort leg long leg"
Images from CUB 200 dataset
Analyzing the text: instance specific properties
Different properties are revealed for different instances
bird familyfur color
beak shapetail size
beak color
beak sizetail size
wing colorhead colorleg color
beak colorsitting vs. flyingfeather color
tail sizeleg color
Analyzing the text: instance specific properties
Frequency of usage is a measure of its discriminability
Analyzing the text: instance specific properties
Frequency of usage is a measure of its discriminability
red rudder vs. white rudderpointy nose vs. round nose
sentence pairs
Analyzing the text : discovering a lexicon
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
red rudder vs. white rudderpointy nose vs. round nose
sentence pairs
Analyzing the text : discovering a lexicon
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
ruddernose
parts
{red, white}{pointy, round}
modifiers
red rudder vs. white rudderpointy nose vs. round nose
sentence pairs
red rudder vs. white rudderpointy nose vs. round nose
sentence pairs
Analyzing the text : discovering a lexicon
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
ruddernose
parts
{red, white}{pointy, round}
modifiers
red rudder vs. white rudderpointy nose vs. round nose
sentence pairs
Key ideanouns : words that repeatmodifiers : words that are differentEach sentence has only one noun and modifier
Analyzing the text : discovering a lexicon
Sentence alignment
yellowbeak
blackandwhitebeak
Used in NLP to initialize translation tables (IBM models)
Analyzing the text : discovering a lexicon
Sentence alignment
yellowbeak
blackandwhitebeak
Used in NLP to initialize translation tables (IBM models)
GoalsDiscover a lexicon of parts, modifiers and part-modifier relations
Modifiers should be shared across attributes Estimate the frequency of each attribute
A generative model of sentence pairs
A generative model of sentence pairs
z
a
⇡
f
e
N
J
tI
✓
⌦ �
A generative model of sentence pairs
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
A generative model of sentence pairs
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs
A generative model of sentence pairs
zbeak-color
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs
A generative model of sentence pairs
zbeak-color
ttopic:colortopic:beak
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs
A generative model of sentence pairs
yellowbeak
NULL
ezbeak-color
ttopic:colortopic:beak
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs
A generative model of sentence pairs
yellowbeak
NULL
ezbeak-color
ttopic:colortopic:beak
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs a
A generative model of sentence pairs
yellowbeak
NULL
eblackandwhitebeak
fzbeak-color
ttopic:colortopic:beak
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs a
A generative model of sentence pairs
yellowbeak
NULL
eblackandwhitebeak
fzbeak-color
ttopic:colortopic:beak
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs a
Initialize part and modifier topics using word alignments
Parts, modifiers and attributes of airplanes
wheelwheels plane
engineengines rudder
wingswing
frontback nose facing body tail GLOBAL
onetwono
singlethreedoublefour
colorblackskylight
whiteblueordinarycoloreddark
whitegreenwhitered
pointyroundflat
pointedsharppointsquare
propellerpassenger
jetonly
militarycargo
smallbiglarge
medium
whiteredbluegreenyellowgrayorangebrown
topbottommiddledown
openclosedopenedclose
rightleft
slightly
onnearoff
200 images, 1000 random pairsImages from airliners.net
Parts, modifiers and attributes of airplanes
wheelwheels plane
engineengines rudder
wingswing
frontback nose facing body tail GLOBAL
onetwono
singlethreedoublefour
colorblackskylight
whiteblueordinarycoloreddark
whitegreenwhitered
pointyroundflat
pointedsharppointsquare
propellerpassenger
jetonly
militarycargo
smallbiglarge
medium
whiteredbluegreenyellowgrayorangebrown
topbottommiddledown
openclosedopenedclose
rightleft
slightly
onnearoff
200 images, 1000 random pairsImages from airliners.net
1
Parts, modifiers and attributes of airplanes
wheelwheels plane
engineengines rudder
wingswing
frontback nose facing body tail GLOBAL
onetwono
singlethreedoublefour
colorblackskylight
whiteblueordinarycoloreddark
whitegreenwhitered
pointyroundflat
pointedsharppointsquare
propellerpassenger
jetonly
militarycargo
smallbiglarge
medium
whiteredbluegreenyellowgrayorangebrown
topbottommiddledown
openclosedopenedclose
rightleft
slightly
onnearoff
200 images, 1000 random pairsImages from airliners.net
12
Parts, modifiers and attributes of airplanes
wheelwheels plane
engineengines rudder
wingswing
frontback nose facing body tail GLOBAL
onetwono
singlethreedoublefour
colorblackskylight
whiteblueordinarycoloreddark
whitegreenwhitered
pointyroundflat
pointedsharppointsquare
propellerpassenger
jetonly
militarycargo
smallbiglarge
medium
whiteredbluegreenyellowgrayorangebrown
topbottommiddledown
openclosedopenedclose
rightleft
slightly
onnearoff
200 images, 1000 random pairsImages from airliners.net
12 3
Parts, modifiers and attributes of airplanes
wheelwheels plane
engineengines rudder
wingswing
frontback nose facing body tail GLOBAL
onetwono
singlethreedoublefour
colorblackskylight
whiteblueordinarycoloreddark
whitegreenwhitered
pointyroundflat
pointedsharppointsquare
propellerpassenger
jetonly
militarycargo
smallbiglarge
medium
whiteredbluegreenyellowgrayorangebrown
topbottommiddledown
openclosedopenedclose
rightleft
slightly
onnearoff
200 images, 1000 random pairsImages from airliners.net
12 3localized vs. global
Parts, modifiers and attributes of birds
bird wingsfeatherfeathers tail beak like body leg legs eyes neck head in fur GLOBAL
longshortsmalllargebig
vpointypointedpointbendbendedpointlyslightly
sparrowducksparowcroweagledovepigeonhumming
kiteparrot
brownblueyellowgrayredgreenspottedashlight
blackwhiteorange
fatslimsilmlean
sharproundflat
normalshapedcurvedbluntlittle
roundedordinaryoval
200 images, 1600 pairs1 image per category
CUB 200 dataset
beak-size, wing-color, tail-size, body color, bird type
Parts, modifiers and attributes of people
handhands hair facing
snapsnape
thepicture spectacles in glass bag watch glasses GLOBAL
towardsleftright
backwardsforwardsidewards
manlady
womanboygirlladies
babyadultchildkid
childrenadults
shirttshirtjacketdresscoattshirts
wearingnot
smilinghaving
asiancaucasianafricansasianslatin
western
sidebackfront
backsidefrontaltowardturnturnedrear
sidewaysuprightus
blackwhitebluebrownblonderedgreengrayyellowcoloredpinkorange
singlecouple2non
doublegroupcouplesmany3
darklightfairteeskyshowbrightlighterdesignmediumthick
fatslim
normalthinleanfit
averageskinny
femalemale
fullhalfof
sleeveonlyfully
sleevelessjeensclosepartialrain
thinningtorsowaist
indooroutdoordoorhomelystage
insideoutsideout
handed
longshortsmallshot
womenmen
womens
youngoldoldermiddlematureelderly
twoonethreeboth
somebodyweight
sittingstandingwalkingridingcyclingsleepingdancingdriving
nowith
withoutalonebald
400 images, 1600 pairsRandom images from
PASCAL VOC 11
location, shirt vs tshirt, hair length, gender
Conclusions
• Discriminative description is an effective way to elicit a lexicon of attributes that are useful for fine-grained distinction
• Simple analysis of sentence pairs can help discover
• a lexicon of parts
• a lexicon of modifiers
• relationships between parts and modifiers (attributes)
• the relative frequency of these in a dataset
• A useful tool to bootstrap annotation collection process
top related