attributes for classifier feedback amar parkash and devi parikh
TRANSCRIPT
Attributes for Classifier Feedback
Amar Parkash and Devi Parikh
You can teach a child by examples...
And on, and on, and on…
Is this a giraffe? No.
Is this a giraffe? Yes.
Is this a giraffe? No.
And on, and on, and on…
Proposed Active Learning Scenario
I think this is a giraffe. What do you think?
No, its neck is too short for it to be a giraffe.
Ah! These must not be giraffes
either then.
[Animals with even shorter necks]
……
Current belief Focused feedbackKnowledge of the world
Feedback on one, transferred to many
• Learner learns better from its mistakes• Accelerated discriminative learning with few examples
Communication
Need a language that is• Machine understandable• Human understandable
Attributes!• Mid-level shareable• Visual • Semantic
Proposed Active Learning
Concepts to teach:
C1 C2 CK…
Unlabeled pool of images
Classifiers:
h1 h2 hK…
[Label-feedback][Attributes-based feedback]
Attribute predictors:
a1 a2 aM…
[Predicted Label]
Any feature spaceAny discriminative learning algorithm
Relative Attributes[Parikh and Grauman, ICCV 2011]
Openness
Unlabeled pool of images
Attribute predictors:
a1 a2 aM…
Image featuresParameters
Attributes-based Feedback
Unlabeled pool of images
Unlabeled pool of images
No,It is too open to be a forest
Attribute predictors:
a1 a2 aM…
Forest
Openness
Not Forest
Attributes-based Feedback
Unlabeled pool of images
Unlabeled pool of images
No,It is too open to be a forest
Attribute predictors:
a1 a2 aM…
Forest
Classifiers:
h1 h2 hK…
Not Forest
Proposed Active Learning
Concepts to teach:
C1 C2 CK…
Unlabeled pool of images
Classifiers:
h1 h2 hK…
[Label-feedback][Attributes-based feedback]
Attribute predictors:
a1 a2 aM…
[Predicted Label]
Label-based Feedback
• Not our contribution
• Experiment with different scenarios
• Benefits of attributes-based feedback– Small when label-based is very informative– Large when label-based feedback is weak
Unlabeled pool of images
Forest
Label-based Feedback
• Accept: Yes, this is a forest.– Strong: It is not anything else
Example: Classification– Weak: It can be other things
Example: Annotation
Label-based Feedback
• Reject: – Strong: No, it is a coast.
Example: Classification with few classes– Weak: No, this is not a forest.
Example: Large-scale classificationExample: Biased binary classification
• 4 different scenarios in experiments
Unlabeled pool of images
Forest
Datasets
• Datasets and relative attribute predictors from [Parikh and Grauman, ICCV 2011]
Datasets
• Faces: – 8 celebrity categories– 11 attributes (chubby, white, etc.)
[Kumar et al., ICCV 2009]
Datasets
• Scenes:– 8 categories– 6 attributes (open, natural, etc.)
[Oliva and Torralba, IJCV 2001]
Settings
• Feedback from MTurk• Features:– Raw image features (gist, color)– Attribute scores
• Category classifiers: SVM with RBF kernel• Results on– 2 datasets x 2 features x 4 label-feedback scenarios– Show 2 here, rest in paper.
This image doesn’t have enough perspective to be a street scene.
0 10 20 30 40 5025
35
45
55
65
75
Baseline
Proposed (Human)
Proposed (Oracle)
Results
Faces, Attribute features, Strong label-feedback
# iterations
Accu
racy
0 10 20 30 400
10
20
30
40
Baseline
Proposed (Human)
Proposed (Or-acle)
Results
Scenes, Image features, Weak label-feedback
# iterations
Accu
racy
1/4th !
More results in the paper.
Conclusion
• Attributes for providing classifier feedback• Novel learning paradigm with enhanced
human-machine communication• Discriminative learning + domain knowledge• Learning with few examples
• Connections to semi-supervised learning– Shrivastava, Singh and Gupta: Up Next!
Thank you!
Backup Slides
Negative Feedback
• Many reasons come together to make a concept– Hard to describe why an image is a concept
• One reason can break a concept– Easier to describe why an image is not a concept
• (Arguably) waste to give feedback when right
Discriminative– No domain knowledge
is conveyed– Many training images– Discriminative model– Classification in any
feature space– State-of-art
performance
Related WorkZero-shot learning– Convey domain
knowledge – Zero training images– Generative model– Classification in
attribute space– Performance
compromised
Proposed paradigm– Convey domain
knowledge to transfer– Few training images– Discriminative model– Classification in any
feature space– Performance
maintained
bear
turtle rabbit
furry
big
[Lampert et al., CVPR 2009]
C
Smili
ng
Age
S
J H
C is younger than H C smiles more than H
MM is younger than J
M smiles more than J
[Parikh and Grauman, ICCV 2011]++
+–
– –
Related Work
Focused discrimination– Mining of hard negatives [Felzenszwalb 2010]– To understand classifier [Golland 2001]– Here, supervisor provides the discriminative
direction by verbalizing semantic knowledge
[Golland, NIPS 2001][Felzenszwalb et al., CVPR 2008]
Related Work
What we are not doing:• Collecting deeper annotations of images
• Segmentation masks [Russell 2008], Parts [Farhadi 2010], Pose [Bourdev 2009], Attributes [Kumar 2009]• We use attributes for broad propagation of category
labels to unlabeled images
Related Work
What we are not doing:• Actively interleaving attribute annotations
• Object & attributes [Kovashka 2011], Image, boxes & segments [Vijaynarsimhan 2008], Parts & attributes [Wah 2011], etc.• Human-in-the-loop at test time [Branson 2010]• Our supervisor provides additional information at
training time which is leveraged for better category models
Related Work
• Rationales– Human feature selection in NLP [Raghavan 2005]– Spatial and attribute rationales [Donahue 2011]– Restricted to classification in attribute space– We can operate in any feature space
[Donahue and Grauman, ICCV 2011]
Imperfect Attribute Predictors
• In the end, all images labeled with ground truth
• Discriminative training can deal with outliers• Attributes are pre-trained, and so unlikely to
be severely flawed • Experiments: used predictors directly from
Parikh and Grauman, 2011
Large-scale Classification
• Categories may require expert knowledge, attributes need not
• Can show a few exemplars to verify category
0 10 20 30 40 5025
35
45
55
65
75
Baseline
Proposed (Human)
Proposed (Oracle)
Zero-shot
Results
Faces, Classification, Attribute features
(overestimate)
# iterations
Accu
racy
Label-based Feedback
Classification Large-scale Classification
Annotation
StreetOutdoor
CityGrayscale
….
Biased Binary Classification
• Simulate the different scenarios in responsesBenefits of attributes-based feedback• Small when label-based is very informative• Large when label-based feedback is weak
0 10 20 30 4042444648505254
Baseline
Proposed (Human)
Proposed (Or-acle)
Results
Scenes, Annotation, Image features
# iterations
Accu
racy
0 5 10 15 2025
35
45
55
65
75
Baseline
Proposed (Human)
Proposed (Or-acle)
Results
Faces, Biased binary classification, Attribute features
# iterations
Accu
racy
More results in the paper.
Traditional Active Learning
Concepts to teach:
C1 C2 CK…
Unlabeled pool of images
Classifiers:
h1 h2 hK…
[Label]Highest Entropy