adaptation for objects and attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · adaptation...
TRANSCRIPT
![Page 1: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/1.jpg)
Adaptation for Objects and Attributes
Kristen GraumanDepartment of Computer Science
University of Texas at Austin
With Adriana Kovashka (UT Austin), Boqing Gong (USC), and Fei Sha (USC)
![Page 2: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/2.jpg)
Learning-based visual recognition
Last 10+ years: impressive strides by learningappearance models (usually discriminative).
Annotator
Training images
New imageCAR
CAR
NOT CAR
Image features
CAR!
![Page 3: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/3.jpg)
Typical assumptions
1. Test set will look like the training set.2. Human labelers “see” the same thing.
![Page 4: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/4.jpg)
Mismatched domains
TRAIN TESTFlickr YouTube
![Page 5: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/5.jpg)
TRAIN TESTCatalog images Mobile phone photos
Mismatched domains
![Page 6: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/6.jpg)
TRAIN TESTImageNet PASCAL VOC
Mismatched domains
![Page 7: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/7.jpg)
TRAIN TESTImageNet PASCAL VOC
“It is worthwile to note that, even with 140Ktraining ImageNet images, we do not perform as well aswith 5K PASCAL VOC training images.”
– Perronnin et al. CVPR 2010
Mismatched domains
![Page 8: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/8.jpg)
Problem: Poor cross-domain generalization• Different underlying distributions
• Overfit to datasets’ idiosyncrasies
Possible solution: Unsupervised domain adaptation
Mismatched domains
![Page 9: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/9.jpg)
SetupSource domain (with labeled data)
Target domain (no labels for training)
ObjectiveLearn classifier to work well on the target
Unsupervised domain adaptation
Different distributions
![Page 10: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/10.jpg)
Much recent research
Correcting sampling bias
[Shimodaira, ’00]
[Huang et al., Bickel et al., ’07][Sugiyama et al., ’08]
[Sethy et al., ’06]
[Sethy et al., ’09][This work]
Adjusting mismatched models
[Evgeniou and Pontil, ’05]
[Duan et al., ’09][Duan et al., Daumé III et al., Saenko et al., ’10]
[Kulis et al., Chen et al., ’11]
+-
---++
+
----++
Inferring domain-invariant features
[Pan et al., ’09]
[Blitzer et al., ’06] [Gopalan et al., ’11][Chen et al., ’12][Daumé III, ’07]
[Argyriou et al, ’08] [Gong et al., ’12][Muandet et al., ’13]
+++-
- +-+- +
![Page 11: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/11.jpg)
Existing methods attempt to adapt allsource data points, including “hard” ones.
Problem
Source Target
![Page 12: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/12.jpg)
Automatically identify the “most adaptable” instances
Use them to create series of easier auxiliary domain adaptation tasks
Our idea
[Gong et al., ICML 2013]
ProblemExisting methods attempt to adapt allsource data points, including “hard” ones.
![Page 13: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/13.jpg)
Landmarks are labeled source instances distributed similarly to the targetdomain.
Landmarks
Source
Target[Gong et al., ICML 2013]
![Page 14: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/14.jpg)
Landmarks are labeled source instances distributed similarly to the targetdomain.
Roles:Ease adaptation difficulty
Provide discrimination (biased to target)
Source
Target
Landmarks
[Gong et al., ICML 2013]
![Page 15: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/15.jpg)
Landmarks
Target
Source
1 Identify landmarksat multiple scales.
Key steps
Coarse
Fine-grained
[Gong et al., ICML 2013]
![Page 16: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/16.jpg)
2 Construct auxiliary domain adaptation tasks
3
Obtain domain-invariant features
4
Predict target labels
Key steps
[Gong et al., ICML 2013]
![Page 17: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/17.jpg)
Objective
Identifying landmarks
Source
Target[Gong et al., ICML 2013]
![Page 18: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/18.jpg)
Maximum mean discrepancy (MMD)
Empirical estimate [Gretton et al. ’06]
a universal RKHS
kernel function induced by
the l-th landmark (from the source domain)
[Gong et al., ICML 2013]
![Page 19: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/19.jpg)
Integer programming
where
Method for identifying landmarks
[Gong et al., ICML 2013]
![Page 20: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/20.jpg)
Convex relaxation
Method for identifying landmarks
[Gong et al., ICML 2013]
![Page 21: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/21.jpg)
Gaussian kernelsHow to choose the bandwidth?
Our solution:Examine distributions at multiple granularities
Multiple bandwidthsmultiple sets of landmarks
Scale for landmark similarity?
[Gong et al., ICML 2013]
![Page 22: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/22.jpg)
Landmarks at multiple scales
22
Headphone Mug
target
TargetS
ource
6σ=20σ=2
-3σ=2
Unselected
[Gong et al., ICML 2013]
![Page 23: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/23.jpg)
2 Construct auxiliary domain adaptation tasks
Key steps
[Gong et al., ICML 2013]
![Page 24: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/24.jpg)
Constructing easier auxiliary tasks
Target
Source
Landmarks
At each scale σ
Intuition: distributions are closer (cf. Theorem 1)
[Gong et al., ICML 2013]
![Page 25: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/25.jpg)
At each scale σ
Intuition: distributions are closer (cf. Theorem 1)
New target
New source
Landmarks
Constructing easier auxiliary tasks
[Gong et al., ICML 2013]
![Page 26: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/26.jpg)
- Integrate out domain changes-Obtain domain-invariant representation [Gong, et al. ’12]
Each task provides new basis of features via geodesic flow kernel (GFK):
Constructing easier auxiliary tasks
[Gong et al., CVPR 2012]
![Page 27: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/27.jpg)
2 Construct auxiliary domain adaptation tasks
3
Obtain domain-invariant features
MKL
Key steps
![Page 28: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/28.jpg)
Multiple kernel learning on the labeled landmarks
Arriving at domain-invariant feature space
Discriminative loss biased to the target
Combining features discriminatively
![Page 29: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/29.jpg)
2 Construct auxiliary domain adaptation tasks
3
Obtain domain-invariant features
4
Predict target labels
Key steps
![Page 30: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/30.jpg)
Four vision datasets/domains on visual object recognition[Griffin et al. ’07, Saenko et al. 10’]
Four types of product reviews on sentiment analysisBooks, DVD, electronics, kitchen appliances [Biltzer et al. ’07]
Experiments
![Page 31: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/31.jpg)
Cross-dataset object recognition
![Page 32: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/32.jpg)
Cross-dataset object recognition
![Page 33: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/33.jpg)
Cross-dataset object recognition
![Page 34: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/34.jpg)
Datasets as domains?
Domain 1 Domain 2
Domain 3
Domain 4Domain 5
ASSUMED
![Page 35: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/35.jpg)
Datasets as domains?
Domain 1 Domain 2
Domain 3
Domain 4Domain 5
Domain 5Domain 6
Domain 7Domain 8
Domain 9
Domain 10
REALITY
![Page 36: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/36.jpg)
Datasets as domains?
Domain 1 Domain 2
Domain 3
Domain 4Domain 5
Domain 5Domain 6
Domain 7Domain 8
Domain 9
Domain 10
Dataset != DomainCross-dataset adaptation is suboptimal
REALITY
![Page 37: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/37.jpg)
NLP: Language-specific domains
Speech: Speaker-specific domains
Vision: ??pose-specific? illumination-specific? occlusion? image resolution? background?
Challenges:Many continuous factors vs. few discreteFactors overlap and interact
How to define a domain?
![Page 38: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/38.jpg)
Discovering latent visual domains
Maximum distinctiveness
Maximum learnabilityDetermine K with domain-wise cross-validation
MMD
where
[Gong et al., NIPS 2013]
We propose to discover domains – “reshaping” them to cross dataset boundaries
![Page 39: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/39.jpg)
Discovered domain I
Discovered domain II
Results: discovering domains
[Gong et al., NIPS 2013]
![Page 40: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/40.jpg)
33
34
35
36
37
38
39
40
41
42
42
43
44
45
46
47
48
49
50
Domains=datasets
Hoffman et al. 2012
Discovered domains (ours)
Cross-datasetobject recognition
Cross-viewpointaction recognition
Domain I Domain II
Domains=datasets
Hoffman et al. 2012
Discovered domains (ours)
Results: discovering domainsAc
cura
cy
Accu
racy
![Page 41: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/41.jpg)
Summary so far
landmarkslabeled source instances distributed similarly to the target
auxiliary tasks provably easier to solve
discriminative loss despite unlabeled target
reshaping datasets to latent domains
discover cross-dataset domains
maximally distinct & learnable
![Page 42: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/42.jpg)
Typical assumptions
1. Test set will look like the training set.2. Human labelers “see” the same thing.
![Page 43: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/43.jpg)
Visual attributes
• High-level semantic properties shared by objects• Human-understandable and machine-detectable
brown
indoors
outdoors flat
four-legged
high heel
redhas-ornaments
metallic
[Oliva et al. 2001, Ferrari & Zisserman 2007, Kumar et al. 2008, Farhadi et al. 2009, Lampert et al. 2009, Endres et al. 2010, Wang & Mori 2010, Berg et al. 2010, Branson et al. 2010, Parikh & Grauman 2011, …]
![Page 44: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/44.jpg)
Standard approach
Learn one monolithic model per attribute
Vote on labels
“formal”
“not formal”
Annotator A
Annotator B Annotator C
![Page 45: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/45.jpg)
Problem
Formal? User labels: 50% “yes”50% “no” or
More ornamented? User labels: 50% “first”20% “second”30% “equally”
There may be valid perceptual differences within an attribute.
Binary attribute Relative attribute
![Page 46: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/46.jpg)
Overweight?or just
Chubby?
Fine-grained meaning
Imprecision of attributes
![Page 47: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/47.jpg)
Is formal?
= formal wear for a conference? OR
= formal wear for a wedding?
Context
Imprecision of attributes
![Page 48: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/48.jpg)
Is blue or green?
English: “blue”
Russian: “neither” (“голубой” vs. “синий”)
Japanese: “both”(“青” = blue and green)
Cultural
Imprecision of attributes
![Page 49: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/49.jpg)
But do we need to be that precise?
Yes. Applications like image search require that user’s perception matches system’s predictions.
[WhittleSearch, Kovashka et al. CVPR 2012]
“less formal than these”
“white high heels”
![Page 50: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/50.jpg)
Our idea
[Kovashka and Grauman, ICCV 2013]
• Treat learning perceived attributes as an adaptation problem.
• Adapt generic attribute model with minimal user-specific labeled examples.
• Obtain implicit user-specific labels from user’s search history
![Page 51: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/51.jpg)
Vote on labels
“formal”
“not formal”
“formal”
“not formal”
“formal”
“not formal”
[Kovashka and Grauman, ICCV 2013]
Our idea
![Page 52: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/52.jpg)
• Adapting binary attribute classifiers:
Learning adapted attributes
J. Yang et al. ICDM 2007.
Given user-labeled data
and generic model ,
![Page 53: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/53.jpg)
• Adapting relative attribute rankers:
Learning adapted attributes
Given user-labeled data
and generic model ,
B. Geng, et al. TKDE 2010.
![Page 54: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/54.jpg)
Collecting user-specific labels
• Explicitly from actively requested labelsSeek labels on uncertain and diverse images
• Implicitly from search historyoTransitivity
oContradictions
“My target is…
less formal than
more formal than “
implies
![Page 55: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/55.jpg)
Inferring implicit labels
more sporty
… … … …
“Target is more sporty than B”
“Target is less sporty than A”
less sporty
… … … …
A
B
User’s feedback history can reveal mismatch in perceived and predicted attributes
![Page 56: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/56.jpg)
User’s feedback history can reveal mismatch in perceived and predicted attributes
more sporty
… … … …
“Target is more sporty than B”
A C
B
more feminine (~ less sporty)
… … … …
“Target is more feminine than A”
Inferring implicit labels
![Page 57: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/57.jpg)
SUN Attributes:14,340 scene images
12 attributes:“sailing”, “hiking”,
“vacationing”, “open area”, “vegetation”, etc.
Datasets
Shoes:14,658 shoe images;
10 attributes: “pointy”, “bright”, “high-heeled”, “feminine” etc.
57
![Page 58: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/58.jpg)
Adapted attribute accuracy
• 3 datasets• 22 attributes• 75 total users
![Page 59: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/59.jpg)
Adapted attribute accuracy
• 3 datasets• 22 attributes• 75 total users
![Page 60: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/60.jpg)
Adapted attribute accuracy
• 3 datasets• 22 attributes• 75 total users
![Page 61: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/61.jpg)
Adaptation approach most accurately captures perceived attributes
Adapted attribute accuracy
[Kovashka and Grauman, ICCV 2013]
![Page 62: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/62.jpg)
Which images most influence adaptation?
pointy open bright ornamented shiny high-heeled
long formal sporty feminine
sailing vacationing hiking camping socializing shopping
vegetation clouds natural light cold open area horizon far62
![Page 63: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/63.jpg)
SUN – Binary Attributes – “Vacationing”
Shoes – Relative Attributes – “Formal”
gene
ricad
apte
d less more
gene
ricad
apte
d
less more
Visualizing adapted attributes
![Page 64: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/64.jpg)
0
10
20
30
40
50
60
70
Shoes-Binary SUN
generic generic+ user-exclusive user-adaptive
Mat
ch ra
tePersonalizing image search
with adapted attributes“white shiny heels”
“shinier than ”
![Page 65: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/65.jpg)
71.5
72
72.5
73
73.5
74
74.5
Shoes-Relative
explicit labels only+contradictions+transitivity
Perc
entil
e ra
nk
Impact of implicit labels
![Page 66: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/66.jpg)
Summary
• Practical concerns if learning visual categories:Test images can look different from training images!People do not perceive image labels universally!
• Domain adaptation methods help address themLandmark-based unsupervised adaptationReshaping datasets into latent domainsAdapt generic models to account for user-specific perception of attributes
![Page 67: Adaptation for Objects and Attributesgrauman/slides/grauman-iccv2013-adaptation.pdf · Adaptation for Objects and Attributes. Kristen Grauman. Department of Computer Science](https://reader030.vdocuments.us/reader030/viewer/2022040402/5e7e03f52a4b19725b5c8c1c/html5/thumbnails/67.jpg)
References• Attribute Adaptation for Personalized Image Search. A. Kovashka
and K. Grauman. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Sydney, Australia, December 2013.
• Reshaping Visual Datasets for Domain Adaptation. B. Gong, K. Grauman, and F. Sha. In Proceedings of Advances in Neural Information Processing Systems (NIPS), Tahoe, Nevada, December 2013.
• Connecting the Dots with Landmarks: Discriminatively Learning Domain-Invariant Features for Unsupervised Domain Adaptation. B. Gong, K. Grauman, and F. Sha. In International Conference on Machine Learning (ICML), Atlanta, GA, June 2013.
• Geodesic Flow Kernel for Unsupervised Domain Adaptation. B. Gong, Y. Shi, F. Sha, and K. Grauman. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, June 2012.