visual processing for social media
DESCRIPTION
Visual Processing for Social Media. Andrew C. Gallagher Tsuhan Chen September 30, 2012 Cornell University. TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A A A. Outline. Social Media Overview Visual Processing Overview - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/1.jpg)
1
Visual Processing for Social Media
Andrew C. GallagherTsuhan Chen
September 30, 2012
Cornell University
![Page 2: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/2.jpg)
Outline
Social Media Overview Visual Processing Overview Social Media Insights Within the Image Social Media Insights From Sharing
2
![Page 3: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/3.jpg)
3
Now, pictures of people Examples of how social data has helped
understand images of people Some things I’ve learned about people
from computer vision
![Page 4: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/4.jpg)
4
Understanding images of people
![Page 5: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/5.jpg)
5
What the computer sees
![Page 6: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/6.jpg)
6
Faces in the lab
[Turk et al., Cog. Neuro. 1991]
[Belhaumer et al., PAMI 1997]
[Wiskott et al., PAMI, 1997] [Lucey et al., IJCV 2007]
[Blanz et al., PAMI 2003]
[Kanade, Kyoto U. 1973]
![Page 7: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/7.jpg)
8
Is this a family?
![Page 8: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/8.jpg)
9
The Loop
Images and Computer Vision
What we know about people
![Page 9: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/9.jpg)
10
Understanding Images of People
Describe people: How tall? How old? Identify people: Who? Why are they together? Exploit the same context humans use!
![Page 10: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/10.jpg)
11
Understanding Images of People
Capture Context Social Context
July 2, 20058:27 PMLat: 42.2902Long: 85.5361
June 25, 200510:50 AMLat: 42.3202Long: 85.1261
![Page 11: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/11.jpg)
12
Understanding Images of People
Capture Context Social Context
Adult male height: 177 cmAdult female height: 163 cmMLE mother-child: 27 yearsMLE husband-wife: 2 yearsMLE |sibling-sibling|: 6 years
![Page 12: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/12.jpg)
13
What is social context?Social Context: information about people and their society that is useful for understanding images. Distributions of ages and genders in social
groups Social relationships Face position in a group image First name popularity over time Anthropometric measurements
![Page 13: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/13.jpg)
14
Group Images What we know and learn about people:
Group dynamics Computer vision task:
Measuring age, gender, of each person in a group
![Page 14: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/14.jpg)
15
Images of Groups
Identify age and gender Recognize certain group events Consider context and appearance
[A. Gallagher, T. Chen, CVPR 2009]
![Page 15: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/15.jpg)
16
Contextual Features
Absolute face position Size, position relative neighbor and group Minimal spanning tree degree
![Page 16: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/16.jpg)
17
Evidence of Social Context
Relative positions of nearest neighbors depends on the social relationship
Mean distance is 306 mm
Neighbors Male to Female Other to Baby
![Page 17: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/17.jpg)
19
Evidence of Social Context
Samples of faces based on image locationRandom samples
![Page 18: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/18.jpg)
20
Use All Context
5080 images with 28,231 faces Classification improves with more
contextual features
![Page 19: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/19.jpg)
21
Appearance Features
Project face into Fisher Space Nearest neighbor density estimation
Gender subspaceNearest neighbors
![Page 20: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/20.jpg)
22
Gender Estimation
Context Appearance Combined
![Page 21: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/21.jpg)
24
Context and Appearance
Context contributes more when appearance is weak.
Context Appearance Combined
All Faces Small Faces
![Page 22: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/22.jpg)
25
Context for Scene Geometry
Find the face vanishing lineEstimated horizon from face positionsManually labeled horizon
![Page 23: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/23.jpg)
26
Context for Dining Event
Group Structure = Activity
![Page 24: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/24.jpg)
27
Row Segmentation
[A. Gallagher, T. Chen, ICME 2009]
![Page 25: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/25.jpg)
29
Row Segmentation
![Page 26: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/26.jpg)
30
Row Segmentation
![Page 27: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/27.jpg)
31
Social Relationship Retrieval
Spouses
Mother-Child
[G. Wang, A. Gallagher, J. Luo, D. Forsyth, ECCV 2010]
![Page 28: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/28.jpg)
32
Names as Context What we know and learn about people:
Government census data Computer vision task:
Matching names to faces. Guessing age and gender.
![Page 29: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/29.jpg)
33
First names capture information about age and gender
First names are social context
Person A and Person B
First Names as Context
Mildred and Lisa
1900 1920 1940 1960 1980 20000
0.01
0.02
0.03
0.04
0.05
0.06
0.07
Birth Year
Pro
babi
lity
Probability of Birth Year
LisaMildred
1900 1920 1940 1960 1980 20000
0.01
0.02
0.03
0.04
0.05
0.06
0.07
Birth Year
Pro
babi
lity
Probability of Birth Year
MildredLisaNoraPeytonLinda
Source: U.S. Social Security Administration
[A. Gallagher, T. Chen, CVPR 2008]
![Page 30: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/30.jpg)
34
First names and appearance? Tom_101 Ben_165 Caleb_337 Andrew_233 Brian_116 Zachary_431
1953 1956 2003 1984 1962 1996
Abigail_194 Heather_224 Alejandra_152 Juanita_192 Ethel_165 Gertrude_532002 1970 1977 1947 1926 1924
![Page 31: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/31.jpg)
35
Gertrude_531924
Sort by Expected Age Tom_101
1953Ben_165
1956
Caleb_3372003
Andrew_2331984
Brian_1161962
Zachary_4311996
Abigail_1942002
Heather_2241970
Alejandra_1521977
Juanita_1921947
Ethel_1651926
![Page 32: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/32.jpg)
36
First Names as Context
Mildred and Lisa
Name
Birth Year
AgeFeatures
Gender
GenderFeatures
Image-Based Features
![Page 33: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/33.jpg)
37
More Context = Better Results
Appearance First Name Full Model
![Page 34: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/34.jpg)
38
Recognition from First Name
The model improves name assignment, age estimation, and gender classification
![Page 35: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/35.jpg)
39
Learning about people
39
Images and Computer Vision
What we know about people
![Page 36: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/36.jpg)
40
Group Images How close do people stand in group
photos? Computer vision answer: 306 mm Sociology’s “Personal Space”: 457 mm
Do people suspend personal space needs during photograph?
![Page 37: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/37.jpg)
41
Group Images: Gender Prior How do people end up in a group photo
anyhow?
![Page 38: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/38.jpg)
42
Group Images: Gender Prior Bernoulli world?
Implicit prior, IID: Let’s look at the data!
![Page 39: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/39.jpg)
43
“Group Shots”
Number of Females
Gender Distribution of 6 peopleBinomial Distribution
Number of Females 0 1 2 3 4 5 6
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0 1 2 3 4 5 60
0.05
0.1
0.15
0.2
0.25
0.3
0.35
?
0 1 2 3 4 5 60
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Number of Females
?
“Family”
Actu
al D
istrib
ution
s
Genders of people in a image are not independent!
![Page 40: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/40.jpg)
44
Group Shot Analysis Standing Order Frequency for 4 people (2
male, 2 female):0.13
0.11
0.19
0.13
0.30
0.15
But why?
![Page 41: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/41.jpg)
45
Learning about people
4545
Images and Computer Vision
What we know about people
(what they do and think!)
![Page 42: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/42.jpg)
46
Social Context Data Summary U.S. Social Security First Name Database
6693 first names, birth years, gender U.S. CDC National Center for Health
Statistics Physical growth tables Birth rates and other birth statistics Family structure statistics
Farkas, 1994 Facial anthropometric measurements
![Page 43: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/43.jpg)
47
Conclusions Social context is useful for interpreting
single images or image collections Social context is learned from images or
other public sources Learning about people improves our
understanding of images of people
![Page 44: Visual Processing for Social Media](https://reader035.vdocuments.us/reader035/viewer/2022081603/56815aad550346895dc84f75/html5/thumbnails/44.jpg)
48
Related Publications J. Xin, A. Gallagher, L. Cao, J. Luo, J. Han
The Wisdom of Social Multimedia: Using Flickr For Prediction and Forecast
ACM MM 2009
G. Wang, A. Gallagher, J. Luo, D. Forsyth
Seeing People in Social Context: Recognizing People and Social Relationship
ECCV 2010
A. Gallagher, A. Blose, T. Chen
Jointly Estimating Demographics and Height with a Calibrated Camera
ICCV 2009
A. Gallagher, T. Chen Using Context to Recognize People in Consumer Images IPSJ Trans. on Comp. Vis. and Apps., 2009
A. Gallagher, T. Chen Understanding Images of Groups of People CVPR 2009
A. Gallagher, T. Chen Finding Rows of People in Group Images ICME 2009
A. Gallagher, C. Neustaedter, J. Luo, L. Cao, T. Chen
Image Annotation Using Personal Calendars as Context ACM MM 2008
A. Gallagher, T. Chen Estimating Age, Gender and Identity using First Name Priors
CVPR 2008
A. Gallagher, T. Chen Clothing Cosegmentation for Recognizing People CVPR 2008
P. Singla, H. Kautz, J. Luo, A. Gallagher
Discovery of Social Relationships in Consumer Photo Collections Using Markov Logic
CVPR SLAM 2008
A. Gallagher, T. Chen Using a Markov Network to Recognize People in Consumer Images
ICIP 2007
A. Gallagher, M. Das, A. Loui
User-Assisted People Search in Consumer Image Collections ICME 2007
A. Gallagher, T. Chen Using Group Prior to Identify People in Consumer Images CVPR SLAM 2007