![Page 1: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/1.jpg)
Forever YoungATributetotheGrandmaster&ARecountofPersonalJourney
Jiebo LuoUniversity of Rochester
![Page 2: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/2.jpg)
2
Why Do We Need to Do Research?
![Page 3: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/3.jpg)
3
Why Do We Need to Do Research?
![Page 4: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/4.jpg)
4
Why Do We Need to Do Research?
![Page 5: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/5.jpg)
5
Learned from the GrandmasterCuriosity
Open mindedNew problemsNew techniques
Passion Love, focus - sustained
ScholarshipUnaffected by noise, hypeUncompromised integrity
Never too old to learn
![Page 6: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/6.jpg)
6
PhD ThesisImage Processing
Image and Video Codingwavelet transform
Scene-adaptive Coding primitive/budding scene understanding
![Page 7: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/7.jpg)
7
Kodak Years • Intelligent Image Processing and Semantic Image Understanding
– Automatic Redeye Removal
– Automatic Image Orientation Detection
– Image Classification
– Image Annotation
– Geotagging
– Context + Content
![Page 8: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/8.jpg)
From Classification to Description Recognizing Realistic Actions from Videos "in the Wild"UCF-11 to UCF-101(CVPR 2009)
Similarity btw Videos Cross-Domain Learning
Visual Event Recognition in Videos by Learning from Web Data (CVPR2010 Best Student Paper)
Heterogeneous Feature Machine For Visual Recognition(ICCV 2009)
![Page 9: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/9.jpg)
Image Captioning with Semantic Attention
• Motivations– Real-world Usability
• Help visually impaired people, learning-impaired– Improving Image Understanding
• Classification, Objection detection– Image Retrieval
1. a young girl inhales with the intent of blowing out a candle2. girl blowing out the candle on an ice cream
1. A shot from behind home plate of children playing baseball
2. A group of children playing baseball in the rain
3. Group of baseball players playing on a wet field
![Page 10: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/10.jpg)
Key Elements
• Additional textual information– Leverage noisy titles, tags or captions (Web)
![Page 11: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/11.jpg)
Key Elements
• Additional textual information– Leverage noisy titles, tags or captions (Web)– Leverage visually similar nearest neighbor images
![Page 12: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/12.jpg)
Key Elements
• Additional textual information– Leverage noisy titles, tags or captions (Web)– Leverage visually similar nearest neighbor images– Incorporate success of low-level tasks
• Visual attribute detection
![Page 13: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/13.jpg)
Attention Model on Attributes
• Instead of using the same set of attributes at every step
• At each step, select the attributes (attention)
m mtmt kKwatt ),(
)softmax VK(wTtt
))],,(;([),( 11 tttttt hKwattxfhxfh
![Page 14: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/14.jpg)
Overall Framework
• Training with a bilinear/bilateral attention model
ht
pt
xt
v
{Ai}
Yt~
RNN
Image
CNN
AttrDet 1
AttrDet 2
AttrDet 3
AttrDet N
t = 0
Word
![Page 15: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/15.jpg)
Performance
• MS-COCO Image Captioning Challenge
![Page 16: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/16.jpg)
TGIF: A New Video Dataset and Benchmark
![Page 17: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/17.jpg)
Examples
a skate boarder is doing trick on his skate board.
a gloved hand opens to reveal a golden ring.
a sport car is swinging on the race playground
the vehicle is moving fast into the tunnel
![Page 18: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/18.jpg)
Machine Generated Sentence Examples
![Page 19: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/19.jpg)
19
Two Most Important Social Signals (IMO)• User
• Sentiment
![Page 20: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/20.jpg)
• Cross‐modality Consistent Regression
Joint Visual-Textual Sentiment Analysis
![Page 21: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/21.jpg)
21
Infer User Demographics & Interests
![Page 22: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/22.jpg)
• Social interactions and social activities• Public health surveillance• Web sentiment analysis and trend prediction• Cyber terrorism, extremism, and activism• Fads and infectious ideas• Marketing intelligence analytics • Traffic and human mobility patterns• Human and environment• Social unrest, protest and riot
Understanding the Pulse of Society
![Page 23: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/23.jpg)
23
•Correctly classified flattering expressions
•Correctly classified unflattering expressions
•Correctly classified neutral expressions
• Acts like a prism to reveal the spectrum of opinions
• Competitive Vector Autoregressive Model
2012: Calling the Swing States
![Page 24: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/24.jpg)
2016: Fine-Grained Campaign Analysis
![Page 25: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/25.jpg)
2016: Shifting Tide? Too Close to Call
25
August 26, 2016
September 15, 2016
![Page 26: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/26.jpg)
Social Multimedia
Visual Data
Textual Data
Computer Vision
NLP
User Demographics
User Activities
Behavior Patterns
Using Social Multimedia to Study Social Problems
![Page 27: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/27.jpg)
Time Pattens of Underage Alcohol Use
Temporal Patterns of Underage Drinking
![Page 28: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/28.jpg)
Drug Use Pattern from Instagram
![Page 29: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/29.jpg)
29
When We Were Young
29
![Page 30: Forever Young: A Tribute to the Grandmaster through a recount of Personal Journey](https://reader031.vdocuments.us/reader031/viewer/2022020410/58aa4f571a28ab10578b4f21/html5/thumbnails/30.jpg)
30
Forever Young
30
Forever young,I want to be forever young.Do you really want to live forever?Forever, and ever
Forever young,I want to be forever young.Do you really want to research forever?Forever, and ever
Image Processing, Computer Vision, Multimedia, Social Media, Big Data, …
(A younger version of an old song)
……
Let’s Celebrate the Forever Young Huang Academic Tree!