![Page 1: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/1.jpg)
Advanced Computer Vision
Devi Parikh
Electrical and Computer Engineering
![Page 2: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/2.jpg)
Plan for today
• Topic overview
• Introductions
• Course overview: – Logistics– Requirements
• Please interrupt at any time with questions or comments
![Page 3: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/3.jpg)
Computer Vision
• Automatic understanding of images and video
– Computing properties of the 3D world from visual data (measurement)
– Algorithms and representations to allow a machine to recognize objects, people, scenes, and activities. (perception and interpretation)
– Algorithms to mine, search, and interact with visual data (search and organization)
Kristen Grauman
![Page 4: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/4.jpg)
What does recognition involve?
Fei-Fei Li
![Page 5: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/5.jpg)
Detection: are there people?
![Page 6: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/6.jpg)
Activity: What are they doing?
![Page 7: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/7.jpg)
Object categorization
mountain
building
tree
banner
vendorpeople
street lamp
![Page 8: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/8.jpg)
Instance recognition
Potala Palace
A particular sign
![Page 9: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/9.jpg)
Scene and context categorization
• outdoor
• city
• …
![Page 10: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/10.jpg)
Attribute recognition
flat
graymade of fabric
crowded
![Page 11: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/11.jpg)
Why recognition?
• Recognition a fundamental part of perception– e.g., robots, autonomous agents
• Organize and give access to visual content– Connect to information – Detect trends and themes
• Where are we now?
Kristen Grauman
![Page 12: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/12.jpg)
We’ve come a long way…
![Page 13: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/13.jpg)
We’ve come a long way…
![Page 14: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/14.jpg)
We’ve come a long way…
![Page 15: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/15.jpg)
Posing visual queries
Kooaba, Bay & Quack et al.
Yeh et al., MIT
Belhumeur et al.
Kristen Grauman
![Page 16: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/16.jpg)
Exploring community photo collections
Snavely et al.
Simon & SeitzKristen Grauman
![Page 17: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/17.jpg)
http://www.darpa.mil/grandchallenge/gallery.asp
Autonomous agents able to detect objects
Kristen Grauman
![Page 18: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/18.jpg)
We’ve come a long way…
Fischler and Elschlager, 1973
![Page 19: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/19.jpg)
We’ve come a long way…
![Page 20: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/20.jpg)
We’ve come a long way…
Dollar et al., BMVC 2009
![Page 21: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/21.jpg)
Still a long way to go…
Dollar et al., BMVC 2009
![Page 22: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/22.jpg)
Dollar et al., BMVC 2009
![Page 23: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/23.jpg)
Dollar et al., BMVC 2009
![Page 24: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/24.jpg)
Challenges
![Page 25: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/25.jpg)
Challenges: robustness
Illumination
Object pose
ViewpointIntra-class appearance
Occlusions
Clutter
Kristen Grauman
![Page 26: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/26.jpg)
Challenges: context and human experience
Context cues
Kristen Grauman
![Page 27: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/27.jpg)
Challenges:context and human experience
Context cues Function Dynamics
Video credit: J. DavisKristen Grauman
![Page 28: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/28.jpg)
Challenges: scale, efficiency
• Half of the cerebral cortex in primates is devoted to processing visual information
• ~20 hours of video added to YouTube per minute
• ~5,000 new tagged photos added to Flickr per minute
• Thousands to millions of pixels in an image
• 30+ degrees of freedom in the pose of articulated objects (humans)
• 3,000-30,000 human recognizable object categories
Kristen Grauman
![Page 29: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/29.jpg)
Challenges: learning with minimal supervision
MoreLess
Cropped to
object, parts and
classes labeled
Classes labeled,
some clutter
Unlabeled,
multiple
objects
Kristen Grauman
![Page 30: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/30.jpg)
Slide from Pietro Perona, 2004 Object Recognition workshop
![Page 31: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/31.jpg)
Slide from Pietro Perona, 2004 Object Recognition workshop
![Page 32: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/32.jpg)
Recognizing flat, textured objects (like books, CD
covers, posters)
Reading license plates, zip codes, checks
Fingerprint recognition
Frontal face detection
What kinds of things work best today?
Kristen Grauman
![Page 33: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/33.jpg)
Inputs in 1963…
L. G. Roberts, Machine Perception of Three Dimensional Solids, Ph.D. thesis, MIT Department of Electrical Engineering, 1963.
Kristen Grauman
![Page 34: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/34.jpg)
Personal photo albums
Surveillance and security
Movies, news, sports
Medical and scientific images
Slide credit; L. Lazebnik
… and inputs today
![Page 35: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/35.jpg)
… and inputs today
Images on the Web Movies, news, sports
916,271 titles
10 mil. videos, 65,000 added daily
350 mil. photos, 1 mil. added daily
1.6 bil. images indexed as of summer 2005
Satellite imagery City streets
Slide credit; L. Lazebnik
Understand and organize and index all this data!!
![Page 36: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/36.jpg)
Introductions
• What is your name?• Which program are you in? How far along?• What is your research area and current project about?
– Take a minute to explain it to us– In a way that we can all follow
• Have you taken a computer vision course before? Machine learning or pattern recognition?
• What are you hoping to get out of this class?
![Page 37: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/37.jpg)
This course
• ECE 5984• TR 3:30 pm to 4:45 pm• Hutcheson (HUTCH) 207• Office hours: by appointment (email)
• Course webpage: http://filebox.ece.vt.edu/~S14ECE5984/(Google me My homepage Teaching)
![Page 38: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/38.jpg)
This course
• Focus on current research in computer vision
• High-level recognition problems, innovative applications.
![Page 39: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/39.jpg)
Goals
• Understand state-of-the-art approaches
• Analyze and critique current approaches
• Identify interesting research questions
• Present clearly and methodically
![Page 40: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/40.jpg)
Expectations
• Discussions will center on recent papers in the field [15%]
• Paper reviews each class [25%]– Can have 3 late days over the course of the semester
• Presentations (2-3 times) [25%]
– Papers and background reading
– Experiments
• Project [35%]No “Assignments”,
Exams, etc.
![Page 41: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/41.jpg)
Prerequisites
• Course in computer vision
• Courses in machine learning is a plus
![Page 42: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/42.jpg)
Paper reviews
• For each class – Review one paper in detail– Review one paper at a high-level– (Reduced from last time I offered this course)
• Email me reviews by noon (12:00 pm) the day of the class
• Skip reviews the classes you are presenting.
![Page 43: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/43.jpg)
Paper review guidelines• One page• Detailed review:
– Brief (2-3 sentences) summary – Main contribution– Strengths? Weaknesses? – How convincing are the experiments? Suggestions to improve them?– Extensions? Applications?– Additional comments, unclear points
• High-level review:– Problem being addressed– High-level intuition/idea of approach
• Relationships observed between the papers we are reading• Will pick on students in class during discussions• Write in your own words• Write well, proof read
![Page 44: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/44.jpg)
Paper presentation guidelines
• Papers
• Experiments
![Page 45: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/45.jpg)
Papers• Read selected papers in topic area and look at
background papers as necessary• Well-organized talk, 45 minutes• What to cover?
– Topic overview, motivation– For selected papers:
• Problem overview, motivation• Algorithm explanation, technical details• Experimental set up, results• Strengths, weaknesses, extensions
– Any commonalities, important differences between techniques covered in the papers.
• See class webpage for more details.
![Page 46: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/46.jpg)
Experiments
• Implement/download code for a main idea in the paper and evaluate it:– Experiment with different types of training/testing data sets
– Evaluate sensitivity to important parameter settings
– Show an example to analyze a strength/weakness of the approach
– Show qualitative and quantitative results
![Page 47: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/47.jpg)
Tips
• Look up papers and authors. Their webpage may have data, code, slides, videos, etc.– Make sure talk flows well and makes sense as a whole.– Cite ALL sources.
• Don’t forget the high-level picture.
• Give a very clear and well-organized and thought out talk.
• Will interrupt if something is not clear
![Page 48: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/48.jpg)
Tips• Make sure you are saying everything we need to
know to understand what you are saying.
• Make sure you know what you are talking about.
• Think about your audience.
• Make your talks visual (images, video, not lots of text).
![Page 49: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/49.jpg)
ProjectsPossibilities:
– Extension of a technique studied in class– Analysis and empirical evaluation of an
existing technique– Comparison between two approaches– Design and evaluate a novel approach– Be creative!
Can work with a partner
Talk to me if you need help with ideas
![Page 50: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/50.jpg)
Project timeline• Project proposals (1 page) [10%]
– March 6th
• Mid-semester presentations (10 minutes) [20%]– March 27th and April 1st
• Final presentations (20 minutes) [35%]– April 24th to May 6th
• Project reports (4 pages) [35%]– May 12th
– Could serve as a first draft of a conference submission!
![Page 51: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/51.jpg)
Implementation
• Use any language / platform you like
• No support for code / implementation issues will be provided
![Page 52: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/52.jpg)
Miscellaneous
• Best presentation, best project and best discussion prizes!– We will vote– Dinner
• Feedback welcome and useful
![Page 53: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/53.jpg)
Coming up• Read the class webpage
– Schedule is up– Tour of schedule
• Select 6 dates (topics) you would like to present – Email me by Wednesday (tomorrow)– Webpage shows how many people have already signed
up for a topic– Select those that have fewer selections
• Overview of my research on Thursday– How many of you were at the ECE grad seminar in
November?
![Page 54: Advanced Computer Vision Devi Parikh Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062305/56649ddf5503460f94ad8b4d/html5/thumbnails/54.jpg)
Questions?
See you Thursday!