selim aksoy department of computer engineering bilkent ...saksoy/courses/cs484-spring2010/...cs 484,...

97
Introduction Selim Aksoy Department of Computer Engineering Department of Computer Engineering Bilkent University [email protected]

Upload: others

Post on 05-Aug-2021

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Introduction

Selim Aksoy

Department of Computer EngineeringDepartment of Computer Engineering

Bilkent University

[email protected]

Page 2: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

What is computer vision?

� Analysis of digital images by a computer.

Stockman and Shapiro: making useful decisions about real � Stockman and Shapiro: making useful decisions about real physical objects and scenes based on sensed images.

� Trucco and Verri: computing properties of the 3D world from one or more digital images.

� Ballard and Brown: construction of explicit, meaningful

CS 484, Spring 2010 ©2010, Selim Aksoy 2

� Ballard and Brown: construction of explicit, meaningful description of physical objects from images.

� Forsyth and Ponce: extracting descriptions of the world from pictures or sequences of pictures.

Page 3: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Why study computer vision?

� Possibility of building intelligent machines is fascinating.

� Capability of understanding the visual world is a prerequisite for such machines.

� Much of the human brain is dedicated to vision.

CS 484, Spring 2010 ©2010, Selim Aksoy 3

� Humans solve many visual problems effortlessly, yet we have little understanding of visual cognition.

Page 4: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Why study computer vision?

� Fast growing collections and many useful applications.

CS 484, Spring 2010 ©2010, Selim Aksoy 4

� Goals of vision research:� Give machines the ability to understand scenes.

� Aid understanding and modeling of human vision.

� Automate visual operations.Adapted from CSE 455, U of Washington

Page 5: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Applications

� Medical image analysis� Security

Biometrics

� Industrial inspection, quality control

Document analysis� Biometrics� Surveillance� Tracking� Target recognition

� Remote sensing� Robotics

� Document analysis

� Multimedia

� Assisted living

� Human-computer interfaces

CS 484, Spring 2010 ©2010, Selim Aksoy 5

� …

Page 6: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Medical image analysis

CS 484, Spring 2010 ©2010, Selim Aksoy 6

http://www.clarontech.com

Page 7: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Medical image analysis

CS 484, Spring 2010 ©2010, Selim Aksoy 7

http://www.clarontech.com

Page 8: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Medical image analysis

CS 484, Spring 2010 ©2010, Selim Aksoy 8

http://www.clarontech.com

Page 9: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Medical image analysis

CS 484, Spring 2010 ©2010, Selim Aksoy 9

Adapted from CSE 455, U of Washington

3D imaging: MRI, CTImage guided surgery

Grimson et al., MIT

Page 10: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Medical image analysis

Cancer detection and grading

CS 484, Spring 2010 ©2010, Selim Aksoy 10

Page 11: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Medical image analysis

Slice of lung

CS 484, Spring 2010 ©2010, Selim Aksoy 11

Adapted from Linda Shapiro, U of Washington

Page 12: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Medical image analysis

CS 484, Spring 2010 ©2010, Selim Aksoy 12

Page 13: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Biometrics

CS 484, Spring 2010 ©2010, Selim Aksoy 13

Adapted fromAnil Jain,

Michigan State

Page 14: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Biometrics

CS 484, Spring 2010 ©2010, Selim Aksoy 14

Adapted from Anil Jain, Michigan State

Page 15: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Surveillance and tracking

CS 484, Spring 2010 ©2010, Selim Aksoy 15

University of Central Florida, Computer Vision Lab

Page 16: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Surveillance and tracking

CS 484, Spring 2010 ©2010, Selim Aksoy 16

Adapted from Octavia Camps, Penn State

Page 17: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Surveillance and tracking

CS 484, Spring 2010 ©2010, Selim Aksoy 17

Adapted from Martial Hebert, CMU

Page 18: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Surveillance and tracking

CS 484, Spring 2010 ©2010, Selim Aksoy 18

University of Central Florida, Computer Vision Lab

Generating traffic patterns

Page 19: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Surveillance and tracking

CS 484, Spring 2010 ©2010, Selim Aksoy 19

Adapted from Martial Hebert, CMU, andMasaharu Kobashi, U of Washington

Tracking in UAV videos

Page 20: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Vehicle and pedestrian protection

CS 484, Spring 2010 ©2010, Selim Aksoy 20

http://www.mobileye-vision.com

Lane departure warning, collision warning, traffic sign recognition,pedestrian recognition, blind spot warning

Page 21: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Smart cars

CS 484, Spring 2010 ©2010, Selim Aksoy 21

Adapted from CSE 455, U of Washington

Page 22: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Forest fire monitoring system

CS 484, Spring 2010 ©2010, Selim Aksoy 22

Adapted from Enis Cetin, Bilkent University

Early warning of forest fires

Page 23: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Land cover classification

CS 484, Spring 2010 ©2010, Selim Aksoy 23

Page 24: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Land cover classification

CS 484, Spring 2010 ©2010, Selim Aksoy 24

Page 25: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Object recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 25

Page 26: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Object recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 26

Recognition of buildings and building groups

Page 27: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Content-based retrieval

CS 484, Spring 2010 ©2010, Selim Aksoy 27

Finding similar regions: airports

Page 28: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Robotics

CS 484, Spring 2010 ©2010, Selim Aksoy 28

Adapted from CSE 455, U of Washington

Page 29: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Robotics

CS 484, Spring 2010 ©2010, Selim Aksoy 29

Adapted from Steven Seitz, U of Washington

Page 30: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Autonomous navigation

CS 484, Spring 2010 ©2010, Selim Aksoy 30

Michigan State UniversityGeneral Dynamics Robotics Systems

http://www.gdrs.com

Page 31: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Industrial automation

CS 484, Spring 2010 ©2010, Selim Aksoy 31

Color Vision Systemshttp://www.cvs.com.au

Automatic fruit sorting

Page 32: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Industrial automation

Industrial robotics;bin picking

CS 484, Spring 2010 ©2010, Selim Aksoy 32

http://www.braintech.com

Page 33: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Postal service automation

CS 484, Spring 2010 ©2010, Selim Aksoy 33

General Dynamics Robotics Systemshttp://www.gdrs.com

Page 34: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Optical character recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 34

Digit recognition, AT&T labshttp://www.research.att.com/~yann

Adapted from Steven Seitz, U of Washington

License place recognition

Page 35: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Document analysis

CS 484, Spring 2010 ©2010, Selim Aksoy 35

Adapted from Shapiro and Stockman

Page 36: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Document analysis

CS 484, Spring 2010 ©2010, Selim Aksoy 36

Adapted from Linda Shapiro, U of Washington

Page 37: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Sports video analysis

CS 484, Spring 2010 ©2010, Selim Aksoy 37

http://www.hawkeyeinnovations.co.ukTennis review system

Page 38: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Scene classification

CS 484, Spring 2010 ©2010, Selim Aksoy 38

Page 39: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Organizing image archives

CS 484, Spring 2010 ©2010, Selim Aksoy 39

Adapted from Pinar Duygulu, Bilkent University

Page 40: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Photo tourism: exploring photo collections

CS 484, Spring 2010 ©2010, Selim Aksoy 40

Building 3D scene models from individual photos

Adapted from Steven Seitz, U of Washington

Page 41: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Content-based retrieval

CS 484, Spring 2010 ©2010, Selim Aksoy 41

Page 42: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Content-based retrieval

CS 484, Spring 2010 ©2010, Selim Aksoy 42

Page 43: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Content-based retrieval

CS 484, Spring 2010 ©2010, Selim Aksoy 43

Online shopping catalog search

http://www.like.com

Page 44: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Face detection and recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 44Adapted from CSE 455, U of Washington

Page 45: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Object recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 45

Adapted from Rob Fergus, MIT

Page 46: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

3D scanning

CS 484, Spring 2010 ©2010, Selim Aksoy 46

Adapted from Linda Shapiro, U of Washington

Page 47: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

3D reconstruction

CS 484, Spring 2010 ©2010, Selim Aksoy 47

Adapted from David Forsyth, UC Berkeley

Page 48: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

3D reconstruction

CS 484, Spring 2010 ©2010, Selim Aksoy 48

Adapted from David Forsyth, UC Berkeley

Page 49: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Motion capture

CS 484, Spring 2010 ©2010, Selim Aksoy 49

Adapted from Linda Shapiro, U of Washington

Page 50: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Visual effects

CS 484, Spring 2010 ©2010, Selim Aksoy 50

Adapted from CSE 455, U of Washington

Page 51: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Mozaic

CS 484, Spring 2010 ©2010, Selim Aksoy 51

Adapted from David Forsyth, UC Berkeley

Page 52: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Mozaic

CS 484, Spring 2010 ©2010, Selim Aksoy 52

Adapted from David Forsyth, UC Berkeley

Page 53: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Critical issues

� What information should be extracted?

� How can it be extracted?

� How should it be represented?

How can it be used to aid analysis and

CS 484, Spring 2010 ©2010, Selim Aksoy 53

� How can it be used to aid analysis and understanding?

Page 54: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Challenge

� What do you see in the picture?

� A hand holding a man

� A hand holding a shiny sphere

� An Escher drawing

CS 484, Spring 2010 ©2010, Selim Aksoy 54

Adapted from Octavia Camps, Penn State

Page 55: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Perception and grouping

Subjective contours

CS 484, Spring 2010 ©2010, Selim Aksoy 55

Page 56: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Perception and grouping

Subjective contours

CS 484, Spring 2010 ©2010, Selim Aksoy 56

Adapted from Michael Black, Brown University

Page 57: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Perception and grouping

CS 484, Spring 2010 ©2010, Selim Aksoy 57

Adapted from Gonzales and Woods

Page 58: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Perception and grouping

CS 484, Spring 2010 ©2010, Selim Aksoy 58

Adapted from Gonzales and Woods

Page 59: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS
Page 60: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

CS 484, Spring 2010 ©2010, Selim Aksoy 60Copyright A.Kitaoka 2003

Page 61: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Perception and grouping

Occlusion

CS 484, Spring 2010 ©2010, Selim Aksoy 61

Adapted from Michael Black, Brown University

Page 62: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Perception and grouping

� The shape of junctions constrains the possible interpretations of the interpretations of the scene.

� Ambiguous: paint and surface boundaries can be confused.

CS 484, Spring 2010 ©2010, Selim Aksoy 62

Adapted from Michael Black, Brown University

Page 63: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Challenges 1: view point variation

CS 484, Spring 2010 ©2010, Selim Aksoy 63

Michelangelo 1475-1564

Adapted from L. Fei-Fei,R. Fergus, A. Torralba

Page 64: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Challenges 2: illumination

CS 484, Spring 2010 ©2010, Selim Aksoy 64

Adapted from L. Fei-Fei, R. Fergus, A. Torralba

Page 65: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Challenges 3: occlusion

Magritte, 1957

CS 484, Spring 2010 ©2010, Selim Aksoy 65

Adapted from L. Fei-Fei,R. Fergus, A. Torralba

Page 66: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Challenges 4: scale

CS 484, Spring 2010 ©2010, Selim Aksoy 66

Adapted from L. Fei-Fei,R. Fergus, A. Torralba

Page 67: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Challenges 5: deformation

CS 484, Spring 2010 ©2010, Selim Aksoy 67

Xu, Beihong 1943

Adapted from L. Fei-Fei, R. Fergus, A. Torralba

Page 68: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Challenges 6: background clutter

Klimt, 1913

CS 484, Spring 2010 ©2010, Selim Aksoy 68

Adapted from L. Fei-Fei,R. Fergus, A. Torralba

Page 69: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Challenges 7: intra-class variation

CS 484, Spring 2010 ©2010, Selim Aksoy 69

Adapted from L. Fei-Fei, R. Fergus, A. Torralba

Page 70: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Recognition

� How can different cues such as color, texture, shape, motion, etc., can be used texture, shape, motion, etc., can be used for recognition?

� Which parts of image should be recognized together?

� How can objects be recognized without focusing on detail?

CS 484, Spring 2010 ©2010, Selim Aksoy 70

focusing on detail?

� How can objects with many free parameters be recognized?

� How do we structure very large model bases?

Page 71: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Color

CS 484, Spring 2010 ©2010, Selim Aksoy 71

Adapted from Martial Hebert, CMU

Page 72: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Texture

CS 484, Spring 2010 ©2010, Selim Aksoy 72

Adapted from David Forsyth, UC Berkeley

Page 73: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Original Images Color Regions Texture Regions Line Clusters

Segmentation

CS 484, Spring 2010 ©2010, Selim Aksoy 73

Adapted from Linda Shapiro, U of Washington

Page 74: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Segmentation

CS 484, Spring 2010 ©2010, Selim Aksoy 74

Adapted from Jianbo Shi, U Penn

Page 75: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Shape

CS 484, Spring 2010 ©2010, Selim Aksoy 75

Model database

Recognized objects

Adapted from Enis Cetin, Bilkent University

Page 76: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Motion

CS 484, Spring 2010 ©2010, Selim Aksoy 76

Adapted from Michael Black, Brown University

Page 77: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 77

Adapted from Michael Black, Brown University

Page 78: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 78

Adapted from Michael Black, Brown University

Page 79: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 79

Adapted from Michael Black, Brown University

Page 80: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 80

Adapted from Michael Black, Brown University

Page 81: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 81

Adapted from Michael Black, Brown University

Page 82: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 82

Page 83: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Recognition

CS 484, Spring 2010 ©2010, Selim Aksoy 83

Adapted from David Forsyth, UC Berkeley

Page 84: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Detection

CS 484, Spring 2010 ©2010, Selim Aksoy 84

Adapted from David Forsyth, UC Berkeley

Page 85: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Detection

CS 484, Spring 2010 ©2010, Selim Aksoy 85

Adapted from David Forsyth, UC Berkeley

Page 86: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Detection

CS 484, Spring 2010 ©2010, Selim Aksoy 86

Adapted from Michael Black, Brown University

Page 87: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Parts and relations

CS 484, Spring 2010 ©2010, Selim Aksoy 87

Adapted from Michael Black, Brown University

Page 88: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Parts and relations

CS 484, Spring 2010 ©2010, Selim Aksoy 88

Adapted from Michael Black, Brown University

Page 89: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Context

CS 484, Spring 2010 ©2010, Selim Aksoy 89

Adapted from Antonio Torralba, MIT

Page 90: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Context

CS 484, Spring 2010 ©2010, Selim Aksoy 90

Adapted from Antonio Torralba, MIT

Page 91: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Context

CS 484, Spring 2010 ©2010, Selim Aksoy 91

Adapted from Derek Hoiem, CMU

Page 92: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Context

CS 484, Spring 2010 ©2010, Selim Aksoy 92

Adapted fromDerek Hoiem, CMU

Page 93: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Stages of computer vision

� Low-levelimage � imageimage � image

� Mid-levelimage � features / attributes

Image analysis / image understanding

CS 484, Spring 2010 ©2010, Selim Aksoy 93

� High-levelfeatures � “making sense”, recognition

Page 94: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

sharpening

Low-level

sharpening

CS 484, Spring 2010 ©2010, Selim Aksoy 94

blurring

Adapted from Linda Shapiro, U of Washington

Page 95: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

Canny

Low-level

original image edge image

Mid-level

CS 484, Spring 2010 ©2010, Selim Aksoy 95

ORT

edge image circular arcs and line segments

datastructure

Adapted from Linda Shapiro, U of Washington

Page 96: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

K-meansclustering

Mid-level

clustering

(followed byconnectedcomponentanalysis)

CS 484, Spring 2010 ©2010, Selim Aksoy 96

original color image regions of homogeneous color

datastructure

Adapted from Linda Shapiro, U of Washington

Page 97: Selim Aksoy Department of Computer Engineering Bilkent ...saksoy/courses/cs484-Spring2010/...CS 484, Spring 2010 ©2010, Selim Aksoy 89 Adapted from Antonio Torralba, MIT Context CS

low-level

Low-level to high-level

edge image

consistent

low-level

mid-level

CS 484, Spring 2010 ©2010, Selim Aksoy 97

consistentline clustershigh-level

Adapted from Linda Shapiro, U of Washington