3D Computer Vision
and Video Computing SensorsSensors
Instructor: Zhigang ZhuCity College of New York
CSc I6716Fall 2006
3D Computer Vision and Video Computing
Topic 2 in Part 1: Sensors
3D Computer Vision
and Video Computing AcknowledgementsAcknowledgements
The slides in this lecture were adopted and modified from lectures by
Professor Allen HansonUniversity of Massachusetts at Amherst
3D Computer Vision
and Video Computing SensorsSensors
Static monocular reflectance data (monochromic or color) Films Video cameras (with tapes) Digital cameras (with memory)
Motion sequences (camcorders) Stereo (2 cameras) Range data (Range finder) Non-visual sensory data
infrared (IR) ultraviolet (UV) microwaves
Many more
3D Computer Vision
and Video Computing The Electromagnetic SpectrumThe Electromagnetic Spectrum
Visible Spectrum
700 nm 400 nm
C = f
f
3D Computer Vision
and Video Computing The EyeThe Eye
The Retina: rods (low-level light, night vision) cones (color-vision) synapses optic nerve fibers
Sensing and low-level processing layer 125 millions rods and cones feed into 1 million nerve fibers Cell arrangement that respond to horizontal and vertical lines
Retina
RodsCones
3D Computer Vision
and Video Computing Film, Video, Digital CamerasFilm, Video, Digital Cameras
Black and White (Reflectance data only) Color (Reflectance data in three bands - red, green, blue)
3D Computer Vision
and Video Computing Color ImagesColor Images
Blue Green Red
‘Dimensions’ of an Image
Spatial (x,y)Depth (no. of components)Number of bits/channelTemporal (t)
Pixel
Spatial Resolution
Spectra Resolution
Radiometric Resolution
Temporal Resolution
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Cargo inspection using Gamma Rays
Mobile Vehicle and Cargo Inspection System (VACIS®)
Gamma rays are typically waves of frequencies greater than 1019 Hz
Gamma rays can penetrate nearly all materials and are therefore difficult to detect
Courtesy:Science Applications International Corporation (SAIC),
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Cargo inspection using Gamma Rays
Mobile Vehicle and Cargo Inspection System (VACIS®)
Gamma rays are typically waves of frequencies greater than 1019 Hz
Gamma rays can penetrate nearly all materials and are therefore difficult to detect
Courtesy:Science Applications International Corporation (SAIC),
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Cargo inspection using Gamma Rays
Mobile Vehicle and Cargo Inspection System (VACIS®)
Gamma rays are typically waves of frequencies greater than 1019 Hz
Gamma rays can penetrate nearly all materials and are therefore difficult to detect
Courtesy:Science Applications International Corporation (SAIC),
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Chandra X-Ray Satellite
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
From X-Ray images to 3D Models: CT Scans
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Flower Patterns in Ultraviolet
Dandelion - UV
Po
ten
till
a
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Messier 101 in Ultraviolet
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Traditional images
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Non-traditional Use of Visible Light: Range
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Scanning Laser Rangefinder
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
IR: Near, Medium, Far (~heat)
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
IR: Near, Medium, Far (~heat)
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
IR: Finding chlorophyll -the green coloring matter of plants that functions in photosynthesis
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Microwave Imaging: Synthetic Aperture Radar (SAR)
San Fernando Valley Tibet: Lhasa River
Thailand: Phang Hoei RangeAthens, Greece
Red: L-band (24cm)
Green: C-band (6 cm)
Blue:C/L
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Radar in Depth: Interferometric Synthetic Aperture Radar - IFSAR(elevation)
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Low Altitude IFSAR
IFSAR elevation, automatic, in minutes
Elevation from aerial stereo, manually, several days
3D Computer Vision
and Video Computing Across the EM SpectrumAcross the EM Spectrum
Radio Waves (images of cosmos from radio telescopes)
3D Computer Vision
and Video Computing Stereo GeometryStereo Geometry
P(X,Y,Z)
f = focal length
Optical Center
pr(x,y)
Film plane
pl(x,y)
Optical Center
f = focal length
Film plane
LEFT CAMERA RIGHT CAMERA
B = Baseline
3D Computer Vision
and Video Computing Stereo GeometryStereo Geometry
LEFT IMAGE RIGHT IMAGE
Disparity = xr - xl
P
Pr(xr,yr)Pl(xl,yl)
≈ depth
3D Computer Vision
and Video Computing Stereo ImagesStereo Images
A Short Digression
StereoscopesStereoscopes
3D Computer Vision
and Video Computing Range SensorsRange Sensors
Light Striping
David B. Cox, Robyn Owens and Peter HartmannDepartment of BiochemistryUniversity of Western Australia http://mammary.nih.gov/reviews/lactation/Hartmann001/
3D Computer Vision
and Video Computing Why is Vision Difficult?Why is Vision Difficult?
Natural Variation in Object Classes: Color, texture, size, shape, parts, and relations
Variations in the Imaging Process Lighting (highlights, shadows, brightness, contrast) Projective distortion, point of view, occlusion Noise, sensor and optical characteristics
Massive Amounts of Data 1 minute of 1024x768 color video = 4.2 gigabytes
(Uncompressed)
3D Computer Vision
and Video Computing The Need for KnowledgeThe Need for Knowledge
Knowledge
Function
Context
Shape
SpecificObjects
GenericObjects
Structure Size
Shape
Motion
Purpose
Variation
3D Computer Vision
and Video Computing
And as something else…..
‘To interpret something is to give it meaning in context.’
ContextContext
3D Computer Vision
and Video Computing Vision System ComponentsVision System Components
…..at the low (image) level, we need Ways of generating initial descriptions of the image data Method for extracting features of these descriptions Ways of representing these descriptions and features Usually, cannot initially make use of general world
knowledge
IMAGE(numbers)
DESCRIPTION(symbols)
3D Computer Vision
and Video Computing
….at the intermediate level, we need Symbolic representations of the initial descriptions Ways of generating more abstract descriptions from the initial ones (grouping) Ways of accessing relevant portions of the knowledge base Ways of controlling the processing
Intermediate level processes should be capable of being used top-down (knowledge-directed) or bottom-up (data-directed)
IMAGEIINTERMEDIATEDESCRIPTIONS
KNOWLEDGE
Vision System ComponentsVision System Components
3D Computer Vision
and Video Computing Vision System ComponentsVision System Components
….at the high (interpretation) level, we need Ways of representing world knowledge
Objects Object parts Expected scenarios (relations) Specializations
Mechanisms for Interferencing Beliefs Partial matches
Control Information Representations of
Partial interpretations Competing interpretations Relationship to the image descriptions