physiology of vision: a swift overview 16-721: learning-based methods in vision a. efros, cmu,...

Physiology of Vision: a swift overview

16-721: Learning-Based Methods in VisionA. Efros, CMU, Spring 2009

Some figures from Steve Palmer

ThalamusLGN

Striatecortex(V1)

DorsalStream

Parietalvisualcortex

Temporalvisualcortex

Eye Opticnerve

Extrastriatecortex

VentralStream

Image Formation

Digital Camera

The Eye

Film

Monocular Visual Field: 160 deg (w) X 135 deg (h)Binocular Visual Field: 200 deg (w) X 135 deg (h)

Point of observation

Figures © Stephen E. Palmer, 2002

What do we see?

3D world 2D image

Point of observation

What do we see?

3D world 2D image

Painted backdrop

The Plenoptic Function

Q: What is the set of all things that we can ever see?

A: The Plenoptic Function (Adelson & Bergen)

Let’s start with a stationary person and try to parameterize everything that he can see…

Figure by Leonard McMillan

Grayscale snapshot

is intensity of light • Seen from a single view point

• At a single time

• Averaged over the wavelengths of the visible spectrum

(can also do P(x,y), but spherical coordinate are nicer)

P()

Color snapshot


• At a single time

• As a function of wavelength

P()

Spherical Panorama

All light rays through a point form a ponorama

Totally captured in a 2D array -- P()Where is the geometry???

See also: 2003 New Years Eve

http://www.panoramas.dk/fullscreen3/f1.html

http://www.panoramas.dk/fullscreen3/f1.html

http://www.geocities.com/michaelschernau/pano/travelGal.html

A movie


• Over time


P(,t)

Space-time images

x

y

t

Holographic movie

is intensity of light • Seen from ANY viewpoint

• Over time


P(,t,VX,VY,VZ)

The Plenoptic Function

• Can reconstruct every possible view, at every moment, from every position, at every wavelength

• Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality! Not bad for a function…

P(,t,VX,VY,VZ)

The Eye is a camera

The human eye is a camera!• Iris - colored annulus with radial muscles

• Pupil - the hole (aperture) whose size is controlled by the iris

• What’s the “film”?– photoreceptor cells (rods and cones) in the retina

The Retina

Cross-section of eye

Ganglion cell layer

Bipolar cell layer

Receptor layer

Pigmentedepithelium

Ganglion axons

Cross section of retina

Retina up-close

Light

© Stephen E. Palmer, 2002

Cones cone-shaped less sensitive operate in high light color vision

Two types of light-sensitive receptors

cone

rod

Rods rod-shaped highly sensitive operate at night gray-scale vision

Rod / Cone sensitivity

The famous sock-matching problem…


Distribution of Rods and Cones.

0

150,000

100,000

50,000

020 40 60 8020406080

Visual Angle (degrees from fovea)

Rods

Cones Cones

Rods

FoveaBlindSpot

# R

ecep

tors

/mm

2

Night Sky: why are there more stars off-center?

Electromagnetic Spectrum

http://www.yorku.ca/eye/photopik.htm

Human Luminance Sensitivity Function

Why do we see light of these wavelengths?


.

0 1000 2000 3000

En

erg

y

Wavelength (nm)

400 700

700 C

2000 C

5000 C

10000 C

VisibleRegion

…because that’s where theSun radiates EM energy

Visible Light

The Physics of Light

Any patch of light can be completely describedphysically by its spectrum: the number of photons (per time unit) at each wavelength 400 - 700 nm.

400 500 600 700

Wavelength (nm.)

# Photons(per ms.)



.

# P

ho

ton

s

D. Normal Daylight

Wavelength (nm.)

B. Gallium Phosphide Crystal

400 500 600 700

# P

ho

ton

s

Wavelength (nm.)

A. Ruby Laser

400 500 600 700

400 500 600 700

# P

ho

ton

s

C. Tungsten Lightbulb

400 500 600 700

# P

ho

ton

s

Some examples of the spectra of light sources



Some examples of the reflectance spectra of surfaces

Wavelength (nm)

% P

hoto

ns R

efle

cted

Red

400 700

Yellow

400 700

Blue

400 700

Purple

400 700


The Psychophysical Correspondence

There is no simple functional description for the perceivedcolor of all lights under all viewing conditions, but …...

A helpful constraint: Consider only physical spectra with normal distributions

area

Wavelength (nm.)

# Photons

400 700500 600

mean

variance



Mean Hue

yellowgreenblue

# P

hoto

ns

Wavelength



Variance Saturation

Wavelength

high

medium

low

hi.

med.

low# P

hoto

ns



Area Brightness#

Pho

tons

Wavelength

B. Area Lightness

bright

dark



.

400 450 500 550 600 650

RE

LAT

IVE

AB

SO

RB

AN

CE

(%

)

WAVELENGTH (nm.)

100

50

440

S

530 560 nm.

M L

Three kinds of cones:

Physiology of Color Vision

• Why are M and L cones so close?

Retinal Processing


Single Cell Recording

Microelectrode

Amplifier

Time

Electrical response(action potentials)

mV


Single Cell Recording


Retinal Receptive Fields

Receptive field structure in ganglion cells:On-center Off-surround

Stimulus condition Electrical response

Time

Response


Receptive field structure in ganglion cells:On-center Off-surround

Stimulus condition Electrical response

Time

Response



RF of On-center Off-surround cells

Receptive FieldNeural Response

Center

Surround

On Off

Response Profile

on-center

off-surround

Horizontal Position

FiringRate



RF of Off-center On-surround cells

Receptive Field

Horizontal Position

on-surround

off-center

Response Profile

FiringRate



Center

Surround

On Off

Neural Response

Surround

Center

Receptive field structure in bipolar cells

Light



Receptive field structure in bipolar cells

Receptors

Bipolar Cell

A. WIRING DIAGRAM

HorizontalCells

Direct excitatory component (D)

B. RECEPTIVE FIELD PROFILES

LIGHT

Direct Path

Indirect Path

Indirectinhibitory

component (I)

D + I




Visual CortexVisual Cortex

aka:Primary visual cortexStriate cortexBrodman’s area 17

Cortical Area V1

ThalamusLGN

Striatecortex(V1)

DorsalStream

Parietalvisualcortex

Temporalvisualcortex

Eye Opticnerve

Extrastriatecortex

VentralStream

Cortical Receptive Fields

Single-cell recording from visual cortex

David Hubel & Thorston Wiesel© Stephen E. Palmer, 2002


Single-cell recording from visual cortex

TimeTime



Three classes of cells in V1

Simple cells

Complex cells

Hypercomplex cells



Simple Cells: “Line Detectors”

A. Light Line Detector

Horizontal Position

FiringRate

B. Dark Line Detector

Horizontal Position

FiringRate



Simple Cells: “Edge Detectors”

C. Dark-to-light Edge Detector

Horizontal Position

FiringRate

D. Light-to-dark Edge Detector

Horizontal Position

FiringRate



Constructing a line detector

Receptive Fields

Retina LGN

Center-Surround Cells

Simple Cell

CorticalArea V1



Complex Cells

STIMULUS NEURAL RESPONSE

Time

00o



Complex Cells


Time

060o



Complex Cells


Time

090o



Complex Cells


Time

0120o



Constructing a Complex Cell

Simple Cells

Cortical Area V1

Complex CellReceptive Fields

Retina



Hypercomplex Cells

Time




Time


“End-stopped” Simple Cells



Constructing a Hypercomplex Cell

Receptive Fields

RETINA CORTICAL AREA V1

Complex Cell End-stopped Cell


Mapping from Retina to V1Mapping from Retina to V1

Why edges?

So, why “edge-like” structures in the Plenoptic Function?

physiology of vision: a swift overview 16-721: learning-based methods in vision a. efros, cmu,...

Documents

figures stephen

single timeas

normal distributions

single view pointat

function of wavelengthpq

single view pointover

single timeaveraged

deg w x