Vision, Video

and Virtual Reality Omnidirectional VisionOmnidirectional Vision

Lecture 6Omnidirectional Cameras

CSC 59866CDFall 2004

Zhigang Zhu, NAC 8/203Ahttp://www-cs.engr.ccny.cuny.edu/~zhu/

Capstone2004/Capstone_Sequence2004.html

Vision, Video

and Virtual Reality Lecture OutlineLecture Outline Applications

Robot navigation, Surveillance, Smart rooms Video-conferencing/ Tele-presence Multimedia/Visualization

Page of Omnidirectional Vision (Many universities and companies….) http://www.cis.upenn.edu/~kostas/omni.html

Design Requirements 360 degree FOV, or semi-sphere or full sphere in one snapshot Single effective viewpoint Image Resolutions – one or more cameras? Image Sharpness – optics as well as geometry

Several Important Designs Catadioptric imaging : mirror (reflection) + lens ( refraction) Mirrors: Planar, Conic, Spherical, Hyperboloidal, Ellipsoidal, Paraboloidal Systematic design ( S. Nayar’s group)

Calibrations Harder or simpler?

Vision, Video

and Virtual Reality Sensor DesignSensor Design Catadioptric imaging :

mirror (reflection) + lens ( refraction) Theory of Catadioptric Image Formation ( S. Nayar’s group)

"A Theory of Single-Viewpoint Catadioptric Image Formation" , Simon Baker and Shree K. Nayar ,International Journal of Computer Vision, 1999.

Mirrors Planar Conic, Spherical Hyperboloidal, Ellipsoidal Paraboloidal

Cameras (Lens) Perspective (pinhole) or orthogonal (tele-centric lens) projection One or more?

Implementations Compactness - size, support, and installation Optics – Image sharpness, reflection, etc.

Vision, Video

and Virtual Reality Planar MirrorPlanar Mirror

Panoramic camera system using a pyramid with four (or more) planar mirrors and four (or more) cameras (Nalwa96) has a single effective viewpoint

4 camera design and 6 camera prototype:

FullView - Lucent Technology http://www.fullview.com/

6 cameras

Mirror pyramid

Vision, Video

and Virtual Reality Planar MirrorPlanar Mirror

Panoramic camera system using a pyramid with four (or more) planar mirrors and four (or more) cameras (Nalwa96) has a single effective viewpoint

P1

P2

Viewpoint of the Virtual camera

Geometry of 4 camera approach: four separate cameras in 4 viewpoints can generate images with a single effective viewpoint

Vision, Video

and Virtual Reality Planar Mirror ApproachPlanar Mirror Approach

A single effective viewpoint More than one cameras High image resolution

Vision, Video

and Virtual Reality Planar Mirror ApproachPlanar Mirror Approach

A single effective viewpoint More than one cameras High image resolution

Vision, Video

and Virtual Reality Conic MirrorConic Mirror Viewpoints on a circle semispherical view except occlusion Perspective projection in each direction Robot Navigation (Yagi90, Zhu96/98)

viewpoint

pinhole

Vision, Video

and Virtual Reality Spherical MirrorSpherical Mirror

Viewpoints on a spherical-like surface Easy to construct (Hong91 -UMass )

Intersection of incoming rays are along this lineLocus of

viewpoints

Vision, Video

and Virtual Reality Hyperboloidal MirrorHyperboloidal Mirror Single Viewpoint

if the pinhole of the real camera and the virtual viewpoint are located at the two loci of the hyperboloid

Semi-spherical view except the self occlusion

pinhole

P1

viewpoint

P2

Rotation of the hyperbolic curve generates a hyperboloid

Vision, Video

and Virtual Reality Hyperboloidal MirrorHyperboloidal Mirror ACCOWLE Co., LTD, A Spin-off at Kyoto University

http://www.pluto.dti.ne.jp/~accowle1/   Spherical Mirror Hyperbolic Mirror

Image: High res. in the top

Vision, Video

and Virtual Reality Ellipsoidal MirrorEllipsoidal Mirror Single Viewpoint

if the pinhole of the real camera and the virtual viewpoint are located at the two loci of the ellipsoid

Semi-spherical view except the self occlusion

pinhole

viewpoint

P1

P2

Vision, Video

and Virtual Reality Panoramic Annular Lens

panoramic annular lens (PAL)- invented by P. Greguss* 40 mm in diameter, C-mount* view: H: 360, V: -15 ~ +20* single view point (O)

- geometric mathematical model for image transform & calibration

p p1

pinhole

P1

P

B

O

C

Ellipsoidal mirror

Hyperboloidal mirror

Vision, Video

and Virtual Reality Panoramic Annular Lens

panoramic annular lens (PAL)- invented by P. Greguss* 40 mm in diameter, C-mount* view: H: 360, V: -15 ~ +20•single view point (O)•C-Mount to CCD Cameras

Image: High res. In the bottom

Vision, Video

and Virtual Reality Cylindrical panoramic un-warping

Circular to cylindrical transformationafter eliminating radial distortion

Two Steps:

(1). Center determination

(2) Distortion rectification

2-order polynomial approximation

Vision, Video

and Virtual Reality Paraboloidal MirrorParaboloidal Mirror

Semi-spherical view except the self occlusion Single Viewpoint at the locus of the paraboloid, if

Tele-lens - orthographic projection is used Mapping between image, mirror and the world invariant to

translation of the mirror. This greatly simplifies calibration and the computation of perspective images from paraboloidal images

P1

viewpoint

tele-lens

P2

Vision, Video

and Virtual Reality Paraboloidal MirrorParaboloidal Mirror

Remote Reality – A Spin-off at Columbia University

http://www.remotereality.com/

Camcorder Web Camera Back to Back : Full Spherical View

Vision, Video

and Virtual Reality Paraboloidal MirrorParaboloidal Mirror

Remote Reality – A Spin-off at Columbia University

http://www.remotereality.com/

Vision, Video

and Virtual Reality Catadioptric Camera CalibrationCatadioptric Camera Calibration

Omnidirectional Camera Calibration – Harder or Easier? In general, the reflection by the 2nd order surface makes

the calibration procedure harder However, 360 view may be helpful

Paraboloidal mirror + orthogonal projection Mapping between image, mirror and the world invariant to

translation of the mirror. Projections of two sets of parallel lines suffice for intrinsic

calibration from one view C. Geyer and K. Daniilidis, "Catadioptric Camera calibration",

In Proc. Int. Conf. on Computer Vision, Kerkyra, Greece, Sep. 22-25, pp. 398-404, 1999.

Vision, Video

and Virtual RealityImage Properties of Paraboloid System Image Properties of Paraboloid System

The Image of a Line is a circular arc if the line is not parallel to the optical axis Is projected on a (radial) line otherwise

Dual Vanishing Points There are two VPs for each set of parallel lines, which are

the intersections of the corresponding circles Collinear Centers

The center of the circles for a set of parallel lines are collinear

Vanishing Circle The vanishing points of lines with coplanar directions* lie

on a circle ( all the lines parallel to a common plane)

(Assuming aspect ratio = 1)

Vision, Video

and Virtual RealityImage Properties of Paraboloid System Image Properties of Paraboloid System

The Image Center Is on the (“vanishing”) line connecting the dual vanishing

points of each set of parallel lines Can be determined by two sets of parallel lines

Projection of a Line with unknown aspect ratio Is an elliptical arc in the general case

The Aspect Ratio Is determined by the ratio of the lone-short axes of the

ellipse corresponding to a line Intrinsic Calibration

Estimate aspect ratio by the ratio of ellipse Estimate the image center by the intersection of vanishing

lines of two sets of parallel lines in 3-D space

(with aspect ratio)

Vision, Video

and Virtual RealityCalibration of Paraboloid System Calibration of Paraboloid System

The Image Center Is on the (“vanishing”) line connecting the dual vanishing

points of each set of parallel lines Can be determined by two sets of parallel lines

Vision, Video

and Virtual RealityCalibration of Paraboloid System Calibration of Paraboloid System

The Image Center Yellow “vanishing” line of horizontal set of parallel lines Pink “vanishing” line of vertical set of parallel lines

The Vanishing Circle (Red dotted) The vanishing points of lines with coplanar directions ( on a plane in this example)

Projected to the plane of the calibration pattern

Vision, Video

and Virtual Reality NextNext

Next: Features

END