Panoramic imaging

Camera projectionsRecall the plenoptic function:

At any point in space, there is a full sphere of possible incidence directions , coveredby , . A regular camera captures the incident rays from some region arounda “forward” direction, and projects these directions onto rectilinear image coordinates inthe image plane by a linear perspective projection, or something reasonably close. This type ofprojection limits the field of view to strictly less than 180 degrees. Most cameras in fact have afield of view (FOV) that is only around 45 degrees. A lens with a field of view of 90 degrees isconsidered a very wide angle lens. The planar perspective projection is in fact unsuitable forwide angles. Objects at the image edges are projected in a very oblique direction towards theplane, and will have their proportions heavily distorted. A projection with a FOV close to 180degrees will spend most of its pixels for objects at the extreme edge of the view, whereas theimportant parts of the image are most probably in the middle.

For very wide angle lenses, the planar projection is abandoned in favor of the fisheye projection.A fisheye projection is a projection through a projection reference point, just like the planar per-spective projection, but it is performed against a sphere instead of a plane. The projection ref-erence point is at the center of the sphere. The surface of the sphere is then mapped to a planar

The scene from above (the small black object in the mid-dle marks the camera position for the following views)

45 degrees FOV(normal view)

90 degrees FOV(very wide angle view)

150 degrees FOV (extreme) 170 degrees FOV (strange) 175 degrees FOV (useless)

Figure 1: Planar perspective projections with increasing field of view

I x y z ϕ θ λ t, , , , , ,( )

x y z, , ϕ θ,0 ϕ 2π≤ ≤ 0 θ π≤ ≤

u v,( )

image, oftan so that the angle of incidence is mapped linearly to the radial distance to the imagecenter. Fisheye projections can have a very large field of view, 180 degrees or even more. Theprojection as such lends itself to a full 360 degrees field of view, even though it is difficult todesign actual lenses with a FOV significantly larger than 180 degrees.

Panorama mappingsThere are of course infinitely many ways of mapping from to when recording animage. Planar perspective projection and fisheye projection are just two examples. The planarperspective projection happens to be practical for most applications, and when it is inappropri-ate, the fisheye mapping will often do the job. Other mappings can be useful, though. In partic-ular, it is sometimes desirable to capture image data in every possible direction from one singlepoint of view.

An image with a full 360 degrees field of view is called a panoramic image, or a panorama.Recording and storing such images presents a mapping problem. All image recording and stor-age media that exist today are essentially flat. Digital images are perhaps not flat in the normalsense of the word, but they are two-dimensional data structures with an equidistant rectilinearmapping of image position to pixel coordinates, which really implies a flat image. Themapping for a panorama can be chosen in a number of different ways. The probem is equivalentto the cartographic problem of making a flat map of the whole Earth. Perhaps the most straight-forward mapping is to map directly onto . This is called a spherical mapping. An-other option is to exclude the top and bottom parts of the sphere and project the remainderagainst a cylinder. Such an image is called a cylindrical mapping. A third possibility is to mapthe environment sphere onto the six faces of a cube, and store the panorama as six square imag-es. Quite logically, this mapping is called a cubical mapping. Each of these three mappings havetheir benefits.

Figure 2: A fisheye projection with 180 degrees FOV

ϕ θ,( ) u v,( )

u v,( )

ϕ θ,( ) u v,( )

Figure 3: Common panorama mappings. From top to bottom: spherical, cylindrical, cubical.

Other mappings exist. One relatively convenient way of storing a panorama would be to storetwo 180 degree fisheye images, each covering one hemisphere. It is also possible to use a fish-eye mapping extended to 360 degrees field of view. Such mappings are irregular and highlynon-uniform, but they are sometimes used for environment maps, and recently also for imagebased lighting. The light probe mapping described and used by Debevec et al [http://www.de-bevec.org] is in fact a 360 degree fisheye projection, and the dual paraboloid environment map-ping described by Heidrich and Seidel [http://www.cs.ubc.ca/~heidrich/Papers/index.html] is akind of dual fisheye projection.

Panorama remappingsOne of the advantages with digital images over traditional image media is that they can easilybe non-uniformly spatially resampled (warped) in a totally arbitrary manner. It is therefore pos-sible to remap panoramas from one particular projection to any other projection, but some cau-tion is advisable, because the resampling might throw away significant amounts of data.Different panorama mappings have different sampling densities, and because there is simply noway of making a perfectly uniform mapping from the surface of a sphere to a plane, the sam-pling density will always vary somewhat over the panoramic image. Converting a panoramafrom one mapping to another means resampling an image from one irregular, curvilinear coor-dinate system to another, which is a lot more tricky than resampling between regular, rectilinearcoordinate systems. When dealing with panoramas, it is even more important than with regularimages to use a higher resolution for the input image than the final output image, and to avoidrepeated remapping operations. Using software with a good resampling algorithm can make abig difference. Commercial image editing tools are often insufficient. A very high quality re-sampling package is Panorama Tools by Helmut Dersch [http://www.fh-furtwangen.de/~der-sch/]. It is free, and it is designed for tricky panoramic remappings.

Panorama acquisitionPanoramic images are not a new invention. Panorama cameras have been available since at leastthe beginning of the 20th century, even though they have never had a big market. There are sev-eral different principles for acquiring a panorama image. Which one is best depends heavily onthe application. Each of the methods presented below has its its own merits.

Scanning panorama camerasClassic panorama cameras, and some modern digital versions as well, operate by the scanningslit principle. The shutter opening of the camera is a vertical slit, the camera rotates on a tripodduring the exposure, and the film is fed past the slit with a speed that matches the translation ofthe scene as a result of the camera rotation. The width of the slit can be adjusted to change theaperture. Scanning slit panoramic cameras use standard film, and capture a 360 degree pano-rama horizontally on a wide film strip. Their vertical field of view, however, is comparable toa regular camera and therefore significantly less than 180 degrees. The panoramas obtained inthis manner are cylindrical panoramas. Scanning slit cameras can also capture panoramas withless than 360 degrees horizontal field of view, simply by stopping before one full turn is com-pleted.

One-shot, single view panorama camerasIt is possible to design optical systems that directly capture the full environment sphere. A fish-eye lens with a 360 degrees field of view is impossible to build, but by using curved mirrors a360 degrees field of view can be achieved.

The simplest setup uses a reflective sphere. An image of a reflective sphere contains direct re-flections from all (or at least most) incidence directions. Such images can be used straight offfor panoramas, but their bad sampling uniformity makes it advisable to remap them before stor-age. A reflective sphere image may be remapped to a 360 degree fisheye projection by a resa-mpling in the radial direction.

One-shot, multi-view panorama camerasAnother way of designing a camera that captures a panorama in a single shot is to use multiplelenses and acquire images from all lenses simultaneously, either by using multiple cameras, orby using mirrors or prisms to combine images from several lenses in a a single frame. Two 180degree fisheye images are enough to cover an entire sphere, but many fisheye lenses have prob-lems with the image quality towards the eges, with significant defocus, color aberration, flareand light falloff. For better image quality, four fisheye lenses in a tetrahedral configuration canbe used, but that of course requires twice as much hardware. A cubical panorama can be aquiredby six cameras facing front, back, left, right, up and down, each with a 90 degree field of view.90 degrees FOV is an unusually wide angle lens which is not readily available as inexpensivestandard equipment, but such lenses do exist.

A problem which becomes apparent with single-shot panoramas is that a photographer cannothold the camera, or even be near it, without being in clear view in the shot. This can be solvedeither by remote control or a timer release. The camera mount will still present a problem,though. Sometimes it is OK to leave it in the image, other times some manual retouching of theimage might be required. Scanning or multi-shot panorama acquisition methods make it easierto keep the photographer and the camera mount out of view.

Panoramic movie camerasOne-shot panorama cameras (single- or multi-view) can of course capture movies instead of stillimages. Panoramic film or video sequences have found some applications, and a couple ofpanorama video systems are on the market.

Figure 4: An image of a reflective sphere. Background masked black for clarity.

Multi-shot panoramasInstead of acquiring several images at once, a single camera can be used to capture each part ofa panorama in sequence. A 180 degree fisheye lens ideally requires only two shots, even thoughthree or four might be needed to better avoid edge problems. A fisheye lens makes the methodquite covenient, but even a regular planar projection lens with a smaller field of view can beused, along with a good tripod mount, to capture a dozen or more images in different directions.It takes some cutting and pasting to combine the images to a panorama, but it requires no specialcamera equipment. Furthermore, the image quality is excellent, because the method uses manyhigh quality source images, each with a very uniform sampling. The recombination of severalimages to a panorama is called stitching. Software tools exist to assist panorama photographersin stitching, and even make the process more or less automatic.

One disadvantage of stitching, apart from that it is cumbersome, is that the images are not takensimultaneously. If the scene changes or the light conditions change while the images are ac-quired, there will be problems stitching them together to a consistent panorama. Effects fromstrong onlight, such as glare and lens flare, will also present a problem, since glare and flare willonly appear in images where a light source is visible, and not in adjacent images in the sequence.

Panorama viewersA panorama can be viewed in two quite different ways. Either the entire panorama is displayedon a surface that encloses the viewer, and lets him or her look around freely. This method is suit-able for large audiences, but it requires expensive display equipment and a lot of open space. Amore common method is to use an interactive computer program to remap part of the panoramato a planar perspective projection with a smaller field of view, and giving the user control overthe direction of view, and possibly also the field of view. This gives the viewer an impressionof being in a scene and looking around with a camera, which is often enough to invoke a senseof immersion.

Remapping a panoramic projection to a planar perspective projection is not a simple operation,and an image contains a lot of data. It is in fact not until recently that computers have been ableto handle that kind of heavy computations with a reasonable speed and sufficient image quality.The recent development of inexpensive 3-D graphics accelerators also presents an alternativemapping method: the panorama can be placed as a texture on the inside of a three-dimensional

Figure 5: Unfinished stitched panorama (6 images placed, each with 45 degrees FOV)

shape that matches the panorama projection (a sphere, a cylinder or a cube), and the viewingcan then be taken care of by a hardware accelerated rendering of the view from a virtual cameraplaced at the center. Cubical panoramas are particularly suitable for this, because their projec-tion fits nicely into today’s polygon and texture rendering pipelines.

Commercial applicationsThere are actually not that many commercial applications for panoramic images, but now thatcomputers are able to handle them more easily and with high quality, we will probably see moreof it in the near future.

A few computer games using panoramic images have been released over the years, the most re-cent one being Myst III: Exile. In Exile, the panorama viewer is enhanced to simulate glare, andparts of the scene are moving, which gives a very immersive and compelling effect.

For quite a few years by now, Apple Computers [www.apple.com] has been one of the leadersin the development of panoramic imaging with their software Quicktime VR. They deserve amention for providing reasonably open standards, while some others have been trying to markettheir software as secret magic. A particularly sad mix of marketing hype, prohibitively expen-sive software licensing and repeated and dubious attacks on free software developers comesfrom the company IPIX [www.ipix.com]. Their products are not bad, but they have made a lotof enemies over the years.

As panoramic imaging becomes more commonplace, it will be easier for companies to make aliving on selling their products, but right now, commercial panoramic imaging is a struggle be-tween only a few actors for shares of a too small market.

Stefan Gustavson, 2002-01-22