Real-Time Spherical Videos from a Fast Rotating Camera

Frank Nielsen, Alexis Andre and Shigeru Tajima

Sony Computer Science Laboratories Inc, Japan

Ecole Polytechnique, France

June 26th , 2008

Outline• Problem• Related Work• Two Prototypes:

–Line Sensor•Workflow•Results

–Area Sensor•Workflow•Results

Three core problems

• We want to generate (1) full spherical (2) videos at (3) high framerates (>25fps).

Main Contribution

• Framework for acquisition of full spherical videos with a special tailored fast-rotating camera

• High quality smooth image (no seams)

Problem 1: Full-Spherical Images

• Two traditional ways:–Stitching of a small number of images

taken with a fish eye lens (Quicktime VR® )

–Succession of 1-pixel wide vertical strips (Spheron™ )

Inherent problems

• Use of step motors for automatic acquisition– Slow– Need to rewind (avoid cable twist)

• Rough calibration is ok: a slight seam is acceptable (for discrete stitch)

Problem 2: Full-Spherical Videos

• Use of multiple cameras (Sony Fourth View)– Parallax problems

• Virtual panoramic videos (Agarwala et al.)– Wide field of view– Very low frame rate (wrt. the rotation speed)

Our approach

• Unique camera, rotating very fast

(1800rpm – 30fps)– No parallax problems (MCOP image)– No calibration between various cameras

• Connection– Slip rings -> no need for a step motor– Gigabit Ethernet connection with remote control

Challenge

• 1D rotation angles unknown:– Register the data: we need to find the

rotation angle:

– Then, extract frames every revolution

θ(t) ?

θ(t) = t mod(2π)

System Setup

Slip rings for Ethernet Connection

Camera Platform

Movie No parallax barrier

Line Area Sensor

• Black and white CCD • very fast sensor (67000 lines per second)• Smooth spherical image

(not unique center of projection, MCOP)

-> calibration not required

• Only 1/3 of the vertical axis (lens/sensor)• Choosing appropriate lens (tailored lens)

Raw Data: Plain unrolling

Acquisition speed may not be constant

Raw Data

Raw Data

One round One Round

Frame Registration

• Based on current data and past data

• We look for similar bands of pixels

(Dynamic programming-like search)– Only works if most of the scene is static

Registration

SSD

Score

ReferencePoint

CurrentPoint

Registration

SSD

Score

ReferencePoint

CurrentPoint

Registration

SSD

Score

ReferencePoint

CurrentPoint

Registration

SSD, or normalized mutual information, etc.

Registration

Minimum: One round!

Registration

Best Match

ReferencePoint

CurrentPoint

Registration

NextPoint Search Zone (expand)

Registration: Frame Extraction

Yields potential for super-resolution (non-uniform sampling)

Result movie (cropped vertically)

Results

• Rotation speed might not be uniform across one revolution:– Slight jittering/tripod

• Changes in speed affect resolution– In a production setting, speed must be controlled

carefully.

Issues

• Not robust to dynamic scenes

• Fails on homogenous scenes (ambiguity)

• Use of a small visual hint help registration– Bright LED at a specific position (visual pollution)

Area Sensor

• Align the nodal point with the rotation axis to avoid parallax (similar to spherical images problems)

• 24-bit color VGA frames• horizontal resolution fixed• need to control shutter time and/or rotation

speed to get enough overlap

Key difference: Camera can be calibrated exactly on its nodal point(multi-camera cannot)

“Traditional Approach”

• Stitch frames: need to register each frame with the previous ones

• Horizontal shift depends on the rotation speed and on the shutter time

• But resolution is fixed (programmable ROI)

Workflow for 2D stitching

Registration for two frames

Defish (=map to env. map.)

Registration: crude w/o blending

Actually perform only this in coordinate systems (no intermediate images)

Obvious seam

Smooth blending

Better methods: Laplacian multi-spectral, Poisson blending, etc.(but we are concerned with real-time systems…)

Panoramic roll: never-ending sequence

Artefact comes from the model of fish-eye projection. Calibrate rayel!

Results (movie)

Inherent Problems

• Lens imperfections / resolution decrease near the edges of the lens

• Perfect fisheye model or calibration is needed• Motion Blur

Imperfect Fisheye Model

• Slight seam– On static images: can be ignored

(error propagated to the edges of the image)– On movies: disturbing moving seam

Motion Blur

• Lighting conditions force us to use long exposure time (wrt. the hardware):– Motion blur appears– Ideally horizontal motion blur– At the desired speeds, unrecoverable blur

(horizontal) Motion blur

0rpm 150rpm 250rpm

Summary

• Path to high-frame rate full-spherical videos is promising– Better hardware is needed– H/W technologies is already here

• Classical framework– Established techniques– Optimization available

Future Work

• Horzontal deblur

• Registration in more dynamic scenes

• ROI selection

• Applications for AI applications

ETVC08Emerging Trends in

Visual Computing

Information geometry

Computer vision

Computer graphics

Machine learning

Computational Geometry

Registration is free