"digital gimbal: rock-steady video stabilization without extra weight!," a presentation...

Copyright © 2016 FotoNation 1

DIGITAL GIMBAL Rock-Steady Video Stabilization Without Extra Weight!

Dr. Petronel Bigioi May 3, 2016


IMAGING

Wide Expertise

Strong Innovation

Top Quality

SOFTWARE

Always Real Time

Platform Agnostic

Full Flexibility

HARDWARE

Proven

Low Power

High Performance

QA

Advanced Data Acquisition

Multi-million Image Database

Testing Automation Tools

FotoNation in a Nutshell


FotoNation in Numbers

550+ 1000+ 60% 2.5+ 450+

imaging patents man years of experience high end smartphone penetration billion units shipped million devices / year

understand ● enhance ● accelerate


DIGITAL GIMBAL

A state-of-the-art, energy-efficient image stabilization solution

for high frequency vibration by FotoNation


Problem Statement

High quality videos 4K @ 60 FPS are accepted

as norm in cameras on the fly and move

Low frequency motion is not the only problem

anymore

HIGH and VERY HIGH frequency motion due to

motors and activities need to be addressed

including motions with horizon lock


Mechanical Gimbal Solution — Current State-of-the-Art to

Attenuate Motors Vibration Effects

ADVANTAGES

DISADVANTAGES

WHAT IS IT? • Mechanical assembly built to counteract vibration effects

• Uses gyros in close loop to control motors

• Have large range of movement

• Cost!!!

• Large & inconvenient

• Heavy

• Consume power

• Reduce flight time due to extra weight

• Slow reaction time


Digital Gimbal Solution — FotoNation’s Technology to

Deal with Vibrations (Low and High Frequency)

ADVANTAGES

DISADVANTAGES

WHAT IS IT?

• Algorithms removing the need for mechanical assembly; uses gyro to measure high

frequency vibration and digital image processing for frame to frame registration to reverse

the effects in the image/frames

• Low cost

• No extra weight

• Low power

• Ultra fast reaction time

• Advanced image stabilisation

• Rolling shutter correction

• Limited range of movement


Image Stabilization Requirements

Image stabilization has always been a subject of great interest especially when associated with camera modules on the move; today’s solutions are influenced by system requirements (cost) as well as jitter specific problems (usage).

Performance System requirements

good sampling quality

stable frame rate

small error estimations

low power consumption

future proof (2K, 4K, 8K)

roll (x-y rotation)

panning

out of plane rotations

rolling shutter distortions

large moving objects


The Rolling Shutter Problem

Hand jitter vibration patterns Drone/motorbike jitter vibration patterns

Rolling shutter effects — caused by slow

camera motion during line by line exposure

and frame readout

Rolling shutter effects — caused by rapid

camera motion during line by line exposure

and frame readout

Have low frequency (up to 15Hz) Have high frequency (hundreds of Hz)


ISP

Inertial

Sensor

Motion

Filtering

Camera

projection

model

Motion

Correction

Image

Corrected

Image

Correction

Grid

Motion

Estimation

Image

Sensor

Legend:

Memory Buffer

Digital Gimbal Components

HW Module App Processor & DDR

256 independent motion vectors; IMU samples

acquisition and sync; sensor fusion

(IMU + frame to frame)

Corrects lens distortion, roll, pan & complex

rolling shutter in 1 pass

Adaptive motion filtering & rolling

shutter effects estimation

System Architecture and Workflow


Motion Estimation — Synchronization

• Accurate synchronization of inertial and video data is essential

• Hardware core used to read inertial data (IMU) and assign precise timestamps,

with same time source for frames and IMU samples

• Two possible implementations

• Pure HW implementation — ensures minimum latency and highest timing

accuracy, low power consumption

• Hybrid implementation — more flexible, but higher power. Requires high

sampling frequency to minimize motion measurement errors


SOF

Interrupt SOF

Interrupt Sample with

FSYNC marker

Sample with

FSYNC marker

unknown time

periods

Maximum unknown period is equal to IMU sample interval. Very inaccurate for low sampling frequencies

Hybrid Synchronization Timing Accuracy Limit


Bias — a constant value added to the measured signal

Motion Filtering — IMU calibration


Sampled IMU data needs to be filtered (4x rule) in order to avoid aliasing

Motion Filtering — Data Filtering


Use sensor fusion to track camera

orientation over time

Using gyroscope alone to track camera

orientation leads to error accumulation

Acceleration and magnetic field lack short-term accuracy but provide good long-term reference frame

Gyroscope provides short-

term accurate changes in

the camera orientation

Motion Filtering — Sensor Fusion


• The real camera orientation is tracked using quaternion arithmetic

• The required correction is the difference between the real and the estimated

trajectory

• Our motion filters adapts to the camera motion in order to produce naturally

smooth experience and react fast to the intentional motion

Motion Filtering


• Stabilisation margin is limited by the sensor size. Sudden camera movements can cause

optimal correction to exceed available correction margin. Excessive motion needs to be

limited to the available correction margin

• Limits imposed on 2D projection must control 3D rotation represented as 4D quaternion —

difficult task. Problem gets even more complicated for highly distorted rectilinear lenses

(left) or fisheye lenses (right)

Motion Filtering — Motion Limitation


tim

e

reference line

• Orientation of the camera in the first line is the

reference point

• Each consecutive line is transformed to counteract

camera movement up to the given line

• Inertial samples and the video frames must be

precisely synchronized

• Camera projection must be accurately modeled

Motion Correction — Rolling Shutter Correction


Camera frame

reference

vector

Gravity vector

from

accelerometer

Correction

amount

After correction

vectors are

aligned

The horizon locking feature keeps the horizon line in the middle of the

frame and horizontal regardless of the drone orientation.

Motion Correction — Horizon Locking Option


Motion Correction With “Follow Me” Feature /

Owner Tracking


FUTURE

PROOF

FAST TO

DEPLOY EFFICIENTLY

DESIGNED

In a seamless way that is simultaneously

A DEDICATED

HARDWARE — IPU

Low gate count implementation

to offload heavy image and video

computation. Most difficult tasks

completed locally and then passing

processed data to host CPU

How Do We Deliver This?


OBJECT DETECTION ENGINE

~ 1M gates, 240 kB Sram, 54 mW

Multi-Core CPU

GPU DSP ISP

3G/4G Baseband

Memory & I/Os IPU

Video Encoder/Decoder

MOTION PROCESSING ENGINE

~ 125K gates, 9.8 kB Sram, 2 mW

DISTORTION CORRECTION ENGINE

~ 390K gates, 48 kB Sram, 18 mW

4K@60FPS, 28nm technology

Typical AP

IPU Components Overview


• Precise lens modeling allows for lens distortion correction and re-projection.

• Any type of correction is possible (rectilinear to perspective, fisheye to

perspective, distorted fisheye to perfect fisheye, fisheye to cylindrical, arbitrary

input projection to arbitrary output projection and freeform image warping).

• All distortion corrections are done at the same time with stabilisation and high

frequency rolling shutter correction (single pass).

• Adding distortion correction has no impact on system performance.

• Frames are resampled only once to ensure maximum image quality.

• Lowest possible power consumption (18mW for 4k@60fps correction in 28n

technology).

Motion Correction — Uses FotoNation’s DCE

(Distortion Correction Engine)


DCE GPU

Designed to work with

rectangular texture mapping

for 2D image transformations

Native support for

bicubic resampling

Scalable with increasing input

size and frame rata

Small gate count

Low power consumption

Low bandwidth thanks to

specific cache design

Designed to support generic

triangular texture mapping

for 3D scenes rendering

Bi-cubic resampling requires

extra processing power &

time

Designed to work at display

resolution

Large gate count

High power consumption

High bandwith

Motion Correction — Done DCE (Distortion Correction

Engine) Part of FotoNation’s IPU (Image Processing Unit)


INPUT OUTPUT

Correction — DCE example

Corrected by FotoNation DCE


Correction — DCE example

INPUT OUTPUT

Corrected by FotoNation DCE


FotoNation EIS / High Frequency Stabilization


• Digital Gimbal components (synchronization, motion filtering and

correction) are part of FotoNation’s IPU (Image Processing Unit) for

optimum low power high performance implementation.

• IPU incorporates additional units to enable more features such as owner

recognition and tracking combined with image stabilization (e.g. high

performance object detection and tracking, high performance face

detection and tracking and face recognition).

Conclusion


Thank You


FotoNation EIS / A No Trade-off User Experience