Stereo — 2D to 3D on an FPGA

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas

Massachusetts Institute of Technology

baxelrod, xnie, asbiswas

November 10, 2015

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 1 / 23

What is Stereo

Monocular

Binocular

two perspectives

Algorithmic Idea

Computationally Difficult

2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23

What is Stereo

Monocular

Binocular

two perspectives

Algorithmic Idea

Computationally Difficult

2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23

What is Stereo

Monocular

Binocular

two perspectives

Algorithmic Idea

Computationally Difficult

2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23

What is Stereo

Monocular

Binocular

two perspectives

Algorithmic Idea

Computationally Difficult

2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23

What is Stereo

Monocular

Binocular

two perspectives

Algorithmic Idea

Computationally Difficult

2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23

What is Stereo

Monocular

Binocular

two perspectives

Algorithmic Idea

Computationally Difficult

2x HD camera

4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23

What is Stereo

Monocular

Binocular

two perspectives

Algorithmic Idea

Computationally Difficult

2x HD camera4 million pixels

4.3 Trillion possible correspondencesEven smart algorithms require a lot of power

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23

What is Stereo

Monocular

Binocular

two perspectives

Algorithmic Idea

Computationally Difficult

2x HD camera4 million pixels4.3 Trillion possible correspondences

Even smart algorithms require a lot of power

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23

What is Stereo

Monocular

Binocular

two perspectives

Algorithmic Idea

Computationally Difficult

2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23

Overall System Layout

cam1

cam2

preprocessing 1

preprocessing 2 ddr buffer sgm ddr buffer rendering

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 3 / 23

Camera Pipeline

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 4 / 23

Camera Pipeline

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 5 / 23

Why Rectification?

Intrinsic optical distortion

Improper alignment of the cameras (rotation, translation)

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 6 / 23

Why Rectification?

Intrinsic optical distortion

Improper alignment of the cameras (rotation, translation)

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 6 / 23

Rectification

Calibrate cameras offline (MATLAB)

Acquire rotation, translation matrix coefficients

Acquire intrinsic distortion parameters

Apply the matrix transformations to streamed images (real-time)

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23

Rectification

Calibrate cameras offline (MATLAB)

Acquire rotation, translation matrix coefficients

Acquire intrinsic distortion parameters

Apply the matrix transformations to streamed images (real-time)

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23

Rectification

Calibrate cameras offline (MATLAB)

Acquire rotation, translation matrix coefficients

Acquire intrinsic distortion parameters

Apply the matrix transformations to streamed images (real-time)

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23

Rectification

Calibrate cameras offline (MATLAB)

Acquire rotation, translation matrix coefficients

Acquire intrinsic distortion parameters

Apply the matrix transformations to streamed images (real-time)

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23

Rectification: Example

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 8 / 23

Camera Pipeline

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 9 / 23

Gaussian Filtering

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 10 / 23

Gaussian Filtering

Reduce noise in the image

Convolution: weighted sum ofsurrounding pixels

Separate horizontal and verticalpasses

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 11 / 23

Camera Pipeline

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 12 / 23

Disparity

A shift to the left of an imagefeature when viewed in the rightimage

Disparity cost: associatedmatching cost between twopixels

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 13 / 23

Census Transform

Used to compute disparity matching costs

5x5 window for each pixel to represent the information from thesurroundings of the pixel

produces a bit stream

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 14 / 23

Left Image Right Image

Left Image Right Image

Depth/Disparity Map

Minimize Global Energy

Minimize Global Energy

Minimize Global Energy

D is a disparity image

Minimize Global Energy

Similarity Costs for Each Pixel

Minimize Global Energy

Small Disparity Change (Smoothness)

Minimize Global Energy

Small Disparity Change (Smoothness)

Small Penalty P1

Minimize Global Energy

Object Boundary

Minimize Global Energy

Object Boundary : Large Penalty P2

2D Global Minimization

2D Global Minimization

NP­Complete Problem

1D Optimization● Dynamic Programming

1D Optimization● Dynamic Programming● Minimize Cost along Horizontal Lines

1D Optimization● Dynamic Programming● Minimize Cost along Horizontal Lines

1D Optimization● Dynamic Programming● Minimize Cost along Horizontal Lines

Streaking Issues

Left Image Right Image

Streaking Issues

Use Multiple Directions

Use Multiple Directions

Use Multiple Directions

r varies over 8 directions

Use Multiple Directions

r varies over 8 directions

Total Cost :

Semi Global Matching

Left Image Right Image

No Streaking!!!

Streaming Design

Total Cost :

Depth Map Rendering

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 1 / 3

Point Cloud Rendering

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 3

Timeline

WEEKS: 0 1 2 3 4

Pre-Processing

Camera Capture

Rectification Parameters

Rectification

Census Transform

Gaussian Filtering

Pre-Processing Complete

Semi-Global Matching

CPU Testing

Streaming CPU Test

HLS First Pass

HLS Second Pass

HLS Testing

Memory Architecture

DMA

Reverse DMA

Integration

Integration/Testing

Post-Processing

Project Complete

Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 3 / 3