![Page 1: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/1.jpg)
Stereo — 2D to 3D on an FPGA
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas
Massachusetts Institute of Technology
baxelrod, xnie, asbiswas
November 10, 2015
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 1 / 23
![Page 2: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/2.jpg)
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
![Page 3: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/3.jpg)
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
![Page 4: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/4.jpg)
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
![Page 5: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/5.jpg)
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
![Page 6: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/6.jpg)
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
![Page 7: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/7.jpg)
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera
4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
![Page 8: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/8.jpg)
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels
4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
![Page 9: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/9.jpg)
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondences
Even smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
![Page 10: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/10.jpg)
What is Stereo
Monocular
Binocular
two perspectives
Algorithmic Idea
Computationally Difficult
2x HD camera4 million pixels4.3 Trillion possible correspondencesEven smart algorithms require a lot of power
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 23
![Page 11: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/11.jpg)
Overall System Layout
cam1
cam2
preprocessing 1
preprocessing 2 ddr buffer sgm ddr buffer rendering
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 3 / 23
![Page 12: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/12.jpg)
Camera Pipeline
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 4 / 23
![Page 13: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/13.jpg)
Camera Pipeline
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 5 / 23
![Page 14: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/14.jpg)
Why Rectification?
Intrinsic optical distortion
Improper alignment of the cameras (rotation, translation)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 6 / 23
![Page 15: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/15.jpg)
Why Rectification?
Intrinsic optical distortion
Improper alignment of the cameras (rotation, translation)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 6 / 23
![Page 16: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/16.jpg)
Rectification
Calibrate cameras offline (MATLAB)
Acquire rotation, translation matrix coefficients
Acquire intrinsic distortion parameters
Apply the matrix transformations to streamed images (real-time)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23
![Page 17: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/17.jpg)
Rectification
Calibrate cameras offline (MATLAB)
Acquire rotation, translation matrix coefficients
Acquire intrinsic distortion parameters
Apply the matrix transformations to streamed images (real-time)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23
![Page 18: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/18.jpg)
Rectification
Calibrate cameras offline (MATLAB)
Acquire rotation, translation matrix coefficients
Acquire intrinsic distortion parameters
Apply the matrix transformations to streamed images (real-time)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23
![Page 19: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/19.jpg)
Rectification
Calibrate cameras offline (MATLAB)
Acquire rotation, translation matrix coefficients
Acquire intrinsic distortion parameters
Apply the matrix transformations to streamed images (real-time)
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 7 / 23
![Page 20: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/20.jpg)
Rectification: Example
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 8 / 23
![Page 21: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/21.jpg)
Camera Pipeline
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 9 / 23
![Page 22: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/22.jpg)
Gaussian Filtering
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 10 / 23
![Page 23: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/23.jpg)
Gaussian Filtering
Reduce noise in the image
Convolution: weighted sum ofsurrounding pixels
Separate horizontal and verticalpasses
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 11 / 23
![Page 24: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/24.jpg)
Camera Pipeline
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 12 / 23
![Page 25: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/25.jpg)
Disparity
A shift to the left of an imagefeature when viewed in the rightimage
Disparity cost: associatedmatching cost between twopixels
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 13 / 23
![Page 26: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/26.jpg)
Census Transform
Used to compute disparity matching costs
5x5 window for each pixel to represent the information from thesurroundings of the pixel
produces a bit stream
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 14 / 23
![Page 27: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/27.jpg)
Left Image Right Image
![Page 28: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/28.jpg)
Left Image Right Image
Depth/Disparity Map
![Page 29: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/29.jpg)
Minimize Global Energy
![Page 30: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/30.jpg)
Minimize Global Energy
![Page 31: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/31.jpg)
Minimize Global Energy
D is a disparity image
![Page 32: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/32.jpg)
Minimize Global Energy
Similarity Costs for Each Pixel
![Page 33: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/33.jpg)
Minimize Global Energy
Small Disparity Change (Smoothness)
![Page 34: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/34.jpg)
Minimize Global Energy
Small Disparity Change (Smoothness)
Small Penalty P1
![Page 35: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/35.jpg)
Minimize Global Energy
Object Boundary
![Page 36: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/36.jpg)
Minimize Global Energy
Object Boundary : Large Penalty P2
![Page 37: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/37.jpg)
2D Global Minimization
![Page 38: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/38.jpg)
2D Global Minimization
NPComplete Problem
![Page 39: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/39.jpg)
1D Optimization● Dynamic Programming
![Page 40: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/40.jpg)
1D Optimization● Dynamic Programming● Minimize Cost along Horizontal Lines
![Page 41: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/41.jpg)
1D Optimization● Dynamic Programming● Minimize Cost along Horizontal Lines
![Page 42: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/42.jpg)
1D Optimization● Dynamic Programming● Minimize Cost along Horizontal Lines
![Page 43: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/43.jpg)
Streaking Issues
Left Image Right Image
![Page 44: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/44.jpg)
Streaking Issues
![Page 45: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/45.jpg)
Use Multiple Directions
![Page 46: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/46.jpg)
Use Multiple Directions
![Page 47: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/47.jpg)
Use Multiple Directions
r varies over 8 directions
![Page 48: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/48.jpg)
Use Multiple Directions
r varies over 8 directions
Total Cost :
![Page 49: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/49.jpg)
Semi Global Matching
Left Image Right Image
![Page 50: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/50.jpg)
No Streaking!!!
![Page 51: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/51.jpg)
Streaming Design
![Page 52: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/52.jpg)
Total Cost :
![Page 53: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/53.jpg)
Depth Map Rendering
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 1 / 3
![Page 54: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/54.jpg)
Point Cloud Rendering
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 2 / 3
![Page 55: Stereo 2D to 3D on an FPGA - MITweb.mit.edu/6.111/www/f2015/projects/asbiswas_Project... · 2015-11-13 · Stereo | 2D to 3D on an FPGA Brian Axelrod, Sheena Nie, Amartya Shankha](https://reader030.vdocuments.us/reader030/viewer/2022040307/5ed1acccef23f6521d326f28/html5/thumbnails/55.jpg)
Timeline
WEEKS: 0 1 2 3 4
Pre-Processing
Camera Capture
Rectification Parameters
Rectification
Census Transform
Gaussian Filtering
Pre-Processing Complete
Semi-Global Matching
CPU Testing
Streaming CPU Test
HLS First Pass
HLS Second Pass
HLS Testing
Memory Architecture
DMA
Reverse DMA
Integration
Integration/Testing
Post-Processing
Project Complete
Brian Axelrod, Sheena Nie, Amartya Shankha Biswas (MIT) Short title November 10, 2015 3 / 3