xing mei ; xun sun ; mingcai zhou ; shaohui jiao ; haitao wang ; xiaopeng...
DESCRIPTION
On Building an Accurate Stereo Matching System on Graphics Hardware. Xing Mei ; Xun Sun ; Mingcai Zhou ; Shaohui Jiao ; Haitao Wang ; Xiaopeng Zhang Samsung Advanced Institute of Technology, China Lab Computer Vision Workshops, 2011 IEEE. Outline. Introduction Related Works - PowerPoint PPT PresentationTRANSCRIPT
Xing Mei ; Xun Sun ; Mingcai Zhou ; Shaohui Jiao ;
Haitao Wang ; Xiaopeng Zhang Samsung Advanced Institute of Technology, China Lab
Computer Vision Workshops, 2011 IEEE
On Building an Accurate Stereo Matching System on
Graphics Hardware
Outline• Introduction• Related Works• Algorithmn• CUDA Implementation• Experimental Results• Conclusion
Introduction
IntroductionDense two-frame stereo matching
• Compute a disparity map from stereo images.• Broad applications: 3D reconstruction, view interpolation
Related Works
Related Works• Local methods• Compute each pixel’s
disparity independently over a local support region.• Fast but inaccurate.
•Global methods• Solve the stereo problem in
an energy minimization process.• Accurate but slow due to
time-comsuming global optimizer.(GC,BP)
Related Works• Propagation-based methods• Produce quasi-dense or dense disparity results from a set of seed pixels.
• Relatively fast but sensitive to early wrong matches• use segmented regions as guided propagation unit• expensive cost
Related Works• Introduce a simple guided unit for propagation
: pixel-wise 1D line segments. • No image segmentation required here.• Simple, fast and accurate
Algorithmn
Algorithmn• Framework
Multi-step Disparity Refinement
Scanline Optimization
Cross-based Cost Aggregation
AD-Census Cost InitializationInput: Stereo images
Output: Disparity map
Algorithmn
Multi-step Disparity Refinement
Scanline Optimization
Cross-based Cost Aggregation
AD-Census Cost InitializationInput: Stereo images
Output: Disparity map
Disparity Cost Computing• Cost mesure : AD, BT, gradient-based measures, non-
parametric transforms(rank/census[3])......• Combination : SAD + gradient[6] , AD + Census• AD (Absolute Distance)• Constant color assumption• Repetitive structures
• Census • Encodes local image structures • Textureless regions
[3] H. Hirschmuller and D. Scharstein. “Evaluation of stereo matching costs on images with radiometric differences.”IEEE TPAMI, 31(9):2009.[6] A. Klaus, M. Sormann, and K. Karner. “Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure.” ICPR,2006.
AD-Census Cost Initialization+• p : pixel• d : level• >> a robust function on variable 𝑐
•
• pd = (x-d,y) in the right image
• : Hamming distance[22]
[22] R. Zabih and J. Woodfill. “Non-parametric local transforms for computing visual correspondence.” In Proc. ECCV, 1994.
d
Left I Right I
Census Transform
1 1 0 0 0
1 1 0 0 0
1 1 X 0 0
0 0 0 1 1
1 1 1 1 1
121 130 26 31 39
109 115 33 40 30
98 102 78 67 45
47 67 32 170 198
39 86 99 159 210
1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 0 0 1 1 1 1 1 1 1Census transform window :
Census Hamming Distance
• Left image
• Right image
Hamming Distance = 3
1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 0 0 1 1 1 1 1 1 1
1 1 1 0 0 1 1 0 0 1 0 1 0 0 0 0 0 1 1 1 1 1 1 1
XOR
0 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0
AD-Census Cost Initialization
+
• > >> a robust function on variable 𝑐
AD-Census Cost Initialization
• AD-Census measure produces proper disparity results for both repetitive structures and textureless regions.
Algorithmn
Multi-step Disparity Refinement
Scanline Optimization
Cross-based Cost Aggregation
AD-Census Cost InitializationInput: Stereo images
Output: Disparity map
• Cross construction
• Line ending points P1, P2 for P are located when rule 1 or 2 are violated:R1: Color self-similarity in the line region: smooth depth assumption
R2: Arm length limitation: avoid over-smoothness
Cross-based Cost Aggregation[23]
[23] K. Zhang, J. Lu, and G. Lafruit. “Cross-based local stereo matching using orthogonal integral images.” IEEE TCSVT,2009.
Cross-based Cost Aggregation
Cross-based Cost Aggregation• Enhance cross construction(use pixel p’s left arm and the endpoint pixel pl as an example)
• • •
Cross-based Cost Aggregation• Cost aggregation
• Run this step for 4 iterations to get stable cost values. • For iteration 1 and 3, aggregated horizontally and then vertically. • For iteration 2 and 4, aggregated vertically and then horizontally.
• Reduce the errors at depth discontinuities.
Cross-based Cost Aggregation
• Our aggregation method can better handle large textureless regions and depth discontinuities.
Cross-based Cost Aggregation
[21] K.-J. Yoon and I.-S. Kweon. “Adaptive support-weight approach for correspondence search.” IEEE TPAMI, 2006.[23] K. Zhang, J. Lu, and G. Lafruit. “Cross-based local stereo matching using orthogonal integral images.” IEEE TCSVT,2009.
Algorithmn
Multi-step Disparity Refinement
Scanline Optimization
Cross-based Cost Aggregation
AD-Census Cost InitializationInput: Stereo images
Output: Disparity map
Scanline Optimization[2]
• 4 scanline optimization processes are performed independently.• 2 horizontal directions• 2 vertical directions
𝐶2
𝐶𝑟
𝐶𝑟𝐶𝑟
𝐶𝑟
[2] H. Hirschmuller. Stereo processing by semiglobal matching and mutual information.” IEEE TPAMI, 2008.
r : directionp-r : the previous pixel along the same direction𝑃1, 𝑃2 : penalize the disparity changes between neighboring
pixels. (𝑃1 ≤ 𝑃2 ) [8]
Scanline Optimization
[8]S. Mattoccia, F. Tombari, and L. D. Stefano. “Stereo vision enabling precise border localization within a scanline optimization framework.” In Proc. ACCV, pages 517–527, 2007.
pp-rr
Scanline Optimization• The final cost :
• The disparity with the minimum 𝐶2 value is selected as pixelp’s intermediate result.
𝐶2
𝐶𝑟
𝐶𝑟𝐶𝑟
𝐶𝑟
Algorithmn
Multi-step Disparity Refinement
Scanline Optimization
Cross-based Cost Aggregation
AD-Census Cost InitializationInput: Stereo images
Output: Disparity map
Multi-step Disparity Refinement• Outlier Handling• Outlier Detection• Iterative Region Voting• Proper Interpolation
• Depth Discontinuity Adjustment• Sub-pixel Enhancement
Outlier Handling--Detection• The outliers : 𝐷𝐿(p) != 𝐷R(p − (𝐷𝐿(p), 0)) • Outliers are further classified into occlusion and mismatch
points
• p intersect its epipolar line and 𝐷R is checked• If no intersection p is labelled as “occlusion”, otherwise “mismatch”
Outlier Handling--Iterative Region Voting• Construct cross-based regions and a robust voting scheme
•
• Sp : • 𝜏𝑆, 𝜏𝐻 : threshold values
• 5 iterations
dd
Outlier Handling--Proper Interpolation• occlusion• The pixel with the lowest disparity value is selected for interpolation• It’s most likely comes from the background
• mismatch points• The pixel with the most similar color is selected for interpolation.
Depth Discontinuity Adjustment• For each pixel p on the disparity edge, two pixels p1, p2 from
both sides of the edge are collected. • 𝐷𝐿(p) is replaced by 𝐷𝐿(p1) or 𝐷𝐿(p2) if one of the two pixels
has smaller matching cost than 𝐶2(p,𝐷𝐿(p)).
𝐷𝐿(P1
)𝐷𝐿(P) 𝐷𝐿(P2
)
• Quadratic polynomial interpolation
•
•
• With 3*3 median filter
Sub-pixel Enhancement[20]
[20] Q. Yang, L. Wang, R. Yang, H. Stewenius, and D. Nister. “Stereo matching with color-weighted correlation, hierarchical belief propagation and occlusion handling.” IEEE TPAMI, 2009.
Multi-step Disparity Refinement
• The average error percentages after performing each refinement step.
CUDA Implementation
CUDA Implementation• Compute Unified Device Architecture (CUDA) is a
programming interface for parallel computation tasks on NVIDIA graphics hardware.• The computation task is coded into a kernel function. • The allocation of the threads is controlled with two hierarchical
concepts: grid and block.• A kernel creates a grid with multiple blocks, and each block
consists of multiple threads. Kernel
Grid
Block
Thread
Thread
…
Block …
Grid …
CUDA Implementation• Cost Initialization: • Parallelize with × 𝑊 𝐻 threads. • Organize into a 2D grid and the block size is set to 32× 32. • Each thread computes a cost value for a pixel at a given disparity. • For census transform, a square window is require for each pixel, which
requires loading more data into the shared memory for fast access.Kernel
Grid
Block
Thread
…
32X32 …
Grid
CUDA Implementation• Cross-based Cost Aggregation: • A grid with × 𝑊 𝐻 threads.• Cross construction : block size is 𝑊
or 𝐻 to efficiently handle a scan line • Cost aggregation : block size is 32X32• Data reuse with shared memory is
considered in both steps.
CUDA Implementation• Scanline Optimization: • This step is different, because the process is sequential in the scanline direction and
parallel in the orthogonal direction.• 𝑊 × 𝐷 or × 𝐻 𝐷 threads
• Disparity Refinement: • 𝑊 × 𝐻 threads
Experimental Results
Experimental Results• Device : A PC with Core 2 Duo 2.20GHz CPU and NVIDIA
GeForce GTX 480 graphics card• Settings parameters:
• Source : Middlebury http://vision.middlebury.edu/stereo/ HHI database(book arrival) Microsofy i2i database(Ilkay)
Experimental Results
• The GPU-friendly system brings an impressive 140× speedup.• The average proportions of the GPU running time for the four
computation steps are 1%, 70%, 28% and 1% respectively. • The iterative cost aggregation step and the scanline
optimization process dominate the running time.
Tsukuba Venus Teddy Cones
CPU 2.5 4.5 15 15
GPU 0.015 0.032 0.095 0.094
Experimental Results
• First row: disparity maps generated with our system. • Second row: disparity error maps with threshold 1. • Errors in unoccluded and occluded regions are marked in black and gray respectively.
Experimental Results
Experimental Results• video
Experimental Results
Snapshots on ’book arrival’ stereo video
Experimental Results
Snapshots on ’Ilkay’ stereo video
Conclusion
Conclusion• Contributions • Present a near real-time stereo system with accurate disparity results.• Combine some known techniques without sacrificing performance and
parallelism to obtain the high quality disparity map.
• Future works• Improve to apply it in real world applications • Robust parameter setting methods