High-Resolution Stereo Matching based on Sampled Photoconsistency ComputationChloe LeGendre1, Konstatinos Batsos2, Philippos Mordohai21USC Institute for Creative Technologies 2Stevens Institute of Technology

Motivation • State-of-the-art binocular stereo matching algorithms do not often handle even moderately high resolution images. • For such images, exhaustive photoconsistency computations at every pixel can be avoided, since they are redundant and computationally expensive.• Our pixel-wise sampling routines and superpixel-based disparity hypothesis propagation exploit such redundancies for large computational savings at a small loss of depth accuracy.

Method: Sampled Photoconsistency Stereo• Vertical alignment of stereo pair images• Oversegmentation of reference image into superpixels using SLIC

• Photoconsistency computation for random subsample S within each superpixel

• Plane fitting for each superpixel using RANSAC

• Propagation of planes to neighboring superpixels for N iterations, scored by photoconsistency on a random subsample of the pixels V.

Acknowledgements:This research has been supported in part by a Google Research Award and by the National Science Foundation award #1637761.

Results: Middlebury 2014 Stereo Benchmark• Full Resolution Images (15 training / 15 testing pairs)(average 5.2 MP images; max. disparity of 256-800)

Algorithm Dense Training Dense Test Dense

SPS (ours) y 30.0 21.1 4.33 29.1 19.6 4.77

SGM n 31.7 22.1 10.30 35.8 25.3 13.40 LPS y 33.7 26.2 7.14 27.6 20.3 5.28

SGBM n 36.5 27.4 2.79 38.0 28.4 3.69PFS y 30.3 19.9 5.66 43.0 32.2 10.20ELAS n 38.5 26.6 0.56 44.4 32.3 0.56TSGO y 55.0 31.3 11.4 55.9 39.1 8.26ICSG n 47.9 37.7 31.9 55.2 45.6 36.20

bad 1.0 bad 2.0 time/MP bad 1.0 bad 2.0 time/MP

Table 1. Weighted average error, compared with other algorithms. Ours has highest accuracy compared with all methods using full resolution images, on default benchmark parameter of 2.0 disparity level error. Timing reported for a multi-threaded implementation of our method.SPS Parameters: S = 5%, V = 25%, N = 3 plane propagation iterations.

ground truth N = 0 N = 3

ground truth

left image S = 100%, N = 0

S = 5%, N = 0

S = 100%, N = 3, V = 25%

S = 5%, N = 3, V = 25%

Experiments on Sampling Rates and Iterations

0.0 0.2 0.4 0.6 0.8 1.0

22.0

22.5

23.0

23.5

24.0

24.5

25.0

Sampling rate

Bad

2.0

(%)

●●

●

●

●

●

●

1 Iter3 Iter5 Iter

N = 1, 3, 5

Superpixel-based Plane Propagation

S = V

0.0 0.2 0.4 0.6 0.8 1.0

22.0

22.5

23.0

23.5

24.0

24.5

25.0

Initial sampling rate, s

Bad

2.0

(%)

●●●

●●

●

●

● v = 10%v = 25%v = 100%

N = 3Error rates vs. Runtimes(single-threaded impl.)

N = 3 iterations of plane propagation allows for a smaller subsample S of initial pixels for disparity estimation, while yielding similar accuracy.

Possible speed - accuracy tradeoff with user-specified utility function.

left image disparity map left image disparity map