IEEE 2015 Conference on Computer Vision and Pattern

Recognition

DASC: Dense Adaptive Self-Correlation for Multi-modal and Multi-spectral Correspondence Seungryong Kim, Dongbo Min, Bumsub Ham, Seungchul Ryu, Minh N. Do, and Kwanghoon Sohn

Download the code at http://seungryong.github.io/DASC/

Introduction DASC Descriptor Experimental Results

Conclusion

Motivation • In multi-modal and multi-spectral image, conventional

descriptors often fail to estimate correspondence

Goal • To establish dense correspondence for those images

NIR

Visible

No-flash

Flash

Low exp.

High exp.

Blur

Sharp

Randomized Receptive Field Pooling • Unlike center-biased max pooling in LSS descriptor, the DASC descriptor incorporates

randomized receptive field pooling

Sampling Pattern Learning • Exploit supports vector machine (SVM)

with linear kernel • Features: • Energy function for SVM

Small Support Window Similarity • Adaptive self-correlation measure

Efficient Computation • Asymmetric weights • Reference-biased sampling pairs

• Approximated adaptive self-correlation

Middlebury Stereo Benchmark

Multi-modal and Multi-spectral Image Pairs

Background

Img

1

Img

2

color, gradient, structural similarity

RGB Image NIR Image

Matching Cost A Matching Cost B Matching Cost C

Challenge on Multi-modal Images • Nonlinear photometric deformation, e.g., gradient

reverses and intensity order variations

LSS descriptor DASC descriptor

Middlebury multi-modal SINTEL

1 2 2 2, , ,exp ( ) / 2m l m l m l rr d d

2( ) || || max(0,1 ( ))m mmv v y r

• Intuitions Center-biased pooling

sensitive to noise Randomness of BRIEF

LSS, bin ( )max ( , )

ii l j ld i j

LSS descriptor

DASC descriptor

, , , , ,( , ), ,i l i l i l i l i l id s t s t

, ,,

2 2, ,

( )( )

( , ){ ( )} { ( )}

s s t t s s t ts t

s s s s t t t ts t

f f

s tf f

, ,,

2 2, , ,

,

( )( )

( , )( ) ( )

i i i i j i ji j

i i i i i i j i ji i j

f f

i jf f

, 's s

, ,( ) ( , - ), i l i ltj i si i

• Robust estimation

( , ) max(exp( (1 ( , )) / ), )s t s t • The correlation is normalized

with norm of all

( , )s t

l

Img1 Img2

RSNCC

BRIEF

DAISY

LSS

DASC

RGB-NIR flash-no flash different exposure blur-sharp

• Robust novel descriptor called the DASC for multi-modal correspondence • Leverages adaptive self-correlation and randomized receptive pooling • Efficient computation with fast edge-aware filters

Limitation of Conventional Descriptors • Image gradient (SIFT) or intensity comparison (BRIEF)

cannot capture coherent matching evidence

Img1 Img2 ANCC BRIEF

SIFT LSS DASC Ground Truth Illumination Exposure