robust feature point matching ohne.ppt [kompatibilitätsmodus]

Robust Feature Point MatchingRobust Feature Point MatchingRobust Feature Point Matching Robust Feature Point Matching in General Multiin General Multi--Image SetupsImage Setups

WSCG 2011

A i S ll M i Ei M MAnita Sellent, Martin Eisemann, Marcus MagnorTU Braunschweig, Germany

GeneralGeneral Sets of ImagesSets of ImagesUnsynchronized => no epipolar geometry

GeneralGeneral Sets of ImagesSets of Images

Unordered => no trackingCommon field of view

2Robust Feature Point Matching Anita Sellent

Consistent Feature MatchingConsistent Feature MatchingConsistent Feature MatchingConsistent Feature MatchingSparse featuresFeatures correspond to 3D points => consistency

3Robust Feature Point Matching Anita Sellent

Three Image Optical FlowThree Image Optical FlowDense correspondences

Three Image Optical FlowThree Image Optical Flow

High accuracyBrightness constancySmall displacements

4Robust Feature Point Matching Anita Sellent

[Sellent et al.: Consistent Optical Flow for Stereo Video, ICIP 2010]

OutlineOutlineOutlineOutlineThree Image Feature Matchingg gCombination of Feature Matching and Optical Flow

5Robust Feature Point Matching Anita Sellent

OutlineOutlineOutlineOutlineThree Image Feature MatchingCombination of Feature Matching and Optical Flow

6Robust Feature Point Matching Anita Sellent

Three Image MatchingThree Image MatchingThree Image MatchingThree Image MatchingUnordered, unsynchronized imagesCommon field of viewAny features (SIFT, SURF, FAST, Harris,…)

7Robust Feature Point Matching Anita Sellent

Three Dimensional Assignment Three Dimensional Assignment P blP blProblemProblemSets of equal, finite cardinalityNP-hardApproximation algorithms exist[Crama, Spieksma:Approximation algorithms for the 3 dimensional assignment problem, 1992]

8Robust Feature Point Matching Anita Sellent

Step 1Step 1Step 1Step 1(Robust) nearest neighbor matchingSymmetry

3I

1I 2I

9Robust Feature Point Matching Anita Sellent

Step 2+3Step 2+3Step 2+3Step 2+3Combine distance function

S

),(),(),(ˆ323131 ffdffdffd +=

Symmetry3I

1I 2I

10Robust Feature Point Matching Anita Sellent

Step 4Step 4Step 4Step 4Restart from all directions

11Robust Feature Point Matching Anita Sellent

Step 5Step 5Step 5Step 5Only accept consistent solutions

12Robust Feature Point Matching Anita Sellent

Results: Scene Art with SIFT Results: Scene Art with SIFT FFFeaturesFeatures

Two Image Matching Three Image MatchingTwo Image Matching1444 Matches53 % wrong matches

Three Image Matching603 Matches11 % wrong matches

13Robust Feature Point Matching Anita Sellent

Removed 703 wrong matches at the cost of 138 correct matches

Results: Scene Laundry with Results: Scene Laundry with SURF FSURF FSURF FeaturesSURF Features

Two Image Matching Three Image MatchingTwo Image Matching675 Matches69 % wrong matches

Three Image Matching193 Matches29 % wrong matches

14Robust Feature Point Matching Anita Sellent

Removed 410 wrong matches at the cost of 72 correct matches

Results: Scene Stonemill with Results: Scene Stonemill with H i CH i CHarris CornersHarris Corners

Two Image Matching Three Image MatchingTwo Image Matching225 Matches50 % wrong matches

Three Image Matching133 Matches28 % wrong matches

15Robust Feature Point Matching Anita Sellent

Removed 75 wrong matches at the cost of 17 correct matches

Results: Real World Scenes Results: Real World Scenes i h SIFT Fi h SIFT Fwith SIFT Featureswith SIFT Features

Two Image Matching Three Image MatchingTwo Image Matching Three Image Matching

16Robust Feature Point Matching Anita Sellent

Three Image Feature MatchingThree Image Feature MatchingThree Image Feature MatchingThree Image Feature MatchingNo prior knowledge requiredMatch features consistently on three imagesTrade off number of matches for increased quality of the matchesSparse

17Robust Feature Point Matching Anita Sellent

OutlineOutlineOutlineOutlineThree Image Feature MatchingCombination of Feature Matching and Optical Flow

18Robust Feature Point Matching Anita Sellent

Combination of Feature Combination of Feature M hi d O i l FlM hi d O i l FlMatching and Optical FlowMatching and Optical Flow

Features Optical Flow◦ Sparse◦ Low accuracy

Optical Flow◦ Dense◦ High accuracyy

◦ Large motion◦ Invariant to color

g y◦ Small motion◦ Based on brightness

changes◦ …

gconstancy ◦ …

19Robust Feature Point Matching Anita Sellent

Feature Matches as Motion Feature Matches as Motion P iP iPriorPriorDefine motion prior only where )(xPmatches are foundAssign weight proportional to )(xμmatching costAdd weighted motion prior to optical flow d t tdata-term

20Robust Feature Point Matching Anita Sellent

Results: Results: A E d i EA E d i EAverage Endpoint ErrorAverage Endpoint Error

2 image OF 2 image OF &2 image features

3 image OF 3 image OF &3 image features

Stonemill 4 53 23 16 3 81 3 52Stonemill 4.53 23.16 3.81 3.52Waving 1.03 31.39 0.97 0.92Art 10.62 84.82 9.34 8.70Books 14.60 55.37 6.43 4.85

21Robust Feature Point Matching Anita Sellent

[Sellent et al.: Consistent Optical Flow for Stereo Video, ICIP 2010]

Flow Fields: Scene StonemillFlow Fields: Scene StonemillFlow Fields: Scene StonemillFlow Fields: Scene Stonemill

Ground-truth

2 image OF 2 image OF &2 image features

3 image OF &3 image features

3 image OF

22Robust Feature Point Matching Anita Sellent

Flow Fields: Scene WavingFlow Fields: Scene WavingFlow Fields: Scene WavingFlow Fields: Scene WavingGround-truth

2 image OF 2 image OF &2 image features

3 image OF &3 image features

3 image OF

23Robust Feature Point Matching Anita Sellent

SummarySummarySummarySummaryConsistency on three images can improve sparse and dense correspondences without further knowledge

N lib i◦ No calibration◦ No synchronization

M lti l di i◦ Multiple dimensionsFor high quality feature matches the accuracy of optical flow can be improved byaccuracy of optical flow can be improved by the inclusion of features

24Robust Feature Point Matching Anita Sellent

Future WorkFuture WorkFuture WorkFuture WorkWhat is the best number of images to be used?◦ Number of visible/ detected features◦ Quality of the matching

25Robust Feature Point Matching Anita Sellent

Thank You for Your Attention!Thank You for Your Attention!Thank You for Your Attention!Thank You for Your Attention!

http://graphics.tu-bs.de

This work has been funded by the German Science Foundation

26Robust Feature Point Matching Anita Sellent

This work has been funded by the German Science Foundation, DFG MA2555/4-1

Consistent Optical FlowConsistent Optical FlowConsistent Optical FlowConsistent Optical Flow

Brightness ConstancyBrightness Constancy)()( 2,121 wxIxI +≈

0ˆ22,12,1 ≈−ww

Smoothness Assumption

0ˆ 2,1

r≈∇w

Estimate dense flow between 3 images simultaneously

27Robust Feature Point Matching Anita Sellent

[Sellent et al.: Consistent Optical Flow for Stereo Video, 2010]

Estimate dense flow between 3 images simultaneously

Synchronized CamerasSynchronized CamerasSynchronized CamerasSynchronized Camerassp

ace

time

28Robust Feature Point Matching Anita Sellent

[Hartley, Zisserman: Multiple View Geometry in Computer Vision, 2003]

Unsynchronized CamerasUnsynchronized CamerasUnsynchronized CamerasUnsynchronized Camerassp

ace

time

[Veenman, Reinders, Backer: Establishing Motion Correspondence

29Robust Feature Point Matching Anita Sellent

Using Extended Temporal Scope, 2003]