dense image matching - challenges and potentials (keynote 3d-arch 2015)

Dense Image Matching

Challenges and Potentials

Konrad Wenzel

6th 3D-Arch Workshop, 25th of February 2015, Avila, Spain

Eiger, 20cm GSD

Rottenburg

» Panasonic DMC GX-1System Camera, 16MP

» 14mm lens, uncalibrated

» 2 images per second

» East façade of tower

• 152 images

• True OrthophotoGSD 3.5mm

Rottenburg

» Orthophoto drawing

02.03.2015 Konrad Wenzel, University of Stuttgart, ifp 6

Gyrocopter & IGI DigiCAM

» IGI DigiCAM 50MP

» 131 images, GSD 6cm

7

Carved stone from the Temple of Heliopolis, Egypt

» 30 images (12MP), Nikon D2X02.03.2015 10


Konrad Wenzel, University of Stuttgart, ifp» 0.5mm GSD

GlacierResult from a single DMC II stereo pair

02.03.2015 13

Glacier, DMCII Stereo Pair

GlacierResult from a single DMC II stereo pair


Glacier, DMCII Stereo Pair

Trento, 10cm GSD, 80/60

Stuttgart, 5cm GSD, DMCII, 80/60, 2 strips

True Ortho

Purpose & Motivation

» Integration of groups of aligned range images

• Range image: each pixel contains depth

• Depth cameras (TOF, Structured light)

• Dense image matching (e.g. SGM)

• Polar & triangulating Laserscanners

Extraction of consistent , optimal surface models

24

Perth, 5cm GSD

Perth, 5cm GSD, 80/70 ov.

Image MatchingPixel correspondence search for depth estimation

Dense image matching – one 3D sample for each pixel

28

Image Matching

Find corresponding pixels

29

Image Matching

3D Point P

x, x‘: Image correspondence between image 1 and image 2

Projection centers (EO+IO)

Viewing rays

① Image Matching Correspondence x x‘

② Exterior + Interior orientation Viewing rays

③ Intersection of viewing rays 3D Point P30

Image Matching

31

Image Matching

Depth estimation for each pixel

Dense image matching (Stereo)

32

Image Matching

More than two views

Multi-View Stereo

Challenges

» Ambiguities• Repetition of grey values

» Noise and weak texture• Shadows

» Discontinuities• Edges and details

» Computational complexity• Suitability for production

Matching for one image pair

Stereo

StereoGreat overview – the Middlebury Stereo Page

» D. Scharstein and R. Szeliski. (2002) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.

» Datasets

» Overview of methods

» Automatic benchmark

» http://vision.middlebury.edu/stereo/

Data fusionExploit redundancy

Image space

» Use epipolar relations

» Corresponding measurementsfrom image matching

+ fast data access

+ balance actual measurement

+ topology is available

-- relation limited to matching

(weak on small baselines)36

Object space

Use actual 3D data

e.g. analysis in local neighborhood

+ local geometry is analyzed

+ indepedent validation

+ no image matching required

-- expensive data access

-- topology is challenging

P

xb

xm1 xm2

d

Stereo Approach: Normalized Cross Correlation (NCC)

» Compare local mask for each pixel (e.g. 9 x 9 pixels)

» NCC: „sliding normalized dot product“

» High correlation match

Image source: https://siddhantahuja.wordpress.com/tag/normalized-cross-correlation/

StereoApproach: Scanline Optimization

» Dynamic programming

» Consistency alongepipolar line

streaking effect

Image source: Behzad Salehian ; Abolghasem A. Raie ; Ali M. Fotouhi ; Meisam Norouzi (2013).

Efficient interscanline consistency enforcing method for dynamic programming-based dense stereo matching algorithms

StereoApproach: Belief Propagation

» Message passing algorithm

» Usable for global optimization

» Popular:

• Bayesian networks

• Markov random fields

» Similar: graph cuts

Exact minimum solution

Computationally rather expensive

Image source: Klaus, A., Sormann, M., & Karner, K. (2006, August).

Segment-based stereo matching using belief propagation and a self-adapting

dissimilarity measure. In Pattern Recognition, 2006. ICPR 2006.

StereoApproach: Semi-Global Matching

• Matching: dense, intensity-based

• Global: optimization approach using a global model

• Semi: approximation fast numerical solution

Castle Neuschwanstein, Bavaria, Germany source: Hirschmüller, Heiko (2005) – Accurate and efficient stereo processing by Semi Global Matching an Mutual Information

Intensity image Disparity image using a

correlation matching method

Disparity image using

Semi Global Matching


» SGM Optimization approach: disparities similar to neighboring pixels are preferred

Assignment of costs for each possible disparity on each pixel

• Costs for the similarity of the grey value ( similar low costs )

• Additional costs for disparity jumps forces smooth surfaces41

Disparity along a path L in the image

02.03.2015


» Recursive cost aggregation on paths through the image

Base image, pixel

pi

Match image, pixel qi,j

Minimal costs

Costs c(pi ,qi,j)

• Problem: SGM cost structures require large amount of memory

• Solution:

• Reduce disparity search ranges to a tube around actual surface

• Coarse-to-fine approach: Initialize search ranges /tubes using low resolution imagery

Fast

Low memory requirements

x [pix]

dis

pa

rity

[pix]

x [pix]

dis

pa

rity [pix]

Rothermel, M., Wenzel, K., Fritsch, D., Haala, N. (2012).

SURE: Photogrammetric Surface Reconstruction from Imagery.

StereoApproach: Semi-Global Matching – tSGM variation

4402.03.2015

StereoApproach: Semi-Global Matching – tSGM variation

Multi-View Stereo

Matching on multiple views

» Steve Seitz, Brian Curless, James Diebel, Daniel Scharstein, Richard SzeliskiA Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms, CVPR 2006, vol. 1, pages 519-526.

» http://vision.middlebury.edu/mview/

StereoGreat overview – the Middlebury Multi-View Stereo Page

http://vision.middlebury.edu/mview/

» Yasutaka Furukawa and Jean Ponce, Accurate, Dense, and Robust Multi-ViewStereopsis, CVPR 2007

» Steps:

• Match: find features

• Expand: grow patches

• Filter: using visibility constraint

» Mesh using regulation constraints

» Available Open Source as PMVS

Multi-View StereoApproach: Grow patches around feature points

» Deseilligny, M. P., & Clery, I. (2011). Apero, an open source bundle adjusmentsoftware for automatic calibration and orientation of set of images. 3D Arch 2011

» Multi-Stereo matching for one reference image (available as Open Source: MICMAC)

» Graph cut & dynamic programming optimization

Multi-View StereoApproach: Multi-stereo matching

Multi-View StereoApproach: Depth maps and Volumeteric Range Image Integration

» M. Goesele, N. Snavely, B. Curless, H. Hoppe, S. Seitz (2007). Multi-view stereo for community photo collections, ICCV 2007

» Grow sparse points from SfM

» Estimate refined depth maps withphotoconsistent normals

» Integration using Volumetric Range Image Integration

Brian Curless and Marc Levoy, Stanford University (1996): A Volumetric Method for Building Complex Models from Range Images.

http://grail.cs.washington.edu/software-data/vrip/

50


Korcz, D. (2011). Volumetric Range Image Integration. Diplomathesis, ifp, University of StuttgartDataset: Middlebury Multi-View Stereo

evaluation, Temple

1. Build volumetric space

entity: voxel, a volumetric pixel

2. Project range image into voxel space

3. Compute Signed Distance Field

4. Extract optimal surface

51Korcz, D. (2011). Volumetric Range Image Integration. Diplomathesis, ifp, University of Stuttgart




Signed Distance Field for „Dino“ dataset

» Iso-Surface extraction using Marching Cubes algorithm

» Hole filling by space carving method

54


» [Zach et al., 2007]: simple averaging of signed distance fields without furtherregularization causes inconsistencies

• due to frequent changes of sign

introduction of additional regularization force

energy minimization

uses total variation norm (TV-L1), [Rudin et al, 1992]

Smoothness term allows

• regulatization

• noise suppression

• outlier rejection

55

[Rudin et al., 1992] Rudin, L. I., Osher, S., and Fatemi, E. (1992).

Nonlinear total variation based noise removal algorithms.

[Zach, 2008] Zach, C. (2008). Fast and High Quality Fusion of Depth Maps.

Multi-View StereoDepth maps and Volumeteric Range Image Integration + Total Variation

56

Multi-View StereoDepth maps and Volumeteric Range Image Integration + Total Variation

Source: Korcz, D. (2011). Volumetric Range Image Integration.

» Vu, H.; Keriven, R.; Labatut, P. and Pons, J.-P (2009). Towards high-resolution large-scale multi-view stereo. CVPR, 2009

» Rough dense point cloud through normalized cross correlation (NCC)

» minimum s-t cut global optimization with visibility filtering

» Mesh refinement with photo consistency

Multi-View StereoApproach: rough point cloud and mesh refinement

» Rothermel, M., Wenzel, K., Fritsch, D., Haala, N. (2012). SURE: Photogrammetric Surface Reconstruction from Imagery.

» Approach:

1) Stereo matching using tSGM

2) Multi-ray triangulation

3) Object space fusion, e.g.

• DSM

• Volumetric point cloud filtering

• Meshing

Multi-View StereoApproach: stereo matching, multi-ray triangulation, object space fusion

P

xb

xm1xm2

d

Images

Epipolar images

Disparity images

SURE: Multi-View Stereo Triangulation

» Redundant measurements across stereo pairsallow outlier elimination claiming geometric consistency

Stereo Multi-view Multi-view Multi-view

> 1-fold > 2-fold > 3-fold1-fold

SURE: Multi-View Stereo Triangulation

» Improvement of surface noise

Stereo

1-fold

Multi-view

> 1-fold

SURE: Point Cloud Fusion for 2.5D Surfaces

» Fusion of 3D point clouds to 2.5D surface models

3D point cloud stereo

matching

3D point clouds multi-view

stereo matching

Fusion of point clouds to

2.5D surface model

Munich – DMCII, 10cm GSD [EuroSDR Matching Test]

True Ortho

Traditional Orthophoto

65Courtesy of Sven Briels - burokarto.nl

Low overlap conditions (60/40), 3.5cm GSD

SURE True Orthophoto

66Courtesy of Sven Briels - burokarto.nl

Low overlap conditions (60/40), 3.5cm GSD (processed at 7cm GSD, 9cm ortho)

SURE: Quadtree based on DSM - Example

02.03.2015 Schwerin_2012 67

Munich [EuroSDR Matching Test, DMCII, 10cm GSD, 80/80 ov.]

Textured DSM (coming soon)

Braunschweig, 80/60 Nadir DSM

Postprocessing & retexturing by

SURE: Out-of-core point cloud filtering

» Retrieve locally densest point cloud removal of redundancy

» Validate points keep only points, which are

validated by other point clouds

» Adapt resolution locally

» Scalable – out-of-core octree

Wenzel, K., Rothermel, M., Fritsch, D., & Haala, N. (2014). Filtering of Point Clouds from Photogrammetric Surface Reconstruction

SURE: Out-of-core point cloud filtering(IGI DigiCam Penta)

Imagery courtesy of Aerowest GmbH

„Bathymetric Photogrammetry“


Bathymetric Photogrammetry?

02.03.2015 76Courtesy of Alfons Krismann,

University of Hohenheim


Bathymetric Photogrammetry?

02.03.2015 Konrad Wenzel, University of Stuttgart, ifp 77Courtesy of Alfons Krismann,



SURE: Observation of abliation in plasma flow

» Plasma channel

» 2500 degrees

» Mirrors

» Sub-mm accuracy

Loehle, S., Staebler, T., Reimer, T., & Cefalu, A. (2014) Photogrammetric Surface Analysis of Ablation Processes in High Enthalpy Air Plasma Flow.

SURE: Depth map fusion (Meshing)

» Rothermel, M.; Haala, N.; Fritsch, D. (2014) Generating oriented pointsets from redundant depth maps using restricted quadtrees.

» Adapt depth map resolution using Restricted Quadtree

» Choose locally optimal depth map while forcing visibility constraints

3D Meshing Example: Fusion + Poisson

02.03.2015 Schwerin_2012 82

Mesh Patches + Poisson

3D Meshing Example: Fusion + Poisson

02.03.2015 Schwerin_2012 83

dense image matching - challenges and potentials (keynote 3d-arch 2015)

Science

image correspondence

university of stuttgart

dense image matchingchallenges

image pairstereo stereogreat

pixel28image matching

stereo pair02

gsd 6cm702

d data