10/26/2014

1

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING

STATE-OF-THE-ART

in DENSE IMAGE MATCHING

Fabio REMONDINO

3D Optical Metrology (3DOM)

Bruno Kessler Foundation (FBK)

Trento, Italy

Email: remondino@fbk.eu

http://3dom.fbk.eu

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING

Bruno Kessler Foundation (FBK)

- Born in 1962 as public research center of the Autonomous Province of Trento (Italy)

- Research primarily in the fields of ICT, Materials & Sensors

- More than 350 researchers and developers, 7 laboratories, 26 research units, 11 spin-offs

- Research units and labs: Biosensors, Bioelectronics, Optical sensors, Radiation detectors,

Renewable energies, Intelligent interfaces, Predictive models, Technologies for Vision, 3D

Optical Metrology, etc.

- Hosting BSc, MSc and PhD students

- Master and PhD programs jointly held with international universities

10/26/2014

2

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING

3D Optical Metrology (3DOM) unit

10 researchers

Applied research in the fields of Photogrammetry, Laser scanning and Remote Sensing, 3D Mapping, Geomatics

Hosting BSc, MSc and PhD students (2010: 4 guests; 2011: 9 guest; 2012: 7 guests; 2013: 9 guests, 2014: 4)

Exchange agreement with int. universities for hosting student (ENSG Paris, CGT Siena, etc.)

Funding from local (PAT, private sector, etc.) and international grants (EU, BMBF, private sector, etc.)

Organization of int. events and summerschools for knowledge and technology transfer

Collaborations with national and international universities and private companies

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 4

3D RECORDING & PROCESSING CHAIN

SOFTWARE &

ALGORITHMS

SENSORS &

PLATFORMS

VISUALIZATION,

SEGMENTATION,

GIS, SHARING, etc

2D-3D RESULTS DATA

10/26/2014

3

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 5

IMAGE-BASED PIPELINE

Camera calibration

&

Image triangulation

(Dense)

3D reconstruction

Polygonal 3D

model, maps,

orthoimages, etc.

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 6

Establishment of image correspondences between 2 or more images

Delivery of 3D coordinates from 2D image correspondences

Automated vs interactive procedure

IMAGE MATCHING

OB (X0, Y0, Z0, w, f , k)

X

Y

Z

Reference system

X

Y

X

Y

image A

image B

object space

OA (X0, Y0, Z0, w, f , k)

xA, yA xB, yB

X, Y, Z

10/26/2014

4

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 7

Establishment of image correspondences between 2 or more images

Delivery of 3D coordinates from 2D image correspondences

Automated vs interactive procedure

Natural features vs targets

Single point measurement vs every pixel, i.e. sparse vs dense point cloud

Statistical information of the matched 2D points and derived 3D coordinates

IMAGE MATCHING

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 8

1950’s :

Analogue image

matching and

stereo-plotter

1990’s :

Digital stereo

processing systems

1980’s :

LSM & Multi-photo matching

1960’s:

First digital cross-

correlation

2000’s :

Close range

photogrammetry

(convergent images)

1990’s :

Multi-view stereo

reconstruction

1980:

Stereo matching

2000’s :

Dense reconstruction

http://www.mtzgeo.com/history.cfm

PHOTOGRAMMETRY

COMPUTER VISION

A LOOK AT THE PAST

10/26/2014

5

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 9

A LOOK AT THE PAST

Automated 3D reconstruction of the

Bamiyan Buddha from amateur images (2003)

[Gruen, A., Remondino, F., Zhang, L., 2003:

Image-based reconstruction of the Great Buddha of Bamiyan, Afghanistan. Proc. of Videometrics VII, Vol. 5013, pp. 129-13]

Automated DSM generation from SPOT5

stereo-pair over Bamiyan, Afghanistan (2004)

[Gruen, A., Remondino, F., Zhang, L., 2004:

The Bamiyan Valley: Landscape Modeling for Cultural Heritage Visualization & Documentation. Int. Archives of PRS & SIS,

Vol. XXXVI(5/W1)]

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 10

Generally dense matching on the entire image (up 1:1)

Semi-Global Matching, graph-cut, patch-based, optimal flow, least squares, etc.

Multi-image vs stereo-pairs

Image-based vs object-based (volumetric)

Area-based vs feature-based

Local vs global methods

IMAGE MATCHING ALGORITHM

Algorithm developments (1)

10/26/2014

6

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 11

Open-source solutions: MicMac, PMVS

Commercial solutions inside different tools: ERDAS-LPS (eATE), Photoscan,

Inpho Match-T, Pix4D, etc.

Stand-alone commercial solution: SURE

Comparable geometric results as laser scanning / LiDAR

IMAGE MATCHING ALGORITHM

Algorithm developments (2)

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 12

Trento (Italy), Geo-Eye stereo-pair, 1m resolution DSM

DENSE IMAGE MATCHING RESULTS

10/26/2014

7

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 13

DENSE IMAGE MATCHING RESULTS

Marseille, France: Vexcel Ultracam-X, 10 cm GSD – 20 cm DSM

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 14

DENSE IMAGE MATCHING RESULTS

Paris, France: Oblique imagery, 10 cm GSD – 20 cm DSM

10/26/2014

8

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 15

DENSE IMAGE MATCHING RESULTS

Graz, Austria: Oblique imagery (Osprey camera), 10 cm GSD

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 16

DENSE IMAGE MATCHING RESULTS

UAV image block, avg GSD: 2-3 cm

10/26/2014

9

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 17

ca 2 x 1.5 x 1 m

ca 600 images

avg GSD: 0.2 mm

avg image scale: 1:25

Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.)

DENSE IMAGE MATCHING RESULTS

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 18

Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.)

DENSE IMAGE MATCHING RESULTS

Dense matching algorithm delivered ca 50 mil. punti

10/26/2014

10

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 19

QUALITY CONTROL: intersection angles

DENSE IMAGE MATCHING RESULTS

Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.)

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 20

QUALITY CONTROL: redundancy (numb. of images contributing to the 3D point)

DENSE IMAGE MATCHING RESULTS

Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.)

10/26/2014

11

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 21

DENSE IMAGE MATCHING RESULTS

QUALITY CONTROL: precision of the computed 3D points (avg: 0.06 mm)

Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.)

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 22

Spouses Sarcophagus, Etruscan masterpiece (ca 530 B.C.)

DENSE IMAGE MATCHING RESULTS

10/26/2014

12

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 23

DENSE IMAGE MATCHING – Comparisons (1)

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 24

DENSE IMAGE MATCHING – Comparisons (2)

10/26/2014

13

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 25

Repeated pattern

Illumination and scale changes (oblique imagery)

Noisy 3D data due to low quality images

Mismatches

Irrelevant points

How to go from dense point clouds to more useful and structured 3D objects

Large quantity of data (dense point clouds) to be managed

DENSE IMAGE MATCHING – Open issues

F. Remondino – STATE OF THE ART IN DENSE IMAGE MATCHING 26

…THANK YOU…

Fabio Remondino

3D Optical Metrology unit

FBK Trento, Italy

http://3dom.fbk.eu

Acknowledgments

3D Optical Metrology unit, FBK Trento

References:

• Remondino, F., Spera, M.G., Nocerino, E., Menna, F., Nex, F., 2014:

State of the art in high density image matching. The Photogrammetric Record, Vol.

29(146), pp. 144-166

• Rupnik, E., Nex, F., Remondino, F., 2014:

Oblique multi-camera systems – Orientation and dense matching issues. EuroCOW

Proceedings

• Remondino, F., El-Hakim, S., Gruen, A., Zhang, L., 2008:

Turning images into 3D models - Development and performance analysis of image

matching for detailed surface reconstruction of heritage objects. IEEE Signal

Processing Magazine, Vol. 25(4), pp. 55-65