cluster-based 3d reconstruction of aerial videoieee-hpec.org/2012/index_htm_files/sawyer.pdfscott...
Post on 30-Jul-2020
0 Views
Preview:
TRANSCRIPT
Scott Sawyer (scott.sawyer@ll.mit.edu) MIT Lincoln Laboratory
HPEC ’12 12 September 2012
Cluster-based 3D Reconstruction of Aerial Video
This work is sponsored by the Assistant Secretary of Defense for Research and Engineering under Air Force contract FA8721-05-C-0002. Opinions, interpretations, conclusions, and recommendations are those of the author and are not necessarily endorsed by the United States Government.
Cluster SfM - 2 SMS 09/12/12
Given a collection of photos…
Cluster SfM - 3 SMS 09/12/12
…determine their 3D structure
Cluster SfM - 4 SMS 09/12/12
…determine their 3D structure
Cluster SfM - 5 SMS 09/12/12
• Feature-based 3D Reconstruction • Reconstructing Aerial Video • Cluster-based Structure from Motion • Results
Outline
Cluster SfM - 6 SMS 09/12/12
• Feature-based 3D Reconstruction • Reconstructing Aerial Video • Cluster-based Structure from Motion • Results
Outline
Cluster SfM - 7 SMS 09/12/12
“Structure from Motion” (SfM)
Feature Extraction
Feature Matching
Sparse Reconstruction
Dense Reconstruction
Photo Collection
3D Point Cloud
Applications:
3D Maps Robotic Navigation Video Geo-registration
Cluster SfM - 8 SMS 09/12/12
Feature Extraction
Feature Extraction
Feature Matching
Sparse Reconstruction
Dense Reconstruction
Photo Collection
3D Point Cloud
Scale Invariant Feature Transform (SIFT)
• Identifies distinctive keypoints in an
image and provides 128-dimensional descriptor vector
• Based on finding local extrema in scale space
• Invariant to scale, orientation and lighting conditions (to an extent) and aspect angles up to 30 degrees
• Published by David Lowe in 1999 and widely used in computer vision
Cluster SfM - 9 SMS 09/12/12
Feature Matching
Feature Extraction
Feature Matching
Sparse Reconstruction
Dense Reconstruction
Photo Collection
3D Point Cloud
Euclidean Distance Approximate Nearest
Neighbors (ANN)
• Euclidean distance between SIFT keypoint descriptors used to determine matches
• Tree-based approximation reduces computation
• SIFT points matched between each pair of images – O(n2)
Cluster SfM - 10 SMS 09/12/12
Sparse Reconstruction
Feature Extraction
Feature Matching
Sparse Reconstruction
Dense Reconstruction
Photo Collection
3D Point Cloud
Projective Structure from Motion
• Given m images and n 3D points
(with point correspondences)
• Estimate m projection matrices (Pi, i = 1, …, m) and n 3D points (Xj, j = 1, …, n) from the mn correspondences (xij)
• Iteratively minimize re-projection error (“bundle adjustment”):
( )2
1 1,),( ∑∑
= =
=m
i
n
jjiijDE XPxXP
x1j
x2j
x3j
Xj
P1
P2
P3
Cluster SfM - 11 SMS 09/12/12
Dense Reconstruction
Feature Extraction
Feature Matching
Sparse Reconstruction
Dense Reconstruction
Photo Collection
3D Point Cloud
Identify Patches of Rigid Structure
• Mature software (“PMVS2”)
forms patched-based reconstruction (open source, parallel)
• Removes spurious and non-rigid points
• This step not included in benchmark analysis
Cluster SfM - 12 SMS 09/12/12
• Feature-based 3D Reconstruction • Reconstructing Aerial Video • Cluster-based Structure from Motion • Results
Outline
Cluster SfM - 13 SMS 09/12/12
Lincoln Laboratory performed a campaign of aerial data collections in 2011: • Aerial platform circles ground
subject of interest • Camera gimbal remains fixed on
ground point • Video frames synchronized to
GPS • Challenge: Processing must keep
up with rate of collection (e.g. one-hour mission timeline)
Reconstructing Aerial Video
Cluster SfM - 14 SMS 09/12/12
Sample Video
Cluster SfM - 15 SMS 09/12/12
Bundler
Bundler (v. 0.4) by Noah Snavely
• Works on unordered image collections (no assumptions about input)
• Sequential implementation (single-core)
• Optimization solution considered baseline (truth) 0
20
40
60
100 200 300 400 500
Run
time
(hou
rs)
Image Set Size
Bundler (single-core)
Feat. Extr. Feat. Match Bundle Adj.
Cluster SfM - 16 SMS 09/12/12
VisualSFM
VisualSFM (v. 0.5.15) by Changchang Wu
• Restructures optimization to better enable parallelization of dense matrix kernels
• Feature extraction and bundle adjustment use GPU acceleration (CUDA)
• Multi-threaded feature matching (pThreads) 0
10
20
30
100 200 300 400 500
Run
time
(hou
rs)
Image Set Size
VisualSFM (multi-core & GPU)
Feat. Extr. Feat. Match Bundle Adj.
Run on quad-core Intel Xeon E5620 (2.4 GHz), 6 GB RAM, nVIDIA Quadro FX1800 (64 compute cores)
Cluster SfM - 17 SMS 09/12/12
• Feature-based 3D Reconstruction • Reconstructing Aerial Video • Cluster-based Structure from Motion • Results
Outline
Cluster SfM - 18 SMS 09/12/12
LLGrid Computing Environment
Scheduler assigns each job to a single
core (no threading or shared memory)
Job submission
Jobs may or may not be on the same node
Inter-process communication supported via file system or network
Number of cores used varied experimentally up to 64
Custom Python framework for
submitting jobs, mapping data and
synchronizing between steps
Master Node
Scheduler
Compute Nodes
Network File System (Lustre)
Cluster SfM - 19 SMS 09/12/12
Cluster-based SfM Overview
1. Feature Extraction
2. Feature Matching
3. Reconstruction
4. Point Cloud Fusion
Job 1 Job 2 Job Np
JPEG JPEG Photo
Remap
…
Single Job
Partial Point Clouds
Final Point Cloud
Matched Feature Graph
SIFT keypoints
Remap
Job 1 Job 2 Job Np …
Job 1 Job M …
Runtime Growth
O(N)
O(N2)
O(N2/M)
JPEG JPEG Photo JPEG JPEG Photo
Remap
Job 1 Job 2 Job Np …
Combine Results
Step
O(P2/M)
N: Input photo count; M: Partition count; P: Sparse point count
Cluster SfM - 20 SMS 09/12/12
Feature Matching
• Matching is most expensive step, growing O(N2) with number of photos
• Results of matching represented as graph G:
• Vertices V represent SIFT points
• Edges E represent SIFT matches
Photo ID
1 2 3 4 N
P
hoto
ID
1 0 1 1 1 … 1 2 0 0 1 1 … 1 3 0 0 0 1 … 1 4 0 0 0 0 1
…
N-1 0 0 0 0 1 N 0 0 0 0 0
Matching Pair Matrix
If (i,j) = 1, then match photo i to photo j
Matching Graph (notional)
(1,10)
(2,12) (3,15)
(2,20)
(4,75)
(1,50) (5,81)
Vertices described by tuple (Photo ID, Keypoint ID)
Cluster SfM - 21 SMS 09/12/12
Flight path partitioned using block-cyclic mapping:
• N photos mapped to M partitions
• Assume photo set covers one revolution
• Angular gap between photos should be less than 30 degrees (to help with SIFT matching)
• Photos per partition (N/M) must be greater than 25
• Block size and partition count set automatically
Bundle Adjustment Data Mapping
Reconstructing partial photo sets independently creates challenges: • Point clouds are in different arbitrary coordinate systems
• Many points are duplicated across the partial point clouds
• 3D points have shorter view lists (fewer iterative improvements) Flight P
ath Less than 30°
Example: Block size: 2; M=3
Cluster SfM - 22 SMS 09/12/12
Each 3D point has a view list comprised of SIFT keypoints (vertices in G). View lists from different partitions will always be disjoint because photos are mapped to exactly one partition.
Identifying Duplicate Points
(1,10)
(2,12)
(3,15)
(4,75)
(5,81)
View list A
View list B
Subgraph of G
3D points are considered duplicates if any vertices in B are in the neighborhood of A.
Cluster SfM - 23 SMS 09/12/12
Determine rotation, scale and translation based on duplicate points:
• Robustly eliminate outlier 3D points
• Solve for transformation matrix that minimizes error
• Combine duplicate points as a weighted average
Transforming to a Common Space
Sparse point clouds after independent bundle
adjustment on 4 partitions
Fused sparse point cloud after transformation
Cluster SfM - 24 SMS 09/12/12
Final Point Cloud
Cluster SfM - 25 SMS 09/12/12
• Feature-based 3D Reconstruction • Reconstructing Aerial Video • Cluster-based Structure from Motion • Results
Outline
Cluster SfM - 26 SMS 09/12/12
Runtime Performance
0
20
40
60
100 200 300 400
Run
time
(hou
rs)
Image Set Size
Bundler (single-core)
Feat. Extr. Feat. Match Bundle Adj.
0
10
20
30
100 200 300 400
Run
time
(hou
rs)
Image Set Size
VisualSFM (multi-core & GPU)
Feat. Extr. Feat. Match Bundle Adj.
0
0.2
0.4
0.6
0.8
1
100 200 300 400
Run
time
(hou
rs)
Image Set Size
Cluster-based SfM (64 cores)
Feat. Match & Extr. Bundle Adj. Fuse
Cluster SfM - 27 SMS 09/12/12
Speed-Up & Scalability
Cluster SfM Scalability (400 photos)
Cluster Size 8 16 32 64
Partitions 8 16 16 16 Frame rate (per min.) 1.2 2.4 4.3 7.6
Points per Frame (x103) 2.0 .87 .87 .87
0
5
10
15
20
25
30
35
40
45
0 16 32 48 64
Spee
d-up
(t(1
)/t(Np)
)
Number of Processors (Np)
Cluster-based SfM Speed-up
N=75
100
200
300
400
Cluster SfM - 28 SMS 09/12/12
Reconstruction Quality
RMS Error of Reconstructed Points (meters) Input size (N) 100 200 300 400
VisualSFM 0.20 7.15 1.05 0.44
Cluster-based SfM
4 Partitions 0.10
8 Partitions 0.27
12 Partitions 0.26
16 Partitions 0.24
0
50
100
150
200
250
300
50 100 150 200 250 300 350 400
Poin
ts (x
1,00
0)
Image Set Size
Sparse Point Cloud Density
Bundler
VisualSFM
Cluster (N/M=25)
Cluster SfM - 29 SMS 09/12/12
• Creating 3D reconstructions of photo collections scales well to the cluster – 64-core cluster shows 14x speedup over multi-core workstation with
GPU
• Our algorithm effectively parallelizes reconstruction for the case of aerial video around a ground point
• For algorithms that aren’t “pleasantly parallel”, deep knowledge or application and computing environment necessary to parallelize
• Future work: – Reduce feature matching complexity for different types of input sets – Point cloud geo-registration by matching against reference data
(e.g. LIDAR or satellite imagery)
Summary
Cluster SfM - 30 SMS 09/12/12
Scott Sawyer (scott.sawyer@ll.mit.edu)
SIGMA Team:
• Karl Ni
• Alex Vasile
• Nick Armstrong-Crews
• Ethan Phelps
• Nadya Bliss
• Prof. Noah Snavely (Cornell University)
Acknowledgements
top related