2nd international workshop on point cloud processing stuttgart...
TRANSCRIPT
Universität Stuttgart
ifpifp
Deriving Semanticsfrom Textured Meshes
2nd International Workshop on Point Cloud Processing
Stuttgart, December 04-05, 2019
ifpDominik Laupheimer
ifpifpUniversität Stuttgart
Textured Meshes
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
A mesh is a wired point cloud.
2019/12/05 2
ifpifpUniversität Stuttgart
• Reduced Memory Consumption
• Good Compression Behavior (Noise Reduction)
• Waterproof Surface Representation
Explicit Topology
Texture
• Unambiguous Normal Calculation
2019/12/05 3
Motivation: Why Meshes?
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
ifpifpUniversität Stuttgart
• “Fusing” Data Representation
Imagery
Point Clouds
• Use-cases
Viewshed and Flood Analysis
City Models
Visualization + VR
2019/12/05 4
Motivation: Why Meshes?
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
ifpifpUniversität Stuttgart
“Meshes are comprehensive maps for literally the whole world!”
Motivation: Why Meshes?
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
5
Snapshot of Google’s 3D representation (mesh).
ifpifpUniversität Stuttgart
Aim: Semantic Segmentation of Meshes
Machine
Learning
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
6
ifpifpUniversität Stuttgart
ALS Point Cloud Benchmarks / Public Data Sets Mesh Benchmarks / Public Data Sets
ISPRS 3D Semantic Labeling (Vaihingen, V3D)
RoofN3D (TU Berlin)
AHN3 (the Netherlands)
GRSS Data Fusion Contest
(Track 4: 3D Point Cloud Classification)
?
Availability of Ground Truth Data
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
7
ifpifpUniversität Stuttgart
ALS Point Cloud Benchmarks / Public Data Sets Mesh Benchmarks / Public Data Sets
ISPRS 3D Semantic Labeling (Vaihingen, V3D)
RoofN3D (TU Berlin)
AHN3 (the Netherlands)
GRSS Data Fusion Contest
(Track 4: 3D Point Cloud Classification)
Only Indoor Scenes
Availability of Ground Truth Data
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
8
ifpifpUniversität Stuttgart
Data Acquisition
Ground Truth Generation
2019/12/05
LiDAR: 800 pts/m², footprint Ø < 3 cm
Photogrammetry: GSD @ 3.7 mm (nadir), 2.3 cm (oblique)
Cramer, M.; Haala, N.; Laupheimer, D.; Mandlburger, G. & Havel, P. , 2018:
Ultra-high precision UAV-based LiDAR and Dense Image Matching. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLII-1, 115-120.
DOI: 10.5194/isprs-archives-XLII-1-115-2018
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
9
ifpifpUniversität Stuttgart
Ground Truth Generation
2019/12/05
✋✎
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
10
LiDAR Point Cloud
Textured Mesh
Labeled Mesh
ifpifpUniversität Stuttgart
Labeled Ground Truth
2019/12/05
0. building mass/facade 1. roof 2. impervious surface 3. green space
4. mid and high vegetation 5. vehicle 6. chimney/antenna 7. clutter
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
11
0 1 2 3 4 5 6 7
ifpifpUniversität Stuttgart
Aim: Semantic Segmentation of Meshes
Machine
Learning
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
12
ifpifpUniversität Stuttgart
Methodology
2019/12/05
• Ground Truth Generation
• Feature Calculation
Geometric & radiometric
Multi-scale contextual features
• Train Classifier
Multi-Branch 1D CNN*
RF
• Inference/Evaluation
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
13
* George, D., Xie, X. & Tam, G., 2018:
3D Mesh Segmentation via Multi-branch 1D Convolutional Neural Networks. Graphical Models, 96, 1-10.
ifpifpUniversität Stuttgart
Feature Calculation
Height above Ground
Density Horizontality
2019/12/05 14
Texture
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
ifpifpUniversität Stuttgart
Network Architecture
Feature-Based Multi-Branch 1D CNN
2019/12/05
George, D., Xie, X. & Tam, G., 2018:
3D Mesh Segmentation via Multi-branch 1D Convolutional Neural Networks. Graphical Models, 96, 1-10.
PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
15
Scale 0
Scale 1
Scale 2
ifpifpUniversität Stuttgart
RF Prediction250 trees, depth: 25
Training Time: 6.6h
Accuracy: 79.01%
Inference Time: 45.83s
Results
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
16
Textured Mesh~300.000 faces
Ground TruthLabel Noise
1D CNN Prediction9.3 million parameters
Training Time: < 15min
Accuracy: 79.87%
Inference Time: 14.69s
ifpifpUniversität Stuttgart
Feature-Based Multi-Branch 1D CNN
Comparison of Feature Influence
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
17
Textured Mesh Prediction
(all features)
Prediction
(geometry only)
Prediction
(texture only)
Ground Truth
ifpifpUniversität Stuttgart
Feature-Based Multi-Branch 1D CNN
Comparison of Feature Influence
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
18
Textured Mesh Prediction Using
Geometric Features Only
Prediction Using
Geometric & Radiometric
Features
ifpifpUniversität Stuttgart 2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
19
Conclusion
• Alternative 3D Data Representation: Meshes
Data fusion
Good georeferencing/relative orientation of LiDAR and imagery necessary
• Ground Truth Generation
• Pipeline for Mesh Generation and Semantic Segmentation
Overall accuracy: ~80 %
Detection of buildings and mid/high vegetation works well
• Features
Geometry > Radiometry
Radiometry matters!
ifpifpUniversität Stuttgart
• Ground Truth Generation
Avoid label noise
Crowdsourcing
• Georeferencing
Hybrid georeferencingGlira, P.; Pfeifer, N.; Mandlburger, G., 2019:
Hybrid Orientation of Airborne LIDAR Point Clouds and Aerial Images.
ISPRS Annals of Photogrammetry, Remote Sensing and
Spatial Information Sciences, Volume IV-2/W5, 2019, pp.567-574.
DOI: 10.5194/isprs-annals-IV-2-W5-567-2019
• Features
Incorporate LiDAR features
Incorporate texture explicitly
Future Work
2019/12/05PCP19 - Deriving Semantics from Textured Meshes.
D. Laupheimer, N. Haala
20