range imaging from calibration to modeling - tu wien · range imaging from calibration to modeling...
TRANSCRIPT
![Page 1: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/1.jpg)
Range Imaging
From Calibration to Modeling
Norbert Pfeifer, Sajid Ghuffar, Willi Karel,
Camillo Ressl, Stefan Niedermayr
Institute of Photogrammetry and Remote Sensing
Vienna University of Technology
![Page 2: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/2.jpg)
Why range cameras ?
Background 1 is Photogrammetry
• Passive brightness images, stills
• Object reconstruction by intersection of multiple rays to corresponding points
• Accuracy 1:10.000 – 1:100.000 (++)
• Resolution: diffraction limited
Background 2 is Laser Scanning
• Active ranging, hemispherical, sequential
• Object reconstruction from multiple scans for complex objects
• Accuracy 1:10.000 – 1:100.000
• Beam divergence near diffraction limit, sampling distance function of time
Range Imaging
• Active range imaging, in comparatively narrow FoV
• Accuracy 1:100 – 1:1.000
• High frame rate
![Page 3: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/3.jpg)
Geodata collection
• Geometric parameters of urban furniture/vegetation
e.g.: tree height, DBH, height of first living branch
• Enrichment of existing GeoDB
• Object static, camera dynamic
• Advantage: Ease of use, automating different manual procedures
See also: B. Jutzi, KIT, Karlsruhe
Building interiors, 3D BIM
• „As used“ documentation of buildings: 3D models of rooms
• Ontop floorplan backbone
• Object static, camera dynamic
• Advantage: Very high automation possible, cost-effective
See also: J. Böhn, UCL, London
GEO domain applications 1/2
![Page 4: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/4.jpg)
Fast geomorphological processes
• Landslide, mudslide, embankment failures
• Boundary values for better process understanding
• Object dynamic, camera static
• Advantage: range imaging as enabling technology
Traffic monitoring
• Cars, trains, etc.
• Objects dynamic, camera static (or on moving platform)
• Advntage: high reliability
See also: many others
GEO domain applications 2/2
![Page 5: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/5.jpg)
From calibration to modeling – Presentation outline
Scattering compensation Laboratory
Calibration Laboratory
Orientation Real scenes under advantageous conditions
Modeling Real scenes under advantageous conditions
Overall aim (medium term goal)
Scattering compensation Lab measurements/not required(?)
Calibration model identification Lab measurements
Calibration parameters + orientation Real scenes, on-the-job
Modeling Real scenes
![Page 6: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/6.jpg)
Scattering
Echo of the emitted optical signal is scattered to some extent over the sensor due to multiple reflections within the camera:
• i.e. between the lens, the optical filter, and the sensor
• “Lens Flare” in conventional photogrammetry
Distance distortions for far away objects can be up to 1m or greater
Ongoing instrumental developments, but effects also reported in new generation (Lichti, UCalgary)
![Page 7: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/7.jpg)
Modelling the Point Spread Function
Experimental Setup: point like target in foreground scatters light over
darker background
Investigating the depedence of PSF on the parameters:
• Target size
• Target distance
• Target position
• Integration time
PSF from deconvolution
![Page 8: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/8.jpg)
Spatially variant Point Spread Function
The intensity of scattering distortion changes with angle and distance
from the principal point
Shape of the scattering halo remains constant
[m]
![Page 9: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/9.jpg)
Evaluating the PSF
Applying the correction model to real test scenes
Compensating for distance and amplitude distortion in a combine
deconvolution algorithm
Overall error reduced, but compensation partly overshootes
![Page 10: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/10.jpg)
Calibration in Lab, handheld movies
Reference plane with control points
• Avoids scattering and object space multi path
• Very simple object model
Spatial resection only from amplitude image
Exterior and interior orienation estimated
for each frame
Target tracking fully automated
Significant intrinsic parameters
• Principal point coordinates
• Focal length
• Radial distortion of 3rd order
• Principle point depends on camera attitude
![Page 11: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/11.jpg)
Calibration in Lab, handheld movies
Reference plane with control points
Spatial resection
Significant intrinsic parameters
Systematic error = range observation – reference range
Advantage: very large data volume (e.g. 6000 frames)
Disadvantage
• Low amplitudes for large distances
• Motion blur for large distances
![Page 12: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/12.jpg)
Calibration in Lab – stills
Larger distances due to stills (max. range)
Experimental environment similar
• Control points cover larger volume
Target identification automated
• But computationally more intensive
Number of frames smaller (e.g. 850)
• But better SNR
amplitudes – observed range – range std.dev. – systematic range error
![Page 13: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/13.jpg)
Calibration results
Harmonic range errors according to modulation wavelength (10cm span)
Hyperbolic-type range error for amplitudes (20cm span)
Range error as function of position in sensor plane (50cm span)
Range error increases with integration time (3cm span)
0 1 2 3 4 5 6 7 8-0.1
-0.08
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
0.08
0.1
Observed dist. [m]
Der.
min
us o
bs.
dis
t. [
m]
Der. minus obs. dist. corr. 4 all but obs. dist. , corr. model
offset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs
0 1 2 3 4 5 6 7 80
2
4
6
8
10
12x 10
4
count
0 2000 4000 6000 8000 10000 12000 14000 16000 18000-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
Amplitude []
Der.
min
us o
bs.
dis
t. [
m]
Der. minus obs. dist. corr. 4 all but amplitude , corr. model
offset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs
0 2000 4000 6000 8000 10000 12000 14000 16000 180000
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5x 10
5
count
10 30 60 90 120 150 180 210 255-0.005
0
0.005
0.01
0.015
0.02
0.025
0.03
0.035
int. time []
der.
min
us o
bs.
dis
t. [
m]
Der. minus obs. dist. corr. 4 all but int. time , corr. model
offset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs
![Page 14: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/14.jpg)
Orientation and Modeling
Example 1
• Handheld movie
• Focus on orienation
• Weak object modeling to support orientation
• Scene requirements: containing planes (for noise reduction)
Example 2
• Simulation of range cameras with improved accuracy
• Focus on modeling
• Orienatition solved as necessary task for modeling extended scenes
• Scene requirements: brightness differences in object
![Page 15: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/15.jpg)
Orientation (and Modeling)
Scence contains large objects with simple geometric description (planes)
Objects found automatically by (video) segmentation
Subsequent scenes of movie transformed onto each other
• Scene n: described as set of planes
• Scene n+1: select „plane segment points“ close to planes of scene n
• Euclidean transformation of (n+1) to (n) [ ICP-type orientation with filters ]
![Page 16: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/16.jpg)
Orientation (and Modeling)
Video Segmentation
![Page 17: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/17.jpg)
Orientation (and Modeling)
Transformation of new to previous scene(s)
![Page 18: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/18.jpg)
(Orientation and) Modeling
Video data acquisition including still scenes (tripod)
• Similarity between stills ~ 75% (rotation by ¾ FoV)
Noise reduction in still sequences by averaging
Point correspondence between images throught SIFT
Orientation
• Using direction and range observations
• Tracking points through multiple images
• Global orientation in the final stage
Modeling as subsequent step
![Page 19: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/19.jpg)
(Orientation and) Modeling
Orientation fully automated
Amplitudes Object space in camera view
![Page 20: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/20.jpg)
(Orientation and) Modeling
Modeling
Filter erroneous points (low intensity, very short distances, corona points)
Triangulate (Geomagic), smooth triangulation, automated hole filling
![Page 21: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/21.jpg)
(Orientation and) Modeling
Residuals of
selected points
vs. smoothed
triangulation
up to 5cm
![Page 22: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/22.jpg)
Conclusions 1/2
Scattering
• Instrumental developments reduce problems
• Because range observations are used directly,
complete removal remains a research topic
On the job calibration
• Feasible
• Lab environment
• Reduction of systematic errors by 50-90%
Orienation
• Feasible for hand held movies
• Scene requirements in examples (planes or markers or noise reduction)
Modeling
• Random noise reduction necessary
• Tight integration with calibration and orientation seems possible
EO device may become necessary (MEMS IMU, etc.)
![Page 23: Range Imaging From Calibration to Modeling - TU Wien · Range Imaging From Calibration to Modeling Norbert Pfeifer, Sajid Ghuffar, Willi Karel, ... Scattering compensation Lab measurements/not](https://reader035.vdocuments.us/reader035/viewer/2022062908/5ae421007f8b9a5b348e4d6a/html5/thumbnails/23.jpg)
Outlook 2/2
Ongoing projects
• Acclimatization (MESA and PMDTec)
• Warm-Up and other influcences (MESA, together with Lichti)
• Orientation without scence requirements
Dreams and Wishes
1. Increase device stability (weight!)
2. Increase insensitivty to background light: outdoor applications
3. Increase resolution
4. Increase maximum range
5. Increase accuracy