Download - Multispectral and Multimodal 3d-Imaging
1
Gunther Notni1,2
5d-imaging Multispectral and Multimodal 3d-Imaging
1 Technical University Ilmenau,
Group for Quality Assurance and Industrial Image Processing,
Ilmenau
2 Fraunhofer Institute for Applied Optics and Precision Engineering IOF,
Jena
2
3d-imaging ◼ 3d surface measurement - established method
◼ Different 3d-sensor principles (passive and
active stereo, ToF, triangulation, laser
scanning, …)
◼ Application fields:
◼ industrial quality and process control
◼ humane-machine interaction
◼ robot control
◼ CAD/CAM technologies
◼ medical diagnostics
◼ security - face- / fingerprint ID
◼ forensics
◼ archaeology / paleontology
◼ arts, cultural heritage preservation
- 2 -07.06.2018
3
3d-imaging: pattern projection technique
▪ standard setup – active triangulation system (active stereo)
◼ 3d-imaging → point-cloud (x, y, z)
unique coding of surface points
→ many projection patterns required
◼ Gray code
◼ sinusoidal fringes
◼ variable dot pattern
◼ …
→ mainly limited to static objects/scenes
- 4 -
4
3d-imaging with projection of aperiodic sinusoidal fringes
◼ Active stereo based arrangement
◼ Projection of Aperiodic Sinusoidal Fringes
→ temporal correlation of patterns
𝜌 =σ𝑖=1𝑁 𝐼𝑖
1 − 𝐼 1 𝐼𝑖2 − 𝐼 2
σ𝑖=1𝑁 𝐼𝑖
1 − 𝐼 12
σ𝑖=1𝑁 𝐼𝑖
2 − 𝐼 22
maximum of NCC
Albrecht, Michaelis: IEEE Trans. Pattern Recogn. 14, 845–849 (1998)
Heist, Kühmstedt, Notni: Patent DE 10 2013 013 791 (2013)
Heist et al.: Opt. Eng. 53(11), 112208 (2014)
- 5 -07.06.2018
Correlation of a sequence of
fringes i.e. the stacks of the
intensity values between the
cameras
5
Temporal correlation of aperiodic fringe patternsStack of
intensity at
pixel pi
Camera 1
Advantages:
◼ No requirement on accuracy of pattern or knowledge on phase-(shift)
◼ Low influence of variability of fringe contrast or modulation
◼ No synchronization
→ alternative projection techniques instead of DMD/LCoS
Camera 2
pixel pi
Search at camera 2
- 6 -07.06.2018
6
◼GOBO Projection
Projection principle(side view):
lamp
ellipsoidal
reflector
GOBO
focusing lens
zoom lens image
GOBO
wheel
Heist et al.: Opt. Lasers Eng. 87, 90–96 (2016)
Heist et al.: Patent DE 10 2015 208 285 (2015)
High-Speed Broadband Projection
◼ High-Speed (up to 100kHz) and
broadband projection of aperiodic
sinusoidal fringes
◼ GOBO wheel with
aperiodic binary
fringes (front view):
- 19 -07.06.2018
◼ GOBO-projection system:
GOes Before Optics or
Graphical Optical BlackOut -
heat resistent mask
7
◼ generating an aperiodic sinusoidal fringe pattern by means of
slightly defocusing and rotating the GOBO wheel:
defocusing
1000 rpm, 90 µs
GOBO rotation
◼ GOBO Projection
High-Speed Broadband Projection
- 20 -07.06.2018
aperiodic sinusoidal aperiodic binary aperiodic defocused
8
4d-imaging
▪ 4d measurement of an airbag inflation: >1000 3d-frames/s
◼ temporal resolved (high-speed) 3d-imaging (x, y, z, t)
temporal 3d-point cloudmeasurement
scene
- 7 -07.06.2018
Heist, Lutzke, Schmidt, Dietrich, Kühmstedt, Tünnermann, Notni; Opt. and Lasers in Engineering 87 (2016) 90-96
◼ pattern projector: high power
gas discharge lamp
◼ 1MPixel camera resolution
(Photron FASTCAM SA-X2),
image frame rate 12kHz
◼ no synchronization between
projector and cameras
necessary
◼ measurement field from 0.5 x
0.5 m² up to several square
meters (>2 x 2 m²)
9
5d-imaging
• x, y, z:surface shape
• t: at video rate
• λ1, …, λn: spectral information or one more (special) wavelength of light
◼ 5d-imaging: spatial and spectral 3d-imaging (x, y, z, t, λ1, …, λn)
Classical
3d-imaging
White-light and/or special NIR-wavelength
Multispectral
and
Multimodal
3d-imaging 5d - imaging
3d-imaging at different wavelength / huge spectral range
Visuell
0,4 – 0,7
SWIR
1,1 – 2,7
MIR
3 – 5
FIR
5 – 14
UV
0,2- 0,4
NIR
0,7 – 1,1
- 9 -07.06.2018
10
Multispectral and Multimodal 3d-Measurement
▪ Approaches: λ1, …, λn
Multispectral
techniques
Multimodal
techniques
projector
n spectral resolved
3d-images
at any point in time
- 10 -07.06.2018
one 3d-image
+ n mapped
spectral textures
11
Active 3d-Sensor
Filter-
wheelLcOS ?
• Spectral resolution!
• Lateral resolution!
• High frame rate (>50Hz)!
• Broadband or spectral independent
pattern projection!
• Spectral range (UV, VIS, NIR, … FIR)!
Multispectral camera Projector
Snap-
shot
mosaicDMD ?
Multispectral and Multimodal 3d-imaging
Other ? Other ?
- 11 -07.06.2018
Requirements:
12
Multispectral camera
- 13 -07.06.2018
Snap-Shot Mosaic Camera Filter-Wheel Camera
- onCMOS monolitic (IMEC/XIMEA)
- on-glas wafer (Silios)
- 4, 9, 16, 25 spectral bands
- different spectral ranges (f.e. 465-630nm; 600-
875 nm or 675-975 nm)
- typically band width: 10nm
- up 170 fps at 8-bit resolution
- reduced resolution: f.e. 25 bands - 409x217pixel
- 12 configurable spectral filters (400 –
950nm)
- up to 10 fps (depends on sensor and
rotation)
- resolution: > 1MP Images: XIMEA, IMEC
13
◼ microlens-array →compact system and high number of channels
◼ linear variable filter (LVF) in front of apertures →dedicated wavelengths
◼ baffle array → suppresscrosstalk betweenadjacent channels
Working principle
1 channel
cover glass
image sensor
microlens-array (MLA)
linear variable filter(LVF)
baffle array
- 15 -07.06.2018
▪ Lens diameter: 500µm
Microlens-array based multispectral camera
14
Microlens-array based multispectral camera
◼ 66 spectral bands
◼ spatial sampling (per channel): 400x400 pixels
◼ wavelength range: 450-880 nm
◼ total track length: 7.2 mm
◼ F/# 7, focal length: 3.65 mm
◼ FOV: 68°
◼ angular sampling: 0.12°
Final optical system60 mm
60 mm28 mm
Raw image of Multispectral camera: 11x6 spectral channels
- 16 -07.06.2018
15
Pattern projection
◼ Requirements for
multispectral 3d-measurement
DMD – Digital Micro mirror
Device
LCoS – Liquid Crystal on
Silicon
◼ High frame projection rate
(>>100Hz)
◼ Broadband, spectral
independent pattern
projection / modulation
◼ Spectral range (UV, VIS, NIR,
… FIR)
Near UV, VIS, NIR, SWIR VIS
◼ Normally used image modulators
- 18 -07.06.2018
16
◼ GOBO Projection – GOBO wheel
Heist et al.: Opt. Lasers Eng. 87, 90–96 (2016)
Heist et al.: Patent DE 10 2015 208 285 (201
Brahm et.al. European Optical Society Biennial Meeting (2018)
High-Speed Broadband Projection
- 19 -07.06.2018
Metal wheel with laser cutted slots
◼ Adaptive free-form mirror
◼ 3D projection technology for a wide range over the electromagnetic spectrum (UV – VIS – IR)
Light source
17
12 spectral channels: 400 nm – 1050 nmZhang, C.; Rosenberger, M.; Breitbarth, A.; Notni, G IEEE 2016 Proc. S. 267-272
Filter-wheel-camera 2
Projector
Broadband Gobo-
Projector
Light source: high-
power halogen lamp
(15.000lm)
Filter-wheel-camera 1
Multispectral 3d-measurement
◼ Experimental set-up
- 24 -07.06.2018
18
RGB 700 nm 750 nm 800 nm
850 nm 900 nm 950 nm
Zhang, C.; Rosenberger, M.; Notni, G.innteract conference "3D SENSATION", Chemnitz, Juni (2016)
Multispectral 3d-measurement
◼ Multispectral 3d-measurement of a human hand
Possible application:
◼ Robotic surgery
◼ To find the blood vessels
(veins) and muscles
- 25 -07.06.2018
19
3d-fluorescence imaging
◼ Basic arrangement
◼ Aim: measurement of the spatial distribution of contamination
→ 3d control of cleaning robots
◼ 3d-measurement and fluorescenceexcitation usingUV-LED
07.06.2018
Pattern Projector
Multispectral Camera 1
Multispectral Camera 2
object
UV-LED
20
550 nm
fluorescence image
3d-fluorescence image
3d localisationof oil spots
detectionof oil spots
3d-fluorescence imaging: experimental result
3d-data
- 40 -07.06.2018
21
Agriculture, Biology – Precision Farming
22
Nicolaï et al.: Postharvest Biol. Technol. 46(2), 99–118 (2007)
Zhang, Li, Zhang: Spectrosc. Int. J. 27(2), 93–105 (2012)
Peñuelas et al.: Int. J. Remote Sensing 18(13), 2869–2875 (1997)
- 43 -07.06.2018
Precision farming - investigation of form and
spectral signature during drying processes
◼ Multispectral 3d-measurement of a leaf
23
t = 0 mint = 90 mint = 210 min
950
Ro
bj(a
.u.)
750 800 850 900
λ (nm)t = 0 min t = 90 min t = 210 min
Nicolaï et al.: Postharvest Biol. Technol. 46(2), 99–118 (2007)
Zhang, Li, Zhang: Spectrosc. Int. J. 27(2), 93–105 (2012)
Peñuelas et al.: Int. J. Remote Sensing 18(13), 2869–2875 (1997)
◼ Multispectral 3d-measurement of a citrus plant during water absorption
Possible application:
◼ for the evaluation of fruit and vegetable quality or for the determination of leaf water content
- 43 -07.06.2018
Precision farming - leaf water content
24
◼ collection of historical relief globes (Haeckel Museum Jena)
◼ picture: relief globe from 1885; created by the geographic-artistic institution of Ludwig Julius Heymann in Berlin
◼ offer a promising non-invasive option for the analysis and classification of historical measurement objects
◼ Classification using spectral and 3d-data (in progress)
λ (nm)650 700 750 800 850 900 950
Liang: Applied Physics A 106(2), 309–323 (2012)
Fischer, Kakoulli: Stud. Conserv. 51, 3–16 (2006)
Balas et al.: J. Cult. Herit. 4, Supplement 1, 330–337 (2003)
◼ Multispectral 3d-measurement of a historical globe
07.06.2018
Cultural heritage preservation
25
◼ Motivation: Uncooperative Objects in Visible Wavelength Range (VIS)
reflective surfaces
© Edmund Optics
transparent objects
© Edmund Optics
translucent objects
© Edmund Optics
with high contrast or texture
black objects
© AUDI
uncooperative assemblies
change of spectral range: use
of physical properties
3d-measurement at special wavelength
- 47 -07.06.2018
26
Stack of
N thermal imagesN projections
3d-data
MWIR camera image of thermal pattern:
◼ Scanning from heating - MWIR 3d-sensor
- 48 -07.06.2018
3d-measurement at special wavelength
27
◼ Setup
system parameters:
◼ optical output power: 80 W (=10,6µm)
◼ working distance: 375 mm
◼ baseline distance cameras: 190 mm
◼ triangulation angle: 28.4°
Scanning from heating - MWIR 3d-sensor
MWIR-
camera 1
MWIR-
camera 2
GOBO-
mask
Objekt
- 49 -
28
3d point cloudglass bulb (VIS) thermal image (MWIR)
◼ measurement field: Ø 150 mm
◼ spatial resolution: 400 µm
◼ thermal contrast: 1.2 K
◼ Irradiation time: 1 sec.
◼ 3d-measurement of transparent objects
Scanning from heating - MWIR 3d-sensor
- 51 -07.06.2018
29
Eyeglasses with plastic lensorthodontic brace(Fa. Invisalign)
3D point cloud
3D point cloud
thermal image (MWIR)thermal image (MWIR)
Scanning from heating - MWIR 3D-sensor
- 52 -07.06.2018
◼ 3d-measurement of transparent objects
30
Conclusion
◼ Multispectral and Multimodal 3d-imaging, i.e. the use of multi-spectral
cameras in 3d-sensor technology opens up new possibilities in:
▪ precision farming
▪ medicine
▪ cultural heritage preservation
▪ production / object identification
▪ 3d-fluorescene imaging
◼ The transition to the far infrared spectral range in pattern projection i.e.
scanning by heating enables the 3d-measurement of transparent and
translucent objects
07.06.2018
31
Acknowledgements
◼ Technical University Ilmenau (Ilmenau):
Maik Rosenberger, Chen Zhang, Paul-Gerald Dittrich, Karl Reichwald
◼ Fraunhofer IOF (Jena):
Peter Kühmstedt, Stefan Heist, Peter Lutzke, Christian Bräuer-Burchardt, Ingo Gebhardt,
Daniel Höhne, Patrick Dietrich, Robert Brunner, Robert Brünning, Martin Landmann
This work was mainly supported by
BMBF: Zwanzig20 Allianz 3Dsensation,
Innoprofile Qualimess Next generation
07.06.2018