1

Gunther Notni1,2

5d-imaging Multispectral and Multimodal 3d-Imaging

1 Technical University Ilmenau,

Group for Quality Assurance and Industrial Image Processing,

Ilmenau

2 Fraunhofer Institute for Applied Optics and Precision Engineering IOF,

Jena

2

3d-imaging ◼ 3d surface measurement - established method

◼ Different 3d-sensor principles (passive and

active stereo, ToF, triangulation, laser

scanning, …)

◼ Application fields:

◼ industrial quality and process control

◼ humane-machine interaction

◼ robot control

◼ CAD/CAM technologies

◼ medical diagnostics

◼ security - face- / fingerprint ID

◼ forensics

◼ archaeology / paleontology

◼ arts, cultural heritage preservation

- 2 -07.06.2018

3

3d-imaging: pattern projection technique

▪ standard setup – active triangulation system (active stereo)

◼ 3d-imaging → point-cloud (x, y, z)

unique coding of surface points

→ many projection patterns required

◼ Gray code

◼ sinusoidal fringes

◼ variable dot pattern

◼ …

→ mainly limited to static objects/scenes

- 4 -

4

3d-imaging with projection of aperiodic sinusoidal fringes

◼ Active stereo based arrangement

◼ Projection of Aperiodic Sinusoidal Fringes

→ temporal correlation of patterns

𝜌 =σ𝑖=1𝑁 𝐼𝑖

1 − 𝐼 1 𝐼𝑖2 − 𝐼 2

σ𝑖=1𝑁 𝐼𝑖

1 − 𝐼 12

σ𝑖=1𝑁 𝐼𝑖

2 − 𝐼 22

maximum of NCC

Albrecht, Michaelis: IEEE Trans. Pattern Recogn. 14, 845–849 (1998)

Heist, Kühmstedt, Notni: Patent DE 10 2013 013 791 (2013)

Heist et al.: Opt. Eng. 53(11), 112208 (2014)

- 5 -07.06.2018

Correlation of a sequence of

fringes i.e. the stacks of the

intensity values between the

cameras

5

Temporal correlation of aperiodic fringe patternsStack of

intensity at

pixel pi

Camera 1

Advantages:

◼ No requirement on accuracy of pattern or knowledge on phase-(shift)

◼ Low influence of variability of fringe contrast or modulation

◼ No synchronization

→ alternative projection techniques instead of DMD/LCoS

Camera 2

pixel pi

Search at camera 2

- 6 -07.06.2018

6

◼GOBO Projection

Projection principle(side view):

lamp

ellipsoidal

reflector

GOBO

focusing lens

zoom lens image

GOBO

wheel

Heist et al.: Opt. Lasers Eng. 87, 90–96 (2016)

Heist et al.: Patent DE 10 2015 208 285 (2015)

High-Speed Broadband Projection

◼ High-Speed (up to 100kHz) and

broadband projection of aperiodic

sinusoidal fringes

◼ GOBO wheel with

aperiodic binary

fringes (front view):

- 19 -07.06.2018

◼ GOBO-projection system:

GOes Before Optics or

Graphical Optical BlackOut -

heat resistent mask

7

◼ generating an aperiodic sinusoidal fringe pattern by means of

slightly defocusing and rotating the GOBO wheel:

defocusing

1000 rpm, 90 µs

GOBO rotation

◼ GOBO Projection

High-Speed Broadband Projection

- 20 -07.06.2018

aperiodic sinusoidal aperiodic binary aperiodic defocused

8

4d-imaging

▪ 4d measurement of an airbag inflation: >1000 3d-frames/s

◼ temporal resolved (high-speed) 3d-imaging (x, y, z, t)

temporal 3d-point cloudmeasurement

scene

- 7 -07.06.2018

Heist, Lutzke, Schmidt, Dietrich, Kühmstedt, Tünnermann, Notni; Opt. and Lasers in Engineering 87 (2016) 90-96

◼ pattern projector: high power

gas discharge lamp

◼ 1MPixel camera resolution

(Photron FASTCAM SA-X2),

image frame rate 12kHz

◼ no synchronization between

projector and cameras

necessary

◼ measurement field from 0.5 x

0.5 m² up to several square

meters (>2 x 2 m²)

9

5d-imaging

• x, y, z:surface shape

• t: at video rate

• λ1, …, λn: spectral information or one more (special) wavelength of light

◼ 5d-imaging: spatial and spectral 3d-imaging (x, y, z, t, λ1, …, λn)

Classical

3d-imaging

White-light and/or special NIR-wavelength

Multispectral

and

Multimodal

3d-imaging 5d - imaging

3d-imaging at different wavelength / huge spectral range

Visuell

0,4 – 0,7

SWIR

1,1 – 2,7

MIR

3 – 5

FIR

5 – 14

UV

0,2- 0,4

NIR

0,7 – 1,1

- 9 -07.06.2018

10

Multispectral and Multimodal 3d-Measurement

▪ Approaches: λ1, …, λn

Multispectral

techniques

Multimodal

techniques

projector

n spectral resolved

3d-images

at any point in time

- 10 -07.06.2018

one 3d-image

+ n mapped

spectral textures

11

Active 3d-Sensor

Filter-

wheelLcOS ?

• Spectral resolution!

• Lateral resolution!

• High frame rate (>50Hz)!

• Broadband or spectral independent

pattern projection!

• Spectral range (UV, VIS, NIR, … FIR)!

Multispectral camera Projector

Snap-

shot

mosaicDMD ?

Multispectral and Multimodal 3d-imaging

Other ? Other ?

- 11 -07.06.2018

Requirements:

12

Multispectral camera

- 13 -07.06.2018

Snap-Shot Mosaic Camera Filter-Wheel Camera

- onCMOS monolitic (IMEC/XIMEA)

- on-glas wafer (Silios)

- 4, 9, 16, 25 spectral bands

- different spectral ranges (f.e. 465-630nm; 600-

875 nm or 675-975 nm)

- typically band width: 10nm

- up 170 fps at 8-bit resolution

- reduced resolution: f.e. 25 bands - 409x217pixel

- 12 configurable spectral filters (400 –

950nm)

- up to 10 fps (depends on sensor and

rotation)

- resolution: > 1MP Images: XIMEA, IMEC

13

◼ microlens-array →compact system and high number of channels

◼ linear variable filter (LVF) in front of apertures →dedicated wavelengths

◼ baffle array → suppresscrosstalk betweenadjacent channels

Working principle

1 channel

cover glass

image sensor

microlens-array (MLA)

linear variable filter(LVF)

baffle array

- 15 -07.06.2018

▪ Lens diameter: 500µm

Microlens-array based multispectral camera

14

Microlens-array based multispectral camera

◼ 66 spectral bands

◼ spatial sampling (per channel): 400x400 pixels

◼ wavelength range: 450-880 nm

◼ total track length: 7.2 mm

◼ F/# 7, focal length: 3.65 mm

◼ FOV: 68°

◼ angular sampling: 0.12°

Final optical system60 mm

60 mm28 mm

Raw image of Multispectral camera: 11x6 spectral channels

- 16 -07.06.2018

15

Pattern projection

◼ Requirements for

multispectral 3d-measurement

DMD – Digital Micro mirror

Device

LCoS – Liquid Crystal on

Silicon

◼ High frame projection rate

(>>100Hz)

◼ Broadband, spectral

independent pattern

projection / modulation

◼ Spectral range (UV, VIS, NIR,

… FIR)

Near UV, VIS, NIR, SWIR VIS

◼ Normally used image modulators

- 18 -07.06.2018

16

◼ GOBO Projection – GOBO wheel

Heist et al.: Opt. Lasers Eng. 87, 90–96 (2016)

Heist et al.: Patent DE 10 2015 208 285 (201

Brahm et.al. European Optical Society Biennial Meeting (2018)

High-Speed Broadband Projection

- 19 -07.06.2018

Metal wheel with laser cutted slots

◼ Adaptive free-form mirror

◼ 3D projection technology for a wide range over the electromagnetic spectrum (UV – VIS – IR)

Light source

17

12 spectral channels: 400 nm – 1050 nmZhang, C.; Rosenberger, M.; Breitbarth, A.; Notni, G IEEE 2016 Proc. S. 267-272

Filter-wheel-camera 2

Projector

Broadband Gobo-

Projector

Light source: high-

power halogen lamp

(15.000lm)

Filter-wheel-camera 1

Multispectral 3d-measurement

◼ Experimental set-up

- 24 -07.06.2018

18

RGB 700 nm 750 nm 800 nm

850 nm 900 nm 950 nm

Zhang, C.; Rosenberger, M.; Notni, G.innteract conference "3D SENSATION", Chemnitz, Juni (2016)

Multispectral 3d-measurement

◼ Multispectral 3d-measurement of a human hand

Possible application:

◼ Robotic surgery

◼ To find the blood vessels

(veins) and muscles

- 25 -07.06.2018

19

3d-fluorescence imaging

◼ Basic arrangement

◼ Aim: measurement of the spatial distribution of contamination

→ 3d control of cleaning robots

◼ 3d-measurement and fluorescenceexcitation usingUV-LED

07.06.2018

Pattern Projector

Multispectral Camera 1

Multispectral Camera 2

object

UV-LED

20

550 nm

fluorescence image

3d-fluorescence image

3d localisationof oil spots

detectionof oil spots

3d-fluorescence imaging: experimental result

3d-data

- 40 -07.06.2018

21

Agriculture, Biology – Precision Farming

22

Nicolaï et al.: Postharvest Biol. Technol. 46(2), 99–118 (2007)

Zhang, Li, Zhang: Spectrosc. Int. J. 27(2), 93–105 (2012)

Peñuelas et al.: Int. J. Remote Sensing 18(13), 2869–2875 (1997)

- 43 -07.06.2018

Precision farming - investigation of form and

spectral signature during drying processes

◼ Multispectral 3d-measurement of a leaf

23

t = 0 mint = 90 mint = 210 min

950

Ro

bj(a

.u.)

750 800 850 900

λ (nm)t = 0 min t = 90 min t = 210 min

Nicolaï et al.: Postharvest Biol. Technol. 46(2), 99–118 (2007)

Zhang, Li, Zhang: Spectrosc. Int. J. 27(2), 93–105 (2012)

Peñuelas et al.: Int. J. Remote Sensing 18(13), 2869–2875 (1997)

◼ Multispectral 3d-measurement of a citrus plant during water absorption

Possible application:

◼ for the evaluation of fruit and vegetable quality or for the determination of leaf water content

- 43 -07.06.2018

Precision farming - leaf water content

24

◼ collection of historical relief globes (Haeckel Museum Jena)

◼ picture: relief globe from 1885; created by the geographic-artistic institution of Ludwig Julius Heymann in Berlin

◼ offer a promising non-invasive option for the analysis and classification of historical measurement objects

◼ Classification using spectral and 3d-data (in progress)

λ (nm)650 700 750 800 850 900 950

Liang: Applied Physics A 106(2), 309–323 (2012)

Fischer, Kakoulli: Stud. Conserv. 51, 3–16 (2006)

Balas et al.: J. Cult. Herit. 4, Supplement 1, 330–337 (2003)

◼ Multispectral 3d-measurement of a historical globe

07.06.2018

Cultural heritage preservation

25

◼ Motivation: Uncooperative Objects in Visible Wavelength Range (VIS)

reflective surfaces

© Edmund Optics

transparent objects

© Edmund Optics

translucent objects

© Edmund Optics

with high contrast or texture

black objects

© AUDI

uncooperative assemblies

change of spectral range: use

of physical properties

3d-measurement at special wavelength

- 47 -07.06.2018

26

Stack of

N thermal imagesN projections

3d-data

MWIR camera image of thermal pattern:

◼ Scanning from heating - MWIR 3d-sensor

- 48 -07.06.2018

3d-measurement at special wavelength

27

◼ Setup

system parameters:

◼ optical output power: 80 W (=10,6µm)

◼ working distance: 375 mm

◼ baseline distance cameras: 190 mm

◼ triangulation angle: 28.4°

Scanning from heating - MWIR 3d-sensor

MWIR-

camera 1

MWIR-

camera 2

GOBO-

mask

Objekt

- 49 -

28

3d point cloudglass bulb (VIS) thermal image (MWIR)

◼ measurement field: Ø 150 mm

◼ spatial resolution: 400 µm

◼ thermal contrast: 1.2 K

◼ Irradiation time: 1 sec.

◼ 3d-measurement of transparent objects

Scanning from heating - MWIR 3d-sensor

- 51 -07.06.2018

29

Eyeglasses with plastic lensorthodontic brace(Fa. Invisalign)

3D point cloud

3D point cloud

thermal image (MWIR)thermal image (MWIR)

Scanning from heating - MWIR 3D-sensor

- 52 -07.06.2018

◼ 3d-measurement of transparent objects

30

Conclusion

◼ Multispectral and Multimodal 3d-imaging, i.e. the use of multi-spectral

cameras in 3d-sensor technology opens up new possibilities in:

▪ precision farming

▪ medicine

▪ cultural heritage preservation

▪ production / object identification

▪ 3d-fluorescene imaging

◼ The transition to the far infrared spectral range in pattern projection i.e.

scanning by heating enables the 3d-measurement of transparent and

translucent objects

07.06.2018

31

Acknowledgements

◼ Technical University Ilmenau (Ilmenau):

Maik Rosenberger, Chen Zhang, Paul-Gerald Dittrich, Karl Reichwald

◼ Fraunhofer IOF (Jena):

Peter Kühmstedt, Stefan Heist, Peter Lutzke, Christian Bräuer-Burchardt, Ingo Gebhardt,

Daniel Höhne, Patrick Dietrich, Robert Brunner, Robert Brünning, Martin Landmann

This work was mainly supported by

BMBF: Zwanzig20 Allianz 3Dsensation,

Innoprofile Qualimess Next generation

07.06.2018