Computer Vision and Robotics Group

Institut d’Informàtica i Aplicacions

Jordi PagèsJoaquim Salvi

Overview of coded light projection techniques for automatic 3D

profiling

2

Formació

Presentation outline

1/36

• Introduction• Coded pattern classification

– Time-multiplexing– Spatial codification– Direct codification

• Experimental results• Sub-pixel matching• Conclusions & guidelines

Conclusions

Sub-pixel matching

Experiments

Classification

Introduction

3

Conclusions

Sub-pixel matching

Experiments

Classification

Introduction

Shape acquisition

Stereovision

Encoding/

decoding

Introduction: shape acquisition

2/36

Shape acquisition techniques

Contact

Non-destructive Destructive

CMM Jointed armsSlicing

Non-contact

Reflective Transmissive

Non-optical

Optical

Industrial CT

Microwave radar

Sonar

PassiveActive

Stereo Shading Silhouettes TextureMotion

Shape from X

Imaging radar

Triangulation

Interferometry

(Coded) Structured

light

Moire Holography

Stereo

Source: Brian Curless

4

Conclusions

Sub-pixel matching

Experiments

Classification

Introduction

Shape acquisition

Stereovision

Encoding/

decoding

Introduction: passive stereovision

3/36

• Correspondence problem

• geometric constraints

search along epipolar lines

• 3D reconstruction of matched pairs by triangulation

5

Conclusions

Sub-pixel matching

Experiments

Classification

Introduction

Shape acquisition

Stereovision

Encoding/

decoding

Introduction: active stereo (coded structured light)

4/36

• One of the cameras is replaced by a light emitter

• Correspondence problem is solved by searching the pattern in the camera image (pattern decoding)

6

Conclusions

Sub-pixel matching

Experiments

Classification

Introduction

Shape acquisition

Stereovision

Encoding/

decoding

Introduction: pattern encoding/decoding (I)

5/36

• A pattern is encoded when after projecting it onto a surface, a set of regions of the observed projection can be easily matched with the original pattern. Example: pattern with two-encoded-columns

• The process of matching an image region with its corresponding pattern region is known as pattern decoding similar to searching correspondences

• Decoding a projected pattern allows a large set of correspondences to be easily found thanks to the a priori knowledge of the light pattern

Pixels in red and yellow are directly matched with the pattern columns

Codification using colors

Object scene

7

Conclusions

Sub-pixel matching

Experiments

Classification

Introduction

Shape acquisition

Stereovision

Encoding/

decoding

6/36

Introduction: pattern encoding/decoding (II)

• Two ways of encoding the correspondences: single and double axis codification it determines how the triangulation is calculated

• Decoding the pattern means locating points in the camera image whose corresponding point in the projector pattern is a priori known

Double-axis encoding

Triangulation by line-to-plane intersection

Triangulation by line-to-line intersection

Single-axis encoding

8

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Coded structured light patterns: a classification proposal

7/36

Binary codes Posdamer et al., Inokuchi et al.,

Minou et al., Trobina, Valkenburg and McIvor, Skocaj and Leonardis,

Rocchin et al., …

n-ary codes Caspi et al., Horn and Kiryati, Osawa et al.,…

Gray code + Phase shifting

Bergmann, Sansoni et al., Wiora, Gühring, …

TIME-MULTIPLEXING

Hybrid methods K. Sato, Hall-Holt and Rusinkiewicz, Wang et al., …

Non-formal codification

Maruyama and Abe, Durdle et al., Ito and Ishii, Boyer and Kak, Chen

et al., …

De Bruijn sequences Hügli and Maître, Monks et al.,

Vuylsteke and Oosterlinck, Salvi et al. Lavoi et al., Zhang et al., …

SPATIAL CODIFICATION

M-arrays Morita et al., Petriu et al., Kiyasu et al., Spoelder et al., Griffin and Yee, Davies and Nixon, Morano et

al., …

Grey levels Carrihill and Hummel, Chazan and Kiryati, Hung, ... DIRECT

CODIFICATION Colour Tajima and Iwakawa, Smutny and Pajdla, Geng, Wust and Capson, T.

Sato, …

9

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Time-multiplexing

8/36

• The time-multiplexing paradigm consists of projecting a series of light patterns so that every encoded point is identified with the sequence of intensities that receives

• The most common structure of the patterns is a sequence of stripes increasing its length by the time single-axis encoding

• Advantages:

– high resolution a lot of 3D points

– High accuracy (order of m)

– Robustness against colorful objects since binary patterns can be used

• Drawbacks:

– Static objects only

– Large number of patterns

Pattern 1

Pattern 2

Pattern 3

Projected over time

Example: 3 binary-encoded patterns which allows the

measuring surface to be divided in 8 sub-

regions

10

Formació

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Time-multiplexing: binary codes (I)

9/36

Posdamer et al.

Inokuchi et al.

Minou et al.

Trobina

Valkenburg and McIvor

Skocaj and Leonardis

Binary codes

Rocchini et al.

Static

Scene applicability Moving

Binary

Grey levels

Pixel depth

Colour

Periodical Coding strategy

Absolute

• Coding redundancy: every edge between adjacent stripes can be decoded by the sequence at its left or at its right

11

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Time-multiplexing: Binary codes (II)

10/36

Pattern 1

Pattern 2

Pattern 3

Projected over time

• Every encoded point is identified by the sequence of intensities that receives

• n patterns must be projected in order to encode 2n stripes

Example: 7 binary patterns proposed

by Posdamer & Altschuler

…

Codeword of this píxel: 1010010 identifies the corresponding pattern stripe

12

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Time-multiplexing: n-ary codes (I)

11/36

3 patterns based on a n-ary code of 4 grey levels (Horn & Kiryati) 64 encoded stripes

• n-ary codes reduce the number of patterns by increasing the number of projected intensities (grey levels/colours) increases the basis of the code

• The number of patterns, the number of grey levels or colours and the number of encoded stripes are strongly related fixing two of these parameters the reamaining one is obtained

Using a binary code, 6 patterns are

necessary necessary to encode

64 stripes

13

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Time-multiplexing: n-ary codes (II)

12/36

Caspi et al. √ √ √

Horn and Kiryati √ √ √ N-ary codes

Osawa et al. √ √ √

Static

Scene applicability Moving

Binary

Grey levels

Pixel depth

Colour

Periodical Coding strategy

Absolute

• n-ary codes reduce the number of patterns by increasing the number of projected intensities (grey levels/colours)

14

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Time-multiplexing: Gray code + Phase shifting (I)

13/36

Gühring’s line-shift technique

• A sequence of binary patterns (Gray encoded) are projected in order to divide the object in regions

• An additional periodical pattern is projected

• The periodical pattern is projected several times by shifting it in one direction in order to increase the resolution of the system similar to a laser scanner

Example: three binary patterns

divide the object in 8 regions

Without the binary patterns we would not be

able to distinguish

among all the projected slits

Every slit always falls in the same

region

15

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

14/36

Bergmann

Sansoni et al.

Wiora

Gray Code + Phase Shift

Gühring

Static

Scene applicability Moving

Binary

Grey levels

Pixel depth

Colour

Periodical Coding strategy

Absolute

• A periodical pattern is shifted and projected several times in order to increase the resolution of the measurements

• The Gray encoded patterns permit to differentiate among all the periods of the shifted pattern

Time-multiplexing: Gray code + Phase shifting (II)

16

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Formació

Time-multiplexing: hybrid methods (I)

15/36

• In order to decode an illuminated point it is necessary to observe not only the sequence of intensities received by such a point but also the intensities of few (normally 2) adjacent points

• The number of projected patterns reduces thanks to the spatial information that is taken into account

• The redundancy on the binary codification

is eliminated

Hall-Holt and Rusinkiewicz technique:

4 patterns with 111 binary stripes

Edges encoding: 4x2 bits (every

adjacent stripe is a bit)

1

1

1

1

1

0

0

0 Pattern 1

Pattern 2

Pattern 3

Pattern 4

Edge codeword: 10110101

17

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

16/36

Time-multiplexing: hybrid methods (II)

K. Sato

Hall-Holt and Rusinkiewicz Hybrid methods

Wang et al.

Static

Scene applicability Moving

Binary

Grey levels

Pixel depth

Colour

Periodical Coding strategy

Absolute

18

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Spatial Codification

17/36

• Spatial codification paradigm encodes a set of points with the information contained in a neighborhood (called window) around them

• The codification is condensed in a unique pattern instead of multiplexing it along time

• The size of the neighborhood (window size) is proportional to the number of encoded points and inversely proportional to the number of used colors

• The aim of these techniques is to obtain a one-shot measurement system moving objects can be measured

• Advantages:

― Moving objects supported

― Possibility to condense the codification to a unique pattern

• Drawbacks:

― Discontinuities on the object surface can produce erroneous window decoding (occlusions problem)

― The higher the number of used colours, the more difficult to correctly identify them when measuring non-neutral surfaces

― Maximum resolution cannot be reached

19

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Spatial codification: non-formal codification (I)

18/36

• The first group of techniques that appeared used codification schemes with no mathematical basis

• Drawbacks:

- the codification is not optimal and often produces ambiguities since different regions of the pattern are identical

- the structure of the pattern is too complex for a good image processing

Maruyama and Abe complex structure

based on slits containing randomly

placed cuts

Durdle et al. periodic pattern

20

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Spatial codification: non-formal codification (II)

19/36

Maruyama and Abe

Durdle et al.

Ito and Ishii

Boyer and Kak

Non-formal codification

Chen et al.

Static

Scene applicability Moving

Binary

Grey levels

Pixel depth

Colour

Periodical Coding strategy

Absolute

21

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

20/36

Spatial codification: De Bruijn sequences (I)

• A De Bruijn sequence (or pseudorrandom sequence) of order m over an alphabet of n symbols is a circular string of length nm that contains every substring of length m exactly once (in this case the windows are unidimensional).

1000010111101001

• The De Bruijn sequences are used to define coloured slit patterns (single axis codification) or grid patterns (double axis codification)

• In order to decode a certain slit it is only necessary to identify one of the windows in which it belongs to

Zhang et al.: 125 slits encoded with a De Bruijn sequence of 8 colors and

window size of 3 slits

Salvi et al.: grid of 2929 where a De Bruijn sequence of 3 colors and window

size of 3 slits is used to encode the vertical and horizontal slits

m=4 (window size)

n=2 (alphabet symbols)

22

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Spatial codification: De Bruijn sequences (II)

21/36

Hügli and Maître

Monks et al.

Vuylsteke and Oosterlinck

Salvi et al.

Lavoie et al.

De Bruijn sequences

Zhang et al.

Static

Scene applicability Moving

Binary

Grey levels

Pixel depth

Colour

Periodical Coding strategy

Absolute

23

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Spatial codification: M-arrays (I)

22/36

• An m-array is the bidimensional extension of a De Bruijn sequence. Every window of wh units appears only once. The window size is related with the size of the m-array and the number of symbols used

0 0 1 0 1 00 1 0 1 1 01 1 0 0 1 10 0 1 0 1 0

Morano et al. M-arry represented with an array of coloured dots

Example: binary m-array of size 46 and window size of 22

M-array proposed by Vuylsteke et al. Represented with shape primitives

Shape primitives used to represent

every symbol of the alphabet

24

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

23/36

Spatial codification: M-arrays (II)Morita et al.

Petriu et al.

Kiyasu et al.

Spoelder et al.

Griffin and Yee

Davies and Nixon

M-arrays

Morano et al.

Static

Scene applicability Moving

Binary

Grey levels

Pixel depth

Colour

Periodical Coding strategy

Absolute

25

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Direct Codification

24/36

• Every encoded pixel is identified by its own intensity/colour

• Since the codification is usually condensed in a unique pattern, the spectrum of intensities/colours used is very large

• Additional reference patterns must be projected in order to differentiate among all the projected intensities/colours:

• Ambient lighting (black pattern)• Full illuminated (white pattern)• …

• Advantages:– Reduced number of patterns– High resolution can be teorically achieved

• Drawbacks:– Very noisy in front of reflective properties of the objects, non-

linearities in the camera spectral response and projector spectrum non-standard light emitters are required in order to project single wave-lengths

– Low accuracy (order of 1 mm)

26

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

25/36

Direct codification: grey levels (I)

• Every encoded point of the pattern is identified by its intensity level

Carrihill and Hummel Intensity Ratio Sensor:

fade from balck to white

Requirements to obtain high resolution

Every slit is identified by its own intensity

• Every slit must be projected using a single wave-length

• Cameras with large depth-per-pixel (about 11 bits) must be used in order to differentiate all the projected intensities

27

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Direct codification: grey levels (II)

26/36

Carrihill and Hummel

Chazan and Kiryati Grey levels

Hung

Static

Scene applicability Moving

Binary

Grey levels

Pixel depth

Colour

Periodical Coding strategy

Absolute

28

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

Direct codification: Colour (I)

27/36

• Every encoded point of the pattern is identified by its colour

Tajima and Iwakawa rainbow

pattern

(the rainbow is generated with a

source of white light passing through a

crystal prism)

T. Sato patterns capable of cancelling the object colour by

projecting three shifted patterns

(it can be implemented with an LCD projector if few colours are projected drawback: the pattern

becomes periodic in order to maintain a good

resolution)

29

Conclusions

Sub-pixel matching

Experiments

Introduction

Classification

Time-multiplexing

Spatial codification

Direct codification

28/36

Direct codification: Colour (II)

Tajima and Iwakawa

Smutny and Pajdla

Geng

Wust and Capson

Colour

T. Sato

Static

Scene applicability Moving

Binary

Grey levels

Pixel depth

Colour

Periodical Coding strategy

Absolute

30

Conclusions

Sub-pixel matching

Classification

Introduction

Experiments

Quantitative results

Qualitative results

Experiments: quantitative results

29/36

TechniqueStDev(m

)3D

PointsResolutio

n (%)*Pattern

s

Time-multiplexin

g

Binary code

Posdamer (128 stripes)

37.6 13013 11.18 9

n-ary codeHorn (64 stripes)

9.6 12988 11.15 5

Phase Shift +

Gray code

Gühring (113 slits)

4.9 27214 23.38 14

Spatical codification

De Bruijn sequence

De Bruijn (64 slits)

13.1 13899 11.94 1

Salvi (2929 grid)

72.3 372 0.32 1

M-arrayMorano

(4545 array)23.6 926 0.80 1

Direct codification

ColourSato (64 stripes)

11.9 10204 8.77 3

Results obtained by reconstructing 30 times two flat panels separated by 40mm. The distance between both panels obtained for each technique was calculated for every reconstruction. The Standard Deviation is indicated.

(*) % of pixels inside a window of 515x226 of the camera image that have been triangulated

31

Conclusions

Sub-pixel matching

Introduction

Experiments

Quantitative results

Qualitative results

Experiments: qualitative results

30/36

Time-multiplexi

ng

Posdamer (128 stripes)

Horn (64 stripes) Gühring (113 slits)

Spatial codificati

on

De Bruijn (64 slits) Salvi (29x29 slits)Morano (45x45 dot

array)

Direct codificati

on

Sato (64 slits)

GühringDe Bruijn

Classification

32

Conclusions

Experiments

Introduction

Sub-pixel matching

Stripe patterns

Other patterns

Classification

Sub-pixel matching: a key point

31/36

• A key point in the accuracy of the 3D measurements is to locate the correspondences between camera image and projector pattern with sub-pixel coordinates

• Qualitative example:

Horse reconstruction with pixel-accuracy triangulation (20000

points)

Horse reconstruction with subpixel-accuracy

triangulation (10000 points)

Camera image

The reflected rays of light fall between

ajdacent pixels in the camera sensor

This must be taken into account when

triangulating the 3D points

Projected pattern

33

Conclusions

Experiments

Sub-pixel matching

Stripe patterns

Other patterns

Classification

Sub-pixel matching: stripe-patterns

32/36

• Only points belonging to the edges between adjacent stripes are decoded and reconstructed. Two possible strategies:

- Intersecting the stripe intensity profile with and adaptative binarization threshold (calculated from two additional images: full illuminated and ambient lighting)

- More accurate: Projecting positive and negative patterns and intersecting the stripe profiles

+

Introduction

34

Conclusions

Experiments

Introduction

Sub-pixel matching

Stripe patterns

Other patterns

Classification

Sub-pixel matching: other patterns

33/36

Arry of dots: the subpixel position of the dots is often calculated with their mass centre not very accurate since the circles are observed like deformed ellipses due to the change of perspective and the measuring surface

Slit patterns: every observed slit can be modelled with a gaussian profile and peak detectors can be applied (like Blais & Rioux detector) very accurate

35

Sub-pixel matching

Experiments

Introduction

Conclusions

Guidelines

Classification

Conclusions

34/36

Types of techniques Time-multiplexing

• Highest resolution• High accuracy• Easy implementation

• Inapplicability to moving objects• Large number of patterns

Spatial codification

• Can measure moving objects• A unique pattern is required

• Lower resolution than time-multiplexing• More complex decoding stage• Occlusions problem

Direct codification• High resolution• Few patterns

• Very sensitive to image noise cameras with large depth-per-pixel required• Sensitive to limited bandwith of LCD projectors special projector devices are usually required• Inapplicability to moving objects

36

Sub-pixel matching

Experiments

Introduction

Conclusions

Guidelines

Classification

Guidelines

35/36

Requirements Best technique

• High accuracy• Highest resolution• Static objects• No matter the number of patterns

Phase shift + Gray code Gühring’s line-shift technique

• High accuracy• High resolution• Static objects• Minimum number of patterns

N-ary pattern Horn & Kiryati Caspi et al.

• High accuracy• Good resolution• Moving objects

De Bruijn pattern Zhang et al.

Computer Vision and Robotics Group

Institut d’Informàtica i Aplicacions

Jordi PagèsJoaquim Salvi

END