optical imaging techniques and applications

2011 AAPM Annual Meeting, Vancouver, 7/31/2011

Optical Imaging

Techniques and Applications

Jason Geng, Ph.D.

Vice President

IEEE Intelligent Transportation Systems Society

[email protected]

mailto:[email protected]


Outline

• Structured light 3D surface imaging concept

• Classification framework of structured light 3D surface

imaging techniques

• Temporal/multi-shot projection techniques

• Sequential Projection

• Spatial/single-shot projection techniques

• Continuously varying pattern projection

• Strip indexing

• Grid indexing

• Hybrid

• Application examples

• Conclusions


Structured Light 3D Surface Imaging

Structured

Light

ProjectorCamera

3D Object in the Scene

B

P

R

qa

Triangulation

Structured Light Projection

R = BSin(q )

Sin(a + q)

B and a are known,

How to get the value of q?

Encode q information into

projection pattern design,

such that the q value

corresponding to each pixel

in the acquired image can be derived and a full frame of 3D surface image can be obtained.


3D surface imaging results: point cloud {Pi=(xi, yi, zi, fi), i=1,2, …, N},

where fi represents the value at the ith surface point in the data set.

Structured Light 3D Surface Imaging

SPIE 2011 Photonics West, San Francisco, CA ,01/26/2011


Structured Light 3D Surface Imaging Techniques

Sequential Projections (Multi-Shots)

Gray Code

Stripe Indexing (Single Shot)

Color Coded Stripes

Grid Indexing (Single Shot)

Binary Code

Phase Shift

Hybrid: Gray code + Phase Shift

Segmented Stripes

Pseudo Random Binary-Dots

Mini-Patterns as Codewords

Continuous Varying Pattern (Single Shot)

Rainbow 3D Camera

Continuously Varying Color Code

Gray Scale Coded Stripes

De Bruijn Sequence

Color Coded Grid

2D Color Coded Dot Array

Hybrid Methods

Str

uctu

red

Lig

ht

Pro

jec

tio

n (

SL

P)

Cla

ssif

ica

tio

n F

ram

ew

ork


Sequential Projections - Binary Patterns

12345

LSB MSB

Horizontal Spatial Distribution

Sequence of

Projection

N patterns -> 2N stripes

•Ishii, I., Yamamoto, K., Doi, K. and Tsuji, T., “High-speed 3D image acquisition using coded structured light projection.” In: IEEE/RSJ International Conference on Intelligent

Robots and Systems (IROS), pp. 925-930.(2007)

•K. Sato and S. Inokuchi, “Range-imaging system utilizing nematic liquid crystal mask,” Proc. Int. Conf. on Computer vision, pp. 657-661 (1987).

•R. J. Valkenburg and A. M. McIvor, “Accurate 3d measurement using a structured light system.” Image and Vision Computing, 16(2), pp. 99–110, (1998).

•J. L. Posdamer and M. D. Altschuler, “Surface measurement by space-encoded projected beam systems.” Computer Graphics &Image Processing, 18(1), pp. 1, 1982

00

00

0

10110

01011


projection pattern design, such

that the q value corresponding

to each pixel in the acquired image can be derived.

R = BSin(q )

Sin(a + q)

Triangulation Pros: Very robust to surface

texture and environmental

illumination

Cons: Slow, unable to acquire 3D image of moving object


Sequential Projections - Gray Coding

N patterns -> MN stripes

•J. L. Posdamer and M. D. Altschuler, “Surface measurement by space-encoded projected beam systems.” Computer Graphics d Image Processing, 18(1), pp. 1, 1982.

•S. Inokuchi, K. Sato and F. Matsuda, “Range-imaging for 3-D object recognition,” Proc. Int. Conf. on Pattern Recognition, pp. 806-808 (1984).

•D. Caspi, N. Kiryati, and J. Shamir, “Range imaging with adaptive color structured light. Pattern Analysis and Machine Intelligence”, 20(5), pp.470–480, (1998).

•W. Krattenthaler, K. Mayer, and H. Duwe, “3D-surface measurement with coded light approach,” In Proceedings O¨ esterr. Arbeitsgem. MustererKennung, volume 12, pp.

103–114, (1993).

Pros: use M distinct levels of intensity (instead of only two in the binary code), to

produce unique coding, reduced number of required projection patterns, faster

N patterns -> MN stripes (v.s 2N stripes in binary)

Cons: Unable to acquire 3D image of moving object

M = 4


Sequential Projections - Phase Shift

x

x

x

x

6

4

2

x

6

4

2

B

Pattern Projector

Reference Plane

Image Sensor

P

Z

L

d

•E. Horn, N. Kiryati, “Toward optimal structured light patterns,” Image and Vision Computing volume 17 (2), pp. 87–97, (1999).

A set of sinusoidal patterns are projected. The variation form of these projected fringe

patterns are similar, except the phase of the fringe patterns are shifted a constant

angle with respect to each other. The minimum number of projections is three. . .

Pros: fairly robust, efficient algorithms, sub-pixel

accuracy

Cons: Phase ambiguity, need multiple projections, unable to acquire 3D image of fast moving object


Sequential Projections - Phase Shift + Gray Coding

Eliminate “ambiguity”

•P. S. Huang and S. Zhang, “A Fast Three-Step Phase Shifting Algorithm,” Appl. Opt., (2006).

•Zhang, S., Yau, S.T. “High-resolution, real-time 3d absolute coordinate measurement based on a phase-shifting method,” Opt. Eng 14, pp. 2644–2649, (2006).

Pros: Accurate, very robust to surface texture and environmental illumination

Cons: Slow, unable to acquire 3D image of moving object


Sequential Projections - Photometric Stereo

R. Woodham, “Photometric method for determining surface orientation from multiple images,” Optical Engineering, 19:1, pp. 134-140, (1980).

Shape from Shading (SfS) by using multiple

images

All light sources be “point light”

Only estimates the local surface orientation

(gradients p,q).

Assumes continuities of 3D surface

needs a “starting point” (a point on object

surface whose (x,y,z) coordinate is known)

Photometric stereo

• a variant approach to Shape from Shading (SfS).

• estimates local surface orientation by using several

images of the same surface taken from the same

viewpoint but under illumination from different directions.

• solves the ill-posed problems in SfS by using multiple

images.

Camera

Surface

Patch

Normal

q1L1(x1,y1,z1)

L2(x2,y2,z2) L3(x3,y3,z3)

L4(x4,y4,z

4)

s(x,y,z)

Intensity

Images

I1, I2, I3, and I4.


Single Shot- Rainbow 3D Camera

Z. J. Geng, “Rainbow three-dimensional camera: new concept of high-speed three-dimensional vision systems”, Opt. Eng. 35(2), pp. 376-383 (1996)

Elegant encoding scheme

High speed 3D imaging

“Infinite” resolution

Single shot – simple

implementation

Low-cost


projection pattern design, such

that the q value corresponding

to each pixel in the acquired image can be derived.

R = BSin(q )

Sin(a + q)

Triangulation


Single Shot- Spatially Varying Color Coding

Red Channel Intensity Variation Pattern

Green Channel Intensity Variation Pattern

Blue Channel Intensity Variation Pattern

Composite Three Color Saw-Tooth Pattern

Z. J. Geng, “Rainbow three-dimensional camera: new concept of high-speed three-dimensional vision systems”, Opt. Eng. 35(2), pp. 376-383 (1996)

Multiple cycles of variation

Improved sensitivity and accuracy



- Stripe Indexing Using Color

Structured

Light

ProjectorCamera

3D Object in the Scene

K. L. Boyer, A. C. Kak, “Color-encoded structured light for rapid active ranging,” IEEE trans. Pattern Anal.Mach. Intell. 9(1), pp.14-28 (1987).

Stripe Indexing

pattern variation is in

horizontal direction

Stripe Indexing using color

a set of distinct colors

Projection angles can be

derived from one-to-one

color-angle correspondence



- Segment Pattern with Random Cuts

M. Maruyama and S. Abe, “Range Sensing by Projecting Multiple Slits with Random Cuts,” IEEE Transaction Patten Analysis a d Machine Intelligence 15(6), pp.647-651

(1993).

Stripe Indexing using Segment

Pattern with Random Cuts

Each stripe has unique

segment cut pattern


derived from one-to-one cut

pattern-angle correspondence

Pros: Simple one shot scheme

Cons: Require continuous

surface, ambiguity occurs when

surface texture “disturbs” the projected pattern



- Repeated Grey Scale Pattern

N. G. Durdle, J. Thayyoor, V. J. Raso, “An improved structured light technique for surface reconstruction of the human trunk,” IEEE Canadian Conference on Electrical

and Computer Engineering, Vol. 2, pp. 874–877 (1998).

Stripe Indexing using Repeated

Gray Scale Pattern

Each stripe has unique grey

scale level


derived from one-to-one grey

scale level -angle

correspondence

Pros: Simple, one-shot scheme

Cons: ambiguity occurs when

surface texture “disturbs” the projected pattern



- Stripe Indexing Based on De Bruijn Sequence

F. J. MacWilliams, N. J. A. Sloane, “Pseudorandom sequences and arrays,” Proceedings of the IEEE 64 (12), pp. 1715–1729 ((1976).

H. Fredricksen, “A survey of full length nonlinear shift register cycle algorithms,” Society of Industrial and Applied Mathematics Review, 24 (2), pp. 195–221 (1982).

H. Hügli, G. Maïtre, “Generation and use of color pseudo random sequences for coding structured light in active ranging,” Proc Industrial Inspection, Vol.1010, pp.75 (1989)

T. Monks, J. Carter, “Improved stripe matching for colour encoded structured light,” 5th Int Conference on Computer Analysis of Images and Patterns, pp. 476–485 (1993).

L. Zhang, B. Curless, S. M. Seitz, “Rapid shape acquisition using color structured light and multi-pass dynamic programming,” Int. Symp 3D Data Processing Visualization

Transmission, Padova, Italy (2002).

R

G

B

Stripe Indexing using pseudo-random color sequence

any sub-sequence is not correlated to any other in the De Bruijn sequence

This unique feature is used in constructing stripe pattern sequence that has unique

local variation patterns without repeating themselves.

Uniqueness simplifies the pattern decoding task.

n = 3, k = 2

As we travel around the

cycle, we encounter each

of the 23 = 8 three-digit

patterns 000, 001, 010,

011, 100, 101, 110, 111

exactly once.

De Bruijn sequence of rank n on an alphabet of size k is a cyclic word in which each

of the kn words of length n appears exactly once as we travel around the cycle.

n=3, k=5


2D Spatial Grid Patterns (Single Shot)

- Pseudo Random Binary Array (PRBA)

J. Le Moigne and A.M. Waxman, Structured Light Patterns for Robot Mobility, IEEE Journal of Robotics and Automation 4(5), 541-548 (1988)

2D Grid Indexing

to uniquely label every sub-window in the

projected 2D pattern, such that the pattern in

any sub-window is unique and fully identifiable

with respect to its 2D position in the pattern

pattern variation is in both horizontal and vertical

directions

Grid Indexing using PRBA

to produce grid locations that can be marked by

dots (or other patterns), such that the coded

pattern of any sub-window in unique.

A PRBA is defined by a n1 by n2 array encoded using a pseudo-random sequence,

such that any k1 by k2 sub-window sliding over the entire array is unique and fully defines the sub-window’s absolute coordinate (i,j) within the array.


2D Spatial Grid Patterns (Single Shot)

- Color Coded Grids

Petriu, E.M., Sakr, Z., Spoelder, H.J.W. and Moica, A., “Object recognition using pseudo-random color encoded structured light.” IEEE Instrumentation and

Measurement. v3. 1237-1241 (2000).

J. Pagès, J. Salvi and C. Matabosch. “Robust segmentation and decoding of a grid pattern for structured light.” 1st Iberian Conf Pattern Recognition/ Image

Analysis, IbPRIA pp 689-696, (2003)

Grid Indexing using colors

to color-code both vertical and

horizontal stripes.

Encoding schemes in two directions

can be same of different

There is no guarantee on the

uniqueness of sub-windows,

colored stripes in both directions can

help the decoding in most situations to

establish the correspondence.

The thin grid lines may not be as

reliable in pattern extraction as other

patterns (dots, squares, etc).


2D Spatial Patterns (Single Shot)

- Mini-Patterns as Codewords

P. M. Gri¦n, L. S. Narasimhan and S. R. Yee, Generation of uniquely encoded light patterns for range data acquisition, Pattern Recognition 25(6), 609-616 (1992).

1 2 3

• Instead of using a pseudo random

binary array, a multi-valued pseudo

random array can be used.

• One can correspond each value with

a mini-pattern as special codeword,

thus forming a grid indexed projection

pattern.

• An example of a three-valued pseudo random array and a set of mini-patterns

codewords.

•Using the specially defined codewords, a multi-valued pseudo random array can

be converted into a projection pattern with unique sub-windows



- 2D Array of Color-Coded Dots

Payeur, P. and Desjardins, D., “Structured Light Stereoscopic Imaging with Dynamic Pseudo-random Patterns.” 6th Int Conf Image Analysis and Recognition (2009).

Desjardins, D. and Payeur, P. 2007. “Dense Stereo Range Sensing with Marching Pseudo-Random Patterns.” Proceedings of the Fourth Canadian Conference on

Computer and Robot Vision (2007).

• 6 x 6 array with sub-window size of 3 x 3 using three codeword (R,G,B).

• Fill the upper left 3x3 corner with a randomly chosen pattern.

• add a 3-element column on the right with random codeword. The uniqueness of the sub-

window is verified before adding such a column.

• Keep adding columns until all are filled with random codeword and uniqueness are verified.

• Similarly, add random rows in the downward direction from the initial sub-window position.

• Afterwards, add new random codeword along the diagonal direction. Repeat these

procedures until all dots are filled with colors.

• Alternative methods of generating

pseudo random array.

• Brute force algorithm to generate an

array that preserve the uniqueness of

sub-windows,

• May not exhaust all possible sub-

window patterns.

• Intuitive to algorithm implementation.



- Hybrid Methods

There are many opportunities to improve specific aspect(s) of 3D

surface imaging system performance by combining more than one

encoding schemes.

Here is just one example


Sequential Projections (Multi-Shots)

Continuously Varying Pattern (Single Shot)


Grid Indexing (Single Shot)

Hybrid Methods

Summary of Typical SLP patterns


Performance Evaluation

of 3D Surface Imaging Systems

Accuracy

Speed

Resolution

Primary Performance Space of 3D Imaging Systems

Primary performance indexes

Field of view (FOV)

Depth of field (DOF)

Stand-off distance

Cost

etc


Thank You !

optical imaging techniques and applications

Documents