optical imaging techniques and applicationsa prba is defined by a n1 by n2 array encoded using a...

24
2011 AAPM Annual Meeting, Vancouver, 7/31/2011 Optical Imaging Techniques and Applications Jason Geng, Ph.D. Vice President IEEE Intelligent Transportation Systems Society [email protected]

Upload: others

Post on 26-Jan-2021

1 views

Category:

Documents


0 download

TRANSCRIPT

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Optical Imaging

    Techniques and Applications

    Jason Geng, Ph.D.

    Vice President

    IEEE Intelligent Transportation Systems Society

    [email protected]

    mailto:[email protected]

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Outline

    • Structured light 3D surface imaging concept

    • Classification framework of structured light 3D surface

    imaging techniques

    • Temporal/multi-shot projection techniques

    • Sequential Projection

    • Spatial/single-shot projection techniques

    • Continuously varying pattern projection

    • Strip indexing

    • Grid indexing

    • Hybrid

    • Application examples

    • Conclusions

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Structured Light 3D Surface Imaging

    Structured

    Light

    ProjectorCamera

    3D Object in the Scene

    B

    P

    R

    qa

    Triangulation

    Structured Light Projection

    R = BSin(q )

    Sin(a + q)

    B and a are known,

    How to get the value of q?

    Encode q information into

    projection pattern design,

    such that the q value

    corresponding to each pixel

    in the acquired image can be derived and a full frame of 3D surface image can be obtained.

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    3D surface imaging results: point cloud {Pi=(xi, yi, zi, fi), i=1,2, …, N},

    where fi represents the value at the ith surface point in the data set.

    Structured Light 3D Surface Imaging

    SPIE 2011 Photonics West, San Francisco, CA ,01/26/2011

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Structured Light 3D Surface Imaging Techniques

    Sequential Projections (Multi-Shots)

    Gray Code

    Stripe Indexing (Single Shot)

    Color Coded Stripes

    Grid Indexing (Single Shot)

    Binary Code

    Phase Shift

    Hybrid: Gray code + Phase Shift

    Segmented Stripes

    Pseudo Random Binary-Dots

    Mini-Patterns as Codewords

    Continuous Varying Pattern (Single Shot)

    Rainbow 3D Camera

    Continuously Varying Color Code

    Gray Scale Coded Stripes

    De Bruijn Sequence

    Color Coded Grid

    2D Color Coded Dot Array

    Hybrid Methods

    Str

    uctu

    red

    Lig

    ht

    Pro

    jec

    tio

    n (

    SL

    P)

    Cla

    ssif

    ica

    tio

    n F

    ram

    ew

    ork

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Sequential Projections - Binary Patterns

    12345

    LSB MSB

    Horizontal Spatial Distribution

    Sequence of

    Projection

    N patterns -> 2N stripes

    •Ishii, I., Yamamoto, K., Doi, K. and Tsuji, T., “High-speed 3D image acquisition using coded structured light projection.” In: IEEE/RSJ International Conference on Intelligent

    Robots and Systems (IROS), pp. 925-930.(2007)

    •K. Sato and S. Inokuchi, “Range-imaging system utilizing nematic liquid crystal mask,” Proc. Int. Conf. on Computer vision, pp. 657-661 (1987).

    •R. J. Valkenburg and A. M. McIvor, “Accurate 3d measurement using a structured light system.” Image and Vision Computing, 16(2), pp. 99–110, (1998).

    •J. L. Posdamer and M. D. Altschuler, “Surface measurement by space-encoded projected beam systems.” Computer Graphics &Image Processing, 18(1), pp. 1, 1982

    00

    00

    0

    10110

    01011

    Encode q information into

    projection pattern design, such

    that the q value corresponding

    to each pixel in the acquired image can be derived.

    R = BSin(q )

    Sin(a + q)

    Triangulation Pros: Very robust to surface

    texture and environmental

    illumination

    Cons: Slow, unable to acquire 3D image of moving object

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Sequential Projections - Gray Coding

    N patterns -> MN stripes

    •J. L. Posdamer and M. D. Altschuler, “Surface measurement by space-encoded projected beam systems.” Computer Graphics d Image Processing, 18(1), pp. 1, 1982.

    •S. Inokuchi, K. Sato and F. Matsuda, “Range-imaging for 3-D object recognition,” Proc. Int. Conf. on Pattern Recognition, pp. 806-808 (1984).

    •D. Caspi, N. Kiryati, and J. Shamir, “Range imaging with adaptive color structured light. Pattern Analysis and Machine Intelligence”, 20(5), pp.470–480, (1998).

    •W. Krattenthaler, K. Mayer, and H. Duwe, “3D-surface measurement with coded light approach,” In Proceedings O¨ esterr. Arbeitsgem. MustererKennung, volume 12, pp.

    103–114, (1993).

    Pros: use M distinct levels of intensity (instead of only two in the binary code), to

    produce unique coding, reduced number of required projection patterns, faster

    N patterns -> MN stripes (v.s 2N stripes in binary)

    Cons: Unable to acquire 3D image of moving object

    M = 4

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Sequential Projections - Phase Shift

    x

    x

    x

    x

    6

    4

    2

    x

    6

    4

    2

    B

    Pattern Projector

    Reference Plane

    Image Sensor

    P

    Z

    L

    d

    •E. Horn, N. Kiryati, “Toward optimal structured light patterns,” Image and Vision Computing volume 17 (2), pp. 87–97, (1999).

    A set of sinusoidal patterns are projected. The variation form of these projected fringe

    patterns are similar, except the phase of the fringe patterns are shifted a constant

    angle with respect to each other. The minimum number of projections is three. . .

    Pros: fairly robust, efficient algorithms, sub-pixel

    accuracy

    Cons: Phase ambiguity, need multiple projections, unable to acquire 3D image of fast moving object

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Sequential Projections - Phase Shift + Gray Coding

    Eliminate “ambiguity”

    •P. S. Huang and S. Zhang, “A Fast Three-Step Phase Shifting Algorithm,” Appl. Opt., (2006).

    •Zhang, S., Yau, S.T. “High-resolution, real-time 3d absolute coordinate measurement based on a phase-shifting method,” Opt. Eng 14, pp. 2644–2649, (2006).

    Pros: Accurate, very robust to surface texture and environmental illumination

    Cons: Slow, unable to acquire 3D image of moving object

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Sequential Projections - Photometric Stereo

    R. Woodham, “Photometric method for determining surface orientation from multiple images,” Optical Engineering, 19:1, pp. 134-140, (1980).

    Shape from Shading (SfS) by using multiple

    images

    All light sources be “point light”

    Only estimates the local surface orientation

    (gradients p,q).

    Assumes continuities of 3D surface

    needs a “starting point” (a point on object

    surface whose (x,y,z) coordinate is known)

    Photometric stereo

    • a variant approach to Shape from Shading (SfS).

    • estimates local surface orientation by using several

    images of the same surface taken from the same

    viewpoint but under illumination from different directions.

    • solves the ill-posed problems in SfS by using multiple

    images.

    Camera

    Surface

    Patch

    Normal

    q1L1(x1,y1,z1)

    L2(x2,y2,z2) L3(x3,y3,z3)

    L4(x4,y4,z

    4)

    s(x,y,z)

    Intensity

    Images

    I1, I2, I3, and I4.

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Single Shot- Rainbow 3D Camera

    Z. J. Geng, “Rainbow three-dimensional camera: new concept of high-speed three-dimensional vision systems”, Opt. Eng. 35(2), pp. 376-383 (1996)

    Elegant encoding scheme

    High speed 3D imaging

    “Infinite” resolution

    Single shot – simple

    implementation

    Low-cost

    Encode q information into

    projection pattern design, such

    that the q value corresponding

    to each pixel in the acquired image can be derived.

    R = BSin(q )

    Sin(a + q)

    Triangulation

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Single Shot- Spatially Varying Color Coding

    Red Channel Intensity Variation Pattern

    Green Channel Intensity Variation Pattern

    Blue Channel Intensity Variation Pattern

    Composite Three Color Saw-Tooth Pattern

    Z. J. Geng, “Rainbow three-dimensional camera: new concept of high-speed three-dimensional vision systems”, Opt. Eng. 35(2), pp. 376-383 (1996)

    Multiple cycles of variation

    Improved sensitivity and accuracy

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Stripe Indexing (Single Shot)- Stripe Indexing Using Color

    Structured

    Light

    ProjectorCamera

    3D Object in the Scene

    K. L. Boyer, A. C. Kak, “Color-encoded structured light for rapid active ranging,” IEEE trans. Pattern Anal.Mach. Intell. 9(1), pp.14-28 (1987).

    Stripe Indexing

    pattern variation is in

    horizontal direction

    Stripe Indexing using color

    a set of distinct colors

    Projection angles can be

    derived from one-to-one

    color-angle correspondence

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Stripe Indexing (Single Shot)- Segment Pattern with Random Cuts

    M. Maruyama and S. Abe, “Range Sensing by Projecting Multiple Slits with Random Cuts,” IEEE Transaction Patten Analysis a d Machine Intelligence 15(6), pp.647-651

    (1993).

    Stripe Indexing using Segment

    Pattern with Random Cuts

    Each stripe has unique

    segment cut pattern

    Projection angles can be

    derived from one-to-one cut

    pattern-angle correspondence

    Pros: Simple one shot scheme

    Cons: Require continuous

    surface, ambiguity occurs when

    surface texture “disturbs” the projected pattern

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Stripe Indexing (Single Shot)- Repeated Grey Scale Pattern

    N. G. Durdle, J. Thayyoor, V. J. Raso, “An improved structured light technique for surface reconstruction of the human trunk,” IEEE Canadian Conference on Electrical

    and Computer Engineering, Vol. 2, pp. 874–877 (1998).

    Stripe Indexing using Repeated

    Gray Scale Pattern

    Each stripe has unique grey

    scale level

    Projection angles can be

    derived from one-to-one grey

    scale level -angle

    correspondence

    Pros: Simple, one-shot scheme

    Cons: ambiguity occurs when

    surface texture “disturbs” the projected pattern

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Stripe Indexing (Single Shot)- Stripe Indexing Based on De Bruijn Sequence

    F. J. MacWilliams, N. J. A. Sloane, “Pseudorandom sequences and arrays,” Proceedings of the IEEE 64 (12), pp. 1715–1729 ((1976).

    H. Fredricksen, “A survey of full length nonlinear shift register cycle algorithms,” Society of Industrial and Applied Mathematics Review, 24 (2), pp. 195–221 (1982).

    H. Hügli, G. Maïtre, “Generation and use of color pseudo random sequences for coding structured light in active ranging,” Proc Industrial Inspection, Vol.1010, pp.75 (1989)

    T. Monks, J. Carter, “Improved stripe matching for colour encoded structured light,” 5th Int Conference on Computer Analysis of Images and Patterns, pp. 476–485 (1993).

    L. Zhang, B. Curless, S. M. Seitz, “Rapid shape acquisition using color structured light and multi-pass dynamic programming,” Int. Symp 3D Data Processing Visualization

    Transmission, Padova, Italy (2002).

    R

    G

    B

    Stripe Indexing using pseudo-random color sequence

    any sub-sequence is not correlated to any other in the De Bruijn sequence

    This unique feature is used in constructing stripe pattern sequence that has unique

    local variation patterns without repeating themselves.

    Uniqueness simplifies the pattern decoding task.

    n = 3, k = 2

    As we travel around the

    cycle, we encounter each

    of the 23 = 8 three-digit

    patterns 000, 001, 010,

    011, 100, 101, 110, 111

    exactly once.

    De Bruijn sequence of rank n on an alphabet of size k is a cyclic word in which each

    of the kn words of length n appears exactly once as we travel around the cycle.

    n=3, k=5

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    2D Spatial Grid Patterns (Single Shot)- Pseudo Random Binary Array (PRBA)

    J. Le Moigne and A.M. Waxman, Structured Light Patterns for Robot Mobility, IEEE Journal of Robotics and Automation 4(5), 541-548 (1988)

    2D Grid Indexing

    to uniquely label every sub-window in the

    projected 2D pattern, such that the pattern in

    any sub-window is unique and fully identifiable

    with respect to its 2D position in the pattern

    pattern variation is in both horizontal and vertical

    directions

    Grid Indexing using PRBA

    to produce grid locations that can be marked by

    dots (or other patterns), such that the coded

    pattern of any sub-window in unique.

    A PRBA is defined by a n1 by n2 array encoded using a pseudo-random sequence,

    such that any k1 by k2 sub-window sliding over the entire array is unique and fully defines the sub-window’s absolute coordinate (i,j) within the array.

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    2D Spatial Grid Patterns (Single Shot)- Color Coded Grids

    Petriu, E.M., Sakr, Z., Spoelder, H.J.W. and Moica, A., “Object recognition using pseudo-random color encoded structured light.” IEEE Instrumentation and

    Measurement. v3. 1237-1241 (2000).

    J. Pagès, J. Salvi and C. Matabosch. “Robust segmentation and decoding of a grid pattern for structured light.” 1st Iberian Conf Pattern Recognition/ Image

    Analysis, IbPRIA pp 689-696, (2003)

    Grid Indexing using colors

    to color-code both vertical and

    horizontal stripes.

    Encoding schemes in two directions

    can be same of different

    There is no guarantee on the

    uniqueness of sub-windows,

    colored stripes in both directions can

    help the decoding in most situations to

    establish the correspondence.

    The thin grid lines may not be as

    reliable in pattern extraction as other

    patterns (dots, squares, etc).

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    2D Spatial Patterns (Single Shot)- Mini-Patterns as Codewords

    P. M. Gri¦n, L. S. Narasimhan and S. R. Yee, Generation of uniquely encoded light patterns for range data acquisition, Pattern Recognition 25(6), 609-616 (1992).

    1 2 3

    • Instead of using a pseudo random

    binary array, a multi-valued pseudo

    random array can be used.

    • One can correspond each value with

    a mini-pattern as special codeword,

    thus forming a grid indexed projection

    pattern.

    • An example of a three-valued pseudo random array and a set of mini-patterns

    codewords.

    •Using the specially defined codewords, a multi-valued pseudo random array can

    be converted into a projection pattern with unique sub-windows

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    2D Spatial Patterns (Single Shot)- 2D Array of Color-Coded Dots

    Payeur, P. and Desjardins, D., “Structured Light Stereoscopic Imaging with Dynamic Pseudo-random Patterns.” 6th Int Conf Image Analysis and Recognition (2009).

    Desjardins, D. and Payeur, P. 2007. “Dense Stereo Range Sensing with Marching Pseudo-Random Patterns.” Proceedings of the Fourth Canadian Conference on

    Computer and Robot Vision (2007).

    • 6 x 6 array with sub-window size of 3 x 3 using three codeword (R,G,B).

    • Fill the upper left 3x3 corner with a randomly chosen pattern.

    • add a 3-element column on the right with random codeword. The uniqueness of the sub-

    window is verified before adding such a column.

    • Keep adding columns until all are filled with random codeword and uniqueness are verified.

    • Similarly, add random rows in the downward direction from the initial sub-window position.

    • Afterwards, add new random codeword along the diagonal direction. Repeat these

    procedures until all dots are filled with colors.

    • Alternative methods of generating

    pseudo random array.

    • Brute force algorithm to generate an

    array that preserve the uniqueness of

    sub-windows,

    • May not exhaust all possible sub-

    window patterns.

    • Intuitive to algorithm implementation.

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    2D Spatial Patterns (Single Shot)- Hybrid Methods

    There are many opportunities to improve specific aspect(s) of 3D

    surface imaging system performance by combining more than one

    encoding schemes.

    Here is just one example

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Sequential Projections (Multi-Shots)

    Continuously Varying Pattern (Single Shot)

    Stripe Indexing (Single Shot)

    Grid Indexing (Single Shot)

    Hybrid Methods

    Summary of Typical SLP patterns

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Performance Evaluation

    of 3D Surface Imaging Systems

    Accuracy

    Speed

    Resolution

    Primary Performance Space of 3D Imaging Systems

    Primary performance indexes

    Field of view (FOV)

    Depth of field (DOF)

    Stand-off distance

    Cost

    etc

  • 2011 AAPM Annual Meeting, Vancouver, 7/31/2011

    Thank You !