a projector based hand-held display system leung man chuen wong kin hong 1
TRANSCRIPT
![Page 1: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/1.jpg)
A Projector Based Hand-held Display System
LEUNG Man ChuenWONG Kin Hong
1
![Page 2: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/2.jpg)
Outline• Introduction• Previous work• System overview• System detail• Implementation and Experimental Results• Limitations and discussions• View dependent projection and application• Conclusion• Question and Answer Session
2
![Page 3: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/3.jpg)
Introduction
3
![Page 4: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/4.jpg)
Introduction• Aim of this project:– Build a movable hand-held display system • Uses an ordinary cardboard as the display• Uses a projector to project display content
• Desired result:
4
![Page 5: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/5.jpg)
Introduction
• Motivation:– Popular forms of graphical interfaces are usually
in fixed positions• Monitor• Projection onto a wall / screen
– Hand-held display device gives users greater freedom of control• Viewing angle, viewing distance
5
![Page 6: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/6.jpg)
Introduction• Motivation
– Screen of hand-held display devices are small• Mobile phone, portable DVD player, iPod…
– Increasing the size of the screen => heavy !– Electronic paper
• Technology not very mature yet• Not available at low cost
– Therefore, an alternative solution is needed!
6
![Page 7: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/7.jpg)
Introduction
• Our approach– Use a projector-camera pair to serve as the input
and output devices– Use a white cardboard as the movable display
surface• Can be light weight• Flexible screen size
7
![Page 8: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/8.jpg)
Introduction
• Contributions– Proposed a computer vision approach to enable the
projector to project display content precisely onto a movable ordinary cardboard in 3D space• Projector-camera calibration method to find the projection
matrix from a 3D point in camera coordinate to 2D projector image coordinate
• Real-time quadrangle detection and tracking algorithm to detect and track the display in real time
• Experimental results show that projection content can be registered onto the hand-held display precisely
• This work has been published in the Proc. Of CVPR2009
8
![Page 9: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/9.jpg)
• Contributions (con’t)– Proposed an application with view dependent
projection• “Hand-held 3D Model Viewer”• Use a head-mounted camera to track the viewing
position• Project views of a 3D model according to user view• Give user an perception of a real object sitting on the
display
9
![Page 10: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/10.jpg)
Previous work
10
![Page 11: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/11.jpg)
Previous work• Sensor based methods
– Dynamic Shader Lamp [Bandyopadhyay et al], magnetic sensor– Free Form Projection Display [Kondo et al], magnetic sensor– PaperWindows [Holman et al], infrared reflective marker– Foldable Interactive Display [Lee et al], IR emitter– Projector Based Tracking [Lee at al], light sensors
– Disadvantages: • Need to attached sensors or special markers onto the projection
surface• Accurate tracking systems are very expensive
– Magnetic location tracker– Vicon Motion Capturing System, for infrared reflective marker
11
![Page 12: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/12.jpg)
Previous work• Vision based methods– Portable Display Screen [Borkowski et al]
• Put the rectangular screen to a particular position for detection• Use Kalman filter to track the motion• Register the image to the portable screen using homographies:• A homography is a 3x3 matrix that match a projective plane to a
projective plane that maps straight lines to straight lines
• Disadvantages:– Hpc is calibrated on an initial plane, not updated at each time step– Mismatch when the screen move out of the calibration plane
12
Hps: projector-screen homographyHcs: camera-screen homographyHpc: projector-camera homography
![Page 13: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/13.jpg)
Previous work• Vision based methods (con’t)
– Active Pursuit Tracking [Gupta et al]• Calibrate Hpc on an initial plane• Put at least four color markers onto the projection screen• Detect these four color markers in camera image• Project four virtual markers onto the projection screen according to
the detected real markers and previous Hpc
• Update Hpc using the image points of the virtual markers
• Disadvantages:– Color markers are needed to be put on
the projection screen– Virtual markers are needed to be
projected onto the screen constantly– Make the projection screen unnecessarily large– Once lose tracking, need to start from the calibration plane again
13
![Page 14: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/14.jpg)
System overview
14
![Page 15: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/15.jpg)
System overview
15
Cardboard size,Camera parametersImage frame
Real-time quadrangle detection and tracking
Pre-warp display content
Projection
Offline calibration of projector-camera pair
Gp
Relative pose (R, T)
Pre-warped display content
![Page 16: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/16.jpg)
System detail
16
![Page 17: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/17.jpg)
Part 1: New projector-camera pair calibration
• What to calibrate?
17
1
]|[
100
0
0
43c
c
c
xvpvp
upup
Z
Y
X
TRdf
df
S
V
U Gp (3x4)
![Page 18: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/18.jpg)
Part 1: Projector-camera pair calibration
1c
c
c
p Z
Y
X
G
S
V
U
18
1. Create image points in known positions
2. Project onto a cardboard with known size
3. Capture image using camera
4. Calculate 3D positions of the corners
6. Calculate 3D positions of projected points
7. Collect a set of corresponding points 8. Solve Gp
![Page 19: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/19.jpg)
Part 2: Quadrangle detection and tracking
• Line feature extraction– Line detector based on Hough transform in
OpenCV• Open source Computer Vision library
– We detect line segments
19
HoughTransformin OpenCV
Input: Image Edge map Line feature map
![Page 20: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/20.jpg)
Part 2: Quadrangle detection and tracking• Quadrangle detection– Select long line segments– Check for every 4 segments– Find the formed quadrangle– Check criteria• Based on property of a quadrangle, angles, ratio of
side lines…
20
![Page 21: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/21.jpg)
Part 2: Quadrangle detection and tracking• Quadrangle tracking– Using particle filter• An state estimation algorithm
– Two main components• State dynamic model• Observation model
– To avoid degeneracy• Re-sampling
– Output• The weighted sum of samples
21
Initial samples setState update
Updated samples
Evaluation function
Weighted samples
Re-sampling
New sample set
![Page 22: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/22.jpg)
Part 2: Quadrangle detection and tracking
• Quadrangle tracking (cont’)– Tracking target: relative pose(R, T) of the
camera and the cardboard• Calculated from detection result
– State at time k:
– State dynamic model• Random walk [Pupilli] based on uniform density
– Observation• Line segment feature map of current frame
22
zyxzyxk tttrrrq
rx : rotation along x-axisry : rotation along y-axisrz : rotation along z-axistx : translation along x-axisty : translation along y-axistz : translation along z-axis
),()|( 111 eqeqUqqp kkkk
p : density function qk-1 : state at time k-1qk : state at time kU : uniform density functione : uncertainty about movement
![Page 23: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/23.jpg)
Part 2: Quadrangle detection and tracking
• Evaluation method– Re-projection
23
![Page 24: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/24.jpg)
Part 2: Quadrangle detection and tracking
• Evaluation method– closest line matching and criteria checking
24
![Page 25: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/25.jpg)
Part 2: Quadrangle detection and tracking
• Evaluation method– Particle content replacement
25
![Page 26: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/26.jpg)
Part 2: Quadrangle detection and tracking
• Evaluation method– Weight update
• According to quadrangle detection criteria
• Re-sampling
• Handling tracking failure– Determination of tracking failure
• Use distribution particles• If variance is greater than a
certain threshold– Lost tracking !– Perform detection again
26
![Page 27: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/27.jpg)
Part 3: Pre-warping and projection
• From tracking result– Get 3D corner positions
• From calibration result– Get projector image points
• Warp display content• Project onto the plane
27
Projector imageImage to be displayed
T T
T
![Page 28: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/28.jpg)
Implementation and experimental results
28
![Page 29: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/29.jpg)
System setup
• Projector camera pair– Projector: 1280x1024– Camera: 320x240
• Testing platform– Dual core processor at 2.16GHz– 1GB RAM
• White cardboard– 351mm x 300mm
29
![Page 30: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/30.jpg)
Calibration of projector-camera pair
• Project one point each time• 32 images are collected• Mark the points manually• Sample images
30
![Page 31: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/31.jpg)
Calibration of projector-camera pair
• Calibration error:– About 4.6pixels
31
pixel
pixel
O : projection points+ : re-projection points
![Page 32: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/32.jpg)
Detection and tracking experiments
32
• High tracking precision
• Robust to partial occlusions
• Robust to dense clutter
• Works well when both camera and cardboard are moving
![Page 33: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/33.jpg)
Projection results
• Display content can be projected precisely onto the cardboard• Match to the cardboard in different poses
33
![Page 34: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/34.jpg)
Processing time• Processing time for line feature extraction– ~16ms
• Processing time of tracking algorithm– Depends on number of line segments used– Depends on number particles
34
![Page 35: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/35.jpg)
Processing time
• Processing time of tracking algorithm– About 30ms/frame for 80 particles and 20 line
segments• About 30fps
• Processing time for image warping– Negligible
• Total processing time– ~ 46 ms– ~20 fps
35
![Page 36: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/36.jpg)
Projection latency• Estimation method:– Move the cardboard from left to right– Stop sharply, wait until the projection image match the
edge
– Repeat for several times– Use an external video camera to capture the process
• Count the number of frames between stopping the motion and the projection image match the edge of the cardboard
• Average about 5 frames, 167 ms 36
![Page 37: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/37.jpg)
View Dependent Projection and Application
37
![Page 38: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/38.jpg)
View dependent projection • Project according to view of the user
• Head pose tracking– Mount a magnetic location tracker on the head of the user
• [Cruz-Neira et al], [Kondo et al]• Expensive• Tracked pose is relative to the receiver, not to the cardboard directly
– Mount a fixed camera in front of the user• Track the eye positions of the user• Track point model head pieces• Not very accurate and easily be occluded
38
![Page 39: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/39.jpg)
View dependent projection• Our approach
– Use a head mounted camera– Apply the proposed quadrangle detection and tracking algorithm– Advantage: accurate and get the relative pose directly
![Page 40: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/40.jpg)
View dependent projection• Application: Hand-held 3D Model Viewer
– Project different views of a 3D model according to head poses– User can change the view by moving the display or his/her head– Projection is distortion compensated– Give the user an illusion of a real object sitting on the display
• Motivation– Traditional way of interacting with a virtual model:
• Mouse and keyboard– Natural way of interacting with a real object:
• Hold the object in the hands and interact with it directly– To simulate such natural way of interacting with objects
![Page 41: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/41.jpg)
Design detail• Create virtual scene
– Implemented using OpenGL– An virtual object sitting on an virtual
cardboard
• Create user view image– Set a virtual camera – Apply tracked head pose– Generate virtual camera image– Equivalent to desired user view
![Page 42: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/42.jpg)
Design detail• Generate projector image
![Page 43: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/43.jpg)
Experimental results
• Use another webcam to be the head-mounted camera
• Move the cardboard and the camera to different relative poses
• Capture the view of this webcam• Result video:
![Page 44: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/44.jpg)
Discussion
44
![Page 45: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/45.jpg)
Discussion• Limitation on projection resolution– Down sampled in the image warping process– Possible solutions :
• Warp physically , need a 6dof optical redirect system• Use a projector with higher resolution
• Limitation on depth of field– About 30 cm in our prototype– Possible solution:
• Use multiple projectors
45
![Page 46: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/46.jpg)
Discussion• Handling projected light– Projected light may affect tracking stability– Solved using property of camera:
• Set the contrast and gain of light to high value• Projected light on cardboard is brighter - > saturated
• Hand-held 3D Model Viewer– Limitation on speed of motion
• Fast motion will break the interaction experience because of projection latency
• Acceptable error:– ~4cm in translation, ~5 degrees in rotation
• Limited motion:– ~24cm / second, ~30 degrees / second
46
![Page 47: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/47.jpg)
Discussions– Limitation on viewing angle
![Page 48: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/48.jpg)
Discussions• Possible extensions
– Use multiple cardboards– Project onto a cube instead of a cardboard
![Page 49: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/49.jpg)
Conclusion
49
![Page 50: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/50.jpg)
Conclusion• Proposed a projector-based hand-held display system• Proposed a novel computer vision approach to guide the projector
to project onto a hand-held display in the 3D space– Calibration method to find the projection matrix from 3D camera
coordinate system to 2D projector image coordinate system– Robust quadrangle detection and tracking algorithm– Precise and can run in real time
• Implementation of the whole system– Project onto the cardboard correctly in real-time
• An interactive application with view dependent projection is proposed– “Hand-held 3D Model Viewer”– Use head-mounted camera to obtain user view to the display– Give 3D perception to user
50
![Page 51: A Projector Based Hand-held Display System LEUNG Man Chuen WONG Kin Hong 1](https://reader034.vdocuments.us/reader034/viewer/2022051618/56649ca35503460f9496405e/html5/thumbnails/51.jpg)
Question and Answer Session
Thanks
51