A Keystone-free Hand-held Mobile Projection System
Li ZhaorongAnd
KH WongReference: Zhaorong Li, Kin-Hong Wong, Yibo Gong, and Ming-Yuen Chang, “An Effective Method for Movable Projector Keystone Correction”, IEEE Transactions on Multimedia, VOL. 13, NO. 1, Feb 2011.
keystone correction 1
Outline
• Introduction• Methodology
– Pro-cam pair calibration– Projection region detection and tracking– Automatic keystone correction
• Experimental results• Conclusion
keystone correction 2
Introduction
keystone correction 3
Aim of this work
• A mobile projector keystone correction method– project keystone free content on a general flat
surface– without adding markings or boundaries on the
surface
keystone correction 4
Aim of this work
• Configuration
keystone correction 5
Aim of this work
• Desired Results
Keystoned projection Corrected projection
keystone correction 6
Motivation
• Mobile projector becomes popular– Flexible projector-screen position– More freedom of display control such as viewing angle,
distance etc– General flat surface can be used as screen– Physical size and cost shrinks quickly– Emergence of mobile devices with embedded projector– Others……
keystone correction 7
Motivation
• Limitation: keystone distortion !– When projecting onto a screen at oblique
position, the projection region becomes a trapezoid instead of a rectangle
– Traditional static projection system also face this problem
– Stringent for mobile projection system
keystone correction 8
Motivation
• Existing keystone correction method– All for static projector system, not suitable for
mobile projection, eg.• Su [1]: requires bounded screen• Li et al[2]: requires bounded screen • Raskar [3]: high computation, not real-time• One-time correction, not continuous
keystone correction 9
Motivation
• Special need for mobile keystone correction– Screen independence
• no specially designed or position-fixed screen should be required
– Continuous processing in real time• Continuous correction instead of one-time correction is
expected
keystone correction 10
Our approach
• Propose an effective method that can continuously correct the keystone distortion on a general markless screen
• Only additional device used is a webcam attached with the projector
keystone correction 11
Our approach
• We add a green frame to the projector screen and project it to the display screen
• Track the green frame use the camera and then automatically correct the keystone correction
keystone correction 12
Contributions
• An artfully designed particle filter tracker for tracking projection region
• An efficient and accurate recovery algorithm for recovering 3d projection region
• A continuous and markless mobile projector keystone correction system
keystone correction 13
Methodology
keystone correction 14
Overview
• Three major modules– Projector-camera pair calibration– Projection region detection and tracking– Automatic keystone correction
3D projectionregion recovery
Projector-camera pair calibration
G, K
Corrected projection
Keystoned projection
Pre-warped image3D rectangle
Keystone correctionProjection region tracking
Camera frame
Flow chart
keystone correction 15
Part 1: Projector-camera pair calibration
• Projector-camera relationship– Projection matrix from 3D camera coordinate system
to projector image plane
• Benefits– Independent from projector movement– No need to estimate explicit parameters– Easy to estimate
keystone correction 16
Projector-camera pair calibration
• Projective model
p c x GXpxcX
G
: 2D projector image pixel
: 3D point in camera frame
: 3x4 projection matrix
: scale factor
keystone correction 17
Projector-camera pair calibration
• Estimation method– Project cross to an ordinary cardboard with
known size– Collect a number of projector-camera
corresponding crosses– Calculate the 3D coordinate of the cross– Estimate G matrix using SVD
keystone correction 18
Part 2: Projection region detection and tracking
• Add a green frame to the projection region• Detect the frame in the initial frame• Track the frame in the subsequent frames
keystone correction 19
Detection• Based on the Canny edge map and Hough line segments• Find a quadrangle satisfying following criteria
– each side of the quadrangle longer than a threshold– opposite sides should have similar lengths– each angle within the range from 30 to 150 degree– the overlapping ratio of the line segments to the four sides of the
formed quadrangle bigger than a threshold– the quadrangle approximately located around the center of the
camera image
keystone correction 20
Tracking
• The relationship between projector screen and the projection region in camera
p c x Hx
Hpxcx
: the homography matrix
: the projector image pixel
: the camera image pixel
keystone correction 21
Tracking
• The homography matrixT
1[ ]d
tn
H J R K
JRtn
Kd
: intrinsic parameter matrix of projector
: rotation matrix of the projector w.r.t camera
: translation vector of the projector w.r.t camera
: normal of the screen
: distance of the screen from the camera: intrinsic parameter matrix of camera
H is dominated by n and d since
J,R,t,K are all fixed
keystone correction 22
Tracking
• Tracking state vector
• Particle filter tracking– State dynamic model– Observation model– Initialization
T[ , ]ds n
keystone correction 23
Tracking
• State dynamic model
1 1 1( | ) ( , )k k k kp Uniform s s s e s e
1,k ks s
e
: the state of k-1 and k frame
: the uncertainty of the movement of the projector
keystone correction 24
Tracking
• Observation model– Re-project each particle to the camera
T1[ ]
d
JtnH JR K
3 3 3 1[ , ] [ , ] JR Jt G G
c 1 p x H x
keystone correction 25
Tracking
• Observation model– Compute likelihood by comparing the re-projected
quadrangle with the edge map• Specifically, check how many edge points on each of four sides• The likelihood of each side is the percentage of the edge points
among all side points• The likelihood of the particle is the sum of likelihoods of four sides
keystone correction 26
Tracking
• Initialization– The detected quadrangle in the detection stage is
used to initialize the particle filter– First, recover the 3D position of the quadrangle
[using the method in Part 3]– Second, compute its normal and distance from the
camera
keystone correction 27
Part 3: Automatic keystone correction
• Three steps– Recover 3D projection region– Look for inscribed rectangle– Pre-warp projection image
keystone correction 28
Recover 3D projection region
• Solve following equation set for each corner of the projection region using SVD
c c x KXp c x GX
: the corner of the projection region in camera
: the 3D corner of the projection region
: the corner of the projector screen
cxpxcX
keystone correction 29
Recover 3D projection region
• Incorporate co-planarity constraint
2 2 2 2T T c T T c4 41 i 2 i 1 i 2 iT c T c T c T c
1 13 i 3 i 3 i 3 i
2c c c c c c1 4 1 2 1 3
i i i iu v
c c
i i
k X k X g X g X
k X k X g X g X
X X X X X X
000000000000000000000000000000000000000000
T T T1 2 3, ,k k kT T T1 2 3, ,g g g
K
G: three rows of
: three rows of
: weight
Use the SVD solution as the initial value
keystone correction 30
Look for inscribed rectangle
• Simple geometric intersection
keystone correction 31
Pre-warp projection image
• Re-project inscribed rectangle into the camera image
• Find homography between the projection and projector screen
• Pre-warp the display image using the homography
keystone correction 32
Pre-warp projection image
Pre-warped projection image Display result
keystone correction 33
Experimental results
keystone correction 34
Devices
• PC with 2.16GHz CPU, 1 GB memory• Optoma mobile projector 1280x1024• Logitech Quickcam Pro 4000 webcam 320x240
keystone correction 35
Projector camera pair calibration
• Error distribution
Back-projection error corresponding to 80% inliers is 4.2 pixels
keystone correction 36
Projection region tracking
• Mean and std tracking error w.r.t different noise levels on simulation data
keystone correction 37
Projection region tracking
• Mean and std tracking error on real data are 3.4 and 3.6 pixels
Trajectory of the real projector movement
keystone correction 38
Keystone correction
• Some real correction results
keystone correction 39
Keystone correction
• The error of keystone correction against different poses of the projector
keystone correction 40
Keystone correction
• Comparison of our keystone correction module with the static projector keystone correction
keystone correction 41
Speed
• 16 fps on our platform• The per-frame processing time is about 60 ms• The pre-warping step occupies most of the
time (about 36 ms)
keystone correction 42
Conclusion
• Proposed an effective mobile keystone correction method
• Mobility and Markinglessness are the most distinguishing features
• Limitation: need to project a green frame• Future work: Use invisible IR LED to eliminate the
green frame
keystone correction 43
Reference• [1] R. Sukthankar, R. Stockton, and M. Mullin, “Smarter presentations:
Exploiting homography in camera-projector systems,” in Intl. Conf. on Computer Vision, 2001.
• [2] B. Li and I. Sezan, “Automatic keystone correction for smart projectors with embedded camera,” in Intl. Conf. on Image Processing, 2004
• [3] R. Raskar and P. Beardsley, “A self-correcting projector,” in Intl. Conf. on Computer Vision and Pattern Recognition, 2001
• [4] Zhaorong Li, Kin-Hong Wong, Yibo Gong, and Ming-Yuen Chang, “An Effective Method for Movable Projector Keystone Correction”, IEEE Transactions on Multimedia, VOL. 13, NO. 1, Feb 2011.
keystone correction 44
Thank you
keystone correction 45