![Page 1: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/1.jpg)
Moveable Interactive Projected Displays Using Projector Based Tracking
1Carnegie Mellon University2Mitsubishi Electric Research Labs3Georgia Tech University
Seattle, WA UIST 2005
Johnny C. Lee1,2
Scott E. Hudson1
Jay W. Summet3
Paul H. Dietz2
![Page 2: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/2.jpg)
UIST 2004 – Automatic Projector Calibration
1. Embedded light sensor in surface.
2. Project patterns to find sensor locations.
3. Pre-warp source image to fit surface.
(video clip 1)
Correspondence between location data between and projected image is free (e.g. no need for calibration with external tracking system)
Transforms passive surfaces into active displays in a practical manner.
Variety of useful applications
![Page 3: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/3.jpg)
Touch Calibration
Everywhere Displays, IBM
Shader Lamps, MERL/UNC
Diamond Touch, MERL
Interactive Whiteboard
Projector-based AR
![Page 4: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/4.jpg)
Focus on Moveable Projected Displays
Goals of this work:1. Achieve interactive tracking rates for hand-held surfaces.2. Reduce the perceptibility of the location discovery patterns.3. Explore interaction techniques supported by this approach.
![Page 5: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/5.jpg)
Display Surface
Constructed from foam core and paper
Touch-sensitivity is provided by a resistive film
Lighter than a legal pad
![Page 6: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/6.jpg)
Tablet PC-like Interaction
Video clip 2
![Page 7: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/7.jpg)
Talk Outline
• Reducing Perceptibility
• Achieving Interactive Rates
• Pattern Size and Shape
• Tracking Loss
• Interaction Techniques/Demos
![Page 8: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/8.jpg)
Talk Outline
• Reducing Perceptibility
• Achieving Interactive Rates
• Pattern Size and Shape
• Tracking Loss
• Interaction Techniques/Demos
![Page 9: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/9.jpg)
Gray Code Patterns
Black and White
Difference is visible to the human eye
![Page 10: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/10.jpg)
Gray Code Patterns
Black and White
Difference is visible to the human eye
![Page 11: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/11.jpg)
Gray Code Patterns
Black and White
Difference is visible to the human eye
![Page 12: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/12.jpg)
Gray Code Patterns
Black and White
Difference is visible to the human eye
![Page 13: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/13.jpg)
Gray Code Patterns
Black and White
Difference is visible to the human eye
![Page 14: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/14.jpg)
Gray Code Patterns
Black and White Frequency Shift Keyed (FSK)
HF LF
Difference is visible to the human eye
Difference is NOT visible to the human eye
![Page 15: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/15.jpg)
Gray Code Patterns
Black and White Frequency Shift Keyed (FSK)
HF LF HF
Difference is visible to the human eye
Difference is NOT visible to the human eye
![Page 16: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/16.jpg)
HF HFLF HF LF
Gray Code Patterns
Black and White Frequency Shift Keyed (FSK)
Difference is visible to the human eye
Difference is NOT visible to the human eye
![Page 17: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/17.jpg)
FSK Transmission of PatternsFSK transmission of the Gray Code patterns makes the
stripped region boundaries invisible to the human eye.Patterns appear to be solid grey squares to observers.Light sensor is able to demodulate the HF and LF
regions into 0’s and 1’sThis is accomplished using a modified DLP projector
![Page 18: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/18.jpg)
Inside a DLP projector
DLP = Digital Light Processing • Many consumer projectors currently use DLP technology• “DLP” is Texas Instruments marketing term for DMD
DMD = Digital Micro-mirror Device
• Each mirror corresponds to a pixel• Brightness corresponds to duty cycle of mirror
Pictures from Texas Instruments literature
![Page 19: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/19.jpg)
Inside a DLP projector
Light source
Color wheel
DMD
Projector Lens
![Page 20: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/20.jpg)
Inside our modified DLP projector
DMD
Projector Lens
Light source
![Page 21: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/21.jpg)
FSK Transmission of Location Patterns
• Removing the color wheel flattens the color space of the projector into a monochrome scale
• Multiple points in the former color space now have the same apparent intensity to a human observer, but are manifested by differing signals.
• The patterns formerly known as “red” and “grey” are rendered as 180Hz and 360Hz signals respectively.
• Monochrome projector is not ideal, but is a proof of concept device until we can build a custom DMD projector.
![Page 22: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/22.jpg)
Talk Outline
• Reducing Perceptibility
• Achieving Interactive Rates
• Pattern Size and Shape
• Tracking Loss
• Interaction Techniques/Demos
![Page 23: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/23.jpg)
Projector Specifications
Infocus X1 (~$800 new)– 800x600 SVGA resolution– 1 DMD chip– 60Hz refresh rate
Full-Screen Location Discovery Time:– 20 images ( log2(# pixels) )
– 333ms at 60Hz
3Hz maximum update rate
![Page 24: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/24.jpg)
Incremental TrackingProject small tracking patterns over the last
known locations of each sensor for incremental offsets
Black masks reduce visibility of tracking patterns
Tracking loss strategies are needed (later)
Smaller area = fewer patterns = faster updates– 32x32 unit grid requiring 10 images– 6Hz update rate
![Page 25: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/25.jpg)
Tracking Demo
Video clip 3
![Page 26: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/26.jpg)
Latency and Interleaving Incremental tracking is a tight feedback loop:
project update project update …
6Hz update rate assumes 100% utilization of the 60 frames/sec the projector can display
System latencies negatively impact channel utilization
Achieving 100% utilization of the projection channel requires taking advantage of the axis-independence of Gray Code patterns.
![Page 27: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/27.jpg)
System Latency – Full X-Y Tracking
X patterns Y patternsProjection:
Software:draw
X-Y patternsupdate& draw
X-Y patterns
X patterns Y patterns
Time
Only 73% utilization
Hardware &OS scheduling
Graphics& Video
![Page 28: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/28.jpg)
System Latency - Interleaved Tracking
X patterns Y patternsProjection:
Software:draw
X patternsupdate& draw
X patterns
Time
drawY patterns
X patterns Y patterns
update& draw
Y patterns
update& draw
X patterns
100% utilization of the projection channel and 12Hz interleaved update
![Page 29: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/29.jpg)
Talk Outline
• Reducing Perceptibility
• Achieving Interactive Rates
• Pattern Size and Shape
• Tracking Loss
• Interaction Techniques/Demos
![Page 30: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/30.jpg)
Tracking Pattern Size
Tracking Area Tracking Rate
32x32 grid 12Hz interleaved
16x16 grid 15Hz interleaved
-75% area +25% rate
Smaller tracking area increases risk of losing sensor (e.g. maximum supported velocity)
log2 relationship makes it hard to gain speed though the use of smaller patterns
![Page 31: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/31.jpg)
Tracking Pattern Size
Pixel DensityDecreases
![Page 32: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/32.jpg)
Tracking Pattern Size
large, coarsetracking pattern small, fine
tracking pattern
Preserves physical size of tracking pattern (cm)Preserves maximum supported velocity (m/s)
Distance is approximated from screen sizeScaling factor is adjustable (precision vs. max velocity): ~2.5mm; 25cm/s
![Page 33: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/33.jpg)
Motion Modeling
v
a
Predicting the motion can be used to increase the range of supported movement (e.g. max acceleration vs. max velocity)
Much of the work in motion modeling is applicable. But, no model is perfect and mis-predictions can lead to tracking loss potentially yielding poorer overall performance.
Models are likely to be application and implementation specific.
![Page 34: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/34.jpg)
Tracking Pattern Shape
We used square tracking patterns due to the axis aligned nature of Gray code patterns.
Patterns with high-radial symmetry are best for general movement in two-dimensions.
Pattern geometry can be optimized for specific applications.
![Page 35: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/35.jpg)
Talk Outline
• Reducing Perceptibility
• Achieving Interactive Rates
• Pattern Size and Shape
• Tracking Loss
• Interaction Techniques/Demos
![Page 36: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/36.jpg)
Detecting Occlusions or Tracking Loss
Causes of tracking loss:
1. occlusions
2. exiting the projection area
3. exceeding the range of motion supported by the tracking patterns
With FSK transmission, tracking loss corresponds to a disappearance of the carrier signal. This allows error detection on a per-bit basis.
Implemented on a low-cost PIC processor as:
1. sudden drop in signal amplitude
2. insufficient amplitude
3. invalid edge count
![Page 37: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/37.jpg)
Lost Tracking Behavior
Single/independent sensors:1. Discard and hope the sensor has not moved2. Perform a full screen discovery process (+333ms)3. Grow the tracking pattern around last location until
reacquired
Multiple sensors of known geometric relationship:1. Try the above three techniques. 2. Compute predicted lost sensor locations using the
locations of the remaining available sensors.
![Page 38: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/38.jpg)
Tracking Loss With Multiple Sensors
video clip 4
![Page 39: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/39.jpg)
Estimating Lost Sensors
Available Sensors
Action
4/4 No estimation needed, compute the 4 point warping homography directly
3/4Measure 6 offsets, compute affine transform to estimate lost sensor from last known location
2/4 Measure 4 offsets, compute simplified transform to estimate lost sensor locations
1/4 Measure 2 offsets, compute 2D translation
0/4 Try full screen discovery
Note: Transformations for each point cannot be implemented as a simple matrix stack because LIFO ordering of sensor loss and re-acquisition is not guaranteed.
![Page 40: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/40.jpg)
Talk Outline
• Reducing Perceptibility
• Achieving Interactive Rates
• Pattern Size and Shape
• Tracking Loss
• Interaction Techniques/Demos
![Page 41: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/41.jpg)
Supported Interaction Techniques
Video clip 5
![Page 42: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/42.jpg)
Supported Interaction Techniques
Magic Lens Focus + Context
Location Aware Displays Input Pucks
Simulated Tablet PC
![Page 43: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/43.jpg)
Conclusion
Unifying the tracking and projection technology greatly simplifies the implementation and execution of applications that combine motion tracking with projected imagery.– Coherence between the location data and projected image is
free.– Does not require an external tracking system or calibration– Simple: Demos were created in about a week
This approach has the potential to change the economics of interactive displays– The marginal cost of each display can be as low as $10 USD– Museum: wireless displays could be handed out to visitors.– Medical Clinic: physical organization of patient charts/folders
![Page 44: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/44.jpg)
Future Work Removing the color wheel was a proof-of-concept work
around. Construct high-speed projector using a DLP development
kit Explore using infrared to project invisible patterns Explore other applications where low-speed positioning is
sufficient. Achieve +18Hz (+36Hz interleaved) tracking with visible
patterns and an unmodified DLP projector using RGB sections.
Using multiple projectors (or steerable projectors) to increase freedom of movement.
![Page 45: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/45.jpg)
Acknowledgements
Johnny Chung Lee
Funded in part by the National Science Foundation under grants IIS-0121560 and IIS-0325351
Funded in part by Mitsubishi Electric Research Labs
![Page 46: Moveable Interactive Projected Displays Using Projector Based Tracking 1 Carnegie Mellon University 2 Mitsubishi Electric Research Labs 3 Georgia Tech](https://reader035.vdocuments.us/reader035/viewer/2022062618/55148632550346ea6e8b4c2a/html5/thumbnails/46.jpg)
Technical Details Infocus X1 ($800) 800x600, 60HzPIC16F819 at 20Mhz, 10bit ADCSensor package < $10 in volume4-wire resistive touch sensitive film IF-D92 fiber optic phototransistors45Bytes/sec for location data25mW during active tracking Latency (77ms – 185ms)