projection-based augmented reality - eth z · projection-based augmented reality alexander cebulla...
TRANSCRIPT
![Page 1: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/1.jpg)
Projection-Based Augmented Reality
Alexander Cebulla [email protected]
16.04.2013 1 Distributed Systems Seminar
![Page 2: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/2.jpg)
The Bigger Picture Overall topic: “Interaction in Intelligent Environments“. “[…] information and communication technology (ICT) disappears as
it becomes embedded into physical objects and the spaces in which we live and work.”
(Intelligent Spaces: The Application of Pervasive ICT (2006))
16.04.2013 2 Distributed Systems Seminar
http://web.media.mit.edu/~raskar/UNC/Office/sketch.gif
![Page 3: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/3.jpg)
What is Augmented Reality? Augmentation of reality with virtual information Information is context-dependent
16.04.2013 3 Distributed Systems Seminar
![Page 4: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/4.jpg)
Optical vs. Video See-Through Displays
16.04.2013 4 Distributed Systems Seminar
![Page 5: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/5.jpg)
Overview
16.04.2013 5 Distributed Systems Seminar
(Spatial Augmented Reality Merging Real and Virtual Worlds Oliver Bimber, Ramesh Raskar)
![Page 6: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/6.jpg)
Indoor Navigation
Earlier Talk: “Handheld Augmented Reality”
16.04.2013 6 Distributed Systems Seminar
Touch-free interface Visualization
![Page 7: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/7.jpg)
Why Smart Phones?
Advantages of smart phones: Relatively cheap Common and available Has a range of sensors already built in: GPS, accelerometer, … Unlimited field of view through moving the smart phone around
16.04.2013 7 Distributed Systems Seminar
Hand-held video see-through displays
![Page 8: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/8.jpg)
Why Smart Phones are not Enough? However:
Small display: Limited field of view, if one does not want to move the phone
all the time User might need to hold / move the device over an extended
period of time Limited resolution:
With video mixing reality has the resolution of the camera.
16.04.2013 8 Distributed Systems Seminar
One solution: Spatial optical see-through display
![Page 9: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/9.jpg)
Spatial Optical See-Through Display
16.04.2013 9 Distributed Systems Seminar
One possible build: Combining a beam splitter with a projector or monitor.
Standard example: Virtual Showcase
Discussed in: The Virtual Showcase as a new Platform for Augmented Reality Digital Storytelling [Bimber2003]
![Page 10: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/10.jpg)
How is One Build?
16.04.2013 10 Distributed Systems Seminar
It is a spatial optical see-through display.
Two designs were discussed
(1) Convex assembly of four half-silvered mirror beam splitters
(2) Projection screen
a) Single CRT projector
b) Four CRT monitors
(3) Controllable light projector
(4) Infrared emitters
(5) Shutter glasses
(6) Electromagnetic tracking device
Virtual Showcase
(b)
![Page 11: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/11.jpg)
16.04.2013 11 Distributed Systems Seminar
Virtual Showcase
http://www.youtube.com/watch?v=qgN8T-iVRL8
![Page 12: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/12.jpg)
Advantages and Problems
16.04.2013 12 Distributed Systems Seminar
Advantages: Provides a high resolution
Even higher with the monitor-based approach. Increased control over environmental factors (lighting, sound, …)
inside and outside the Virtual Showcase. Example: Creation of realistic occlusion effects through pixel-
wise illumination of physical content. Easier eye accommodation (usually a problem with optical see-
through displays, which are distant from the augmented object) Furthermore: Larger screen size
Virtual Showcase
Problems: Limitations in multi-user support
currently only four users can be supported simultaneously Useless for mobile applications
“Retina” displays + HD camera
Video mixing
Price:
![Page 13: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/13.jpg)
Overview
16.04.2013 13 Distributed Systems Seminar
(Spatial Augmented Reality Merging Real and Virtual Worlds Oliver Bimber, Ramesh Raskar)
![Page 14: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/14.jpg)
What are the Main Advantages of Projected AR?
16.04.2013 14 Distributed Systems Seminar
A projector allows planar displays of various shapes. allows non-planar, curved displays.
Can directly project onto physical objects overcomes the restriction that display size dictates device size.
Unlimited field of view Can be transported easily compared to large screens.
![Page 15: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/15.jpg)
How to Make Use of These Advantages?
16.04.2013 15 Distributed Systems Seminar
Identified problems: Self-contained:
• Little to no dependence on infrastructure. Geometric-aware:
• Understanding of the geometry of display surface. • Orientation of projector
Self-configuring: • Ad-hoc integration into a cluster of projectors.
Possible solution: iLamps
Discussed in: iLamps: Geometrically Aware and Self-Configuring Projectors
Can be transported easily compared to large screens. Can directly project onto physical objects
![Page 16: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/16.jpg)
Creation of an Self-Contained Projector
16.04.2013 16 Distributed Systems Seminar
Add:
iLamps
Sensors Ability to communicate with objects and other devices Computing and an interface
![Page 17: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/17.jpg)
Geometric Awareness Through Structured Light
16.04.2013 17 Distributed Systems Seminar
Method: 1. A known light pattern (e.g. parallel stripes, random dots) is projected
onto the scene. 2. From the deformation of the pattern, depth and surface information
can be computed.
Ajmal Saeed Mian, Mohammed Bennamoun and Robyn Owens: 3D model-based free-form object recognition – a review
![Page 18: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/18.jpg)
How to Adapt to Unideal Display Shapes?
16.04.2013 18 Distributed Systems Seminar
Problem: Use of surfaces like room corners or columns. Regardless of the viewpoint, the distortion of the projected image
should be minimized.
iLamps
A possible solution: Measure distortion with conformality (angle preserving)
1. Use the camera to capture images of projected structured light. 2. Generate 3D mesh. 3. Compute a conformal map, which is used as texture map.
![Page 19: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/19.jpg)
How to Adapt to Unideal Display Shapes?
16.04.2013 19 Distributed Systems Seminar
iLamps
http://www.youtube.com/watch?v=7IdHTt6pCjY
![Page 20: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/20.jpg)
Problems
16.04.2013 20 Distributed Systems Seminar
Possible Solution: So far: Only one projector Cluster of projectors (iLamps)
Field of view is still limited Low light intensity
![Page 21: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/21.jpg)
Cluster of Projectors
16.04.2013 21 Distributed Systems Seminar
Does the new projector belong to group? ‘Ready to join’ msg. + light pattern
If seen by any other camera, then quick calibration step.
iLamps
Quick calibration Sequentially: Projection of structured pattern (checkerboard) By each unit (i.e. in parallel):
• Computation of the largest inscribed rectangle. • Checking how own projection fits into this rectangle. • Projection of corresponding image part.
![Page 22: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/22.jpg)
Cluster of Projectors
16.04.2013 22 Distributed Systems Seminar
iLamps
http://www.youtube.com/watch?v=NfB5qwD9z9s
![Page 23: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/23.jpg)
Problems
16.04.2013 23 Distributed Systems Seminar
Possible Solutions: So far: Only one projector 1. Cluster of projectors (iLamps)
Field of view is still limited Low light intensity
Remember: Hand-held displays can be moved around … 2. Steerable displays (Beamatron)
![Page 24: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/24.jpg)
Steerable Displays
16.04.2013 24 Distributed Systems Seminar
Discussed in: Steerable augmented reality with the beamatron
Problems: Geometric-aware:
• Project images ‘correctly’ on arbitrary surfaces • Stabilize projected graphics during movement.
Track the user position
The Beamatron:
![Page 25: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/25.jpg)
Stabilize Projected Graphics During Movement
16.04.2013 25 Distributed Systems Seminar
Solution: Build a circuit board which directly connects to the built-in pan and tilt
sensors of the platform.
Problem: The light platform does not provide any feedback on the pan and tilt
configuration. Real-time knowledge of the platform pose is critical for projecting
stabilized graphics.
The Beamatron
![Page 26: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/26.jpg)
Understanding the Geometry of the Display Surface
16.04.2013 26 Distributed Systems Seminar
Use the mounted Kinect sensor with Kinect Fusion to obtain smoothed depth images:
The Beamatron
![Page 27: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/27.jpg)
Project images ‘correctly’ on arbitrary surfaces
16.04.2013 27 Distributed Systems Seminar
Already have seen ‘Wall-paper’ an image onto a real object. Now: Create the illusion of a real 3D Object.
Requires two rendering passes: 1) Render the real objects along with virtual objects from the point of
view of the user. 2) Use the result as texture map for the rendering of the real
geometry.
The Beamatron
![Page 28: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/28.jpg)
Track the User Position
16.04.2013 28 Distributed Systems Seminar
Solution: Use the array microphone of the Kinect sensor to localize the user as
they speak. Also useful for recognizing commands.
Three Kinect sensors are mounted in the corners of the room. Two horizontally and one vertical. One sensor covers a range of around 100°
Problem: Hard to monitor events outside of the field of view of the Kinect sensor.
The Beamatron
![Page 29: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/29.jpg)
Application: Beamabuggy
16.04.2013 29 Distributed Systems Seminar
The Beamatron
http://www.youtube.com/watch?v=Z4bdrG8S1FM
![Page 30: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/30.jpg)
Overview
16.04.2013 30 Distributed Systems Seminar
(Spatial Augmented Reality Merging Real and Virtual Worlds Oliver Bimber, Ramesh Raskar)
![Page 31: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/31.jpg)
Going Mobile
16.04.2013 31 Distributed Systems Seminar
Until now:
Orientation of the projector
Shape of the real-object
Cluster: Position relative to other projectors
Geometric-awareness
Now:
Position of the projector relative to the scene
![Page 32: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/32.jpg)
Going Mobile
16.04.2013 32 Distributed Systems Seminar
Idea: Spatial awareness: System infers location and orientation of the device in 3D space.
Geometry awareness: System constructs the 3D structure of the world (User, Furniture, Walls, …) around it.
Two approaches: 1) Infrastructure-based: Uses sensors positioned in the environment. 2) Infrastructure-less: All sensors are attached to the projector.
Discussed in: Interactive Environment-Aware Handheld Projectors for Pervasive Computing Spaces
![Page 33: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/33.jpg)
Going Mobile
16.04.2013 33 Distributed Systems Seminar
iLamps
Use of markers allows: • computation of camera and hence projector pose relative to the
markers.
• identifying individual objects.
![Page 34: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/34.jpg)
Going Mobile
16.04.2013 34 Distributed Systems Seminar
iLamps
http://www.youtube.com/watch?v=yJZLIHBr1G0
![Page 35: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/35.jpg)
Infrastructure-Based: RoomProjector
16.04.2013 35 Distributed Systems Seminar
The Projector: Inertial Measurement Unit (IMU)
• Orientation of the device
IR Camera
In the room: Four Kinect cameras mounted at the ceiling at the mid-point of each wall.
Discussed in: Interactive Environment-Aware Handheld Projectors for Pervasive Computing Spaces
![Page 36: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/36.jpg)
Register the Environment
16.04.2013 36 Distributed Systems Seminar
When the room is empty: For each Kinect sensor the average of a number of depth map samples is computed.
From these averages a single fused point cloud is generated.
The point cloud is used to produce a background mesh.
This mesh is used for background subtraction.
RoomProjector
![Page 37: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/37.jpg)
Track the Projector Position
16.04.2013 37 Distributed Systems Seminar
Absolute position in the room: Cover the projector with retro-reflective tape:
Light pattern from the Kinect will be reflected and appears much brighter in the 2D Kinect IR image.
The position of the projector can then be triangulated.
RoomProjector
Orientation of the projector: Provided by the Inertial Measurement Unit
![Page 38: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/38.jpg)
Project Images ‘Correctly’ on Arbitrary Surfaces
16.04.2013 38 Distributed Systems Seminar
Mesh of the room + Absolute position in the room + Orientation Automatic correction of projected content.
RoomProjector
Furthermore: Flashlight-like metaphor: Can associate 2D image with any surface and use the projector to “reveal” it.
![Page 39: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/39.jpg)
Interaction
16.04.2013 39 Distributed Systems Seminar
RoomProjector
Use of a hot (IR) mirror: IR camera and projector are coaxial.
Can segment hand of user.
Interactions happen in the coordinate space of the camera.
http://www.youtube.com/watch?v=frGEzlrhve0
![Page 40: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/40.jpg)
Interaction
16.04.2013 40 Distributed Systems Seminar
RoomProjector
Combine data from IR camera and Kinect sensors:
http://www.youtube.com/watch?v=frGEzlrhve0
![Page 41: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/41.jpg)
Interaction: Objects in Mid-Air
16.04.2013 41 Distributed Systems Seminar
RoomProjector
Virtual image projected onto the surface
How to render virtual objects, which are in mid-air?
Project shadows
Use a mobile plane as viewport into the virtual world
![Page 42: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/42.jpg)
Interaction: Objects in Mid-Air
16.04.2013 42 Distributed Systems Seminar
RoomProjector
http://www.youtube.com/watch?v=frGEzlrhve0
![Page 43: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/43.jpg)
Advantages and Problems
16.04.2013 43 Distributed Systems Seminar
RoomProjector
Advantages: Small sensing window of the projector is overcome.
Problem: Coarseness Only prominent objects can be recovered from the scene.
Hybrid tracking of projector can be noisy and error prone:
camera occlusion
ferrous objects interfering with the IMU
![Page 44: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/44.jpg)
Infrastructure-Less: SLAMProjector
16.04.2013 44 Distributed Systems Seminar
KinectFusion uses its depth data to:
• Building a reconstruction of the environment in real-time
Geometry awareness
• Recover the pose of the projector
Spatial awareness
SLAM: Simultaneous Localization and Mapping Kinect sensor is directly mounted on the projector.
![Page 45: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/45.jpg)
Interaction
16.04.2013 45 Distributed Systems Seminar
SLAMProjector
Interaction similar to those of the RoomProjector are possible.
Furthermore, can move Kinect to object of interest:
• Higher resolution:
• Possible to interact with less prominent objects
Can segment and copy them
• No camera occlusion.
![Page 46: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/46.jpg)
Interaction
16.04.2013 46 Distributed Systems Seminar
SLAMProjector
http://www.youtube.com/watch?v=frGEzlrhve0
![Page 47: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/47.jpg)
Advantages and Problems
16.04.2013 47 Distributed Systems Seminar
RoomProjector
Advantages: High 3D sensing fidelity
Independent from infrastructure
Problem: Shadow interaction: User occludes a large part of the Kinect depth image. Can degrade the tracking quality of the projector.
Location is tracked from the built model, accumulated errors can lead to drift.
![Page 48: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/48.jpg)
Summary
16.04.2013 48 Distributed Systems Seminar
Spatial optical see-through display:
Virtual Showcase
Main advantages: Can directly project onto physical objects of various shapes Overcomes the restriction that display size dictates device size.
Unlimited field of view Can be transported easily compared to large screens.
![Page 49: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/49.jpg)
Summary
16.04.2013 49 Distributed Systems Seminar
How to Make Use of These Advantages? iLamps:
• Self-contained, Geometric-aware and Self-organizing Disadvantages:
Limited field of view Light intensity
Solutions:
Cluster of projectors
Steerable displays
• Beamatron
![Page 50: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/50.jpg)
Summary
16.04.2013 50 Distributed Systems Seminar
Going mobile:
Geometry AND spatial aware
Infrastructure-based: • iLamp: Markers
Only discrete number of surfaces
• RoomProjector: 4 Kinect sensors
Advantage: Extended sensing
Disadvantage: Coarse
Infrastructure-less: • SLAMProjector: Kinect sensor mounted on projector
Advantage: High 3D sensing fidelity
Disadvantage: Drift of camera position
Problem when displaying virtual objects in mid-air.
![Page 51: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/51.jpg)
Future
16.04.2013 51 Distributed Systems Seminar
Combination of smart phones with projectors:
![Page 52: Projection-Based Augmented Reality - ETH Z · Projection-Based Augmented Reality Alexander Cebulla acebulla@student.ethz.ch 16.04.2013 Distributed Systems Seminar 1](https://reader033.vdocuments.us/reader033/viewer/2022053021/604abe188ae4460e9b76495d/html5/thumbnails/52.jpg)
Future
16.04.2013 52 Distributed Systems Seminar
Augmenting a complete room:
http://www.youtube.com/watch?v=N_cKsOe7hLI