hybrid user interfaces for augmented reality
DESCRIPTION
Presentation by Dr. Alex Olwal (KTH) for the Disruptive Code conference, Stockholm 2010.TRANSCRIPT
Hybrid User Interfaces for Augmented Reality
Alex Olwal, Ph.D. olwal.com
Human-Computer Interaction, KTH
Alex Olwal, Ph.D.
2003-2010 KTH Stockholm
2006 Microsoft Research Redmond
2005 University of California Santa Barbara
2002-2003 Columbia University New York
startup Rule Communication Stockholm www.rule.se
startup Harmonic Solutions Stockholm www.harmonic.se
Display Systems
Sensing
Interaction Techniques
WIKITUDE
Layar
Augmented Reality (AR) on mobile phones
X: The Man with the X-Ray Eyes (1963)
RoboCop (1987)
Terminator 3 (2003)
Augmented Reality (AR)
• Empower our senses → enhance perception
• Supplement real
• Seamless
AR Fuse virtual objects with real environment
• View
• Perspective
• Real-time
Head-worn
Ivan Sutherland 1968 5dt HMD Nomad ProView XL40/50 STm AddVisor 150
Head-worn
Handheld, Situated & Hybrid
Everywhere Displays [Pinhanez et al. 2001], iLamp Projectors [Raskar et al. 2003],On-board Mixed Reality Projector [Karitsuka & Sato 2003], Transflective Surfaces [Bimber et al. 2000], Cell phone [Möhring & Bimber 2004], NaviCam [Rekimoto 1997], PDA [Wagner & Schmalstieg 2003]
Ubiquitous computingWeiser 1993
Ubiquitous
• Displays
• Sensing
• Connectivity
Devices
• Tabs, pads & boards
Interactive surfaces & tangible UIs
• Display 2D graphics
• Sense on & above surface
• Detect & augment objects
DigitalDesk [ Wellner 1993 ]
Tangible Bits [ Ullmer & Ishii & Buxton 1997 ]
Augmented Surfaces [ Rekimoto & Saitoh 1999 ]
PlayAnywhere / PlayTogether [ Wilson 2005 / Wilson & Robbins 2006 ]
• Tracked display
• Context-sensitive
• Focus + context
Chameleon [ Fitzmaurice 1993 ]NaviCam [ Rekimoto 1995 ]Augmented Notebook [ Mackay et al. 2002 ]Focus + Context displays [ Baudisch et al. 2002 ]VITA [ Benko et al. 2004 ]Ubiquitous graphics [ Sanneblad & Holmquist 2006 ]
Spatially aware displays
14
NaviCamRekimoto 1995
Hybrid UIs for AR
1 Unobtrusive
2 Sensing in environment
AR, UbiComp, Mobile, Interactive Surfaces, …
Shopping window Microsoft Surface
SurfaceFusionSensing & augmenting everyday objects
GI 2008 [Olwal & Wilson]Proc. Graphics Interface
LightSenseDynamic augmentations of printed media
ISMAR 2006 [Olwal]Proc. IEEE & ACM International Symposium on Mixed & Augmented Reality
Ericsson Demo CenterStockholm 2008
LUMARHybrid 2D + 3D handheld AR
ICAT 2007 [Olwal & Henrysson]Proc. International Conference on Artifical Reality & Teleexistence
Judge’s Choice Award in Nokia “Calling All Innovators” Challenge 2009Mobile World Congress, Barcelona, 13-16 February, 2009
LUMARHybrid 2D + 3D handheld AR
Printed media
Floor plan
2D graphics
Photographs
3D graphics
3D model
Spatially aware handheldsEnhancing interaction with large displays
TEI 2009 [Olwal & Feiner]Proc. International Conference on Tangible & Embedded Interaction
INTERACT 2009 [Olwal]Proc. IFIP TC13 Conference on Human-Computer Interaction
Ericsson Trade Show Events in 2009 / 2010Barcelona, Las Vegas, Boston, Galway, Stockholm, Paris, Amsterdam, San Francisco
352 px
288
px
408 mm
306
mm
800 px
600
px
71 px
95 p
x
camera view
projection & touch input
mobile device 2704 mobile px
2021
mob
ile p
xdevice tracking region
improved visual fidelity & touch input resolution
projected pixels
mobile pixels
240 px
36 mm
48 m
m
320
px
217 px
163
px
px/mm2
0
20
40
60
80
100
120
140
160
180
Projector32"
1024×768
LCD21"
1600×1200
Apple iPhone3.5"
320×480
Sony Ericsson G9002.4"
240×320
Nokia 5800XM3.2"
360×640
Google Nexus One3.7"
480×800
Apple iPhone 43.5"
640x960
ResolutionDenser pixels & sensors on handhelds
Augmenting surface interaction Mobile devices ・ expanded capabilities
INTERACT 2009 [Olwal]Proc. IFIP TC13 Conference on Human-Computer Interaction
Ericsson Trade Show Events in 2009 / 2010Barcelona, Las Vegas, Boston, Galway, Stockholm, Paris, Amsterdam, San Francisco
• Need HybridSurface slide
Augmenting surface interactionCollaboration for multiple users, devices & locations
TEI 2009 [Olwal & Feiner]Proc. International Conference on Tangible & Embedded Interaction
INTERACT 2009 [Olwal]Proc. IFIP TC13 Conference on Human-Computer Interaction
Ericsson Trade Show Events in 2009 / 2010Barcelona, Las Vegas, Boston, Galway, Stockholm, Paris, Amsterdam, San Francisco
Hybrid UIs for AR
1 Unobtrusive
2 Sensing in environment
3 Novel display technology
AR, UbiComp, Mobile, Interactive Surfaces, …
Shopping window Microsoft Surface Microvision HUD Heliodisplay, Mid-air
ASTORTransparent window with autostereoscopic 3D overlays
SPIE 2008 [Olwal, Gustafsson & Lindfors]Proc. SPIE Electronic Imaging, Vol 6804
ISMAR 2005 [Olwal, Lindfors, Gustafsson, Kjellberg & Mattson]Proc. IEEE & ACM Symposium on Mixed & Augmented Reality
SIGGRAPH 2004 Sketches [Olwal, Lindfors & Gustafsson]International Conference on Computer Graphics & Interactive Techniques
workspace
process data
workspace & process data
Unencumbered 3D interactionSupporting walk-up-and-use for public displays
NordiCHI 2008 [Olwal]Proc. Nordic Conference on Human Computer Interaction
eye tracker
touch overlay
transparent 3D display ( holographic optical element )
3D camera
Unencumbered 3D interactionManipulate 3D data using mobile, gestures & touch
NordiCHI 2008 [Olwal]Proc. Nordic Conference on Human Computer Interaction
3DTV Book 2008 [DiVerdi, Olwal, Rakkolainen & Höllerer]Three-Dimensional Television: Capture, Transmission & Display, Springer
VR 2006 [Olwal, DiVerdi, Rakkolainen & Höllerer]Proc. IEEE Virtual Reality
SIGGRAPH 2005 Emerging Technologies [Rakkolainen, DiVerdi, Olwal, Candussi & Höllerer]International Conference on Computer Graphics & Interactive Techniques
Immaterial displaysFace-to-face, reach-through & walk-through interaction
Hybrid UIs for AR
1 Unobtrusive
2 Sensing in environment
3 Novel display technology
4 Hybrid approaches → sensing / displays / interaction
AR, UbiComp, Mobile, Interactive Surfaces, …
BiDi: Flat depth-sensing displaysHirsch, Lanman, Holtzman & Raskar 2009
SecondLightIzadi, Hodges, Taylor, Rosenfeld, Villar, Butler & Westhues 2008
Interactions in the airHilliges, Izadi, Wilson, Hodges, Mendoza & Butz 2009
Nintendo WiimoteCamera + accelerometers + gyros + “sensor” bar
PlayStation MovePS Eye camera + Motion controller
Microsoft KinectDepth camera
Mobile projection
AR marketing
• Relevant
• Content consumption
• Interactivity
• Activity analysis
Rule Communication – www.rule.se
HybridSurface – www.harmonic.se
• Hybrid interaction: Touch surface + mobile devices
• Ultra-thin design
• Mobile World Congress 2009 (Barcelona), CTIA Wireless 2009 (Las Vegas), Ericsson Volvo Ocean Race VIP Pavilion 2009 (Boston, Galway, Stockholm), Ericsson Experience Center
• Next generation for new Maritime Museum Exhibit 2011
Acknowledgements
Organizers
• Disruptive Media
Collaborators
• Department of Production Engineering, KTH
• Center for Medicine, Health and Technology, KTH / KI / KS
• FunkIS (HCI + radiology research)
• Department of Mechatronics
• VITA, Linköping University
• Computer Graphics & UI lab, Columbia University
• ilab, USCB
Funding
• KK foundation
• Swedish Institute for Assistive Technologies
• Engineer’s Science Academy / Innovation Bridge
• Blanceflor Foundation
• Swedish Research Council
Equipment, donations & funding
• Ericsson, Nokia, Microsoft Research, Mitsubishi, Doro, …
Hybrid User Interfaces for Augmented Reality
Alex Olwal, Ph.D.olwal.com
research: olwal.come-marketing: rule.se
new user interfaces : harmonic.se
Hybrid User Interfaces for Augmented Reality
Alex Olwal, Ph.D.olwal.com
research: olwal.come-marketing: rule.se
new user interfaces: harmonic.se