Download - 2013 Lecture 8: Mobile AR
![Page 2: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/2.jpg)
1983 – Star Wars
![Page 3: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/3.jpg)
1999 - HIT Lab US
![Page 4: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/4.jpg)
CPU: 300 Mhz HDD; 9GB RAM: 512 mb Camera: VGA 30fps Graphics: 500K poly/sec
1998: SGI O2 2008: Nokia N95
CPU: 332 Mhz HDD; 8GB RAM: 128 mb Camera: VGA 30 fps Graphics: 2m poly/sec
![Page 5: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/5.jpg)
Mobile Phone AR Mobile Phones
camera processor display
AR on Mobile Phones Simple graphics Optimized computer vision Collaborative Interaction
![Page 6: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/6.jpg)
2005: Collaborative AR
AR Tennis Shared AR content Two user game Audio + haptic feedback Bluetooth networking
![Page 7: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/7.jpg)
![Page 8: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/8.jpg)
Mobile AR History
![Page 9: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/9.jpg)
Evolution of Mobile AR
Wearable AR
Handheld AR Displays
Camera phone
1995 1997 2001 2003 2004
Camera phone - Self contained AR
Wearable Computers
PDAs -Thin client AR
PDAs -Self contained AR
Camera phone - Thin client AR
![Page 10: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/10.jpg)
Handheld Displays Tethered Applications
Fitzmaurice Chameleon (1994) Rekimoto’s Transvision (1995)
Tethered LCD PC Processing and Tracking
![Page 11: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/11.jpg)
Handheld AR Display - Tethered
1995, 1996 Handheld AR ARPad, Cameleon Rekimoto’s NaviCam, Transvision Tethered LCD PC Processing and Tracking
![Page 12: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/12.jpg)
AR Pad (Mogilev 2002) Handheld AR Display
LCD screen Camera SpaceOrb 3 DOF controller Peripheral awareness Viewpoint awareness
![Page 13: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/13.jpg)
Mobile AR: Touring Machine (1997) University of Columbia
Feiner, MacIntyre, Höllerer, Webster
Combines See through head mounted display GPS tracking Orientation sensor Backpack PC (custom) Tablet input
![Page 14: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/14.jpg)
MARS View
Virtual tags overlaid on the real world “Information in place”
![Page 15: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/15.jpg)
Backpack/Wearable AR
1997 Backpack AR Feiner’s Touring Machine AR Quake (Thomas) Tinmith (Piekarski) MCAR (Reitmayr) Bulky, HMD based
![Page 16: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/16.jpg)
PCI 3D Graphics Board
Hard Drive
Serial
Ports
CPU
PC104 Sound Card
PC104 PCMCIA
GPS Antenna
Tracker Controller
DC to DC Converter
Battery
Wearable Computer
GPS RTK correction
Radio
Example self-built working solution with PCI-based 3D graphics
Columbia Touring Machine
Mobile AR - Hardware
![Page 17: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/17.jpg)
1997 Philip Kahn invents camera phone 1999 First commercial camera phone
Sharp J-SH04
![Page 18: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/18.jpg)
Millions of Camera Phones
![Page 19: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/19.jpg)
Handheld AR – Thin Client
2001 BatPortal (AT&T Cambridge) PDA used as I/O device Wireless connection to workstation Room-scale ultrasonic tracking (Bat)
2001 AR-PDA (C Lab) PDA thin graphics client Remote image processing www.ar-pda.com
![Page 20: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/20.jpg)
2003 ARphone (Univ. of Sydney) Transfer images via Bluetooth (slow – 30 sec/image) Remote processing – AR Server
Mobile Phone AR – Thin Client
![Page 21: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/21.jpg)
Early Phone Computer Vision Apps 2003 – Mozzies Game - Best mobile game Optical motion flow detecting phone orientation Siemens SX1 – Symbian, 120Mhz, VGA Camera
2005 – Marble Revolution (Bit-Side GmbH) Winner of Nokia's Series 60 Challenge 2005
2005 – SymBall (VTT)
![Page 22: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/22.jpg)
Computer Vision on Mobile Phone Cameras and Phone CPU sufficient for computer vision
applications Pattern Recognition (Static Processing)
QR Code Shotcode (http://www.shotcode.com/)
Motion Flow (2D Image Processing) GestureTek
- http://www.gesturetekmobile.com/
TinyMotion
3D Pose Calculation Augmented Reality
![Page 23: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/23.jpg)
Handheld AR – Self Contained 2003 PDA-based AR
ARToolKit port to PDA Studierstube ported to PDA AR Kanji Educational App. Mr Virtuoso AR character Wagner’s Invisible Train
- Collaborative AR
![Page 24: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/24.jpg)
Mobile Phone AR – Self Contained 2004 Mobile Phone AR
Moehring, Bimber Henrysson (ARToolKit) Camera, processor, display together
![Page 25: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/25.jpg)
![Page 26: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/26.jpg)
AR Advertising
Txt message to download AR application (200K) See virtual content popping out of real paper advert Tested May 2007 by Saatchi and Saatchi
![Page 27: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/27.jpg)
2008 - Location Aware Phones
Nokia Navigator Motorola Droid
![Page 28: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/28.jpg)
Real World Information Overlay Tag real world locations
GPS + Compass input Overlay graphics data on live video
Applications Travel guide, Advertising, etc
Eg: Mobilizy Wikitude (www.mobilizy.com)
Android based, Public API released
Other companies Layar, AcrossAir, Tochnidot, RobotVision, etc
![Page 29: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/29.jpg)
Layar – www.layar.com
![Page 30: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/30.jpg)
HIT Lab NZ Android AR Platform Architectural Application Loads 3D models
a OBJ/MTL format
Positions content in space GPS, compass
Intuitive user interface toolkit to modify the model
Connects to back end model database
![Page 31: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/31.jpg)
Architecture
Android application Database server
Postgres
Web Interface
Add models
Web application java and php server
![Page 32: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/32.jpg)
1995 Handheld Display: NaviCam, AR-PAD, Transvision
1997 Wearable AR: Touring Machine, AR Quake
2001 Handheld AR – Thin Client: AR-PDA, Bat Portal
2003 Handheld AR – Self contained: Invisible Train
2003 Mobile Phone – 2D Vision: Mozzies, Symball
2003 Mobile Phone – Thin Client: ARphone
2004 Mobile Phone – Self contained: Moehring, Symbian
History of Handheld and Mobile AR
![Page 33: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/33.jpg)
Mobile AR by Weight
Backpack+HMD: …5-8kg
Scale it down: Vesp‘R [Kruijff ISMAR07]: …Sony UMPC 1.1GHz …1.5kg …still >$5K
Scale it down more: Smartphone…$500 …All-in-one …0.1kg …billions of units
1996
2003 2007
![Page 34: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/34.jpg)
2013 State of the Art Handheld Hardware available
PDA, mobile phones, external cameras Sensors: GPS, accelerometer, compass Software Tools are Available
Tracking: ARToolKitPlus, stbTracker, Vuforia Graphics: OpenGL ES Authoring: Layar, Wikitude, Metaio Creator
What is needed: High level authoring tools Content development tools Novel interaction techniques User evaluation and usability
![Page 35: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/35.jpg)
Mobile AR Companies Mobile AR
GPS + compass
Many Companies Layar Wikitude Acrossair PressLite Yelp Robot vision Etc..
![Page 36: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/36.jpg)
$2 million USD in 2010 $732 million USD in 2014
![Page 37: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/37.jpg)
![Page 38: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/38.jpg)
Qualcomm
Acquired Imagination October 2010 - Releases free Android AR SDK Computer vision tracking - marker, markerless Integrated with Unity 3D renderer http://developer.qualcomm.com/ar
![Page 39: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/39.jpg)
Rock-em Sock-em
Shared AR Demo Markerless tracking
![Page 40: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/40.jpg)
Mobile AR Technology
![Page 41: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/41.jpg)
Technology Components Software Platform
Eg studierStube platform
Developer Tools iOS, Android
Tracking Technology Computer vision, sensor based
Mobile Graphics OpenGL ES
Interaction Methods Handheld Interaction
![Page 42: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/42.jpg)
iPhone 4 Apple iOS Faster CPU (1.2GHz) High screen resolution
3.5”, 960x640
camera API Multi-touch Hardware 3D GPS, compass, accelerometer
and gyroscope
![Page 43: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/43.jpg)
Hardware Sensors Camera (resolution, fps)
Maker based/markerless tracking Video overlap
GPS (resolution, update rate) Outdoor location
Compass Indoor/outdoor orientation
Accelerometer Motion sensing, relative tilt
![Page 44: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/44.jpg)
Studierstube ES Framework
Typical AR application framework
Developed at TU Graz
![Page 45: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/45.jpg)
Hardware
OS/Low Level API
Programming Libraries
End User Application
![Page 46: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/46.jpg)
The Studierstube ES framework
Tracking
Platform
Graphics
Content
User Interface - Application
![Page 47: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/47.jpg)
Mobile Graphics
![Page 48: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/48.jpg)
Computer Graphics on Mobile Phones Small screen, limited input options Limited support for accelerated graphics
Most phones have no GPU
Mobile Graphics Libraries OpenGL ES (1.0, 2.0)
- Cross platform, subset of OpenGL - C/C++ low level library
Java M3G - Mobile 3D graphics API for J2ME platform - Object importer, scene graph library - Support from all major phone manufacturers
![Page 49: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/49.jpg)
![Page 50: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/50.jpg)
OpenGL ES Small-footprint subset of OpenGL
OpenGL is too large for embedded devices!
Powerful, low-level API, full functionality for 3D games Can do almost everything OpenGL can Available on all key platforms Software and hardware implementations available
Fully extensible Extensions like in OpenGL
![Page 51: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/51.jpg)
OpenGL ES on mobile devices
![Page 52: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/52.jpg)
Versions Two major tracks
Not compatible, parallel rather than competitive
OpenGL ES 1.x Fixed function pipeline Suitable for software implementations All 1.x are backwards compatible
OpenGL ES 2.x Vertex and pixel shaders using GLSL ES All 2.x will be backwards compatible
![Page 53: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/53.jpg)
Fixed Function (1.x)
h"p://www.khronos.org/opengles/2_X/
![Page 54: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/54.jpg)
Programmable (2.x)
h"p://www.khronos.org/opengles/2_X/
![Page 55: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/55.jpg)
OpenGL ES 1.x vs 2.0
![Page 56: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/56.jpg)
Tracking
![Page 57: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/57.jpg)
Mobile Augmented Reality’s goal
Create an affordable, massively multi-user, widespread platform
© Tinmith, U. of South Australia
![Page 58: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/58.jpg)
How to not do it…
Ka-Ping Yee: Peephole Displays, CHI’03
![Page 59: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/59.jpg)
Tracking on mobile phones Vision-based tracking
Marker-based tracking Model-based natural feature tracking Natural feature tracking in unknown
environments
Sensor tracking GPS, inertial compass, gyroscope
![Page 60: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/60.jpg)
Backpack-based
Höllerer et al. (1999), Piekarski & Thomas al. (2001), Reitmayr & Schmalstieg (2003)
Laptop, HMD Enhanced GPS (DGPS / RTK) + inertial sensor for viewpoint tracking Hand tracking w/ fiducial markers
![Page 61: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/61.jpg)
Tracking for Handheld AR SLIDE 61
Backpack-based 2.
Kalkusch et al., 2002 Video see-through HMD w/ camera Viewpoint Tracking w/ inside-out computer vision using markers ARToolKit markers on walls installed and surveyed manually
![Page 62: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/62.jpg)
Tablet PC / UMPC-based
Schall et al., 2006 Hybrid tracking on UMPC
Camera fiducial marker tracking When no marker in view inertial sensor + UWB tracking
![Page 63: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/63.jpg)
PDA-based 1.
BatPortal (Newman et al., 2001) PDA as thin client (rendering &
tracking on server + VNC) Ultrasonic tracking
SHEEP (MacWilliams et al., 2003)
Tracking by ART (external IR cameras + retroreflective target)
Projection-based AR environ.
![Page 64: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/64.jpg)
non-AR Tracking on Phones
AR-PDA (2003)
Model-based tracking PDA = thin client
tracking on server Not real-time
Mosquito Hunt (2003) Marble Revolution (2004) Pingis (VTT, 2006) Game control w/ optical flow techniques
TinyMotion (2006) GUI control & input
on cell phones w/ image differencing
& block correlation
![Page 65: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/65.jpg)
History of AR Tracking on Phones (1)
2003 ARToolKit on PDA Wagner et at.
2004 3D Marker on Phone Möhring et al.
2005 ARToolKit on Symbian Henrysson et al.
![Page 66: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/66.jpg)
Tracking for Handheld AR SLIDE 66
Fiducial marker tracking on handhelds
Möhring et al., 2004 Henrysson et al., 2006 Wagner et al., 2003
Rohs, 2006 Bucolo et al., 2005
![Page 67: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/67.jpg)
History of AR Tracking on Phones (2)
2005 Visual Codes Rohs et at.
2008 Advanced Marker Tracking Wagner et al.
2008 Natural Feature Tracking Wagner et al.
![Page 68: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/68.jpg)
What can we do on today‘s mobile phones?
Typical specs 600+ MHz ~5MB of available RAM 160x120 - 320x240 at 15-30 Hz camera
Possible to do Marker tracking in 5-15ms Natural feature tracking in 20-50ms
![Page 69: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/69.jpg)
Other Mobile Sensors Orientation
Compass
Relative movement/rotation Accelerometer, gyroscope
Context Audio, light sensor, proximity
Location GPS, A-GPS, Wifi positioning, Cell tower triangulation
![Page 70: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/70.jpg)
Handheld AR Interfaces
![Page 71: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/71.jpg)
Handheld HCI
Consider your user Follow good HCI principles Adapt HCI guidelines for handhelds Design to device constraints Rapid prototyping User evaluation
![Page 72: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/72.jpg)
Sample Handheld AR Interfaces Clean Large Video View Large Icons Text Overlay
![Page 73: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/73.jpg)
Handheld Display vs Fixed Display
Experiment comparing handheld moving, to handheld button input, small fixed display, desktop display, large plasma
Users performed (1) navigation task, (2) selection task Moving handheld display provided greater perceived FOV, higher degree of
presence, faster completion time
J. Hwang, J. Jung, G. Kim. Hand-held Virtual Reality: A Feasibility Study. In proceedings of VRST 2006
![Page 74: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/74.jpg)
Search Task Completion Time
![Page 75: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/75.jpg)
FOV, Presence and Immersion
![Page 76: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/76.jpg)
HandHeld AR Wearable AR
Output: Display
Input
Input & Output
HMD vs Handheld AR Interface
![Page 77: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/77.jpg)
Handheld Interface Properties Handheld interface vs. HMD interface
Display is handheld rather than headworn Much greater peripheral view of real world Display and input device connected Can move device independent of view
Phone Keypad Touch Screen One handed input Keypad only Bimanual interaction
Object based interaction
Two handed input Stylus/touch screen Screen based input/selection
Large screen Limited number of buttons
![Page 78: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/78.jpg)
Handheld Interface Metaphors Tangible AR Lens Viewing
Look through screen into AR scene Interact with screen to interact with AR
content - Eg Invisible Train
Tangible AR Lens Manipulation Select AR object and attach to device Use the motion of the device as input
- Eg AR Lego
![Page 79: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/79.jpg)
![Page 80: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/80.jpg)
Translation Study Conditions
A: Object fixed to the phone (one handed) B: Button and keypad input C: Object fixed to the phone (bimanual) - one hand for rotating tracking pattern
![Page 81: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/81.jpg)
Results – Translation • 9 subjects – within subject design • Timing
• Tangible fastest • twice as fast as keypad
• Survey • Tangible easiest (Q1) • Keypad most accurate (Q2) • Tangible quickest (Q3) • Tangible most enjoyable (Q4)
• Ranking • Tangible favored
A B C
Rank 1.44 2.56 2.0
![Page 82: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/82.jpg)
Conditions A: Arcball B: Keypad input for rotation about
the object axis C: Object fixed to the phone (one handed)
D: Object fixed to the phone (bimanual)
Rotation Study
![Page 83: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/83.jpg)
• Timing • Keypad(B) and Arcball(A) fastest
• No significant survey
differences
A B C D Rank 3.0 2.3 2.4 2.2
Results – Rotation
![Page 84: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/84.jpg)
Collaborative AR
AR Tennis Virtual tennis court Two user game Audio + haptic feedback Bluetooth messaging
![Page 85: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/85.jpg)
TAT Augmented ID
![Page 86: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/86.jpg)
![Page 87: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/87.jpg)
![Page 88: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/88.jpg)
Design Guidelines
Apply handheld HCI guidelines for on-screen content - large buttons, little text input, etc
Design physical + virtual interface elements Pick appropriate interface metaphor
- “handheld lens” approach using handheld motion - Tangible AR for AR overlay
Build prototypes Continuously evaluate application
![Page 89: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/89.jpg)
Mobile AR Browsing
![Page 90: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/90.jpg)
![Page 91: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/91.jpg)
![Page 92: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/92.jpg)
AR Browsers AR equivalent of web browser
Request and serve up content
Commercial outdoor AR applications Junaio, Layar, Wikitude, etc
All have their own language specifications Wikitude – ARML Junaio – XML, AREL
![Page 93: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/93.jpg)
AR Browsers Commercial outdoor AR applications
Junaio, Layar, Wikitude, etc
All have their own language specifications Wikitude – ARML Junaio - XML
Need for common standard Based on existing standards for geo-located content etc Support for dynamic/interactive content Easier to author mobile AR applications Easy to render on AR browsers
![Page 94: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/94.jpg)
Architecture
![Page 95: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/95.jpg)
Common AR Browsers Layar
http://www.layar.com/
Wikitude http://www.wikitude.com/
Junaio http://www.junaio.com
TagWhat http://www.tagwhat.com/
Sekai Camera http://sekaicamera.com/
![Page 96: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/96.jpg)
Nokia City Lens
More recent AR Browser
![Page 97: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/97.jpg)
![Page 98: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/98.jpg)
Junaio - www.junaio.com
![Page 99: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/99.jpg)
![Page 100: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/100.jpg)
![Page 101: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/101.jpg)
![Page 102: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/102.jpg)
Key Features Content provided in information channels
Over 2,000 channels available
Two types of AR channels GLUE channels – visual tracking Location based channels – GPS, compass tracking
Simple to use interface with multiple views List, map, AR (live) view
Point of Interest (POI) based POIs are geo-located content
![Page 103: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/103.jpg)
![Page 104: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/104.jpg)
QR Code Launch
![Page 105: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/105.jpg)
Glue Tracking - Markerless
Search for “instant tracker”
![Page 106: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/106.jpg)
Junaio Interface
![Page 107: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/107.jpg)
Interface
List View, Map View, AR View
![Page 108: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/108.jpg)
![Page 109: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/109.jpg)
Back-end Servers
![Page 110: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/110.jpg)
Data Flow
![Page 111: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/111.jpg)
Search.php <?xml version=\"1.0\" encoding=\"UTF-8\"?> <results>
<poi id=\"1\" interactionfeedback=\"none\"> <name><![CDATA[[Hotel Hello World]]]></name> <description><![CDATA[[This is a beautiful, family hotel and restaurant, just around the
corner. Special Dinner and Rooms available.]]]></description>
<l>37.776685,-122.422771,0</l> <mime-type>text/plain</mime-type>
<icon>http://dev.junaio.com/publisherDownload/tutorial/icon_map.png</icon> <thumbnail>http://dev.junaio.com/publisherDownload/tutorial/thumb.jpg</thumbnail> <phone>555/1234567</phone> <homepage> http://www.hotelaroundthecorner.com </homepage> </poi>
</results>
![Page 112: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/112.jpg)
AR Outcome
![Page 113: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/113.jpg)
Limitations of Plain XML No interactivity
Only simple pop-ups
No user interface Customizations Can only use Junaio GUI elements
No local interactivity Always needs remote server connection
![Page 114: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/114.jpg)
Junaio AREL
![Page 115: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/115.jpg)
AREL Augmented Reality Environment Language
Overcomes limitations of XML by itself Based on web technologies; XML, HTML5, JavaScript
Core Components 1. AREL XML: Static file, specifies scene content 2. AREL JavaScript: Handles all interactions and animation. Any
user interaction send an event to AREL JS 3. AREL HTML5: GUI Elements. Buttons, icons, etc
Advantages Scripting on device, more functionality, GUI customization
![Page 116: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/116.jpg)
![Page 117: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/117.jpg)
![Page 118: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/118.jpg)
![Page 119: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/119.jpg)
Adding Interactivity
![Page 120: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/120.jpg)
Basic Interactivity
Add a button on screen to move virtual character
Use the following HTML: button specification Javascript: Interaction PHP/XML: 3D model
Junaio Tutorial 5 http://www.junaio.com/develop/quickstart/
advanced-interactions-and-location-based-model-3ds/
![Page 121: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/121.jpg)
Server File Structure
HTML – GUI
JavaScript - interactivity
Main Index
PHP - content
![Page 122: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/122.jpg)
search.php – specify Lego Man if(!empty($_GET['l'])) $position = explode(",", $_GET['l']); … //return the lego man $oLegoMan = ArelXMLHelper::createLocationBasedModel3D(
"1", // id "lego man", //title WWW_ROOT . "/resources/walking_model3_7fps.md2", // mainresource WWW_ROOT . "/resources/walking_model.png", // resource $position, // location array(0.2, 0.2, 0.2), // scale new ArelRotation(ArelRotation::ROTATION_EULERRAD, array(1.57,0,1.57)) // rotation
); …
Use local position
Lego model and texture
![Page 123: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/123.jpg)
styles.css – HTML GUI #buttons { position: absolute; bottom: 44px; right: 44px; }
.ipad div { width: 104px; height: 106px; }
#buttons div { background-image: url("../images/button.png"); background-repeat: no-repeat; background-size: 100%;
float: left; }
Button location
Button style
![Page 124: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/124.jpg)
Logic_LBS5.js - JavaScript Create an event listener
setEventListener();
Add functionality to model object Load model from scene Adding model behaviours
Add functionality to GUI objects Define the event listener Bind model behaviours to GUI objects
![Page 125: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/125.jpg)
Result
![Page 126: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/126.jpg)
Authoring Tools
![Page 127: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/127.jpg)
Metaio Creator
Drag and drop Junaio authoring
![Page 128: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/128.jpg)
BuildAR – buildar.com
![Page 129: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/129.jpg)
Sony CSL © 2004
Developing Mobile AR Experiences
![Page 130: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/130.jpg)
![Page 131: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/131.jpg)
![Page 132: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/132.jpg)
![Page 133: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/133.jpg)
![Page 134: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/134.jpg)
![Page 135: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/135.jpg)
![Page 136: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/136.jpg)
![Page 137: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/137.jpg)
![Page 138: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/138.jpg)
![Page 139: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/139.jpg)
![Page 140: 2013 Lecture 8: Mobile AR](https://reader033.vdocuments.us/reader033/viewer/2022042518/54c8c86b4a79591c188b45d0/html5/thumbnails/140.jpg)