LECTURE 2: AR TECHNOLOGY

Mark Billinghurst mark.billinghurst@unisa.edu.au

Zi Siang See zisiang@reina.com.my

November 29th-30th 2015

Mobile-Based Augmented Reality Development

TECHNOLOGY

Augmented Reality Definition

• Defining Characteristics • Combines Real and Virtual Images

• Display Technology • Interactive in real-time

• Interaction Technology • Registered in 3D

• Tracking Technology

DISPLAY

Display Technologies

! Types (Bimber/Raskar 2003) ! Head attached

•  Head mounted display/projector ! Body attached

•  Handheld display/projector ! Spatial

•  Spatially aligned projector/monitor

Display Taxonomy

Types of Head Mounted Displays

Occluded See-thru

Multiplexed

Optical see-through HMD

Virtual images from monitors

Real World

Optical Combiners

Epson Moverio BT-200

▪  Stereo see-through display ($700) ▪  960 x 540 pixels, 23 degree FOV, 60Hz, 88g ▪  Android Powered, separate controller ▪  VGA camera, GPS, gyro, accelerometer

Strengths of optical see-through •  Simpler (cheaper) •  Direct view of real world

•  Full resolution, no time delay (for real world) •  Safety •  Lower distortion

•  No eye displacement •  see directly through display

Video see-through HMD

Video cameras

Monitors

Graphics

Combiner

Video

Vuzix Wrap 1200DXAR

▪  Stereo video see-through display ($1500) ■  Twin 852 x 480 LCD displays, 35 deg. FOV

■  Stereo VGA cameras ■  3 DOF head tracking

Strengths of Video See-Through •  True occlusion

•  Block image of real world

•  Digitized image of real world •  Flexibility in composition •  Matchable time delays •  More registration, calibration strategies

•  Wide FOV is easier to support •  wide FOV camera

Multiplexed Displays

•  Above or below line of sight •  Strengths

•  User has unobstructed view of real world •  Simple optics/cheap

•  Weaknesses •  Direct information overlay difficult •  Display/camera offset from eyeline •  Wide FOV difficult

Google Glass

▪  Monocular see-through multiplexed display ▪  640 x 360 microprojector, 15 degree FOV ▪  5 MP camera, gyro, accelerometer

Display Technology

•  Curved Mirror •  off-axis projection •  curved mirrors in front of eye •  high distortion, small eye-box

•  Waveguide •  use internal reflection •  unobstructed view of world •  large eye-box

See-through thin displays

•  Waveguide techniques for thin see-through displays •  Wider FOV, enable AR applications •  Social acceptability

Opinvent Ora

Lumus DK40

Spatial Augmented Reality

• Project onto irregular surfaces • Geometric Registration • Projector blending, High dynamic range

• Book: Bimber “Spatial Augmented Reality”

Projector-based AR

Examples: Raskar, MIT Media Lab Inami, Tachi Lab, U. Tokyo

Projector

Real objects with retroreflective covering

User (possibly head-tracked)

HMD vs. HMPD

Head Mounted Display Head Mounted Projected Display

CastAR - http://technicalillusions.com/

• Stereo head worn projectors • Interactive wand • Rollable retro-reflective sheet

• Designed for shared interaction

Video Monitor AR

Video cameras Monitor

Graphics Combiner

Video

Stereo glasses

Examples

Handheld Displays

• Mobile Phones • Camera • Display • Input

TRACKING

Objects Registered in 3D

• Registration • Positioning virtual object wrt real world

• Tracking • Continually locating the users viewpoint

•  Position (x,y,z), Orientation (r,p,y)

Tracking Technologies "  Active

•  Mechanical, Magnetic, Ultrasonic •  GPS, Wifi, cell location

"  Passive •  Inertial sensors (compass, accelerometer, gyro) •  Computer Vision

•  Marker based, Natural feature tracking

"  Hybrid Tracking •  Combined sensors (eg Vision + Inertial)

Tracking Types

Magnetic Tracker

Inertial Tracker

Ultrasonic Tracker

Optical Tracker

Marker-Based Tracking

Markerless Tracking

Specialized Tracking

Edge-Based Tracking

Template-Based Tracking

Interest Point Tracking

Mechanical Tracker

Example: Marker tracking •  Available for more than 10 years •  Several open source solutions exist

•  ARToolKit, ARTag, ATK+, etc

•  Fairly simple to implement •  Standard computer vision methods

•  A rectangle provides 4 corner points •  Enough for pose estimation!

Marker Based Tracking: ARToolKit

http://artoolkit.sourceforge.net/

Coordinate Systems

Tracking challenges in ARToolKit

False positives and inter-marker confusion (image by M. Fiala)

Image noise (e.g. poor lens, block

coding / compression, neon tube)

Unfocused camera, motion blur

Dark/unevenly lit scene, vignetting

Jittering (Photoshop illustration)

Occlusion (image by M. Fiala)

Markerless Tracking

Magnetic Tracker Inertial Tracker

Ultrasonic Tracker

Optical Tracker

Marker-Based Tracking

Markerless Tracking

Specialized Tracking

Edge-Based Tracking

Template-Based Tracking

Interest Point Tracking

• No more Markers! #Markerless Tracking

Mechanical Tracker

Natural Feature Tracking

• Use Natural Cues of Real Elements • Edges • Surface Texture •  Interest Points

• Model or Model-Free • No visual pollution

Contours

Features Points

Surfaces

Texture Tracking

Edge Based Tracking • RAPiD [Drummond et al. 02]

•  Initialization, Control Points, Pose Prediction (Global Method)

Line Based Tracking

• Visual Servoing [Comport et al. 2004]

Model Based Tracking

• OpenTL - www.opentl.org • General purpose library for model based visual tracking

Marker vs. natural feature tracking •  Marker tracking

•  ++ Markers can be an eye-catcher •  ++ Tracking is less demanding •  -- The environment must be instrumented with markers •  -- Markers usually work only when fully in view

•  Natural feature tracking •  -- A database of keypoints must be stored/downloaded •  ++ Natural feature targets might catch the attention less •  ++ Natural feature targets are potentially everywhere •  ++ Natural feature targets work also if partially in view

Example: Outdoor Hybrid Tracking

• Combines • computer vision

•  natural feature tracking

•  inertial gyroscope sensors • Both correct for each other

•  Inertial gyro - provides frame to frame prediction of camera orientation

• Computer vision - correct for gyro drift

Robust Outdoor Tracking

• Hybrid Tracking • Computer Vision, GPS, inertial

• Going Out •  Reitmayr & Drummond (Univ. Cambridge)

Reitmayr, G., & Drummond, T. W. (2006). Going out: robust model-based tracking for outdoor augmented reality. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.

Handheld Display

INTERACTION

• Interface Components • Physical components • Display elements

• Visual/audio • Interaction metaphors

Physical Elements

Display Elements Interaction

Metaphor Input Output

AR Interface Elements

AR Design Space

Reality Virtual Reality

Augmented Reality

Physical Design Virtual Design

Interface Design Path

1/ Prototype Demonstration

2/ Adoption of Interaction Techniques from other interface metaphors

3/ Development of new interface metaphors appropriate to the medium

4/ Development of formal theoretical models for predicting and modeling user actions

Desktop WIMP

Virtual Reality

Augmented Reality

Interaction Development

• Information Browsing • Camera movement • Limited interaction

• 3D AR Interaction • HMD, hand tracking • 3D UI/VR techniques • Specialized input devices

Tangible User Interfaces (Ishii 97)

• Augmented Surfaces • Rekimoto 1998 • Multiple projection surfaces • Tangible prop interaction

• i/O Brush (2004) • Ryokai, Marti, Ishii • Sensor enhanced real brush

Other Examples

• Triangles (Gorbert 1998) • Triangular based story telling

• ActiveCube (Kitamura 2000-) • Cubes with sensors

Lessons from Tangible Interfaces

• Benefits • Physical objects make us smart (affordances) • Objects aid collaboration • Objects increase understanding (cognitive artifacts)

• Limitations • Difficult to change object properties • Limited display capabilities (2D surface) • Separation between object and display

Orthogonal Nature of AR Interfaces

Tangible AR Interaction

• AR overcomes limitation of TUIs • enhance display possibilities • merge task/display space • provide public and private views

• TUI + AR = Tangible AR • Apply TUI methods to AR interface design

Tangible AR Design Principles

• Tangible AR Interfaces use TUI principles • Physical controllers for moving virtual content • Support for spatial 3D interaction techniques • Support for multi-handed interaction • Match object affordances to task requirements • Support parallel activity with multiple objects • Allow collaboration between multiple users

VOMAR - Tangible AR Interface • Use of natural physical object to control virtual objects

• Physical objects • Catalog book:

• Turn over the page

• Paddle operation: •  Push, shake, incline, hit, scoop

Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., & Tachibana, K. (2000). Virtual object manipulation on a table-top AR environment. In Augmented Reality, 2000.(ISAR 2000). Proceedings. IEEE and ACM International Symposium on (pp. 111-119). Ieee.

Interaction Evolution

• Browsing Interfaces • simple (conceptually!), unobtrusive

• 3D AR Interfaces • expressive, creative, require attention

• Tangible Interfaces • Embedded into conventional environments

• Tangible AR • Combines TUI input + AR display

AR APPLICATIONS

• Web based AR •  Flash, HTML 5 based AR •  Marketing, education

• Outdoor Mobile AR •  GPS, compass tracking •  Viewing Points of Interest in real world •  Eg: Junaio, Layar, Wikitude

• Handheld AR •  Vision based tracking •  Marketing, gaming

•  Location Based Experiences •  HMD, fixed screens •  Museums, point of sale, advertising

Typical AR Experiences

CityViewAR Application

•  Visualize Christchurch before the earthquakes

User Experience

• Multiple Views

• Map View, AR View, List View

• Multiple Data Types • 2D images, 3D content, text, panoramas

Warp Runner

• Puzzle solving game • Deform real world terrain

Demo: colAR

• Turn colouring books pages into AR scenes • Markerless tracking, use your own colours..

• Try it yourself: http://www.colARapp.com/

What Makes a Good AR Experience?

• Compelling • Engaging, ‘Magic’ moment

• Intuitive, ease of use • Uses existing skills

• Anchored in physical world • Seamless combination of real and digital

Conclusion • AR seamlessly blends real and virtual imagery

•  Interactive in real time, fixed in space

• AR has developed into a mass market technology • Education, engineering, entertainment

• The technologies to create AR are available • Display, tracking, interaction