426 lecture2: ar technology

COSC 426: Augmented Reality

Mark Billinghurst

mark.billinghurst@hitlabnz.org

July 18th 2012

Lecture 2: AR Technology

Key Points from Lecture 1

Augmented Reality Definition   Defining Characteristics [Azuma 97]

 Combines Real and Virtual Images -  Both can be seen at the same time

  Interactive in real-time -  Virtual content can be interacted with

  Registered in 3D -  Virtual objects appear fixed in space

What is not Augmented Reality?

  Location-based services   Barcode detection (QR-codes)   Augmenting still images   Special effects in movies   …   … but they can be combined with AR!

Milgram’s Reality-Virtuality Continuum

Mixed Reality

Reality - Virtuality (RV) Continuum

Real Environment

Augmented Reality (AR)

Augmented Virtuality (AV)

Virtual Environment

Metaverse

AR History Summary   1960’s – 80’s: Early Experimentation   1980’s – 90’s: Basic Research

  Tracking, displays

  1995 – 2005: Tools/Applications   Interaction, usability, theory

  2005 - : Commercial Applications  Games, Medical, Industry

Applications

  Medicine   Manufacturing   Information overlay   Architecture   Museum   Marketing   Gaming

AR Technology

“The product is no longer the basis of value. The

experience is.”

Venkat Ramaswamy The Future of Competition.

experiences

services

products

components

Valu

e

Sony CSL © 2004

Gilmore + Pine: Experience Economy

Function

Emotion

experiences

applications

tools

components

Building Compelling AR Experiences

Tracking, Display

Authoring

Interaction

Usability

experiences

applications

tools

components

Sony CSL © 2004

Building Compelling AR Experiences

Display, Tracking

AR Technology   Key Technologies

 Display   Tracking   Input   Processing

Display

Processing

Input

Tracking

AR Displays

AR Displays

e.g. window reflections

Virtual Images seen off windows

e.g. Reach-In

Projection CRT Display using beamsplitter

Not Head-Mounted

e.g. Shared Space Magic Book

Liquid Crystal Displays LCDs

Head-Mounted Display (HMD)

Primarily Indoor Environments

e.g. WLVA and IVRD

Cathode Ray Tube (CRT) or Virtual Retinal Display (VRD)

Many Military Applications & Assistive Technologies

Head-Mounted Display (HMD)

e.g. Head-Up Display (HUD)

Projection Display Navigational Aids in Cars

Military Airborne Applications

Not Head Mounted (e.g. vehicle mounted)

Primarily Outdoor (Daylight) Environments

AR Visual Displays

Head Mounted Displays

Head Mounted Displays (HMD) -  Display and Optics mounted on Head -  May or may not fully occlude real world -  Provide full-color images -  Considerations

•  Cumbersome to wear •  Brightness •  Low power consumption •  Resolution limited •  Cost is high?

Key Properties of HMD   Field of View

  Human eye 95 degrees horizontal, 60/70 degrees vertical

  Resolution   > 320x240 pixel

  Refresh Rate   Focus

  Fixed/manual

  Power   Size

Types of Head Mounted Displays

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

The image cannot be displayed. Your computer

Occluded See-thru

Multiplexed

Immersive VR Architecture

Head!Tracker

Host !Processor

Data Base!Model

Rendering!Engine Frame!

Buffer

head position/orientation

to network Display!Driver

Non see-thru!Image source

& optics

virtual object

Virtual World

See-thru AR Architecture

Head!Tracker

Host !Processor

Data Base!Model

Rendering!Engine Frame!

Buffer

head position/orientation

to network Display!Driver

see-thru!combiner

Virtual Image superimposed!over real world object

real world

Image source

Optical see-through head-mounted display

Virtual images from monitors

Real World

Optical Combiners

Optical See-Through HMD

Optical see-through HMDs

Sony Glasstron

Virtual Vision VCAP

View Through Optical See-Through HMD

DigiLens

  www.digilens.com

  Compact HOE   Solid state optics   Switchable Bragg Grating   Stacked SBG   Fast switching   Ultra compact

Google Glasses

The Virtual Retinal Display

  Image scanned onto retina   Commercialized through Microvision

  Nomad System - www.mvis.com

Strengths of optical AR   Simpler (cheaper)   Direct view of real world

  Full resolution, no time delay (for real world)   Safety   Lower distortion

  No eye displacement   but COASTAR video see-through avoids this

Video AR Architecture

Head!Tracker

Host !Processor

Graphics!renderer

Digital!Mixer Frame!

Buffer

head position/orientation

to network Display!Driver

Non see-thru!Image source

& optics

Head-mounted camera aligned to

display optics

Video!Processor

Video image of real world

Virtual image inset into video of real world

Video see-through HMD Video cameras

Monitors

Graphics

Combiner

Video

Video See-Through HMD

Video see-through HMD

MR Laboratory’s COASTAR HMD (Co-Optical Axis See-Through Augmented Reality) Parallax-free video see-through HMD

TriVisio   www.trivisio.com   Stereo video input

  PAL resolution cameras

  2 x SVGA displays   30 degree FOV   User adjustable convergence

  $6,000 USD

View Through a Video See-Through HMD

Vuzix Display

  www.vuzix.com   Wrap 920   $350 USD   Twin 640 x 480 LCD displays   31 degree diagonal field of view   Weighs less than three ounces

Strengths of Video AR   True occlusion

  Kiyokawa optical display that supports occlusion

  Digitized image of real world   Flexibility in composition  Matchable time delays  More registration, calibration strategies

  Wide FOV is easier to support

Optical vs. Video AR Summary   Both have proponents   Video is more popular today?

  Likely because lack of available optical products

  Depends on application?  Manufacturing: optical is cheaper  Medical: video for calibration strategies

Eye multiplexed AR Architecture

Head!Tracker

Host !Processor

Data Base!Model

Rendering!Engine Frame!

Buffer

head position/orientation

to network Display!Driver

Virtual Image inset into!real world scene

real world

Opaque!Image source

Virtual Image ‘inset’ into real

Virtual Vision Personal Eyewear

Virtual image inset into real world

Spatial/Projected AR

Spatial Augmented Reality

  Project onto irregular surfaces   Geometric Registration   Projector blending, High dynamic range

  Book: Bimber, Rasker “Spatial Augmented Reality”

Projector-based AR

Examples: Raskar, MIT Media Lab Inami, Tachi Lab, U. Tokyo

Projector

Real objects with retroreflective covering

User (possibly head-tracked)

Example of projector-based AR

Ramesh Raskar, UNC, MERL

Example of projector-based AR

Ramesh Raskar, UNC Chapel Hill

The I/O Bulb

  Projector + Camera   John Underkoffler, Hiroshi Ishii  MIT Media Lab

Head Mounted Projector

  Head Mounted Projector   Jannick Rolland (UCF)

  Retro-reflective Material   Potentially portable

Head Mounted Projector

  NVIS P-50 HMPD   1280x1024/eye   Stereoscopic   50 degree FOV   www.nvis.com

HMD vs. HMPD

Head Mounted Display Head Mounted Projected Display

Pico Projectors

  Microvision - www.mvis.com   3M, Samsung, Philips, etc

MIT Sixth Sense

  Body worn camera and projector   http://www.pranavmistry.com/projects/sixthsense/

Other AR Displays

Video Monitor AR

Video cameras Monitor

Graphics Combiner

Video

Stereo glasses

Examples

Virtual Showcase

  Mirrors on a projection table   Head tracked stereo   Up to 4 users   Merges graphic and real objects   Exhibit/museum applications

  Fraunhofer Institute (2001)   Bimber, Frohlich

Augmented Paleontology

Bimber et. al. IEEE Computer Sept. 2002

Alternate Displays

LCD Panel Laptop PDA

Handheld Displays   Mobile Phones

 Camera  Display   Input

Display Taxonomy

Other Types of AR Display   Audio

  spatial sound   ambient audio

  Tactile   physical sensation

  Haptic   virtual touch

Haptic Input

  AR Haptic Workbench  CSIRO 2003 – Adcock et. al.

Phantom

  Sensable Technologies (www.sensable.com)   6 DOF Force Feedback Device

AR Haptic Interface

  Phantom, ARToolKit, Magellan

AR Tracking and Registration

  Registration   Positioning virtual object wrt real world

  Tracking  Continually locating the users viewpoint

-  Position (x,y,z) -  Orientation (r,p,y)

Registration

Spatial Registration

The Registration Problem   Virtual and Real must stay properly aligned   If not:

  Breaks the illusion that the two coexist   Prevents acceptance of many serious applications

Sources of registration errors   Static errors

 Optical distortions  Mechanical misalignments   Tracker errors   Incorrect viewing parameters

  Dynamic errors   System delays (largest source of error)

-  1 ms delay = 1/3 mm registration error

Reducing static errors   Distortion compensation   Manual adjustments   View-based or direct measurements   Camera calibration (video)

View Based Calibration (Azuma 94)

Dynamic errors

  Total Delay = 50 + 2 + 33 + 17 = 102 ms   1 ms delay = 1/3 mm = 33mm error

Tracking Calculate Viewpoint Simulation

Render Scene

Draw to Display

x,y,z r,p,y

Application Loop

20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms

Reducing dynamic errors (1)

  Reduce system lag   Faster components/system modules

  Reduce apparent lag   Image deflection   Image warping

Reducing System Lag

Tracking Calculate Viewpoint Simulation

Render Scene

Draw to Display

x,y,z r,p,y

Application Loop

Faster Tracker Faster CPU Faster GPU Faster Display

Reducing Apparent Lag

Tracking Update

x,y,z r,p,y

Virtual Display

Physical Display

(640x480)

1280 x 960

Last known position

Virtual Display

Physical Display

(640x480)

1280 x 960

Latest position

Tracking Calculate Viewpoint Simulation

Render Scene

Draw to Display

x,y,z r,p,y

Application Loop

Reducing dynamic errors (2)   Match input streams (video)

 Delay video of real world to match system lag

  Predictive Tracking   Inertial sensors helpful

Azuma / Bishop 1994

Predictive Tracking

Time

Position

Past Future

Can predict up to 80 ms in future (Holloway)

Now

Predictive Tracking (Azuma 94)

More Information •  Mark Billinghurst

– mark.billinghurst@hitlabnz.org •  Websites

– www.hitlabnz.org