FAR: Fachgebiet Augmented Reality Technische Universität München

Head Mounted DisplaysHow to realize ultimate AR experiences?

Yuta Itoh (TUM)

Workshop Visualisierung großer Datenmengen, 9th June 2015

1

FAR: Fachgebiet Augmented Reality

Prof. Gudrun Klinker

2/52

Industrial Augmented Reality3/52

Head-Mounted Displays (HMDs)

Optical See-Through (OST-HMD)

Non Optical See-Through

4/52

OST-HMD in science fiction

From Ironman, 2008 ©Marvel Entertainment

5/52

Everything is perfect in Science Fiction

World sensing

Image recognition

Eye tracking

6/52

7/52

Arianespace, 2009Le Voyage dans la Lune, 1902

1902 20098/52

What is the current statusof the OST-HMD technology?

What’s still missing?

9/52

Hugo Gernsback, TV Glasses

196310/52

Ivan Sutherland, The Sword of Damocles

196811/52

2013-12/52

13/52

14/52

OST-HMDs Overwrite the reality

NVIS nVisor ST60

15/52

Related Work | AR X-ray vision

Avery, Ben, Christian Sandor, and Bruce H. Thomas. "Improving spatial perception for augmented reality x-ray vision." Virtual Reality Conference, 2009. VR 2009. IEEE. IEEE, 2009.

16/52

Related Work | Visual Assistance

Hwang, Alex D., and Eli Peli. "An augmented-reality edge enhancement application for Google Glass." Optometry & Vision Science 91.8 (2014): 1021-1030.

17/52

Issues in OST-HMDs

18/52

Design Factor (non-OST)

Toshiba, 2006 Oculus Rift DK2, 2014Photo from IFIXIT

19/52

Design Factor

Google Glass, 2013Photo by Mikepanhu

20/52

Wide Field of View

“Pinlight Displays: Wide Field of View Augmented Reality Eyeglassesusing Defocused Point Light Sources” Maimone et al., TOG’14

21/52

Wide Field of ViewIssues on OST-HMD

“Pinlight Displays: Wide Field of View Augmented Reality Eyeglassesusing Defocused Point Light Sources” Maimone et al., TOG’14

22/52

Consistency issues in OST-HMDs

Temporal

[Zheng ’14] [Didier ’05]

Visual

[Kiyokawa ’01] [Liu ’08] [Lee ’09]

Spatial

[Azuma ’95] [Tuceryan ’00] [Genc ’02]

Social

Image from Google Glass: Don't Be A Glasshole | Mashable

23/52

Temporal Inconsist. in OST-HMD

“How fast is fast enough? : a study of the effects of latency in direct-touch pointing tasks” Jota et al. CH’13

“latencies down to 2.38 ms are required to alleviate user perception when dragging”

24/52

Temporal Inconsist. in OST-HMDDigital Light Processing Projector

“Minimizing Latency for Augmented Reality Displays: Frames Considered Harmful” Zheng et al. ISMAR’14

25/52

Visual ConsistencyOcclusion, Depth, Color, Shadow, etc..

Lee and woo, 2009

Kiyokawa et al. 2003Liu et al. 2008

26/52

Occlusion (physical “Z-buffer”)27/52

Depth (eye accommodation)

Liu et al. 2008

28/52

Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through-kalibrierung fuer mobile augmented reality assistenzsysteme. Michael Schenk, Hrsg, 6(7).

Spatial Calibration of OST-HMDs29/52

Spatial Calibration of OST-HMDs30/52

Eye-HMD Calibration

3D

2D

31/52

P is a Black Box

Find 3D-2D Projection:

3D2D

32/52

2D

Say, is a Perspective Projection33/52

Find 3D-2D Projection:

In the Eye of the Beholder…

= K*[R t]

34/52

Manual alignment– Intensive user interaction– User-dependent noise

Predictive tracking for augmented realityRT Azuma - 1995 - ronaldazuma.com

35/52

Manual alignmentSPAAM: Single Point Active Alignment Method

– Medium user interaction– User-dependent noise

Single point active alignment method (SPAAM) for optical see-through HMD calibration for ARM Tuceryan, N Navab - Augmented Reality, 2000.(ISAR 2000). …, 2000 - ieeexplore.ieee.orgPractical solutions for calibration of optical see-through devicesY Genc, M Tuceryan, N Navab - … of the 1st International Symposium on …, 2002 - dl.acm.org

N times

36/52

SPAAM (Manual Calibration) video37/52

Idea: user-perspective rendering

Spindler, Martin, Wolfgang Büschel, and Raimund Dachselt. "Use your head: tangible windows for 3D information spaces in a tabletop environment."Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces. ACM, 2012.

38/52

Utilizes 3D Eye Localization– Interaction-free, thus do not bother users– More accurate than a realistic SPAAM setup

Automated alignement

Yuta Itoh, and Gudrun Klinker. "Interaction-free calibration for optical see-through head-mounted displays based on 3d eye localization." 3D User Interfaces (3DUI), 2014 IEEE Symposium on. IEEE, 2014.

39/52

1. Estimate a 2D iris ellipse– Iris detector + Fitting by RANSAC

2. Back project it to 3D circle

Robust real-time pupil tracking in highly off-axis imagesL Świrski, A Bulling, N Dodgson - … of the Symposium on Eye Tracking …, 2012 - dl.acm.org

3D Eye Position Estimation

Image-based Eye Pose and Reflection Analysis for Advanced Interaction Techniques andScene UnderstandingC Nitschke, A. Nakazawa, H. Takemura - CVIM,[ …, 2011 - ci.nii.ac.jp

40/52

Estimated 3D Eye Position[m]

>10 cm

41/52

Wearable eye-trackers

Pupil Labs Tobii

42/52

OST-HMD + eye trackers?43/52

OST-HMD + eye trackers?

A compact, eye-tracked optical see-through head-mounted displayH Hua, C Gao - Proc. SPIE, 2012 - ebooks.spiedigitallibrary.org

44/52

Pupil Center Cornea Reflection

Villanueva ‘08

Cornea reflection

45

HMD Screen resembles a plane

HMDImageScreen

46

Corneal Reflection47

Detected Checkerboard in 2D

Detected grid corners

48

Scene Reflection in the Eye49

Detected and ProjectedCorners

Reprojected Points (3D 2D)

Original Image

50

Estimated Cornea Position

Limbus tracking Corneal Imaging

Plopski, Alexander, et al. "Corneal-imaging calibration for optical see-through head-mounted displays." Visualization and Computer Graphics, IEEE Transactions on 21.4 (2015): 481-490.

51

Optical See-Through HMDs are hard fun

• Consistency issues(Spatial, visual, temporal, etc.)

• Eye sensing will be a hot topic• Big players => big markets

Takeaways:52/52

Appendix

53/52

An ideal eye-HMD system

Eye

Image source

Environment

Combiner (a half mirror)

54/52

HMD-dependent parameters

User-dependent parameters

Image by Soerfm Photo by Laitr Keiows

55

• Shipments of smart watches&glasses– 130 million by 2018

• 10 times than 2013

Dawn of a Commercial Wearable Devices56/52

Innovega iOptik57/52

SimpleNo user interaction

Accurate better than Degraded manual calibrations

Summary of INDICACalibration of OST-HMDs using

3D eye position

58/52

Eye Position Estimation

E<

C1

Eyeballsphere

C3Eyeballorbit

g2

g3

g1

C2

59