![Page 1: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/1.jpg)
The Reality of Augmented Reality: Are we there yet?
Mark Billinghurst
The HIT Lab NZ, University of Canterbury
March 24th 2015
![Page 2: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/2.jpg)
My Childhood Home
![Page 3: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/3.jpg)
The Vision - Auckland
![Page 4: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/4.jpg)
The Road Trip
![Page 5: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/5.jpg)
Happy Trails
![Page 6: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/6.jpg)
The Reality
![Page 7: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/7.jpg)
Mount Messenger
![Page 8: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/8.jpg)
Are We There Yet?
![Page 9: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/9.jpg)
Vision vs. Reality
![Page 10: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/10.jpg)
Lesson’s Learned ! Have a clear driving vision ! Enjoy the journey ! Travel with others ! Have a well equipped vehicle ! The end always seems nearer than it really is
![Page 11: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/11.jpg)
THE AR VISION
![Page 12: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/12.jpg)
The Ultimate Display The ultimate display would .. be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs .. would be confining, and a bullet .. would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked.
Ivan E. Sutherland
Sutherland, I.E. The ultimate display. Information Processing 1965, Proc. IFIP Congress 65, 506-508.
![Page 13: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/13.jpg)
Realizing the Vision ! 3D computer graphics ! Physical input devices ! Speech interaction ! Eye-gaze input ! Kinesthetic force feedback
Sutherland, I. E. (1968, December). A head-mounted three dimensional display. In Proceedings of the December 9-11, 1968, fall joint computer conference, part I (pp. 757-764). ACM.
![Page 14: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/14.jpg)
Star Wars (1970’s-80’s)
! 3D projection ! Collaborative systems
![Page 15: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/15.jpg)
The Terminator (1984 - )
! Scene recognition ! Wearable AR HUD
![Page 16: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/16.jpg)
The Super Cockpit (1980’s)
! Furness - USAF
Furness, T. A. (1986, September). The super cockpit and its human factors challenges. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 30, No. 1, pp. 48-52). SAGE Publications.
![Page 17: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/17.jpg)
Star Trek (1987 – 94)
! HoloDeck, Sensory glasses ! Ubiquitous Computing
![Page 18: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/18.jpg)
Minority Report (2002)
! Natural gesture interaction
![Page 19: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/19.jpg)
Ironman (2008-)
! Natural gesture input ! Wearable AR system
![Page 20: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/20.jpg)
Trends ! 1960’s – 70’s: Display of Information
! Ultimate Displays, Star Wars
! 1980’s Information Overlay ! Terminator, Super cockpit
! 1990’s Environmental Enhancement ! Star Trek HoloDeck
! 2000’s Natural Interaction, Wearable Systems ! Minority Report, Iron-man
![Page 21: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/21.jpg)
ARE WE THERE YET?
![Page 22: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/22.jpg)
2012 – Iron Man 2
![Page 23: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/23.jpg)
To Make the Vision Real.. ! Automatically detecting real environment
! Environmental awareness, Physically based interaction
! Gesture interaction ! Free-hand interaction
! Multimodal input ! Speech and gesture interaction
! Intelligent interfaces ! Implicit rather than Explicit interaction
![Page 24: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/24.jpg)
Environmental Awareness ! AR MicroMachines
! AR experience with environment awareness and physically-based interaction
! Based on MS Kinect RGB-D sensor
! Augmented environment supports ! occlusion, shadows ! physically-based interaction between real and
virtual objects
Clark, A., & Piumsomboon, T. (2011). A realistic augmented reality racing game using a depth-sensing camera. In Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry (pp. 499-502). ACM.
![Page 25: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/25.jpg)
Physics Simulation
! Create virtual mesh over real world
! Update at 10 fps – can move real objects
! Use by physics engine for collision detection (virtual/real)
! Use by OpenScenegraph for occlusion and shadows
![Page 26: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/26.jpg)
Rendering
Occlusion Shadows
![Page 27: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/27.jpg)
Natural Hand Interaction
! Using bare hands to interact with AR content ! MS Kinect depth sensing ! Real time hand tracking ! Physics based simulation model
![Page 28: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/28.jpg)
Skeleton Based Interaction
! 3 Gear Systems ! Kinect/Primesense Sensor ! Two hand tracking ! http://www.threegear.com
![Page 29: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/29.jpg)
AR Rift Display
![Page 30: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/30.jpg)
Skeleton Interaction + AR
! HMD AR View ! Viewpoint tracking
! Two hand input ! Skeleton interaction, occlusion
Piumsomboon, T., Altimira, D., Kim, H., Clark, A., Lee, G., & Billinghurst, M. (2014, September). Grasp-Shell vs gesture-speech: A comparison of direct and indirect natural interaction techniques in augmented reality. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (pp. 73-82). IEEE.
![Page 31: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/31.jpg)
Multimodal Interaction ! Combined speech and gesture input ! Gesture and Speech complimentary
! Speech: modal commands, quantities ! Gesture: selection, motion, qualities
! Previous work found multimodal interfaces intuitive for 2D/3D graphics interaction ! However, few multimodal AR interfaces
![Page 32: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/32.jpg)
Free Hand Multimodal Input
! Use free hand to interact with AR content ! Recognize simple gestures
! Open hand, closed hand, pointing
Point Move Pick/Drop
Lee, M., Billinghurst, M., Baek, W., Green, R., & Woo, W. (2013). A usability study of multimodal input in an augmented reality environment. Virtual Reality, 17(4), 293-305.
![Page 33: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/33.jpg)
Multimodal Architecture
![Page 34: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/34.jpg)
Experimental Setup
Change object shape and colour
![Page 35: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/35.jpg)
Results - Performance ! Average performance time
! Gesture: 15.44s ! Speech: 12.38s ! Multimodal: 11.78s
! Significant difference across conditions (p < 0.01) ! Difference between gesture and speech/MMI
![Page 36: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/36.jpg)
Subjective Results (Likert 1-7)
! User subjective survey ! Gesture significantly worse, MMI and Speech same ! MMI perceived as most efficient
! Preference ! 70% MMI, 25% speech only, 5% gesture only
Gesture Speech MMI
Naturalness 4.60 5.60 5.80
Ease of Use 4.00 5.90 6.00
Efficiency 4.45 5.15 6.05
Physical Effort 4.75 3.15 3.85
![Page 37: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/37.jpg)
Lessons Learned ! Multimodal interaction significantly better than
gesture alone in AR interfaces for 3D tasks ! Shorter task time, more efficient
! Multimodal input was more natural, easier, and more effective that gesture/speech only ! Simultaneous input rarely used
! More studies need to be conducted ! What gesture/speech patterns? Richer input
![Page 38: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/38.jpg)
Intelligent Interfaces
! AR interface + intelligent tutoring system ! ASPIRE constraint based system (from UC) ! Constraints
- relevance cond., satisfaction cond., feedback
Westerfield, G., Mitrovic, A., & Billinghurst, M. (2013). Intelligent Augmented Reality Training for Assembly Tasks. In Artificial Intelligence in Education (pp. 542-551). Springer Berlin Heidelberg.
![Page 39: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/39.jpg)
Intelligent Feedback
! Actively monitors user behaviour ! Implicit vs. explicit interaction
! Provides corrective feedback
![Page 40: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/40.jpg)
Evaluation Results ! 16 subjects, with and without ITS ! Improved task completion
! Improved learning
![Page 41: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/41.jpg)
MagicLeap (2014)
! How much is real?
![Page 42: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/42.jpg)
! Digital Imperceptible from the Real World ! Multi-sensory Display ! Natural unencumbered input ! Seamless blend between real and virtual
! Huge technical and non-technical challenges
The Ultimate AR Display
![Page 43: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/43.jpg)
AR Research Trends ! Zhou 10 year ISMAR survey (1998-2008)
! 276 papers reviewed
! Most researched topics ! (1) Tracking techniques (20%) ! (2) Interaction techniques (15%) ! (3) Calibration and registration (14%) ! (4) AR applications (14%) ! (5) Display techniques (12%)
Feng Zhou, Henry B.L. Duh, Mark Billinghurst. Trends in Augmented Reality Tracking, Interaction and Display: A Review of Ten Years of ISMAR. In proceedings of ISMAR 2008, Cambridge, UK, 15-18th of Sept. 2008.
![Page 44: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/44.jpg)
Tracking
2005 2015
Marker Based ARToolKit Single Sensor Local
Object Based Vuforia Smart Terrain Hybrid Systems Wide area
![Page 45: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/45.jpg)
Input
2005 2015
Tangible AR Custom controller Sensor based
Natural gesture Mobile touch Vision based
![Page 46: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/46.jpg)
2005 2015
Video See-through Desktop Projected Custom
Optical See-through Wearable Handheld Consumer
Display
![Page 47: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/47.jpg)
Rendering
2005 2015
Simple graphics No environmental effects No lighting capture
Photorealistic Scene capture/shading Real time lighting
![Page 48: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/48.jpg)
Better Tools Than Every Before ! Cheap/Free Software
! Vuforia, Junaio, etc
! Wide Range of Displays ! AR, VR, Projected, etc
! Input Devices ! Eye-tracking, depth sensing, body measuring
! New Funding Models ! Kickstarter, Indiegogo, Investor
![Page 49: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/49.jpg)
Research Gaps ! Least Researched Topics
! (6) Evaluation/testing (6%); ! (7) Mobile/wearable AR (6%); ! (8) Visualization (5%); ! (9) AR authoring (4%); ! (10) Multimodal AR (3%); ! (11) Rendering (2%).
![Page 50: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/50.jpg)
Research Opportunities ! Tracking
! 3D scene capture, ubiquitous tracking, middleware
! Display ! Projection displays, retinal displays, contact lens
! Interaction ! Collaborative, Ubiquitous computing, Tangible Input
![Page 51: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/51.jpg)
Is AR the Right Tool ?
! Which tasks are suitable for AR? ! Eg AR vs Map for outdoor navigation
![Page 52: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/52.jpg)
No One Wants to be a Glasshole
! Social impact of AR technology ! Privacy, Social Interactions, Cultural Impact
![Page 53: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/53.jpg)
MY VISION
![Page 54: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/54.jpg)
! sdfs
![Page 55: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/55.jpg)
![Page 56: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/56.jpg)
! sdfgs
![Page 57: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/57.jpg)
! axcvxca
![Page 58: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/58.jpg)
Vision
“Using technology to create shared emotional experiences between users
and so create a deeper sense of empathy and understanding”
![Page 59: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/59.jpg)
![Page 60: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/60.jpg)
Empathic Computing 1. Computing systems that can understand your feelings and emotions 2. Computing systems that help you better understand the feelings of others
![Page 61: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/61.jpg)
Affective Computing
! Ros Picard – MIT Media Lab ! http://affect.media.mit.edu
![Page 62: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/62.jpg)
Appliances That Make You Happy
! Jun Rekimoto – Univ. Tokyo ! Smile detection + smart devices
![Page 63: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/63.jpg)
Can we develop interfaces that allow us to be more empathetic to others?
![Page 64: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/64.jpg)
Movies are like a machine that generates Empathy
Roger Ebert
![Page 65: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/65.jpg)
Empathy Computing Requirements
! Basic Requirements ! Making the technology transparent
! Empathy Definition ! Seeing with the eyes of another ! Hearing with the ears of another ! Feeling with the heart of another
![Page 66: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/66.jpg)
Using AR for Empathy
! Augmented Reality can: ! Remove technology barriers ! Enhance communication ! Change perspective ! Share experiences ! Enhance interaction in real world
![Page 67: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/67.jpg)
Wearable Computer
▪ Wearable computer – Ego-Vision view ▪ camera + processing + display + connectivity
![Page 68: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/68.jpg)
Current Collaboration on Wearables
! First person remote conferencing/hangouts ! Limitations
! Single POV, no spatial cues, no annotations, etc
![Page 69: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/69.jpg)
Sharing Space: Social Panoramas
! Capture and share social spaces in real time ! Enable remote people to feel like they’re with you
![Page 70: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/70.jpg)
Technology
! Google Glass ! Image capture, viewpoint sharing
! Remote device (desktop, tablet) ! Immersive viewing, live annotation
![Page 71: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/71.jpg)
Key Research Questions
! Where is my partner looking? ! Enhanced radar display, Context compass
! How can we interact together? ! Shared pointers, Shared drawing
![Page 72: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/72.jpg)
Sensor Systems/Software
! Different systems developed/evaluated ! SSI framework (emoVoice, etc) ! e-Health, bitalino, arduino bio-sensors ! Wearscript wearable eye-tracker
!
!
![Page 73: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/73.jpg)
Approach Data Capture
Feature Detection
Emotion Recognition
Emotion Representation
Empathic User Interface
Hardware
User Interface
![Page 74: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/74.jpg)
Wearable Interface
! Google Glass + e-Health + Spydroid + SSI ! Measure GSR, pulse oxygen, ECG, pitch ! Share video and audio remotely ! Represent emotions back to Glass user (4 states)
! !
![Page 75: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/75.jpg)
Desktop Interface
! Live video, real time emotion data ! See what sender sees, emotion representation
!
![Page 76: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/76.jpg)
Emotion Representation
! How can we show what you’re feeling? ! Tested - Raw data, Visual tinting, Emotion labeling
!
! !Excited Happy
![Page 77: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/77.jpg)
Early Results ! Colour overlay + video stream preferred
! Easier to understand
! Disagreement over best colour coding ! Different emotional response with diff. stimuli
! Scary movie best
! Remote users felt connected to local user ! Understanding of emotions
![Page 78: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/78.jpg)
Table 1. Correlation Analysis between the local users and the remote users
![Page 79: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/79.jpg)
Analysis
Table 2. Correlation between different streaming scenarios obtained from the feedback of the local and remote users
Experiment 3 (Creating a shared experience while watching an audio Visual Watching an audio visual)
Happy Sad Neutral Excited Case1
(Only video stream) 0.77 0.84 0.85 0.85 Case 2 (Case 1 + Color Overlay) 0.92 0.91 0.85 0.98
Case 3 (Case 2 + Graphical Analysis) 0.86 0.75 0.85 0.94
![Page 80: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/80.jpg)
Capturing Space: Real World Capture
! Hands free AR ! Portable scene capture (color + depth)
! Projector/Kinect combo, Remote controlled pan/tilt
! Remote expert annotation interface
![Page 81: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/81.jpg)
Remote Expert View
![Page 82: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/82.jpg)
Ghostman
! Use AR to capture and overlay your actions into remote persons space ! Eg remote therapy
![Page 83: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/83.jpg)
Looking to the Future
What’s Next?
![Page 84: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/84.jpg)
Brain to Brain Control
! Rajesh Rao, University of Washington ! First Brain to Brain control
![Page 85: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/85.jpg)
System Architecture
![Page 86: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/86.jpg)
Scaling Up
! Seeing actions of millions of users in the world ! Augmentation on city/country level
![Page 87: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/87.jpg)
AR + Smart Sensors + Social Networks
! Track population at city scale (mobile networks) ! Match population data to external sensor data
! medical, environmental, etc
! Mine data to improve social services
![Page 88: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/88.jpg)
![Page 89: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/89.jpg)
![Page 90: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/90.jpg)
Research Challenges ! How to convey emotion? ! How to measure empathy? ! Interface/interaction models? ! How to communicate emotion? ! Scaling up to city/country scale?
![Page 91: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/91.jpg)
CONCLUSIONS
![Page 92: The Reality of Augmented Reality: Are we there yet?](https://reader034.vdocuments.us/reader034/viewer/2022042817/55a626451a28ab0c3c8b48a1/html5/thumbnails/92.jpg)
Take Home Messages ! Have a Vision ! Find people to travel with ! Equip the vehicle ! Enjoy the journey
Most Important: We’re nowhere near the end of interesting research in Augmented Reality