building the matrix: your first vr app (svcc 2016)

1

BUILDING THE MATRIX: YOUR FIRST VR APP@MISSLIVIROSE

2

ABOUT ME

Livi EricksonLivi.Erickson@Microsoft.com

Virtual & Augmented Reality Developer / Evangelist

VR Blog: The Matrix is my Office @MissLiviRose on Twitter

3

BUILDING VIRTUAL REALITY APPLICATIONS

Why Build for Virtual Reality? Understanding the VR Device Ecosystem Breakdown of a VR application Building a VR app: Development Options Design Considerations & Concerns for VR today Q & A

4

WHY BUILD FOR VR?

5

It’s Really, Really Fun.

6

We’re Used to 3D Interactions

7

The Freedom to Redefine Possibilities

Storytelling is significantly more immersive in an environment that surrounds the user

Virtual experiences improve empathy in the viewer

Visualizations of complex data New and improved ways of interacting with the

technical ecosystem that we’ve been building for decades

9

VR DEVICE ECOSYSTEMDESKTOP // MOBILE

PLATFORM GROWTH

2012: Oculus Developer Kit 1

2014: Oculus Developer Kit 2 Cardboard GearVR Innovator Edition 1

2015: GearVR Innovator Edition 2 Gear VR HTC Vive Developer Kit 1 OSVR

2016: Oculus Rift HoloLens Developer Kit HTC Vive Pre HTC Vive PlayStation VR FOVE Developer Kit Gear VR

TODAY’S TECHNOLOGY

Virtual Reality

Fully immersive experience that replaces your physical world

Example: Oculus Rift

Augmented Reality

Overlays digital information about the physical world around you

Example: Google Glass

Mixed Reality

Combines virtual objects and the physical world

Example: Microsoft HoloLens

12

HEAD MOUNTED DISPLAYS

Head Mounted Displays (HMDs) can be desktop-powered devices or mobile-powered devices

Desktop: Separate display device self-containing screen, content provided externally from a separate PC

Mobile: Content provided and displayed by a smart phone inserted into the casing

Standalone: Entirely self-contained headsets

13

DESKTOP VIRTUAL REALITY

Powered from external computersHardware includes separate screensTreated as an external display at the hardware levelSoftware runs on the desktop and is passed to the displayExamples: Oculus Rift, HTC Vive, FOVE

14

MOBILE VIRTUAL REALITY

Headsets are containers that support stereoscopic rendering for mobile phones Applications are typical smartphone apps The display is the phone screen Usually, almost all computational aspects are done on the phone rather than the headset Examples: GearVR, Homido, FreeFly, Cardboard

15

INPUT - CONTROLLERS

Traditional controllers don’t solve the “I can’t see my hands” issues with VR

Navigation is less natural for movement with joysticks due to the sense of being in one place

Limited number of inputs / actions Some apps may rely on specific types of

input controllers and have compatibility problems

16

INPUT - WEARABLES

Devices that are lower profile and can be worn to control various inputs without specific controller input

Example: Nod Ring Example: Myo Bracelet Example: Emotiv Headset

17

HAPTICS

Components to a VR system that provide tactile feedback based on what is going on in the virtual environment

Add to realism of a VR system by allowing the user to physically touch and experience things going on in virtual reality

Example: GloveOne (pictured) has multiple feedback points that mimic the sense of touch based on collisions in a VR game

18

MORE OPTIONS

Gestures via cameras: example: Leap Motion Controller Omnidirectional treadmills Voice commands

19

OVERVIEW OF A VR APPLICATION

20

DESIGNING & PLANNING

Consider the platforms that you’re interested in targeting Understand the input and design considerations for the given platform as you plan out your

experience Outline key scenes and emotions that you’re looking to capture in your app Choose a development option that aligns with your platform of choice

21

Render Loop

Hardware / Input

Update

Game Logic

Lifecycle of a VR App

22

Game Logic• Defines the behavior of your application

components• Models and textures are defined for your

scene• Scripts control behavior of the scene

objects• Generally doesn’t rely on the status of the

hardware, but will likely contain checks for changes

Render Loop

Hardware / Input

Update

Game Logic

23

Hardware / Input• Check on the orientation of the device to

calculate how the camera should render objects in the scene

• Evaluate if there are actions triggering different behaviors in your scene

• Update scene variables or components based on input checks

Render Loop

Hardware / Input

Update

Game Logic

24

Render Loop• Evaluate the scene appearance based on

the orientation of the headset• Apply effects based on changing scene

(shadows, lighting, etc.) dynamically to match the user’s expectation

• Draw to the headset / display• Example: If player has moved under a tree,

update the shadows cast by the player accordinglyPlayback Required Refresh

RateFilm / TV 24 – 48 FPS

Console Gaming 30 FPS

PC 50-60 Hz

VR 90-120 Hz

Render Loop

Hardware / Input

Update

Game Logic

25

DESIGNING FOR VR

26

INTERACTION & DISCOVERY

There are no standards yet in VR for expected behaviors and interfaces Capturing and guiding a user through a process is a new challenge for the medium Agency of a player is critical The amount of control or interaction a user can have in an environment will vary

dependent on experience and platform

27

PRESENCE IN VR

Consider how your user will want to interact with the environment Environment scale will impact how realistic a scene feels to a player Non-autonomous character actions will break the sense of self

28

Eyes: We’re Moving!

Body: No, we aren’t!

Body: We’re moving!

Eyes: No, we aren’t!

29

DISPLAY PERFORMANCE & CONSIDERATIONS

Low Framerates = Motion Sickness “Tearing” around the edges of the display Shadowing (duplication) of scene objects at high framerates Motion blur

30

TOOLS OF THE TRADEUNITY, UNREAL ENGINE, WEBVR (OH MY!)

31

DEVELOPMENT OPTIONS

Native development with Platform SDKs Game Engines:

Unity Unreal CryEngine

Web: WebGL VR support through WebVR Three.JS A-Frame Vizor

32

EXAMPLE NATIVE TOOLS: OCULUS SDK

Add stereoscopic, distortion, and update effects within the graphics pipeline of your application

Render to a texture specifically output to the Oculus device as well as to a non-stereoscopic screen view

Great for custom game engines adding in VR support with the Rift

More customization and control granted Versions available for desktop (Rift) and

mobile (GearVR) Source: developer.oculus.com

33

UNITY

Game Engine from Unity Technologies Visual editor for environment building & scripting Languages: C#, UnityScript (a JS derivative), Boo

34

35

UNITY: VR MODE

Added in 5.1 (current is 5.4) Requires runtime for headsets installed Supports a number of devices built-in, many others have packaged support add-ins

Implementing it:1. Edit -> Project Settings -> Player2. Under “Other Settings” check the ‘Virtual Reality Supported’ box3. Platform dependent on device type

36

UNREAL ENGINE

Game Engine from Epic Games Visual editor for environment building &

scripting Language: C++

37

38

BENEFITS OF GAME ENGINES

Handles many of the hardware-level interactions with devices Templates for different app types are available Implements physics and lighting engines Robust communities for asset creation Additional services for common application needs (networking, multiplayer, awards) Central tooling for cross-platform development

39

DOWNSIDES OF GAME ENGINES

Less granular control over application behaviors Reliant on implementation of SDKs and devices Black box code for engine functionality

40

WEB VR

Web applications with VR support Front-end Rendering:

WebGL Three.JS, Babylon.JS

Desktop & mobile support with one codebase

www.webvr.info

41

BENEFITS OF WEBVR

No installation of application is needed – run in the browser on any device Automatically support mobile and desktop devices Utilize the current set of tools and libraries for JavaScript Easily switch between VR and non-VR mode Offload interfacing with hardware to the browser

42

DOWNSIDES OF WEBVR

Performance hits for non-native graphics Non-standard support for some browsers Less insight and control into bugs with hardware Relies on browser detection of devices

43

PLANNING YOUR EXPERIENCE

44

THINK ABOUT THE FOLLOWING CONSIDERATIONS:

3D Development Scripting Animation & 3D Modeling Audio Testing

45

PLANNING A VR APPLICATION

Evaluate how the medium lends itself to your game or application Minimize the amount of locomotion Choose one platform to target at a time

Read up on best practices, but don't lock things in Prepare to iterate - a lot!

46

IMPLEMENTATION OF A VR APP - MY STRATEGY

Build out the basics in your environment so that you have a rough understanding of what your application components will do

Script the behaviors for gameplay elements & the rest of the application components Finalize the environment with lighting, textures, effects

47

TESTING YOUR VR APP

One person QA is not quite enough! Variations in height, gender, life experiences, comfort with technology, etc. will all impact

your user to a much greater extent in VR Performance is hugely important to prevent motion sickness

48

SAMPLE PROJECTSSOURCE CODE & TUTORIALS AVAILABLE!

49

SPACE PHOTOS IN SPACE

Space Photography Viewer

Built using WebVR

https://github.com/misslivirose/webvrspace

50

BUBBLE BLASTERA mini-game for the HTC Vive

Built with Unity and the SteamVR Toolkit

https://github.com/misslivirose/vive_bubbleblaster

51

STAR WARS VR TEXT EFFECT

https://github.com/misslivirose/SWScrollText

Sample Experience built with World Canvas for VR Text

Made in Unity

52

VISUAL DATA

https://github.com/misslivirose/visualdata

WebVR App for Visualizing Excel Charts

ASP.NET with Three.JS WebVR Site

53

FINAL CONSIDERATIONS

54

TODAY'S CHALLENGES

Input devices are non-standardized across platforms Constant evolution in tools Market Size

55

RESOURCES

Blog: http://livierickson.com/blog for short-form VR development tutorials, reviews, and news

GitHub: http://github.com/misslivirose for all the code in my VR experiments (including Kitten VR!)

ARVR Academy Introductory course materials: http://github.com/arvracademy/intro-academy/

Just A/VR Show: http://justavrshow.com/ Learning Unity: http://livi.link/learnunity

Andrew MollTechnical Evangelist

Doris ChenTechnical Evangelist

John AliotoPrincipal Technical Evangelist

Tobiah ZarlezTechnical Evangelist

Daniel Egan Technical Evangelist

Livi EricksonTechnical Evangelist

Tim ReillyTechnical Evangelist

Jeremy FosterTechnical Evangelist

Tierney WixtedCommunity Evangelist

Annie BubinskiCommunity Evangelist

Shaina HoustonDirector of Community

EvangelismYesenia Alvarez

Community Evangelist

Joe ShireyDirector of Technical

EvangelismBeth Massi

Program ManagerJennelle Crothers

Technical EvangelistRobin Shahan

Content DeveloperJessica Deen

Technical EvangelistJustin Chizer

Microsoft Student Partner

MEET THE TEAM

Paul HackerVisual Studio and Dev

Tech

Mathias BrandewinderVisual Studio and Dev

Tech

Deborah KurataVisual Studio and Dev

Tech

Jeremy ClarkVisual Studio and Dev

Tech

David McCarterVisual Studio and Dev

TechWard Bel

Windows DevelopmentVishal Saxena

Microsoft AzureCraig Berntson

Visual Studio and Dev Tech

Chander DhallVisual Studio and Dev

Tech

Steve EvansVisual Studio and Dev

Tech

MEET OUR MVP/RDS

• Complete the survey for a chance to win Beats by Dre Headphones

• Microsoft Azure Essentials book giveaway for survey completion while supplies last

Survey aka.ms/SVCC

• Connect with our speakers

Visit our booth

• Fill out a session card to enter a raffle for a Ricoh Theta 360 camera

Submit Your Session Cards

MAKE THE MOST OF YOUR EXPERIENCE

59

Q & A@misslivirose