holo basics develop your mixed reality hololens app with unity and visual studio
TRANSCRIPT
DEVELOP YOUR MIXED REALITY APP FOR HOLOLENS
WITH UNITY AND VISUAL STUDIO
HOLOBASICS
“HoloLens Evangelist”
At ETTU I am responsible architecture related business for our project services
department at ETTU. I’m involved as architect at different companies like Planon,
PGGM, Gemeente Amersfoort and others
Alexander Meijers
ABOUT ME Solutions Architect for ETTU
Founder of the Mixed Reality User Group
MixUG: http://www.mixug.nl
Email: [email protected] or [email protected]
Twitter: @ameijers
Blog: http://www.appzinside.com
3INTRO ETTU
TODAY’S TALKAGENDA Mixed Reality
Microsoft HoloLens
Tools
Our first hologram
Sound
Gestures
Spatial Mapping
Occlusion
Slider
Content Ads
MIXED REALITY
HOLOBASICS 5
VIRTUAL REALITY
Generation of realistic images, sounds and other sensations that replicate a real environment or create an imaginary setting
An immersive experience created entirely from computer-generated Content. Also similar to 360 degree video
HOLOBASICS 6
AUGMENTED REALITY
Direct or indirect view of a physical, real-worldenvironment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data
An overlay of content on the real world where that content is not anchored to or part of it
HOLOBASICS 7
MIXED REALITY
Merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time.
Overlay of synthetic content on the real world that is anchored to and interacts with the real world
MICROSOFT HOLOLENS
HOLOBASICS 9
HISTORY OF HOLOLENS
Codenamed Project HoloLens
Chief inventor Alex Kipman Official title is technical fellow
He dreamed up Kinect in at the end of 2007 and set a vision which incorporated HoloLens
It is the start of a transforming world
In the new reality, sensors will be anywhere
A visual computing platform controlled by speech and gesture
Satya Nadella - “The emergence of the next computing interface”
Alex Kipman
HOLOBASICS 10
MICROSOFT HOLOLENSSPECIFICATIONS
Windows 10 device based on 32 bit architecture
64GB flash
2GB memory
1GB Holographic Processor Unity (HPU)
First of its kind
Device is more powerfull than a laptop
No overheating due to warm air flows to the sides
2-3 hours active and 2 weeks standby
Weight 579g
HOLOBASICS 11
SENSORS, OPTICS ANDSPEAKERS
Environment camera
Depthcamera
Videocamera
Spatial SoundWith
speakers
High definitionlenses
HOLOBASICS 12
MICROSOFT HOLOLENS
First Holographic computer running on Windows 10
Contains a CPU, GPU and HPU
Human understanding
Gesture and voice input
Gaze tracking
Spatial sound
Uses wi-fi to connect and has no wires attached
Design for comfort
HOLOBASICS 13
MICROSOFT HOLOLENSSPECIFICATIONS
Contains depth camera Field of vision that spans 120 by 120 degrees
18 sensors flooding the device with terabytes of data every second
Tricks your brain into perceiving holographic images Light engine in which light particles are bouncing million times
Photons enter the two lenses
Ricochet some between layers of blue, green and red glass
Hitting back of the eye
HOLOBASICS 14
HOLOLENS EXAMPLES
HOLOBASICS 15
HOLOLENS EXAMPLES
HOLOBASICS 16
HOLOLENS EXAMPLES
HOLOBASICS 17
HOLOLENS EXAMPLES
HOLOBASICS 18
HOLOLENS EXAMPLES
“DEMONSTRATION OF HOLOLENSAPPLICATIONS”
“HoloBasics by Alexander Meijers”
TOOLS
HOLOBASICS 21
DEVELOPMENT TOOLS
Visual Studio 2017 UWP workload
Game development with Unity workload
Windows 10 SDK (version 1511 or later) There is no separate SDK for HoloLens
Unity 5.6 or Unity 2017.1 Be aware! - Unity 2017.2 beta for immersive headsets
HoloLens emulator HoloLens Emulator (build 10.0.14393.1358)
Hyper-V
Contains DirectX project templates for Visual Studio
HoloLens device
HOLOBASICS 22
DEVELOPMENT TOOLSGITHUB
Microsoft/MixedRealityToolkit-Unity
Ameijers/HoloLens-Unity-Utilities
HOLOBASICS 23
BUILD LIFECYCLE OF AHOLOLENS PROJECT
Create Unity Project Configure HoloLens settings Create scene
Visual Studio
Unity HoloLens
Configure build settings Build and generate Visual Studio
project
Open project with Visual Studio Pair with HoloLens Build & deploy Visual Studio project
Start Application Test & debug Monitor
Unity
HOLOBASICS 24
DEVELOPMENT FORHOLOLENS
HoloLens Device Developer mode needs to be turned on
Pair the device with your development machine
Retrieve IP address of HoloLens Device Ask Cortana
Settings Network & Internet Advanced options
HOLOBASICS 25
WINDOWS DEVICE PORTAL
3D View
Mixed Reality Capture
Performance Performance tracing
System performance
Processes
Apps Maintenance
Crash dumps
Additional tools Logging
File Explorer
Virtual Input
HOLOBASICS 26
UNITYEDITOR
MainCamera
DirectionalLight
Game Objects
Assets
Components
Script
Scene
HOLOBASICS 27
UNITYPREFABS
• Predefined Game Objects
• Contains components with predefined values
• Prefabs used in Scene inherit the components and settings
HOLOBASICS 28
UNITYGAMEOBJECT SCRIPT
• Derived from MonoBehaviour
• Contains methods which are called by Unity through the a broadcast messaging system
• void Awake() – Called when class is instantiated
• void Start() – Called when object is enabled once before the first Update()
• void Update() – Called once per frame
• More methods are available• FixedUpdate, LateUpdate, OnGUI, OnDisable, OnEnabled
HOLOBASICS 29
UNITYMESSAGE SYSTEM
public void BroadcastMessage(string methodName, object parameter = null, SendMessageOptions options = SendMessageOptions.RequireReceiver);
• Used for calling methods on GameObjects
• Calls the methods also on its Children
• Examples
• this.BroadcastMessage(“OnSelect”);
• focusedObject.BroadcastMessage(“OnSelect”);
HOLOBASICS 30
UNITYCOMPONENTS
• Transform• Position of object
• Rotation of object
• Scale (size) of object
• Mesh Renderer & Mesh Filter• Renders the object
• Mostly done using a mesh
• Mesh Collider• Determines the collision for the object
• Script• Perform actions on the object
• Handle messages on an object
HOLOBASICS 31
TOOLS3D MODELING
Blender
Studio 3DS
Max
Many others like
Maya, Rhino3D
Paint 3D
OUR FIRST HOLOGRAM
HOLOBASICS 33
COORDINATE SYSTEMHOLOLENS AND UNITY
forward (x, y, z)Camera (0, 0, 0)Rotation (x, y, z)
Camera (1, -1, 4)Rotation (x, y, z)
HologramObject Pinned
HoloLensmoved to
new location
View directionGaze
HOLOBASICS 34
HOLOGRAMS AND PINNING
Device can project a hologram into a room and keep it locked in position
Called pinnnig
Objects are not moving relative to you but you can move around the object
Allows you to view the object from any angle
“DEMONSTRATION OF PINNED HOLOGRAM”
“HoloBasics by Alexander Meijers”
HOLOBASICS 36
PINNED HOLOGRAMSUMMARY
Basics of Unity and 3D objects
Building an app using prefabs, components and script
Build lifecycle of a HoloLens application using Unity and Visual Studio
Pinned hologram robot in the room continue facing the HoloLens wearer
SPATIAL SOUND
HOLOBASICS 38
SPATIAL SOUNDOVERALL
• Simulates 3D sound using direction, distance and environmental simulations
• Above, below, behind, to the side, etc.
• Attach sound to holographic objects• Works also when object is not in line of
sight
• Used to draw attention
• Audio engine in HoloLens• CPU and memory considerations
• 10-12 spatial sound voices
HOLOBASICS 39
SPATIAL SOUNDTYPES
• Gaze mixing• Highlighting objects
• Play a sound on the object to get the users attention
• Audio Haptics• Reactive audio for touchless interactions
• Play a sound when user selects an object or when his hands appear inside the gesture frame
• Immersion• Ambient sounds surrounding the user
• Support your scene with background sounds or music
HOLOBASICS 40
SPEECHCORTANA
• Start with “Hey Cortana”
• Some thing you can say• Move <app name> here
• Take a picture
• Start or stop recording
• Call <contact>
• How much battery do i have left?
• Restart or Shutdown
• Cortana can be turned on/off
“DEMONSTRATION OF SPATIAL SOUND ADDED TO HOLOGRAM”
“HoloBasics by Alexander Meijers”
HOLOBASICS 42
PINNED HOLOGRAMSPATIAL SOUND
Unity and audio sources
Creating a spatial sound with an audio source
Explained how the speakers of HoloLens adjust based on the location of the sound
GESTURES
HOLOBASICS 44
GESTURESGAZE
• The direction of the HoloLens pointing is called the gaze
• Hits an object or part of the environment
• Uses a Vector3 based value to define its so called “forward” direction
HOLOBASICS 45
GESTURESGESTURE RECOGNIZER
• Recognize input by tracking the position of either or both hands
• Gesture frame (Frustum)
• Recognized input from hands• Bloom
• Press, hold and release
• Following your hand
• Gestures• Hold
• Manipulation
• Navigation
HOLOBASICS 46
GESTURESGESTURE RECOGNIZER
• Connect commands to gestures• GestureRecognizer gestures = new GestureRecognizer();
• gestures.TappedEvent += (source, tapCount, ray) => { … };
• gestures.StartCapturingGestures();
• Events• Tapped
• Select press and release
• Hold (Started, Completed, Canceled)
• Select press beyond system hold threshold
• Navigation (Started, Updated, Completed, Canceled)• Select press with relative movement
• Velocity based continuous scrolling or zooming
• Virtual Joystick
• Manipulation (Started, Updated, Completed, Canceled)
• Select press with absolute movement
• Move, resize or rotate hologram
HOLOBASICS 47
GESTURESSPEECH RECOGNIZER
• Use words or sentences to control your environment
• English language only at the moment
• Connect commands to spoken text• Dictionary<string, System.Action> keywords = new Dictionary<string,
System.Action>();
• keywords.Add(“some text”, () => { … };
• KeywordRecognizer keywords = new KeywordRecognizer(keywords.Keys.ToArray());
• keywordRecognizer.OnPhraseRecognized += OnPhraseRecognized;
• keywordRecognizer.Start();
• OnPhraseRecognized( PhaseRecognizedEventArgs args);• Invoke the action
• keywords[args.text].Invoke();
“DEMONSTRATION OF GESTURES BASED ON HAND AND SPEECH TO INTERACT WITH
HOLOGRAM”
“HoloBasics by Alexander Meijers”
HOLOBASICS 49
PINNED HOLOGRAMGESTURES WITH HAND AND SPEECH
Implemented code for handling the gaze and cursor
Implemented code for adding gestures
Implemented code for adding speech
Control the robot by shooting at it with a click gesture
Control the scene by using words and sentences like “reset”
SPATIAL MAPPING
HOLOBASICS 51
SPATIAL PERCEPTIONHOW DOES IT WORK?
• Spatial perception and mapping
• Rooms
• Different ways of detecting and working with spatial perception
• Device scans your environment and builds a digital model in real time
• It allows HoloLens to see different surfaces, walls and ceiling
• Meshes & planes
• Possible to simulate a physical space• E.g. Projecting a terrain over your floor
Spatial perception is the ability to be aware of
your relationships with the environment around
you and with yourself
HOLOBASICS 52
SPATIAL MAPPING ANDUNDERSTANDING
Device scans your environment and builds a digital model in real time
It allows HoloLens to see different surfaces, walls and ceiling
Possible to simulate a physical space E.g. Projecting a terrain over your floor
“DEMONSTRATION OF LIVE SPATIAL MAPPING AND UNDERSTANDING BY HOLOLENS”
“HoloBasics by Alexander Meijers”
HOLOBASICS 54
PINNED HOLOGRAMSPATIAL MAPPING UNDERSTANDING
Windows Device Portal connected to the HoloLens device
Mapping the room by walking around
Show mapping in the Windows Device Portal
HOLOBASICS 55
SPATIAL MAPPING ANDUNDERSTANDING
Spatial Mapping components in Unity Spatial Mapping Renderer script
Spatial Mapping Collider script
Use an empty GameObject in your scene
Spatial Mapping render state None
Visualization
Occlusion
Cursor on spatial mapping SpatialMappingCollider.layer = 31;
Cursor uses Raycast with layer mask 1<< 31
Rendering material Occlusion material
Visual material
“DEMONSTRATION OF SPATIAL MAPPING USED WITH HOLOGRAM”
“HoloBasics by Alexander Meijers”
HOLOBASICS 57
PINNED HOLOGRAMSPATIAL MAPPING
Using the component SpatialMappingRenderer and SpatialMappingCollider in Unity
Show spatial mapping by using the HoloDeck material/shader
SPATIAL MAPPING & OCCLUSION
HOLOBASICS 59
OCCLUSIONCULLING
Process of determining which surfaces and parts are not visible from a certain viewpoint
Create a more naturally feeling with mixed reality
Holograms can be occluded by Other holograms
Real-life objects in your environment which are spatially mapped
Culling enforces the HoloLens not to (partially) render the object To speed up the rendering process
Objects which are too far away are left out the view (frustum)
Use Spatial Mapping Renderer Set Occlusion material
Set SpatialMappingRenderer.renderState = RenderState.Occlusion;
“DEMONSTRATION OF SPATIAL MAPPING AND OCCLUSION WITH HOLOGRAM FOR TRUE
MIXED REALITY”
“HoloBasics by Alexander Meijers”
HOLOBASICS 61
PINNED HOLOGRAMSPATIAL MAPPING AND OCCLUSION
Using the component SpatialMappingRenderer and SpatialMappingCollider in Unity
Change the way spatial mapping is done for None, Visualization and Occlusion
Placed another robot behind a real-life object and explain occlusion
HOLOBASICS 62
WRAP-UPTAKAWAYS
Unity and Visual Studio are the tools when it comes to developing mixed reality apps for HoloLens
Know your tools like the Windows Device Portal
Understand physics when creating an app for HoloLens
Lot’s of stuff to make your life easy for gestures, speech, spatial mapping and other
Understand the basics and how it works
Lot of stuff available on GitHub like the MixedReality Toolkit to make your life easier
THANKS FOR WATCHING!
HoloBasics code on gitHub - https://github.com/ameijers/HoloBasics
Kampenringweg 45b, Gouda
0182-686 000
[email protected] / [email protected]
Contact me:
facebook.com/alexandermeijers.5
@ameijers
Follow me on: