it next summit - holobasics - develop your mixed reality hololens app with unity and visual studio
TRANSCRIPT
DEVELOP YOUR MIXED REALITY APP FOR HOLOLENS WITH UNITY AND VISUAL STUDIO
ALEXANDER MEIJERSARCHITECT / HOLOLENS EVANGELIST @ ETTU
“HoloLens Evangelist”
I am responsible for architecture related business for our project services
department at ETTU. I’m involved as architect at different companies like Jumbo
Maritime, PGGM, Gemeente Amersfoort and others
Alexander Meijers
ABOUT ME Solutions Architect for ETTU
Founder of the Mixed Reality User Group
MixUG: http://www.mixug.nl
Email: [email protected] or [email protected]
Twitter: @ameijers
Blog: http://www.appzinside.com
3INTRO ETTU
TODAY’S TALKAGENDA Realities explained
Microsoft HoloLens
Building blocks and tools
Tools
Gestures
Sound
Spatial Mapping & Occlusion
Slider
Content Ads
HOLOBASICS 5
REALITIESEXPLAINED
Merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interactin real time.
An overlay of synthetic content on the real world that is anchored to and interacts with the real world
MIXED REALITY
Direct or indirect view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data
An overlay of content on the real world where that content is not anchored to or part of it
AUGMENTED REALITY
Generation of realistic images, sounds and other sensations that replicate a real environment or create an imaginary setting
An immersive experience created entirely from computer-generatedContent. Also similar to 360 degree video
VIRTUAL REALITY
HOLOBASICS 7
MICROSOFT HOLOLENSSPECIFICATIONS
Windows 10 device based on 32 bit architecture
Contains CPU, GPU and HPU
1GB Holographic Processor Unity (HPU)
64GB flash / 2GB memory
Device is more powerful than a laptop or game computer
No overheating due to warm air flows to the sides
2-3 hours active and 2 weeks standby
Weight 579g
Wi-fi
Gestures, Voice input and spatial understanding
HOLOBASICS 8
SENSORS, OPTICS ANDSPEAKERS
Environment camera
Depthcamera
Videocamera
Spatial SoundWith
speakers
High definitionlenses
HOLOBASICS 12
DEVELOPMENT TOOLS
Visual Studio 2017 UWP workload
Game development with Unity workload
Windows 10 SDK (version 1511 or later) There is no separate SDK for HoloLens
Unity 2017.2 or Unity 5.6.x Scripting Runtime Version Experimental .NET4.6. Equivalent
HoloLens emulator HoloLens Emulator (build 10.0.14393.1358)
Hyper-V
Contains DirectX project templates for Visual Studio
HoloLens device
GitHub Microsoft/MixedRealityToolkit-Unity
HOLOBASICS 13
BUILD LIFECYCLE OF AHOLOLENS PROJECT
Create Unity Project Configure HoloLens settings Create scene
Visual Studio
Unity HoloLens
Configure build settings Build and generate Visual Studio
project
Open project with Visual Studio Pair with HoloLens Build & deploy Visual Studio project
Start Application Test & debug Monitor
Unity
HOLOBASICS 14
WINDOWS DEVICE PORTAL
3D View
Mixed Reality Capture
Performance Performance tracing
System performance
Processes
Apps Maintenance
Crash dumps
Additional tools Logging
File Explorer
Virtual Input
HOLOBASICS 16
UNITYPREFABS
• Predefined Game Objects
• Contains components with predefined values
• Prefabs used in Scene inherit the components and settings
HOLOBASICS 17
UNITYGAMEOBJECT SCRIPT
• Derived from MonoBehaviour
• Contains methods which are called by Unity through the a broadcast messaging system
• void Awake() – Called when class is instantiated
• void Start() – Called when object is enabled once before the first Update()
• void Update() – Called once per frame
• More methods are available• FixedUpdate, LateUpdate, OnGUI, OnDisable, OnEnabled
HOLOBASICS 18
UNITYMESSAGE SYSTEM
public void BroadcastMessage(string methodName, object parameter = null, SendMessageOptions options = SendMessageOptions.RequireReceiver);
• Used for calling methods on GameObjects
• Calls the methods also on its Children
• Examples• this.BroadcastMessage(“OnSelect”);
• focusedObject.BroadcastMessage(“OnSelect”);
HOLOBASICS 19
UNITYCOMPONENTS
• Transform• Position of object
• Rotation of object
• Scale (size) of object
• Mesh Renderer & Mesh Filter• Renders the object
• Mostly done using a mesh
• Mesh Collider• Determines the collision for the object
• Script• Perform actions on the object
• Handle messages on an object
HOLOBASICS 21
COORDINATE SYSTEMHOLOLENS AND UNITY
forward (x, y, z)Camera (0, 0, 0)Rotation (x, y, z)
Camera (1, -1, 4)Rotation (x, y, z)
HologramObject Pinned
HoloLensmoved to
new location
View directionGaze
HOLOBASICS 23
GESTURESGAZE
• The direction of the HoloLens pointing is called the gaze
• Hits an object or part of the environment
• Uses a Vector3 based value to define its so called “forward” direction
HOLOBASICS 24
GESTURESGESTURE RECOGNIZER
• Recognize input by tracking the position of either or both hands
• Gesture frame (Frustum)
• Recognized input from hands• Bloom
• Press, hold and release
• Gestures• Hold
• Manipulation
• Navigation
HOLOBASICS 25
GESTURESGESTURE RECOGNIZER
• Connect commands to gestures• GestureRecognizer gestures = new GestureRecognizer();
• gestures.TappedEvent += (source, tapCount, ray) => { … };
• gestures.StartCapturingGestures();
• Events• Tapped
• Select press and release
• Hold (Started, Completed, Canceled)
• Select press beyond system hold threshold
• Navigation (Started, Updated, Completed, Canceled)• Select press with relative movement
• Velocity based continuous scrolling or zooming
• Virtual Joystick
• Manipulation (Started, Updated, Completed, Canceled)
• Select press with absolute movement
• Move, resize or rotate hologram
HOLOBASICS 26
GESTURESSPEECH RECOGNIZER
• Use words or sentences to control your environment
• English language only at the moment
• Connect commands to spoken text• Dictionary<string, System.Action> keywords = new Dictionary<string,
System.Action>();
• keywords.Add(“some text”, () => { … };
• KeywordRecognizer keywords = new KeywordRecognizer(keywords.Keys.ToArray());
• keywordRecognizer.OnPhraseRecognized += OnPhraseRecognized;
• keywordRecognizer.Start();
• OnPhraseRecognized( PhaseRecognizedEventArgs args);• Invoke the action
• keywords[args.text].Invoke();
HOLOBASICS 29
SPATIAL SOUNDOVERALL
• Simulates 3D sound using direction, distance and environmental simulations
• Above, below, behind, to the side, etc.
• Attach sound to holographic objects• Works also when object is not in line of
sight
• Used to draw attention
• Audio engine in HoloLens• CPU and memory considerations
• 10-12 spatial sound voices
HOLOBASICS 30
SPATIAL SOUNDTYPES
• Gaze mixing• Highlighting objects
• Play a sound on the object to get the users attention
• Audio Haptics• Reactive audio for touchless interactions
• Play a sound when user selects an object or when his hands appear inside the gesture frame
• Immersion• Ambient sounds surrounding the user
• Support your scene with background sounds or music
HOLOBASICS 32
SPATIAL PERCEPTIONHOW DOES IT WORK?
• Spatial perception and mapping
• Rooms
• Different ways of detecting and working with spatial perception
• Device scans your environment and builds a digital model in real time
• It allows HoloLens to see different surfaces, walls and ceiling
• Meshes & planes
• Possible to simulate a physical space
• E.g. Projecting a terrain over your floor
Spatial perception is the ability to be aware of
your relationships with the environment around
you and with yourself
“DEMONSTRATION OF LIVE SPATIAL MAPPING AND UNDERSTANDING BY HOLOLENS”
“HoloBasics by Alexander Meijers”
HOLOBASICS 34
SPATIAL MAPPING ANDUNDERSTANDING
Spatial Mapping components in Unity Spatial Mapping Renderer script
Spatial Mapping Collider script
Use an empty GameObject in your scene
Spatial Mapping render state None
Visualization
Occlusion
Rendering material Occlusion material
Visual material
Cursor on spatial mapping SpatialMappingCollider.layer = 31;
Cursor uses Raycast with layer mask 1<< 31
HOLOBASICS 35
OCCLUSIONCULLING
Process of determining which surfaces and parts are not visible from a certain viewpoint
Create a more naturally feeling with mixed reality
Holograms can be occluded by Other holograms
Real-life objects in your environment which are spatially mapped
Culling enforces the HoloLens not to (partially) render the object To speed up the rendering process
Objects which are too far away are left out the view (frustum)
Use Spatial Mapping Renderer Set Occlusion material
Set SpatialMappingRenderer.renderState = RenderState.Occlusion;
“</CODE>”“DEMONSTRATION OF SPATIAL MAPPING AND
OCCLUSION WITH HOLOGRAM FOR TRUE MIXED REALITY”
“HoloBasics by Alexander Meijers”
THANKS FOR WATCHING!
HoloBasics code on gitHub - https://github.com/ameijers/HoloBasics
Kampenringweg 45b, Gouda
0182-686 000
[email protected] / [email protected]
Contact me:
facebook.com/alexandermeijers.5
@ameijers
Follow me on: