kinect interactions: get a grip in c# or html5/ javascript
DESCRIPTION
Kinect Interactions: Get a Grip in C# or HTML5/ Javascript. Ben Lower Program Manager – Kinect for Windows 3-705. Agenda. Periodic Table InteractionStream Sneak Peek of 1.8 SDK New Kinect? Q&A. Periodic Table. CHUCK NORRIS HAD TO DESTROY THE PERIODIC TABLE…. - PowerPoint PPT PresentationTRANSCRIPT
Kinect Interactions: Get a Grip in C# or HTML5/JavascriptBen LowerProgram Manager – Kinect for Windows3-705
Periodic TableInteractionStreamSneak Peek of 1.8 SDKNew Kinect?Q&A
Agenda
Periodic Table
CHUCK NORRISHAD TO DESTROYTHE PERIODIC TABLE…
HE ONLY RECOGNIZES THEELEMENT OF SURPRISE!
AAudio
Waveform
Stream Symbol
Stream Name
Data Type
The Periodic Table of Kinect for Windows
DDepth
DepthImagePixel[]
AAudio
Waveform
IRInfrared
byte[]
CColorbyte[]
ACAcceleromete
rVector4
Data Streams (Eyes & Ears)
IInteractionsInteractionFrame
(UserInfo[])
KFKinect Fusion
Point Cloud/Mesh
FTFace Tracking
FaceTrackFrame
SSpeech
Recognition Events
STSkeleton TrackingSkeleton[]
Recognition Streams (Brains)
Demo: Controls Basics +Interaction Gallery
InteractionStream
Why InteractionStream?Moving to world that is multi-modal
Have experiences feel natural
Consistency of interactions
Interaction StreamTranslates hand position to screen coordinates• Physical Interaction Zone (PhIZ) is shaped to be ergonomic for an
individual independent of position relative to the sensorTrack and recognize press action• Press action based on arm extension• Press distance dependent on starting arm position• Press adjusted to improve accuracy over buttons
Recognize hand grip and release gesture• Computer vision and machine learning algorithm recognize grip gesture• Grip recognized for a wide variety of hand poses and orientations
The PhIZPhysical Interaction Zone (PhIZ)The area in front of the user where there is a spatial mapping relationship between the user’s hand movements in physical space and the cursor movement on the screen.
Features• Ergonomics and Human Movement• Scaling to fit user• Separate PhIZ for each hand• Consistent experience
Targeting & Selection
Cursor Default (L)
Cursor Default (R)
New WayTarget button with hand cursorPress toward button, then releaseCenter “pulls hand in” during press
Old WayTarget button with hand cursorHover, hold, wait (for action)
Press Cursors
Roll Over Press Half Full Press Press Complete
Research showed people like visual cues & feedbackHand “fills up” to indicate press progress
Scrolling & Direct Manipulation
Cursor Default
Grip Over
New WayTarget anywhere w/hand cursorGrip to manipulate
Old WayTarget scroll button w/hand cursorHold and wait for scroll
InteractionStream APIsNative COM based InteractionStream API• Core implementation• Usable with any UI frameworks that can call C functions and COM
objectsManaged .NET Wrapper• Wraps the InteractionStream API for .NET languages, e.g. C#, VB etc.
WPF Adapter – KinectRegion, Cursor and Controls
• Implements IInteractionClient for hit-testing WPF controls• Translates Interaction Frame to WPF events• Implements Cursor to provide feedback for: handedness, press progress
and grip state
Interactions Architecture
NUI_INTERACTION STREAM
KinectInteraction170_*.dll (32 or 64)
DirectXKinect
Support InteractionStream
Microsoft.Kinect.Toolkit.Interaction.dll
.NET/WPF
Kinect-Enabled App
WPF Kinect Controls
Microsoft.Kinect.Toolkit.Controls.dll
XNA Kinect
SupportC++/Win32 Kinect
Controls
Windows Forms Kinect
Controls
C++/DirectX Kinect-Enabled
App
C++/Win32 Kinect-Enabled
App
XNAKinect-Enabled
App
Windows FormsKinect-Enabled
App
Working with InteractionStream
Application
InteractionStream
UI Framework Adapter:IInteractionClient
ProcessSkeleton(skeleton, accelerometer, timestamp)
ProcessDepth(depthData, timestamp)
GetInteractionInfoAtLocation(id, handType, x, y)
new (sensor, interactionClient)
return InteractionInfo{ isGripTarget, isPressTraget,PressAttractionPointX, PressAttractionPointY,PressControlId}
InteractionFrameReady
Integration ConsiderationsPrimary User and Primary Hand• InteractionStream tracks 2 users and 4 hands• Primary User and Primary Hand implemented by KinectRegion• UX Guidance: Avoid 2 handed actions – difficult to learn & master
Cursor – press and grip visualization• Continuous press progress feedback critical to user experience• Press initiates action on release – similar to button press with mouse
Pressable button affordance• Button provides visual feedback when cursor is over pressable button
Sneak Peek: v1.8 SDK
Demo: Background Removal
Demo: Photobombr
Interactions Architecture (1.8)
NUI_INTERACTION STREAM
KinectInteraction170_*.dll (32 or 64)
DirectXKinect
Support InteractionStream
Microsoft.Kinect.Toolkit.Interaction.dll
HTML5/Javascript
Kinect-Enabled AppHTML5/JS Controls
Microsoft.Samples.
Kinect.Webserver.dll
XNA Kinect
SupportC++/Win32 Kinect
Controls
Windows Forms Kinect
Controls
C++/DirectX Kinect-Enabled
App
C++/Win32 Kinect-Enabled
App
XNAKinect-Enabled
App
Windows FormsKinect-Enabled
App
.NET/WPF
Kinect-Enabled AppWPF Kinect
Controls
Microsoft.Kinect.Toolkit.Controls.dl
l
Announcing: v1.8 SDK (September 2013)Background Removal API (green screening w/out the screen)
HTML5/Javascript support for Interactions
Kinect Fusion Color
Plus more…stay tuned
What’s up with the new Kinect?
New Kinect for Windows Sensor• Wider field of view• HD resolution• Track six skeletons• Better ST, audio, infrared• Coming next year
Announcing: Developer Kit ProgramDirect access to our engineering teamExclusive access to pre-release SDKsPrivate access to all API & sample docsPre-release/alpha sensorNew RTM sensor next year
$399 (coming November 2013)Apply now: http://
aka.ms/k4wdevkitapplication
Who should apply?
Already working today to build & deploy NUI experiencesYou are interested in or experimenting with NUIYou want to help push the boundaries of NUIExperienced developer who wants to jump inYou have an awesome idea for a touch-free app
Thank you!Get 1.7 SDK now
Follow our blog to know when 1.8 SDK is released
Apply today for new dev kit program
Kinect for Windows @ Build 2013All Week
• Kinect for Windows Booth (demos, 3D printing with Fusion)
Wednesday, June 26th• 2:00 PM Real-world Machine Learning: How Kinect Gesture Recognition Works, 3-704
Alisson Sol, Bing (formerly Kinect for Windows), North Hall: 123
Thursday, June 27th• 2:00 PM Kinect Interactions: Get a Grip in C# or HTML5/Javascript, 3-705
Ben Lower, Kinect for Windows, North Hall: 131
• 5:00 PM Scanning & Printing the Real-World using Kinect for Windows and Preview of vNext, 3-702
Chris White, Kinect for Windows, North Hall: 131
Kinect for Windows ResourcesDeveloper Kit Program Application
http://aka.ms/k4wdevkitapplication
Developer Center (SDKs, Samples, Videos)http://aka.ms/k4wdev
Product Infohttp://aka.ms/k4w
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.