jancke kinect programming
TRANSCRIPT
![Page 1: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/1.jpg)
Programming with the Kinect for
Windows SDK
![Page 2: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/2.jpg)
What we’ll cover Kinect Sensor
Using Cameras
Understanding Depth Data
Skeletal Tracking
Audio
![Page 3: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/3.jpg)
RGB CAMERA
MULTI-ARRAY MIC MOTORIZED TILT
3D DEPTH SENSORS
![Page 4: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/4.jpg)
SDK Architecture
Video Components Audio Components
Windows
components
1
2
3
5
4
Kernel - mode drivers for Kinect for Windows
DMO codec for mic array
Applications
Kinect for
Windows SDK
User - created
components
USB Hub
Windows Core Audio
and Speech APIs
Device
setup Audio stream control Video stream control
WinUSB device stack USBAudio audio stack WinUSB camera stack
Motor Audio mic array Cameras
NUI API
User Mode
Kernel Mode
Kinect sensor
Hardware
Device
access
![Page 5: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/5.jpg)
SDK Architecture
Video Components Audio Components
Windows
components
1
2
3
5
4
Kernel - mode drivers for Kinect for Windows
DMO codec for mic array
Applications
Kinect for
Windows SDK
User - created
components
USB Hub
Windows Core Audio
and Speech APIs
Device
setup Audio stream control Video stream control
WinUSB device stack USBAudio audio stack WinUSB camera stack
Motor Audio mic array Cameras
NUI API
User Mode
Kernel Mode
Kinect sensor
Hardware
Device
access
![Page 6: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/6.jpg)
SDK Architecture
Video Components Audio Components
Windows
components
1
2
3
5
4
Kernel - mode drivers for Kinect for Windows
DMO codec for mic array
Applications
Kinect for
Windows SDK
User - created
components
USB Hub
Windows Core Audio
and Speech APIs
Device
setup Audio stream control Video stream control
WinUSB device stack USBAudio audio stack WinUSB camera stack
Motor Audio mic array Cameras
NUI API
User Mode
Kernel Mode
Kinect sensor
Hardware
Device
access
![Page 7: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/7.jpg)
SDK Architecture
Video Components Audio Components
Windows
components
1
2
3
5
4
Kernel - mode drivers for Kinect for Windows
DMO codec for mic array
Applications
Kinect for
Windows SDK
User - created
components
USB Hub
Windows Core Audio
and Speech APIs
Device
setup Audio stream control Video stream control
WinUSB device stack USBAudio audio stack WinUSB camera stack
Motor Audio mic array Cameras
NUI API
User Mode
Kernel Mode
Kinect sensor
Hardware
Device
access
![Page 8: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/8.jpg)
SDK Architecture
Video Components Audio Components
Windows
components
1
2
3
5
4
Kernel - mode drivers for Kinect for Windows
DMO codec for mic array
Applications
Kinect for
Windows SDK
User - created
components
USB Hub
Windows Core Audio
and Speech APIs
Device
setup Audio stream control Video stream control
WinUSB device stack USBAudio audio stack WinUSB camera stack
Motor Audio mic array Cameras
NUI API
User Mode
Kernel Mode
Kinect sensor
Hardware
Device
access
![Page 9: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/9.jpg)
Demos
Using Cameras
![Page 10: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/10.jpg)
Understanding Depth Data ImageFrame.Image.Bits
Array of bytes - public byte[] Bits;
Array
– Starts at top left of image
– Moves left to right, then top to bottom
– Represents distance for pixel in millimeters
![Page 11: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/11.jpg)
Calculating Distance 2 bytes per pixel (16 bits)
Depth – Distance per pixel – Bitshift second byte by 8
– Distance (0,0) = (int)(Bits[0] | Bits[1] << 8); – VB (int)(CInt(Bits(0)) Or CInt(Bits(1)) << 8);
DepthAndPlayer Index – Includes Player index – Bitshift by 3 first byte (player index), 5 second byte
– Distance (0,0) =(int)(Bits[0] >> 3 | Bits[1] << 5);
– VB:(int)(CInt(Bits(0)) >> 3 Or CInt(Bits(1)) << 5);
![Page 12: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/12.jpg)
Depth Reference Distance Range: 850 mm to 4000 mm range
Depth value 0 means unknown – Shadows, low reflectivity, and high reflectivity among
the few reasons
Player Index – 0 – No player
– 1 – Skeleton 0
– 2 – Skeleton 1
– …
![Page 13: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/13.jpg)
Demos
![Page 14: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/14.jpg)
Skeleton Data
![Page 15: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/15.jpg)
Joints Maximum two players tracked at once
– Six player proposals
Each player with set of <x, y, z> joints in meters
Each joint has associated state
– Tracked, Not tracked, or Inferred
Inferred - Occluded, clipped, or low confidence joints
Not Tracked - Rare, but your code must check for this state
![Page 16: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/16.jpg)
Skeletal Tracking
![Page 17: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/17.jpg)
Demos
![Page 18: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/18.jpg)
Audio Going Inside of Kinect Four microphone array
with hardware-based
audio processing
– Multichannel echo cancellation (MEC)
– Sound position tracking
– Other digital signal processing (noise suppression
and reduction)
![Page 19: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/19.jpg)
Audio Kinect as a microphone
Kinect for Speech Recognition
![Page 20: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/20.jpg)
Multi-modal Feedback Engagement model
– Mic indicator for speech-enabled menus
– Keyword to engage
Feedback and confirmation
(both passive and active)
![Page 21: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/21.jpg)
Speech Recognition Kinect Grammar available to download
Grammar – What we are listening for
– Code – GrammarBuilder, Choices
– Speech Recognition Grammar Specification (SRGS)
C:\Program Files (x86)\Microsoft Speech Platform
SDK\Samples\Sample Grammars\
![Page 22: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/22.jpg)
Grammar <!-- Confirmation_YesNo._value: string ["Yes", "No"] --> <rule id="Confirmation_YesNo" scope="public"> <example> yes </example> <example> no </example> <one-of> <item> <ruleref uri="#Confirmation_Yes" /> </item> <item> <ruleref uri="#Confirmation_No" /> </item> </one-of> <tag> out = rules.latest() </tag> </rule> </rule>
<!-- Confirmation_Yes._value: string ["Yes"] --> <rule id="Confirmation_Yes" scope="public"> <example> yes </example> <example> yes please </example> <one-of> <item> yes </item> <item> yeah </item> <item> yep </item> <item> ok </item> </one-of> <item repeat="0-1"> please </item> <tag> out._value = "Yes";</tag>
![Page 23: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/23.jpg)
Demos
![Page 24: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/24.jpg)
Resources Kinect Programming Walkthroughs
– http://research.microsoft.com/kinectsdk/
Coding4Fun Kinect Toolkit
– http://c4fkinect.codeplex.com
Kinect SDK Quickstarts – http://channel9.msdn.com/series/KinectSDKQuickstarts
![Page 25: Jancke kinect programming](https://reader031.vdocuments.us/reader031/viewer/2022022410/58e75cdc1a28ab5b2a8b5097/html5/thumbnails/25.jpg)