04/18/23 EEC484/584: Computer Networks 1

EEC-490 EEC-490 Senior Design Senior Design (CE)(CE)Kinect Programming Tutorial 1Kinect Programming Tutorial 1

Wenbing ZhaoWenbing Zhao

wenbing@ieee.org

Kinect Application Development Requirement Hardware: a relatively powerful Windows 7

PC/Laptop (Windows XP not supported) Software

Visual Studio 2010 (or Express) for C# (install this first!)

Kinect for Windows SDK For speech recognition

Speech platform runtime (32-bit & 64-bit) Microsoft speech platform sdk MS Kinect English language pack

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx

Create a new Windows Presentation Foundation app File->New->Project… Select “WPF Application”, name the project “KinectBasic”,

click OK Add reference to Kinect library

Right click References=>Add Reference…

Select Microsoft.Research.Kinect=>OK

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx Define UI

TextBlock Text FontSize

Button Content

Images Leftmost image: change variable name to image Middle image: change variable name to imageCmyk32 Rightmost image: change variable name to depth

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx Add events

When “Start” / “Stop” button is pushed In Properties pane => Events tab In “Click” field: can specify event method name Otherwise, a default method name is used (in the form

buttonName_Click())

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx

MainWindow.xaml.cs Add header include

using Microsoft.Research.Kinect.Nui; Add member variable

In public partial class MainWindow:Window { Runtime kinectNui; int totalFrames = 0; // for frame rate calculation int lastFrames = 0; // frame rate DateTime lastTime = DateTime.MaxValue; // frame rate const int RED_IDX = 2; const int GREEN_IDX = 1; const int BLUE_IDX = 0; byte[] depthFrame32 = new byte[320 * 240 * 4]; // for frame manipulation

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs

Add Kinect init method: private void InitializeNui() {} Declares kinectNui as a Runtime object, which represents the

Kinect sensor instance nui = Runtime.Kinects[0];

Open the video and depth streams, and sets up the event handlers that the runtime calls when a video, depth, or skeleton frame is ready

An application must initialize the Kinect sensor by calling Runtime.Initialize before calling any other methods on the Runtime object kinectNui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex |

RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs, InitializeNui{}

To stream color images: The options must include UseColor Valid image resolutions are Resolution1280x1024 and

Resolution640x480 Valid image types are Color, ColorYUV, and

ColorYUVRaw kinectNui.VideoStream.Open(ImageStreamType.Video, 2,

ImageResolution.Resolution640x480, ImageType.ColorYuv);

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs, InitializeNui{}

To stream depth and player index data: The options must include UseDepthAndPlayerIndex Valid resolutions for depth and player index data are

Resolution320x240 and Resolution80x60 The only valid image type is DepthAndPlayerIndex kinectNui.DepthStream.Open(ImageStreamType.Depth, 2,

ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex);

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx

MainWindow.xaml.cs, InitializeNui{} Init timestamp for frame rate calculation:

lastTime = DateTime.Now; Add event handler to handle incoming frames

kinectNui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(NuiVideoFrameReady);

kinectNui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(NuiDepthFrameReady);

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx

MainWindow.xaml.cs Video frame ready event handler

void NuiVideoFrameReady(object sender, ImageFrameReadyEventArgs e) { PlanarImage Image = e.ImageFrame.Image; image.Source = BitmapSource.Create(

Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null, Image.Bits, Image.Width * Image.BytesPerPixel);

imageCmyk32.Source = BitmapSource.Create( Image.Width, Image.Height, 96, 96, PixelFormats.Cmyk32, null, Image.Bits, Image.Width * Image.BytesPerPixel);

}

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: Depth frame ready event handler

Depth is different because the image you are getting back is 16-bit and we need to convert it to 32-bit

void NuiDepthFrameReady(object sender, ImageFrameReadyEventArgs e) { var Image = e.ImageFrame.Image; var convertedDepthFrame = convertDepthFrame(Image.Bits);

depth.Source = BitmapSource.Create( Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null,

convertedDepthFrame, Image.Width * 4);

CalculateFps();}

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: convert depth frame

Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame that displays different players in different colors

byte[] convertDepthFrame(byte[] depthFrame16){ for (int i16 = 0, i32 = 0; i16 < depthFrame16.Length &&

i32 < depthFrame32.Length; i16 += 2, i32 += 4) { int player = depthFrame16[i16] & 0x07; int realDepth = (depthFrame16[i16 + 1] << 5) | (depthFrame16[i16] >> 3); // transform 13-bit depth information into an 8-bit intensity appropriate // for display (we disregard information in most significant bit) byte intensity = (byte)(255 - (255 * realDepth / 0x0fff));

depthFrame32[i32 + RED_IDX] = intensity; depthFrame32[i32 + BLUE_IDX] = intensity; depthFrame32[i32 + GREEN_IDX] = intensity; } return depthFrame32;}

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: calculate frame rate

void CalculateFps(){ ++totalFrames;

var cur = DateTime.Now; if (cur.Subtract(lastTime) > TimeSpan.FromSeconds(1)) { int frameDiff = totalFrames - lastFrames; lastFrames = totalFrames; lastTime = cur; frameRate.Text = frameDiff.ToString() + " fps"; }}

Kinect Getting Started Basichttp://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: windows loading/closing

private void WindowLoaded(object sender, RoutedEventArgs e){ InitializeNui();}private void WindowClosed(object sender, EventArgs e){ kinectNui.Uninitialize();}

Time to run the app!