Kinect Part II

Anna Loparev

OpenNI API• Actually general purpose API• Need 3 things (http://cs.rochester.edu/courses/297/fall2011/kinect

/openni)– SensorKinect Driver– OpenNI Library– NITE Middleware (gesture support)

• Requirements:– Kinect– Visual Studio 2008/2010– C++/C#

• Requirements (From Tutorial):– Python– SWIG (Allow Python to interface with OpenNI)– OpenCV (Allow Python to work with image data)– NumPy

Features• Recording for debugging• Fully backwards compatible• Under the GNU Lesser General Public License (LGPL)

OpenNI vs Microsoft SDK• Microsoft

– Easy to install– Don’t have to do calibration pose but gets more false positives– Predictive tracking of joints (Good for when sensor losses player)

• Leads to false positives• Uses lots of processing power

– Gets full 1024x768 resolution from camera– Full access to microphone– Support for tilt

• OpenNI– Open source (Allows for commercial use)– Works across all platforms– Hand-only tracking– Gesture recognition– Rotation in joints– Tilt has to be done through motor drivers

Sample Applications (OpenNI)• General

– NiCRead (Prints out depth at center of image)– NiSimpleCreate (C++ version of NiCRead without using SamplesConfig

XML)– NiSimpleRead (C++ version of NiCRead using SamplesConfig XML)– SimpleRead (C# version of NiCRead)– NiSimpleViewer (Depth and video viewer)– SimpleViewer (C# version of NiSimpleViewer)– NiViewer (Shows a bunch of different views and allows recording)– NiUserTracker (Pose detection and skeletal tracker)– UserTracker (C# version of NiUserTracker)– NiUseExtension– NiAudioSample

• Recording– NiBackRecorder (Allows user to save certain number of seconds)– NiRecordRaw– NiRecordSynthetic (Opens recording, does transformation, and re-

records)– NiConvertXToONI (Copies recording)

Sample Applications (NITE)• General

– Sample-Boxes (Highlight boxes based on hand movement and gestures)

– Sample-Boxes.net (C# version of Sample-Boxes)– Sample-CircleControl (Perform a circle gesture and circle is drawn)– Sample-Players (Identifies multiple players and their skeletons)– Sample-PointServer– Sample-PointViewer (Tracks hand and leaves behind trail)– Sample-SceneAnalysis (Identifies people)– Sample-SingleControl (Identifies wave gesture)– Sample-StickFigure (Places skeleton and people in virtual room)– Sample-TrackPad (Clickable grid)

Overview

Overview

NITE Abilities• Focus gestures

– Click– Wave– Swipe left– Swipe right– Raise hand candidate– Hand candidate moved

• Gesture Status– Gesture started– Gesture completed

• Hand tracking• User segmentation (Identify and track people)• Skeletal tracking

Capabilities• Optional extensions (Providers choose if they want to implement

them.)

• Alternative View (Data appears to come from another sensor)• Cropping• Frame Sync (between multiple devices)• Mirror• Pose Detection• Skeleton• User Position (Optimize output depth map)• Error State• Lock Aware (Allows node to be locked outside of context)

Generators• Map Generator• Depth Generator• Image Generator• IR Generator• Scene Analyzer• Audio Generator• Gesture Generator• Hand Point Generator• User Generator

Generators (Cont)• xn::Generator::StartGenerating() - begin generating data• xn::Generator:StopGenerating() - stop generation without

destroying node• xn::Generator::WaitAndUpdateData() - Update data when it

becomes available• xn::Context::Wait[…]UpdateAll() - Update all generators ([…]

determines when)

Simple Program// Set status to OK

XnStatus nRetVal = XN_STATUS_OK;

// Create and initialize new context

xn::Context context;

nRetVal = context.Init();

// Create generators

xn::DepthGenerator depth;

nRetVal = depth.Create(context)

// Start generating data

nRetVal = context.StartGeneratingAll();

Simple Program (Cont)// Do main loop

while (bShouldRun)

{

// Wait for new data

nRetVal = context.WaitOneUpdateAll(depth);

// Make sure update worked

if (nRetVal != XN_STATUS_OK)

{

printf(“Failed updating data: %s\n”, xnGetStatusString(nRetVal));

continue;

}

// DO STUFF

}

Simple Program (Cont)

// Clean-up

context.Shutdown();

OR

depth.Release();

context.Release();

DEMOS

References/More InfoSetup:

http://cs.rochester.edu/courses/297/fall2011/kinect/openni

http://www.codeproject.com/Articles/148251/How-to-Successfully-Install-Kinect-on-Windows-Open.aspx

Extensions:

http://code.google.com/p/simple-openni/

Other:

http://www.openni.org/

http://www.brekel.com/?page_id=671

http://labs.vectorform.com/2011/06/windows-kinect-sdk-vs-openni-2/