Repetition Counting With Microsoft Kinect

Presented by:Jonathan Gurary

Dai Jun

Proposal

Design a Kinect-based system for physical therapy activities

Usable by the end-user in their home No professional staff or facility needed Simple UI anyone can use Similar in performance to expensive 8-

camera setups

Identify and count a variety of motions

Implementation

Hardware: Microsoft Kinect

Software: OpenNI API, executed through Processing

Designed for visual software, communicating with vision sensors

OpenNI

Cross-platform support

-Linux, Mac, Windows Older mod community

-Less updates = more working samples

-Issues are better documented And there's books

-First Kinect SDK books just hit shelves

-OpenNI has a couple books already

Limiting Kinect Problems

Bad guesses Lost a limb? It must be straight at your

side! Mismatched limbs

Crossed legs or arms can “swap” Jitter

Shaky and unstable reads These can happen with no effect on

Confidence

Effects on Repetition Counting

Angles and positions will be collected in error Almost certainly out-of-bounds for the

exercise Best case: Happens during non-critical

portion of motion, no effect Worst case: miss flag, false count,

reset the motion

Our Solution to Erroneous data

Make sure the user knows what the Kinect is seeing

Jerky skeletons, mismatched or lost limbs, all very obvious. Just stop and try again.

Stop looking at data when confidence drops Allows for the split-second guessing

errors common in some motions. Use a tolerance

Mitigate the effects of jittery reads

Flag-Based Motion Identification

Each motion or exercise is broken down and split into five intermediate steps

Example: waving your hand Change in elbow angle of about 50

degrees Step 1: elbow angle reduces 10

degrees Step 2: elbow angle reduces 10 more

degrees

Flag-Based Motion Identification

Every step becomes a flag to be passed Must be passed in order All flags reset (exercise failed) if a body

part goes out of bounds If every flag is passed correctly, count

the action

How Are Flags Determined

Flags are manually input by the coder Can use angles of all major limbs, height of

torso, and distance of hands and feet Potentially (future feature) scale to

user's height

Recording and Playback

Main program records user's entire session automatically

Saved during execution to a .onni file Separate program available for playback of

session. Skeleton filtered out Can pause, fast forward, slow down, or

seek through recording

User Interface Design

User Friendly, easy to use -no programming knowledge required -motion icons to choose -works like a button, but beautiful!

Replay mode -control bar -works like a real media player!

User Interface Design

User Interface Design

Dynamic Flag Setup

Identifying flags by examining Kinect output is tedious

Dynamic program allows the user to “input” flags with their body

Allows any almost any motion to be input on the fly

Data is output to a text file, allowing coder to turn the input into a featured exercise easily

Dynamic Flag Limitations

Impossible to tell what tolerance will be needed

Whole body is captured All limbs must be visible and not

guessed. Exercise is limited to the position

(distance wise) that it was started at Counting isn't always accurate,

especially for small or complex motions

Demo Presentation

Summary

Developed a relatively accurate, user friendly physical therapy program

Count repetitions of exercises Exercises can be set by a professional, added

to the code Dynamic flag setter to identify motion,

then potentially added to exercises by coder

User can view raw footage of exercises

Future Features

Allow the dynamic flag set to ignore some parts of the body

Adjust for big height differences between users

Allow a health professional to easily add a dynamically created exercise to code