MagicWand

Gesture Controls -Alick R Xu, Harshitha Chidananda, Tanuj

MittalFinal Presentation, CS290F - Winter2016

1.Introduction

2.Problem

3.Proposed solution

4.Overviewa. Use cases

b. Innovations

c. Challenges

d. Technical Points

5.Evaluation

6.Conclusion

Contents

INTRODUCTION●Most smartphones are equipped with sensors.

●Leverage accelerometer and gyroscope

●Motion gestures can be used for a variety of applications

●Control laptop using smart phones

What is the problem?

Usability Computer contains a

large amount of data and navigation across

applications is becoming hard.

No gesture apps

There are very few apps which enables controlling something using gestures

Can’t define own gesturesNo application allows

users to define their own gestures and associate

an action with it

Our solution!

Usability Use alternative form of

navigation between applications

Gesture controlled

Controlling applications using gestures

Define own gestures

Users can define own gestures, train it,

associate it with an action and perform

actions with just gestures.

Use cases

Mobile-laptop

DataSend data from mobile

to laptop

Music controlpause, stop, next,

previous

Volume up, down

Mobile

Presentation control

When presenting a slide deck, a user can use their phone to move through the slides.

Learning

ML+gesturesTrain gesture

User inputs their own gestures to perform an action

ChallengesNot many solutions

Time series

Machine learning algorithms

Laptop-mobile communication

Accuracy

Speed

Classify gestures

Overview

DTW

Fast DTW

1nn classifier

Parallel Classifier

Machine Learning Flip

Top

Down

Left

Right

Shake

Whip

Simple Gestures Circle

Rectangle

Square

Infinity

Triangle

Clockwise vs Anti-clockwise

Complex Gestures

Why Dynamic Time Warping?

● Any distance (Euclidean, Manhattan etc.) aligns i-th point on one time series with the i-th point on the other time series

● DTW is a non-linear alignment of time series

“1nn with DTW is exceptionally difficult to beat”*

*Fast Time Series Classification Using Numerosity Reduction (http://alumni.cs.ucr.edu/~xxi/495.pdf)

SolutionAndroid app

Connects to PC app using WiFi

Provides way to create/edit gestures

PC app

Java application

Uses FastDTW with parallel 1nn classifier

Progress from last timeParallelize classifier

Rewrote from python to java for parallelizing

User can define gestures and actions

More gestures

Brief evaluation of performance of classifier

AdvantagesEasy to setup

Full flexibility in defining custom gestures

Complete control over use cases

Not limited by phone’s processing power

Works on any* android phone

Super easy to add support for other mobile platforms

Uses small amount of training data

InnovationsUsers can send data from the phone to the computer

Control the computer with their phone

Users can input their own gesturesTrain gesture

Allows you to do anything cool!

EvaluationOne person trained 7 gestures

Three people tried to use the 7 different gesturesTested with 1nn, 2nn, and 5nn

Tested similar gestures with three users

Tested with a OnePlus One

ResultsOne person trained, three people used

Gestures Tanuj trained

Tanuj (standing)

Alick Harshitha Tanuj trained

Tanuj (standing)

Alick Harshitha Tanuj trained

Tanuj (standing)

Alick Harshitha

flick 1nn 10 10 10 2nn 10 9 (right) 5nn 10 9 (right) 9(flick)

shake 10 106(left) (right)(left)(flick) 10

6(left)(right*3) 9 (left) 10

6(circle, flick, circle,

triangle)9 (left)

left 10 10 10 10 10 10 8(triangle, flick)

10

right 10 9 (left) 10 10 10 10 9 (flick) 10

left infinity 10 10 10 10 10 10 10 8(right,flick)

counter clockwise

circle10 10 9(flick) 10 10 10 10 10 10

triangle 10 10 9(flick) 10 8 (shake*2) 10 107 (shake,

shake, flick)

10

Results (contd.)Testing Similar Gestures with 1nn

Gestures Tanuj Alick Harshitha

square 10 10 6(rectangle)

rectangle 3(square) 7 (square) 9(square)

circle 10 10 10

top-down 10 10 10

whip 10 10 10

Milestones

Feb 10Obtain raw accelerometer/gyroscope readings from phone

Feb 17stream raw data to a PC

Feb 24Detect patterns in data corresponding to certain gestures, and start machine learning training process to recognize these gestures.

March 2Add machine learning capabilities to application, start having gestures be recognized on the PC

March 9User experiments, performance study

LimitationsHave to hold a button to start tracking gesture

Variation in accuracy between users who didn’t train

Only can interact using keystrokes

Contribution and scope

ContributionAccelerometer/gyroscope readings from phoneStream raw data to a PCDetect patterns in data corresponding to certain gesturesStart machine learning training process to recognize these

gestures.User experimentsPerformance study

Scope Make the gesture work without holding phone.Add voice recognition capabilities to open the application

after which any gesture you perform should be associated with some action.

Add mouse controls

ConclusionFast with parallelization

Very accurate

1nn performed well

Not as accurate for short gestures

Requires certain amount of user concentration for similar gestures(Rectangle, Square)

Best performance when user is the trainer

Customizable

Can be used for a variety of applications

Existing gestures and their actions Whip - cmd+space (spotLight)

Circle - space (play/pause music)

Shake - cmd+right (skip to the next song)

Up , down - cmd+up/down (control volume)

Right, left - right/left (control slides)

Demo

Thank You!Questions?