Download - Custom gesture control
MagicWand
Gesture Controls -Alick R Xu, Harshitha Chidananda, Tanuj
MittalFinal Presentation, CS290F - Winter2016
1.Introduction
2.Problem
3.Proposed solution
4.Overviewa. Use cases
b. Innovations
c. Challenges
d. Technical Points
5.Evaluation
6.Conclusion
Contents
INTRODUCTION●Most smartphones are equipped with sensors.
●Leverage accelerometer and gyroscope
●Motion gestures can be used for a variety of applications
●Control laptop using smart phones
What is the problem?
Usability Computer contains a
large amount of data and navigation across
applications is becoming hard.
No gesture apps
There are very few apps which enables controlling something using gestures
Can’t define own gesturesNo application allows
users to define their own gestures and associate
an action with it
Our solution!
Usability Use alternative form of
navigation between applications
Gesture controlled
Controlling applications using gestures
Define own gestures
Users can define own gestures, train it,
associate it with an action and perform
actions with just gestures.
Use cases
Mobile-laptop
DataSend data from mobile
to laptop
Music controlpause, stop, next,
previous
Volume up, down
Mobile
Presentation control
When presenting a slide deck, a user can use their phone to move through the slides.
Learning
ML+gesturesTrain gesture
User inputs their own gestures to perform an action
ChallengesNot many solutions
Time series
Machine learning algorithms
Laptop-mobile communication
Accuracy
Speed
Classify gestures
Overview
DTW
Fast DTW
1nn classifier
Parallel Classifier
Machine Learning Flip
Top
Down
Left
Right
Shake
Whip
Simple Gestures Circle
Rectangle
Square
Infinity
Triangle
Clockwise vs Anti-clockwise
Complex Gestures
Why Dynamic Time Warping?
● Any distance (Euclidean, Manhattan etc.) aligns i-th point on one time series with the i-th point on the other time series
● DTW is a non-linear alignment of time series
“1nn with DTW is exceptionally difficult to beat”*
*Fast Time Series Classification Using Numerosity Reduction (http://alumni.cs.ucr.edu/~xxi/495.pdf)
SolutionAndroid app
Connects to PC app using WiFi
Provides way to create/edit gestures
PC app
Java application
Uses FastDTW with parallel 1nn classifier
Progress from last timeParallelize classifier
Rewrote from python to java for parallelizing
User can define gestures and actions
More gestures
Brief evaluation of performance of classifier
AdvantagesEasy to setup
Full flexibility in defining custom gestures
Complete control over use cases
Not limited by phone’s processing power
Works on any* android phone
Super easy to add support for other mobile platforms
Uses small amount of training data
InnovationsUsers can send data from the phone to the computer
Control the computer with their phone
Users can input their own gesturesTrain gesture
Allows you to do anything cool!
EvaluationOne person trained 7 gestures
Three people tried to use the 7 different gesturesTested with 1nn, 2nn, and 5nn
Tested similar gestures with three users
Tested with a OnePlus One
ResultsOne person trained, three people used
Gestures Tanuj trained
Tanuj (standing)
Alick Harshitha Tanuj trained
Tanuj (standing)
Alick Harshitha Tanuj trained
Tanuj (standing)
Alick Harshitha
flick 1nn 10 10 10 2nn 10 9 (right) 5nn 10 9 (right) 9(flick)
shake 10 106(left) (right)(left)(flick) 10
6(left)(right*3) 9 (left) 10
6(circle, flick, circle,
triangle)9 (left)
left 10 10 10 10 10 10 8(triangle, flick)
10
right 10 9 (left) 10 10 10 10 9 (flick) 10
left infinity 10 10 10 10 10 10 10 8(right,flick)
counter clockwise
circle10 10 9(flick) 10 10 10 10 10 10
triangle 10 10 9(flick) 10 8 (shake*2) 10 107 (shake,
shake, flick)
10
Results (contd.)Testing Similar Gestures with 1nn
Gestures Tanuj Alick Harshitha
square 10 10 6(rectangle)
rectangle 3(square) 7 (square) 9(square)
circle 10 10 10
top-down 10 10 10
whip 10 10 10
Milestones
Feb 10Obtain raw accelerometer/gyroscope readings from phone
Feb 17stream raw data to a PC
Feb 24Detect patterns in data corresponding to certain gestures, and start machine learning training process to recognize these gestures.
March 2Add machine learning capabilities to application, start having gestures be recognized on the PC
March 9User experiments, performance study
LimitationsHave to hold a button to start tracking gesture
Variation in accuracy between users who didn’t train
Only can interact using keystrokes
Contribution and scope
ContributionAccelerometer/gyroscope readings from phoneStream raw data to a PCDetect patterns in data corresponding to certain gesturesStart machine learning training process to recognize these
gestures.User experimentsPerformance study
Scope Make the gesture work without holding phone.Add voice recognition capabilities to open the application
after which any gesture you perform should be associated with some action.
Add mouse controls
ConclusionFast with parallelization
Very accurate
1nn performed well
Not as accurate for short gestures
Requires certain amount of user concentration for similar gestures(Rectangle, Square)
Best performance when user is the trainer
Customizable
Can be used for a variety of applications
Existing gestures and their actions Whip - cmd+space (spotLight)
Circle - space (play/pause music)
Shake - cmd+right (skip to the next song)
Up , down - cmd+up/down (control volume)
Right, left - right/left (control slides)
Demo
Thank You!Questions?