agenda item: vii florida polytechnic university board of ......pi: sherif rashad co-pis: ryan...
TRANSCRIPT
AGENDA ITEM: VII
Florida Polytechnic University
Board of Trustees December 7, 2016
Subject: Faculty/Student Presentation: (Design and Implementation of an Innovative System for Automatic Conversion from American Sign Language to Speech)
Proposed Board Action
No action required- Information only.
Background Information
The main objective of this research project is to design and implement a new practical and reliable system for automatic real-time conversion from ASL to speech. The researchers involved in this project are using a multidisciplinary approach to build a novel system that will be able to capture and analyze hand gesture motions in real-time using motion sensors and machine learning algorithms. The proposed system will be able to learn new signs and to expand and improve the sign language dictionary. This system can have a wide range of applications for healthcare, education, gamification, entrainment, and many other services and applications. Funding Source: Internal Seed Grant, Florida Polytechnic University PI: Sherif Rashad Co-PIs: Ryan Integlia and Elhami Nasr Graduate Student: Rabeet Fatmi Undergraduate Students: Gabriel Hutchison and Xin Wang
Supporting Documentation: Presentation Prepared by: Dr. Sharif Said Rashad, Associate Professor of Computer Engineering, Computer Science
Board of Trustees Meeting 12.07.16 Page 25
Design and Implementation of an Innovative System for Automatic Conversion from
American Sign Language to SpeechPI: Sherif Rashad
Co-PIs: Ryan Integlia and Elhami NasrGraduate Student: Rabeet Fatmi
Undergraduate Students: Gabriel Hutchison and Xin Wang
12/7/2016
Board of Trustees Meeting 12.07.16 Page 26
Introduction
• Gesture recognition is a very efficient means of communication, providing a natural form of expression for the user.
• Millions of hearing-impaired people worldwide communicate through sign languages every day.
• In the same way that voice recognition provides a simple communication platform for most computer users, gesture recognition is a natural means of communication for of hearing-impaired people
Board of Trustees Meeting 12.07.16 Page 27
American Sign Language
• American Sign Language (ASL) is a well-defined collection of gestures.
• ASL gestures are clearly defined as a means of communication, and the meaning of most gestures is very clear since the language is already created for communication between humans.
Board of Trustees Meeting 12.07.16 Page 28
Related Work
• Sensor gloves used to recognize the hand gestures.– The person needs to wear gloves with several sensors to capture the motions. – The main disadvantage of this approach is that it is not practical and it is difficult to
use the sensor gloves in natural settings for many users.
• Most recent research has attempted to invent methods which eliminate this requirement.
• ASL alphabet recognition was introduced using a Microsoft’s Kinect as a low-cost depth camera.
Board of Trustees Meeting 12.07.16 Page 29
Proposed Technique
• Current focus is on using motion sensors and analyzing motion signals• Machine learning techniques are applied to recognize different gestures
Image Processing
Motion Sensors
Imaging Devices
Motion Analysis
Evaluation of Generated Models
Data Fusion
Machine Learning Techniques
Networked Gamification
Text to Speech Modules
Board of Trustees Meeting 12.07.16 Page 30
• Myo Wearable Device:– Medical grade stainless steel EMG
sensors– Highly sensitive nine-axis IMU containing
− Three-axis gyroscope− Three-axis accelerometer− Three-axis magnetometer.
– ARM Cortex M4 Processor.
Myo Wearable Gesture Control Armband Board of Trustees Meeting 12.07.16
Page 31
Data from Myo Armband
• Spatial data:− Orientation data: roll, pitch, and yaw.− Acceleration vector data− Angular velocity data
• EMG data from the eight sensors
Board of Trustees Meeting 12.07.16 Page 32
Sample Data
RollRight PitchRight YawRight RollAccelRight PitchAccelRight YawAccelRight GyroRollRight GyroPitchRight GyroYawRight Class
161.2454335 193.8987434 44.61407429 -0.123046875 -0.31640625 0.938964844 -2.1875 1.25 3.6875 table
152.0511368 188.7658686 33.9266243 -0.078613281 -0.466308594 0.876953125 2.75 1.125 -2.5 above
151.9580826 142.0753625 24.51713538 0.408203125 -0.382324219 0.784667969 34.125 7.5625 -36.5 below
144.5047546 218.5517203 24.50601605 -0.268554688 -0.602539063 0.73828125 0.9375 24.5625 22.3125 drink
110.3546439 148.6133252 353.0980963 0.268554688 -0.901855469 0.330566406 -3.0625 -0.5625 -1.1875 thank you
• A combination of values define what gesture was being performed• Data tells us what feature values explicitly differentiate between classes (words)
Board of Trustees Meeting 12.07.16 Page 33
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
J48 KNN ANN naive Bayes
97.23%96.96%
94.03%
78.18%
Prediction Accuracy
Prediction Accuracy
• What percentage of test data was correctly classified by a classifier• Classifiers used: J48, K nearest-neighbor, Artificial neural network, Naive Bayes
Board of Trustees Meeting 12.07.16 Page 34
Sample Confusion Matrix (J48)
Confusion Matrix describes the performance of a classifier
Board of Trustees Meeting 12.07.16 Page 35
Sample Confusion Matrix (J48)
Words (classes) that are outputted by the classifier
Board of Trustees Meeting 12.07.16 Page 36
Sample Confusion Matrix (J48)
Confusion row for the word ‘table’
Board of Trustees Meeting 12.07.16 Page 37
Sample Confusion Matrix (J48)
Total instances of actual words
Board of Trustees Meeting 12.07.16 Page 38
Sample Confusion Matrix (J48)
Correctly classified instances
Board of Trustees Meeting 12.07.16 Page 39
Android Mobile Application Board of Trustees Meeting 12.07.16
Page 40
Demo
DEMO
Board of Trustees Meeting 12.07.16 Page 41
Future Work
• Add more words to the dictionary
• Work with sentences
• Continue to work on the development of the mobile application
• Include additional sensors (Leap Motion)
Board of Trustees Meeting 12.07.16 Page 42
Thank You
Questions?
Board of Trustees Meeting 12.07.16 Page 43