Kinect Virtual Learning

Team members: Mohamed Hesham Mahmoud Soliman Ahmed Amr Yousef Ahmed Ahmed Nasser Date: 5/7/2015

Agenda• Problem & Solution• Presentation Application • Interviewing Application • Pattern Recognition techniques• Speech Recognition • Demo• Questions ?

Problem• Why Soft Skills ?• Business & Education• Chance !!• Software !!

Existing Solutions• Expensive Courses

• Online Materials

Idea & Solution• Simulate the presentation & interview Processes • train the presenter & interviewee on the correct behavior

Body movements Speech recognition Facial Expression

• Feedback of weakness points

What is Kinect !

Message Delivery • Body Language• Speech • Facial Expression

Project Overview

Presentation Application

Detected Body MistakesReferences

Body Joints

Mistakes Categories

Dynamic Mistakes

Body Language

Body Language

Body Language

Body Language

Body Language

Body Language

Static Mistakes

Body Language

Body Language

Body Language Hand in Pocket

Leaning Body Language

Body Language Hand on waist

Body LanguageUp leg

Presentation Application Flow

Interview Application

References

Detected Face Mistakes

Facial Techniques Face Joints Face Orientation

Face Gestures Touching face and Yawning

Facial Techniques Face Joints Face Orientation

Face Gestures No Eye Contact

Face Gestures

Feeling Shy

Face Gestures Smiling

Face Gestures Cleaning Glasses

Face Gestures Aggressive

Face Gestures Bad Posture

Interview Application Flow

Pattern Recognition Techniques

Classifiers Categories

Rule-Based Classifiers • Conditions

Threshold

• 12 Joints with XYZ Coordinates

Hidden Markov Model Classifiers

• Dataset 50 sample for each mistake

• Models 9 states 200 Observations

• Phases Learning Decoding Evaluate

Rule-Based HMM80

82

84

86

88

90

92

94

96

ArmLegBody center

Classifier Arm Leg Center Type

Rule-Based 92% 88% 90% Static

HMM 95% 85% 93% Static – Dynamic

Results and Statistics

Classifiers Conclusion

• HMM is more accurate than Rule-based and support Dynamic states

• Rule-based is complex to detect specific threshold for different bodies

Gesture Detection Flow

• Not all features are relevant to all gestures.

Each gestures has its own feature vector.

• More than one Gesture can happen at the same time. We group related gestures together under respective limbs.

• Gestures can be related to more than one limb We divide gestures into parts called “states”.

Problems:

Gesture Detection Flow

Right

Arm

Hand Over Hand

Hand On Waist

Hand In PocketLeft Arm

Hand Over Hand

Hand On Waist

Hand In Pocket

Right Leg

Cross Leg

Up Leg

Left Leg

Cross Leg

Up Leg

body

Leaning left

Leaning right

Gesture Detection Flow

Flow :

• Each Gesture has a certain condition on the detected states.• Body consists of 5 Limbs.• Each Limb has a most probable state.• Each state has a classifier object that receives the feature vector.• Each state has its own feature vector.

Gesture Detection Flow

Gesture types :

• Static Gestures : No movement involved Happens when any state from a group of states happen. Happens when all states from a group of states happen.

• Dynamic Gestures : Requires the body to move Happens when a sequence of states happen in a short period of time.

Speech Recognition

Speech Recognition• Speech recognition process

Speech Recognition• Presentation application (Filler Words)

Fillers words and phrases people use to cover verbal gaps—are word crutches. Presenters often use them out of fear.

• The most common fillers are: So, And, All right, Okay, Like, Now, Well, You know,

Right, Um and Uh.

English Test phases

First Phase "Put Question"

Second Phase "Paragraph Test "

Third Phase “Knowledge "

Conclusion• HMM is more accurate than Rule-based Classifiers.• Kinect is the best device to use due to infrared feature.• Kinect V2 is better than Kinect V1 in Joint detection.• Kinect V2 has face Joints property over Kinect V1.

Future Work• Interviewing Enhancement• Try other classifiers seeking better accuracy• Provide the Grammar Builder with more alternatives

Sponsorship

Presentation Demo

Interview Demo