coherence in one-shot gesture recognition for human-robot ...shapiro/ee562/notes/robotics.pdfgesture...

Coherence in One-Shot Gesture Recognition for Human-Robot Interaction

Maru Cabrera

December 3rd 2018

12/3/2018AI FOR ENGINEERS: COHERENCE IN GESTURE RECOGNITION FOR HRI -

MARU CABRERA1

Relevance of Gestures in HRI

vs

❖ Human have the unique ability to quickly adjust their context and learn from very few examples.


MARU CABRERA2

Motivation for One-Shot Gesture Recognition

❖ Lack of a comprehensive method that generalizes gesture recognition from few observations.

❖We focus on the process used to generate a given gesture:❖Cognition❖Learning and Generalization❖Physical execution


MARU CABRERA3

Motivation for Coherence in Gesture Recognition

❖ Explore gesture recognition and understanding when the roles between performer and listener are exchanged.

❖ Including the human aspect within the framework to artificially generate examples.


MARU CABRERA4

Overview of One-Shot Learning Framework

Skeleton Info Gesture example“Gist of

Gesture”Generated Dataset

Train Classifier

HMM

SVM

DTW

CRF

Artificial Generation Methods

Kinect Sensor

Performance Metrics

- Recognition accuracy- Efficiency- Coherency

Forward Method

Backward Method

1

2

3

4

5 6


MARU CABRERA5

Extracting the Gist of the Gesture


MARU CABRERA6

ൟ𝒙𝒒 ∈ {𝒙1 ∪ 𝐼𝑃𝐷 ∩ 𝐼𝑃𝐶 ∪ 𝒙𝑯, 𝑞 = 1,… , 𝑄

2 ≤ 𝑄 ≤ 𝐻

Given 𝑔1𝑖 = {(𝑥1, 𝑦1, 𝑧1 ), . . . , (𝑥𝐻, 𝑦𝐻 , 𝑧𝐻)}

Set of inflection points

𝒙𝒒 = (𝑥𝑞 , 𝑦𝑞 , 𝑧𝑞)

Working Hypothesis:

Compact amount of information stored during cognitive processes of

gesture perception

Validating Extracted Gist of the Gesture


MARU CABRERA7

17 Participants

EEG Power Dynamic wavelets

Motion EEG

Correlation



MARU CABRERA8



MARU CABRERA9




Train Classifier

HMM

SVM

DTW

CRF


Kinect Sensor

Performance Metrics


Forward Method

Backward Method

1

2

3

4

5 6


MARU CABRERA

Artificial Gesture Generation –Forward Approach


MARU CABRERA11

Input:𝒈𝟏𝒔 – 3D hand trajectory of a gesture of class s𝒙𝒔 = (𝒙𝒔,𝒚𝒔,𝒛𝒔) – 3D position of the shoulder

K – Number of artificial trajectories to generate

Variance Estimation based on 𝒙𝒔 and Gaussian Mixture Model (GMM)

Output:

ൟ𝑮𝒔 = { ො𝑔1𝒔, ො𝑔2

𝒔, … ො𝑔𝐾𝒔 – Set of artificial trajectories

for gesture class s

ො𝑔𝑘𝑖 = 𝒜𝑡

෨𝐺𝑖 𝑘 = 1,… , 𝐾 ; 𝑖 = 1,… , 𝑁 ; 𝑡 = 1

ൟ𝑮𝑖 = { ො𝑔1𝑖 , ො𝑔2

𝑖 , … ො𝑔𝑘𝑖 , … , ො𝑔𝑁

𝑖

Artificial Gesture Generation –Backward Approach


MARU CABRERA12

Artificial Gesture Generation –Backward Approach


MARU CABRERA13

Jerk Minimization

Smooth changes in joint space usingthe third derivative of the joint angle

Least Energy Expenditure

Changes in joint space using torqueto calculate economic trajectories

𝑆𝐽𝑒𝑟𝑘 = min

𝑖=1

𝐾

න𝑡1

𝑡2ሸ𝜃𝑖 +⋯+min

𝑖=1

𝐾

න𝑡𝑄−1

𝑡𝑄ሸ𝜃𝑖 𝑆𝐸𝑛𝑒𝑟𝑔𝑦 = min

𝑖=1

𝐾

න𝑡1

𝑡2

𝜏𝑖 × ሶ𝜃𝑖 +⋯+

𝑖=1

𝐾

න𝑡𝑄−1

𝑡𝑄

𝜏𝑖 × ሶ𝜃𝑖

Begins in one IK solution for human arm (k= 7) for one IP 𝓈𝑞𝑣, ends in

solution for different IP 𝓈𝑞+1𝑤

𝓈𝑞𝑣 = 𝜃1

𝑣, 𝜃2𝑣, … , 𝜃𝐾

𝑣

𝓈𝑞+1𝑤 = 𝜃1

𝑤, 𝜃2𝑤, … , 𝜃𝐾

𝑤

Initial

Final

𝑉𝑞 solutions in IP 𝑞 of gesture trajectory

𝑊𝑞+1 solutions in IP 𝑞 + 1

Artificial Gesture Generation –Combined F+B Approach


MARU CABRERA14

Combination at gesture instance level.

Artificially generated observations from both approaches used as training data




Train Classifier

HMM

SVM

DTW

CRF


Kinect Sensor

Performance Metrics


Forward Method

Backward Method

1

2

3

4

5 6


MARU CABRERA

Gesture LexiconMSRC-12


MARU CABRERA16

Contains sequences of human movements (Kinect skeleton)

12 iconic and metaphoric gestures

◦ Gaming commands and media player

Lexicon reduced to 8 gestures◦ Excluded gestures with leg

motions or whole upper body.

Fothergill, S., Mentis, H., Kohli, P., & Nowozin, S. (2012). Instructing people for training gestural interactive systems. In Proceedings of the SIGCHI Conference (pp. 1737-1746). ACM.

Classification Algorithms


MARU CABRERA17

Hidden Markov Models (HMM):

✓ One-vs-all scheme

✓ 5 states left-to-right

✓ Baum-Welch algorithm

Support Vector Machines (SVM):

✓ One-vs-all scheme

✓ Radial Basis Function (RBF) Kernel

✓ MATLAB® library

Conditional Random Fields (CRF):

✓ Multi-class scheme

✓ Samples encoded using BIO: Beginning, Inside, Outside

✓ CRF++ toolkit

Dynamic Time Warping (DTW):

✓ Multi-class scheme

✓ Gesture Recognition Toolkit (GRT)

Training and Features


MARU CABRERA18

1 gesture instance per class was

used to extract inflection points

200 gesture examples per class

generated for training

100 gesture instances per class

from each data set for testing

Classifiers trained and compared

in terms of accuracy (𝐴𝑐𝑐%), and

recognition coherence (𝛾)

Feature vector* 𝑔𝑖 for training:

𝑔𝑖 = {𝒙𝟏 = 𝑥1, 𝑦1, 𝑧1 , … , 𝒙𝑯}

𝑔𝑖 = ሶ𝒙𝟏 𝜷𝟏… ሶ𝒙𝒋 𝜷𝒋… ሶ𝒙𝑯−𝟏 𝜷𝑯−𝟏

ሶ𝒙𝒋 = 𝒙𝒋+𝟏 − 𝒙𝒋 𝜷𝒋 = tan−1𝑦𝑗

𝑥𝑗, tan−1

𝑧𝑗

𝑦𝑗, tan−1

𝑧𝑗

𝑥𝑗

* M. G. Jacob and J. P. Wachs, “Context-based hand gesture recognition for the

operating room,” Pattern Recognition Letters, vol. 36, pp. 196–203, Jan. 2014.




Train Classifier

HMM

SVM

DTW

CRF


Kinect Sensor

Performance Metrics


Forward Method

Backward Method

1

2

3

4

5 6


MARU CABRERA

Performance Metrics

Coherence (g) is defined asthe intersection betweenthe agreement indices (AIx)for humans and machines,whether each agentcorrectly recognized eachgesture or not. 𝛾 =

𝐴𝐼𝑥𝑀 ∩ 𝐴𝐼𝑥𝐻

𝐴𝐼𝑥𝐻


MARU CABRERA20

𝐴% =𝑡𝑜𝑡𝑎𝑙𝑡𝑟𝑢𝑒−ℎ𝑖𝑡𝑠𝑡𝑜𝑡𝑎𝑙𝑠𝑎𝑚𝑝𝑙𝑒𝑠

Accuracy Results


MARU CABRERA21

Forward and Backward approach for One-Shot Gesture Learning compared.

K-fold cross validation scheme with k = 10.

10 𝐴𝑐𝑐% values for m and s for each approach.

Coherence Experiment Setting

❖ Two scenarios were explored with Baxter performing artificially generated gestures:❖ Scenario 1 (MH): Gestures are

recognized by 10 human participants.

❖ Scenario 2 (MM): Gestures are recognized by 4 classification algorithms.


MARU CABRERA22

Coherence Experiment Setting


MARU CABRERA23

Experimental Results


MARU CABRERA24

Takeaway Messages❖ In its application to one-shot learning, the proposed methodhighlights the use of context for gesture recognition from the wayhumans use their bodies.

❖ The obtained results show the performance of the method,demonstrating independence from the selected classification strategy.

❖ The robotic implementation opens a different route towardscoherence in human–robot interaction.

❖ Coherence can be related to gesture classification when humans andmachines interchange the roles of performing and recognizing agesture.

❖ The calculated coherence metric is our main indicator that thegenerated gestures capture human-like variations for all the gestureclasses.


MARU CABRERA25

Maru Cabrera

[email protected]

Coherence in One-Shot Gesture Recognition for Human-Robot Interaction

mailto:[email protected]

coherence in one-shot gesture recognition for human-robot ...shapiro/ee562/notes/robotics.pdfgesture...

Documents