pen-based gesture recognition

USI Seminar, May 30, 2007

Pen-Based Gesture Recognition

Beat Signer

Institute for Information Systems, ETH Zurich

8092 Zurich, Switzerland

http://vub.academia.edu/BeatSigner

USI Seminar, May 30, 2007 Beat Signer, [email protected] 2

Outline

Motivation

Digital Pen and Paper Functionality

Gesture recognition algorithms

Evaluation of algorithms

iGesture framework


Pen-Based User Interfaces

Light Pen Graphics Tablet

PDAs and Tablet PCs Digital Pen and Paper


Palm Graffiti Gestures


Microsoft Application Gestures

scratch-out erase content

triangle insert

square action item

star action item

check check-off

curlicue cut

double-curlicue copy

circle application-specific

double-circle paste

left-semicircle undo

right-semicircle redo

caret past/insert

inverted-caret insert

chevron-left application-specific

chevron-right application-specific

arrow-up application-specific


Customised Gestures


Gesture Design Guidelines

Different gestures should not look too similar

better recognition results

Gestures should be user friendly

intuitivity and usability

Gesture set size

Reuse of gestures

same semantics for different applications

application-specific gestures


Digital Pen and Paper

Developed by Anoto

Different pen manufacturers

Logitech, Maxel, Nokia, ...

Pattern space

60 million km2


Digital Ink Processing

Parse proprietary pen formats

position, timestamp, force, tilt, …

Stroke detection

pen up / down events

time difference between succeeding points

distance between succeeding points, …

Interpolation / Filtering


Note Representation

<note>

<trace>

<point>

<x>59.664</x><y>56.364</y>

<timestamp>1103123178464</timestamp>

<force>224.0</force>

</point>

<point>

<x>59.671</x><y>56.378</y>

<timestamp>1103123178484</timestamp>

<force>227.0</force>

</point>

...

<trace>

...

</note>


Ink Markup Language (InkML), W3C

XML language for representing digital ink

Pen movement data

<trace> and <traceFormat> elements

x,y coordinates, force, angles

Device-specific information

resolution, sample rate, …


Ink Markup Language (InkML)

<ink>

<trace>10 0, 9 14, 8 28, 7 42, 6 56, 6 70, 8 84, 8 98,

8 112, 9 126, 10 140, 13 154, 14 168, 17 182, 18 188

</trace>

<trace>130 155, 144 159, 158 160, 170 154, 179 143,

179 129, 166 125, 152 128, 140 136, 131 149, 126 163

</trace>

...

<inkSource manufacturer="magicomm" model="G303">

<sampleRate uniform="true" value="70"/>

<srcProperty name="weight" value="100" units="g"/>

</inkSource>

...

</ink>


Gesture Recognition Algorithms

Offline recognition algorithms

static image

Online recognition algorithms

spatio-temporal representation

Recognition methods

statistical classification, neural networks, …

Supported gesture types

single-stroke or multi-stroke


SiGeR Algorithm

Simple Gesture Recogniser (Scott Swigart)

Gestures described by regular expressions

N, NE, E, SE, S, SW, W and NW

E

W

NS

(E)+(N)+(W)+(S)+

(NE|E|SE)+(NW|N|NE)+

(SW|W|NW)+(SE|S|SW)+


SiGeR Algorithm ...

Use additional statistical information

Description = Directions [";" Constraints];

Directions = Direction ["," Directions];

Direction = "N"|"NE"|"E"|"SE"|"S"|"SW"|"W"|"NW";

Constraints = Constraint ["AND" Constraints];

Constraint = Operand Operator Operand;

Operator = "EQ"|"NEQ"|"GT"|"GTE"|"LT"|"LTE";

Operand = "N"|"NE"|"E"|"SE"|"S"|"SW"|"W"|"NW"|

"DIAGONAL"|"STRAIGHT"|"PROXIMITY"|"STOPPOINTS";


SiGeR Algorithm ...

Examples

E,N,W,S;STRAIGHT GT 0.8

AND PROXIMITY LT 0.2

E

W

NS

W,SW,SE,NE,N,S;

PROXIMITY GT 0.7

AND STRAIGHT GT 0.3


Rubine Algorithm, 1991

Statistical classification algorithm for single

stroke gestures (training / classification)

A gesture G is represented as vector of P

sample points

iiiiP tyxsssG ,,with , ,... 10

Feature vector f extracted from G

Ffff ,...1


Rubine Features

5

016

2

01

2

015

minmax

minmax4

2

minmax

2

minmax3

2

02

2

02

022

2

02

2

02

021

)(cos

arctan

)()(

)()(

)(sin

)()(

)(cos

f

xxf

yyxxf

xx

yyf

yyxxf

yyxx

yyf

yyxx

xxf

P

PP


Rubine Features …

0113

2

222

0121

22

11

2

1

10

2

1

9

11

11

2

0

22

8

11

5

017

max Let

arctanLet

Let

sin

ttf

t

yxfttt

fff

yxxx

yxyx

yxf

yyyxxx

f

yyf

P

i

iiP

iiii

P

ii

i

P

i

i

P

i

i

iiii

iiiii

P

i

ii

iiiiii

P


Rubine Training / Classification

Training phase

Recognition / classification phase

Optimal

Classifier Fccc www ˆ0ˆˆ ,...,gesture samples

for class c

F

i

iiccc fwwv1

ˆ0ˆˆ


SiGrid Algorithm

Signature algorithm

010000

010010

011010

011000

001010

001010

000010

000000

010001

010011

011011

011011

001001

001011

000011

000001

010101

010111

0111111

011111

001101

001111

000111

000101

010100

010110

011110

011110

001100

001110

000110

000100

110100

110110

111110

111100

101100

101110

100110

100100

110101

110111

111111

111101

101101

101111

100111

100101

110001

110011

111011

111001

101001

101011

100011

100001

110000

110010

111010

111000

101000

101010

10010

100000

Example

8 x 8 Grid

001101

001110

100110

101110

000110

101100

111100

111110

110110


Evaluation Grafitti Numbers

E-Rubine Rubine SiGrid SiGeR

147

3

0

134

15

1

132

15

3

133

4

13

Correct

Error

Reject

Number of gesture classes: 10

Training: 15 examples for each gesture class (collected by 1 person)

Test Samples: 150 (collected by 3 different persons)


Evaluation

RecallPrecision

RecallPrecisonMeasureF

FNTP

TPRecall

FPTP

TPPrecision

2

negative false :FN

positive false :FP

positive true:TP


Evaluation Grafitti Numbers

E-Rubine Rubine SiGrid SiGeR

147

3

0

0.980

1.000

0.990

134

15

1

0.899

0.993

0.944

132

15

3

0.898

0.978

0.936

133

4

13

0.970

0.911

0.940

Correct

F-Measure

Recall

Error

Reject

Precision



Test Samples: 150 (collected by 3 different persons)


Evaluation Grafitti Letters (1)

E-Rubine Rubine SiGrid

334

52

4

0.865

0.988

0.923

280

107

3

0.724

0.989

0.836

273

114

3

0.705

0.989

0.824

Correct

F-Measure

Recall

Error

Reject

Precision



Test Samples: 390 (collected by 3 persons)


Evaluation Grafitti Letters (2)


342

18

3

0.950

0.991

0.970

305

48

10

0.864

0.968

0.913

297

66

0

0.818

1.000

0.900

Correct

F-Measure

Recall

Error

Reject

Precision


Training: 4x4 examples for each gesture class (collected by 4 persons)

Test Samples: 363 (collected by 4 persons)


Evaluation MS Application Gestures


196

4

0

0.980

1.000

0.990

178

19

3

0.904

0.983

0.942

145

32

23

0.819

0.863

0.840

Correct

F-Measure

Recall

Error

Reject

Precision



Test Samples: 200 (collected by 1 person)


Multi-Stroke Gestures


Evaluation Multi-Stroke Gestures


75

0

0

1.000

1.000

1.000

72

3

0

0.960

1.000

0.980

73

2

0

0.973

1.000

0.986

Correct

F-Measure

Recall

Error

Reject

Precision



Test Samples: 75 (collected by the same person)


iGesture Framework

Exercise

download iGesture

analyse the design of the

framework

investigate the existing

algorithms

design and test your own

algorithm

build a small application

based on your recogniser


iGesture Architecture Overview

Common Data Structures

Management

Console

Evaluation

Tools

Recogniser


Gesture Representation


Recognition Algorithm


Persistence


iGesture Test Bench Tab


iGesture Admin Tab


iGesture Test Data Tab


Capture of Samples


Evaluation Tools


Recogniser API

Configuration configuration =

XMLTool.importConfiguration(new File(RUBINE));

GestureSet gestureSet =

XMLTool.importGestureSet(new File(RUBINE_SET));

configuration.addGestureSet(gestureSet);

recogniser = new Recogniser(configuration);

...

ResultSet result = recogniser.recognise(note);


Conclusions

Digital pen and paper technology

Gesture recognition algorithms

SiGeR, Rubine (E-Rubine), SiGrid

Evaluation

precision, recall and F-measure

General iGesture recognition framework

USI Seminar, May 30, 2007 Beat Signer, [email protected]

References

Beat Signer, Ueli Kurmann and Moira C. Norrie,

iGesture: A General Gesture Recognition

Framework, Proc. of ICDAR 2007, 9th Intl.

Conference on Document Analysis and

Recognition, Curitiba, Brazil, September 2007

42

USI Seminar, May 30, 2007 Beat Signer, [email protected]

References ...

Beat Signer, Moira C. Norrie and Ueli Kurmann,

iGesture: A Java Framework for the

Development and Deployment of Stroke-Based

Online Gesture Recognition Algorithms,

Technical Report ETH Zurich, TR561,

September 2007

43


References ...

Dean Rubine, Specifying Gestures by Example,

Proc. of ACM SIGGRAPH '91, 18th Intl.

Conference on Computer Graphics and

Interactive Techniques

iGesture Open Source Framework

http://www.igesture.org

pen-based gesture recognition

Technology

f f usi seminar

specific usi seminar

paper usi seminar

t0 usi seminar

km2 usi seminar

edubeatsigner usi seminar

cos f5usi seminar

multistrokeusi seminar