Text Input: Techniques and Research Tools

Poika Isokoski

at NIT2003

30.2.2003

Background: A Collage of images scanned from: Albertine Gaur. A history of writing. The British Library, London, UK, 2 edition, 1987.

TampereUniversityComputerHumanInteractionGroup

30.1.2003

Text Input: Techniques and Research Tools

Contents

• Introduction

• Historical Notes

• Text Input Methods• Keyboards• Text Recognition• Pointing• Temporal

• Measuring Performance

• More Info

1

30.1.2003

Text Input: Techniques and Research Tools

Introduction

• For some time in the past text input was not a very interesting research topic• Desktop keyboard is so good that it cannot be easily beaten• Additional Inferior text input methods have not been needed

• Mobile computing has changed the situation• Keyboards are difficult to fit in a mobile phone or a PDA.• Handwriting recognition is difficult and writing is slow• Speech recognition is even more difficult• => There is no obvious solution.

• Is this there a real and lasting need for a new writing system(s)?

2

30.1.2003

Text Input: Techniques and Research Tools

Historical Notes

• Interplay of culture and writing• Culture chooses a writing system that best suits it

– need to communicate

– need for information storage

– available technology and materials

– needs of the individuals in power

• Good inventions are meaningless if there is no need for them

3

30.1.2003

Text Input: Techniques and Research Tools

Historical Notes

4

30.1.2003

Text Input: Techniques and Research Tools

Historical Notes

• Separation of interfaces

• For a long time there was only one interface• Written on paper (or whatever material the culture preferred)• Stored on paper• Read from paper

• Pen movement was connected to the form of the written character which again was tightly coupled with the form of the character to be read.

5

30.1.2003

Text Input: Techniques and Research Tools

Historical Notes

• After Gutenberg• Written in printing press (or with a typewriter)• Stored on paper• Read from paper

• Writing motion is not necessarily the same as the form of the characters

6

30.1.2003

Text Input: Techniques and Research Tools

Historical Notes

• Today• Written with text input methods• Stored as bits (magnetic fields, dots on glass, etc.)• Read from text output system.

• None of the tasks are mechanically connected. There is software in between.

• Having separate system for each phase:• Gives more freedom for optimization of each task• Requires more skills from the user of texts

7

30.1.2003

Text Input: Techniques and Research Tools

Historical Notes

M essag e S ticks

K n otted cord s

B ean p ou ch es

M n em on ic d evices

C h in ese

early S u m erian

H ie rog lyp h s

P ic tog rap h ic

Id eas

Jap an ese kan a

K orean

In d ian sc rip ts

S yllab ic

A rab ic

H eb rew

C on son an ta l

P rin tin g

H an d p rin tin g

H an d w rit in g

R om an

C yrillic

G reek

A lp h ab etic

S ou n d s

W rit in g

8

30.1.2003

Text Input: Techniques and Research Tools

Text Input Methods

S H K

P O B ox

T9 (T eg ic)

ea ton i LO PT

iTap (M o toro la)

Te lephone 12 -key

Ins tant T ext

E xce l

S ta rO ff ice

Q W E R TY

C ontex tual

s tandard m u lti

LO P T m u lti

dua l p ress

Te lephone 12 -key

C yr il l ic

M D IT IM

D vorak

Q W E R TY

FA S TA P

G K O S

TW ID D LER

M icrow riter

C H O R D

H a lf-Q W E R TY

C ontex t f ree

K eyboard

O ne dim ens ional

Tw o d im ens ional

M ach ine readab le

B oxed

U nboxed

C urs ive

O n-l ine

O C R

D ynam ic

O ff- line

H um an R eadab le

U n is trokes

G ra ff i ti

A l leg ro

C harac te r level

C irr in

Q u ikw rit ing

oc tave

D asher

W ord /sess ion level

U n istroke

Tex t recogn it ion

Q W E R TY

O P TI

F ITA L Y

C yril l ic

A TO M IK

S oft keyboards

M essageE ase

T -C ube

M enu hyb r ids

D irec t (so f t keyboa rds)

7110 N av i-ro l ler

FO C L

Ind irect

P o in t ing

C ons tra ined

U nconstra ined

S peaker dependent

C ons tra ined

U nconstra ined

S peaker independent

S peech recogn it ion

S ign Language

G estu re recogn it ion

M orse C ode

Tem pora l input

Tex t input

9

S H K

P O B o x

T 9 (T eg ic)

ea ton i LO PT

iT a p (M o toro la)

T e lephon e 12 -key

Ins tant T ext

E xce l

S ta rO ff ice

Q W E R T Y

C on tex tual

s tan da rd m u lti

LO P T m u lti

dua l p re ss

T e lephon e 12 -key

C yr il l ic

M D IT IM

D vorak

Q W E R T Y

F A S T A P

G K O S

T W ID D L ER

M icrow riter

C H O R D

H a lf-Q W E R TY

C onte x t f ree

K eyboa rd

O ne dim ens io nal

T w o d im en s ional

M a ch ine readab le

B oxed

U nbo xed

C urs ive

O n-l ine

O C R

D ynam ic

O ff- line

H um an R ead ab le

U n is trokes

G ra ff i ti

A l leg ro

C harac te r level

C irr in

Q u ikw rit ing

oc ta ve

D asher

W ord /sess ion level

U n istroke

T e x t re co gn it ion

Q W E R T Y

O P T I

F IT A L Y

C yr il l ic

A T O M IK

S o ft ke yboa rds

M e ssageE ase

T -C ube

M e nu hyb r ids

D ire c t (so f t keyboa rds)

711 0 N av i-ro l ler

F O C L

Ind irect

P o in t ing

C ons tra ined

U ncon stra ined

S peake r dep ende nt

C ons tra ined

U ncon stra ined

S peake r inde pend ent

S peech recogn it ion

S ign La nguage

G es tu re recogn it ion

M o rse C o de

T e m pora l input

T e x t input

S H K

P O B ox

T9 (T eg ic)

ea ton i LO PT

iTap (M o toro la)

Te lephone 12 -key

Ins tant T ext

E xce l

S ta rO ff ice

Q W E R TY

C on tex tual

s tandard m u lti

LO P T m u lti

dua l p ress

Te lephone 12 -key

C yr il l ic

M D IT IM

D vorak

Q W E R TY

FA S TA P

G K O S

TW ID D LER

M icrow riter

C H O R D

H a lf-Q W E R TY

C ontex t f ree

K eyboard

O ne dim ens ional

Tw o d im ens ional

M ach ine readab le

B oxed

U nboxed

C urs ive

O n-l ine

O C R

D ynam ic

O ff- line

H um an R eadab le

U n is trokes

G ra ff i ti

A l leg ro

C harac te r level

C irr in

Q u ikw rit ing

oc tave

D asher

W ord /sess ion level

U n istroke

Tex t recogn it ion

Q W E R TY

O P TI

F ITA L Y

C yril l ic

A TO M IK

S o ft keyboards

M essageE ase

T -C ube

M enu hyb r ids

D irec t (so f t keyboa rds)

7110 N av i-ro l ler

FO C L

Ind irect

P o in t ing

C ons tra ined

U nconstra ined

S peaker dependent

C ons tra ined

U nconstra ined

S peaker independent

S peech recogn it ion

S ign Language

G es tu re recogn it ion

M orse C ode

Tem pora l input

Tex t input

30.1.2003

Text Input: Techniques and Research Tools

Keyboards

• Context free• QWERTY• 12-key multi-press• Chord – GKOS as an example

( http://gkos.com )

• Contextual• Instant Text• Microsoft Excel

12

30.1.2003

Text Input: Techniques and Research Tools

Keyboards

• How to measure performance• Speed and error rate (more on these later)• In desktop use: physical stress (dvorak-QWERTY-debate)

• How to model/predict performance• No good solution for multi-finger operation• For one-finger typing use same stuff as with soft-keyboards

(discussed later)• With multi-press/contextual methods consider the number of key

presses, finger travel, and the need for visual feedback.

13

30.1.2003

Text Input: Techniques and Research Tools

Text Recognition

• Machine readable• Bar-codes

• Human Readable• OCR• On-line handwriting recognition• Off-line recognition with information on writing dynamics

images from: http://www.adams1.com/pub/russadam/upccode.htmlhttp://www.adams1.com/pub/russadam/stack.html

14

30.1.2003

Text Input: Techniques and Research Tools

Text Recognition (2)

• Unistrokes• Explicit segmentation by lifting the pen• Character level: Unistrokes,

Graffiti

• Word level: octave

• ( http://www.e-acute.fr/English/manual/manualV1.html (not available since 2002 ))15

30.1.2003

Text Input: Techniques and Research Tools

Text Recognition

• How to measure performance• Speed• Human error rate• Recognition error rate• Need for training (user or algorithms)

• How to model (handwriting)• Models are complicated• Steering law • Models for post-mortem analysis/synthesis (non-predictive)• Model for unistroke writing (simple, but not very accurate)• All these models require some empirical data on the task, therefore

they cannot be used in pure prediction.

16

30.1.2003

Text Input: Techniques and Research Tools

Pointing

• Continuous gesturing (session level unistrokes)

• Dasher (web demo) ( http://wol.ra.phy.cam.ac.uk/djw30/dasher/ )

• Quikwriting (web demo) ( http://mrl.nyu.edu/~perlin/demos/quikwriting.html )

17

30.1.2003

Text Input: Techniques and Research Tools

Pointing

• Direct• Soft-keyboards:

qwerty, fitaly, OPTI (http://www.yorku.ca/

mack/CHI99a.html )

• Menu hybrids • MessageEase ( http://www.exideas.com/ )

• Indirect• FOCL

(http://www.yorku.ca/mack/GI98.html )

Q F U M C K ZO HT

W XAERSBI N D

F1YLGVPJ

space space

space space

QF UM

CK Z

OH

TW

X

A

ER

S B I

N

DYL

G V

PJ

spac

e

18

30.1.2003

Text Input: Techniques and Research Tools

Pointing

• How to measure performance• Speed and error rate

• How to model• Direct: Fitts’ law + statistics on the text to be written• Indirect: number of keypresses (independent KSPC).

19

30.1.2003

Text Input: Techniques and Research Tools

Temporal input

• Morse code

• How to measure performance• Speed and accuracy

• How to model• KSPC

20

30.1.2003

Text Input: Techniques and Research Tools

Fitts’ Law

• Fitts’ law

• T Time for pointing task• a,b determined empirically • A distance to target• W width of the target

More on Fitts’ law at: http://www.yorku.ca/mack/phd.html

)1(log2 W

AbaT

AW

21

30.1.2003

Text Input: Techniques and Research Tools

Fitts’ Law

• Steering Law

• TC Time for steering task C

• a,b empirically determined constants• W(s) width of the steering tunnel at point s• s trajectory being modeled

• Straight tube:

• Circle:

C

C sW

dsbaT

)(

More on Steering law: Johnny Accott and Shumin Zhai, Performance evaluation of Input Devices inTrajectory-based Tasks: An Application of The Steering Law, Proceedings of CHI’99, ACM.

W

AbaTC

W

RbaTC

2

W

A

R

W

22

30.1.2003

Text Input: Techniques and Research Tools

Measuring Performance

• Measuring speed

• What speed?

– Walk-up or expert or something in between?

– Error free or errors included and corrections included?

– Pure writing or in task context?

– The users, are they young, old, blind, one-handed?

– The list is endless. Measure under conditions that represent actual use or are comparable with other studies.

23

30.1.2003

Text Input: Techniques and Research Tools

Measuring Performance

• Measuring error rate• What is an error?

– A character in wrong position?

abba abba

abbba abbba

– How about corrections and corrections withing corrections?

• The best practice:– Compute string distance (levenshtein’s algorithm)

– Compute input/character (dependent KSPC)

– Edit distance gives the number of errors

– KSPC is a measure of the efficiency of the writing method including the effort needed for corrections to achieve the measured error rate.

More at: http://www.yorku.ca/mack/CHI01a.htm (CHI2001 Extended Abstracts)http://www.yorku.ca/mack/nordichi2002-shortpaper.html (NordiCHI)

24

30.1.2003

Text Input: Techniques and Research Tools

Tradeoffs

Speed Accuracy

Minimum input per character Freedom of input

Novice Speed Expert speed

Optimal device use Device Independence

25

30.1.2003

Text Input: Techniques and Research Tools

More Info

• My text input research page:http://www.cs.uta.fi/research/hci/interact/textinput/

• Links to other sites

• Bibliography

• Papers

26