IAT 334

Dialog Design - Gesture & Pen Interfaces, Mobile Devices

1

This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory Abowd, Jim Foley, Elizabeth Mynatt, Jeff Pierce, Colin Potts, Chris Shaw, John Stasko, Bruce Walker, and Melody Moore Jackson. Comments directed to foley@cc.gatech.edu are encouraged. Permission is granted to use with acknowledgement for non-profit purposes. Last revision: October 2007.

IAT 334

Dialog Styles

1. Command languages 2. WIMP - Window, Icon, Menu, Pointer3. Direct manipulation4. Speech/natural language5. Gesture & pen

2

IAT 334

Agenda

PDA overviewPen input styles

3

IAT 334

How to use a PDA

4

IAT 334

Personal Digital Asst. (PDA)

5

Palm Treo

Apple Newton (1993)

Apple iPhone

Blackberry Curve

IAT 334

PDAs

Now ubiquitousSmall displaysOften touch and pen interfacesSmall thumb-based keyboardsRecent Improvements

Wi-Fi, GPS, more memory, better CPU, better OS, BlueTooth

6

IAT 334

Is it a PDA? Phone? GPS? Camera? Computer?

Line between devices is blurred today

Apple iPhone – phone, MP3 player, PDA, camera

Palm Treo 700w – phone, Windows computer, PDA, camera

Nokia N82 – Phone, GPS, 2 cameras, robot brain

7

IAT 334 8

Cally/Callo

IAT 334

No Shredder…

9

IAT 334

Input Options Pen / Stylus is dominant form Main techniques

Free-form ink Soft keyboard Numeric keyboard => text Stroke recognition - strokes not in the shape of characters Hand printing / writing recognition

Sometimes have or can connect keyboard

10

IAT 334

Free-form Ink

Ink is the data, take as is

Human is responsible forunderstanding andinterpretation

Like a sketch pad

11

IAT 334

Soft Keyboards

Common on PDAs and mobile devices

12

IAT 334

Soft Keyboard

Presents a small diagram of keyboard

You click on buttons/keys with pen or finger

QWERTY vs. alphabetical Tradeoffs? Alternatives?

13

Apple iPhone soft keyboard

IAT 334

Numeric Keypad -T9

Tegic Communications developedYou press out letters of your word, it matches

the most likely word, then gives optional choices

Faster than multiple presses per keyUsed in mobile phoneswww.tegic.com/t9

14

IAT 334

Stroke Recognition - GraffitiGraffiti - Unistroke alphabet on Palm PDA

What are yourexperienceswith Graffiti?

Graffiti demo:http://www.youtube.com/watch?v=R-hbjG2hzuk

15

IAT 334

Stroke Recognition - Cirrin

Developed by Jen Mankoff (GT -> Berkeley CS Faculty)

Word-level unistroke techniqueUIST ‘98 paperUse stylus to go

from one letterto the next ->

Nokia N8 does similarwith QWERTY

layout 16

IAT 334 17

QuikWriting

• Break the gesture into octant components:• Start pen in center,– drag in one of 8 directions– drag along edge– drag to center

http://mrl.nyu.edu/~perlin/demos/quikwriting2_1.html

IAT 334

Hand Printing & Hand Writing Recognition

Recognizing letters and numbers and special symbols Lots of commercial systems English, kanji, etc. Not perfect, but people aren’t either!

People - 96% handprinted single characters Computer - >97% is really good

OCR (Optical Character Recognition)

18

IAT 334

Recognition Issues Off-line vs. On-line

Off-line: After all writing is done, speed not an issue, only quality.Work with either a bit map or vector sequence

On-line: Must respond in real-time - but have richer set of features - acceleration, velocity, pressure

Use best-guess pattern matching, including digram, trigram probabilities and word lists to remove ambiguity 1 I l

19

IAT 334

More Issues

Boxed vs. Free-Form inputSometimes encounter boxes on forms

Printed vs. CursiveCursive is much more difficult to impossible

Letters vs. WordsCursive is easier to do in words vs individual letters,

as words create more context

20

IAT 334

Pen Gesture Commands

21

- Might mean delete

- Insert

- Paragraph

Define a series of (hopefully) simple drawing gesturesthat mean different commands in a system

IAT 334

Pen Use Modes

Often, want a mix of free-form drawing and special commands

How does user switch modes?Mode icon on screenButton on penButton on device

22

IAT 334

Error CorrectionHaving to correct errors can slow input

tremendously

StrategiesErase and try againWhen uncertain system shows list of best guesses...

23

IAT 334

A Different Application

Signature verification

But not with a mouse :)

24

IAT 334

Multi-touch interfaces

Apple iPhone

25

Capacitive touchscreen:http://www.apple.com/iphone/features/index.html#touch

Gestures:flick, tap, pinch, un-pinch

http://www.apple.com/iphone/features/index.html#map

IAT 334 26

Pen Videos• Pick-and-Drop by Rekimoto

http://www.youtube.com/watch?v=rFw9aMubL-Y• I-Love-Sketch by Seok-Hyung Bae

http://www.youtube.com/watch?v=5Hd2clwURlQ• Jabberstamp by Hayes Raffle

http://www.youtube.com/watch?v=NIe-XDHcsOE• ShapeWriter iPhone App

http://www.youtube.com/watch?v=TOg91yfvZpo• Marginalia : The Hybrid Textbook

http://www.youtube.com/watch?v=zwYdmsdqqTk