dialog design - gesture & pen interfaces this material has been developed by georgia tech hci...
DESCRIPTION
Many Types of Gesture-based UIs Purpose Text input vs. selecting objects and actions on objects Gestures constrained to 2D surface vs. not constrained (i.e., 3D) Single vs. multiple points One finger vs. multi-touch Sensed via something in/on user’s hand vs. not Stylus on tablet vs. finger on touchpad Data glove vs. video for hand gestures Size of sensing area iPhone – iPad – tablet computer – table-top – wall - room Type/scale of user motor control Finger movements vs. hand movements vs. body movements Work area used by one user vs. more than one UI Design - Georgia Tech 3TRANSCRIPT
![Page 1: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/1.jpg)
Dialog Design - Gesture & Pen Interfaces
This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory Abowd, Jim Foley, Elizabeth Mynatt, Jeff Pierce, Colin Potts, Chris Shaw, John Stasko, and Bruce Walker. Comments directed to [email protected] are encouraged. Permission is granted to use with acknowledgement for non-profit purposes. Last revision: November 2010.
![Page 2: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/2.jpg)
Dialog Styles1. Command languages 2. WIMP - Window, Icon, Menu, Pointer3. Direct manipulation4. Speech/natural language5. Gesture & pen
UI Design - Georgia Tech 2
![Page 3: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/3.jpg)
Many Types of Gesture-based UIs• Purpose
Text input vs. selecting objects and actions on objects• Gestures constrained to 2D surface vs. not constrained
(i.e., 3D)• Single vs. multiple points
One finger vs. multi-touch• Sensed via something in/on user’s hand vs. not
Stylus on tablet vs. finger on touchpad Data glove vs. video for hand gestures
• Size of sensing area iPhone – iPad – tablet computer – table-top – wall - room
• Type/scale of user motor control Finger movements vs. hand movements vs. body
movements• Work area used by one user vs. more than one
UI Design - Georgia Tech 3
![Page 4: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/4.jpg)
Agenda• Look at several points in the gesture
design space PDAs such as iPhone, iPad, Palm Pilot
Text input with some more general usesScroll, resize, rotate
Stylus or fingers (for multi-touch) Multi-touch, shared work space 3d Gestures, video recognition
UI Design - Georgia Tech 4
![Page 5: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/5.jpg)
PDAs – the New Swiss Army Knife
UI Design - Georgia Tech 5
![Page 6: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/6.jpg)
What is a Personal Digital Assistant?• A multi-function, portable device supporting
activities such as Communications by voice, text, video Listen, read, watch – audio, news, books, video Web browsing Calendaring Gaming Social networking Navigating from point A to point B Managing personal information of all sorts Etc etc etc Aka iPhone, iPad – a small but powerful computer
with pen or gesture input
UI Design - Georgia Tech 6
![Page 7: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/7.jpg)
Personal Digital Asst. (PDA)
UI Design - Georgia Tech 7
Palm VII
HP Jornada Handspring VisorCirca 2000
Palm IIIc
Apple Newton (1993)
Been around for a longggg time
![Page 8: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/8.jpg)
Contemporary PDAs
UI Design - Georgia Tech 8
iPhone
Acer Android Tablet
Motorola
Nokia BlackBerry
iPad
![Page 9: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/9.jpg)
Interacting with a PDA• Text input methods
Soft keyboard Free-form ink – no recognition Stylized characters to facilitate recognition Superimposed on keyboard Block printing recognition Script recognition
• Stylus or finger
UI Design - Georgia Tech 9
![Page 10: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/10.jpg)
Soft KeyboardsPros/cons of soft keyboards vs. hard keyboards?Finger vs. stylus soft keyboards?
UI Design - Georgia Tech 10
![Page 11: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/11.jpg)
Free-form Ink• Ink is the data, take as is
• Human is responsible forunderstanding andinterpretation
• Like a sketch pad
UI Design - Georgia Tech 11
![Page 12: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/12.jpg)
Examples• Digital Ink - CMU
Video, CHI ‘98 View it at www. ……..
• Flatland - Xerox PARC Video, CHI ‘99 View it at www. ….
UI Design - Georgia Tech 12
![Page 13: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/13.jpg)
Gesture Recognition - Graffiti• Stylized characters to facilitate
recognition• Graffiti - Unistroke alphabet on Palm
PDA Unistroke = one stroke
UI Design - Georgia Tech 13
![Page 14: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/14.jpg)
Gesture on Keyboard - Cirrin• Developed by Jen Mankoff (GT Ph.D -
> Berkeley -> CMU)• Word-level unistroke technique• UIST ‘98 paper• Use stylus to go
from one letterto the next ->
UI Design - Georgia Tech 14
![Page 15: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/15.jpg)
Gesture on Keyboard - Quikwriting • Developed by Ken Perlin• UIST ‘98 paper
UI Design - Georgia Tech 15
![Page 16: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/16.jpg)
Quikwriting
UI Design - Georgia Tech 16
p l
http://mrl.nyu.edu/projects/quikwriting/
e
Said to be as fast as graffiti, but have to learn more
![Page 17: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/17.jpg)
Gesture Recognition - Shapewriter
• Aka Shark• Live demo on iPad
UI Design - Georgia Tech 17
![Page 18: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/18.jpg)
Gesture Recognition – ShapeWriter/Shark Video
UI Design - Georgia Tech 18
![Page 19: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/19.jpg)
Recognizing Block Printing and Script• Recognizing letters and numbers and
special symbols • Lots of commercial systems• English, kanji, etc.• Not perfect, but people aren’t either!
People - 96% handprinted single characters Computer - >97% is really good
• OCR (Optical Character Recognition)
UI Design - Georgia Tech 19
![Page 20: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/20.jpg)
Recognition Issues• Off-line vs. On-line
Off-line: After all writing is done, speed not an issue, only quality.
Work with either a bit map or vector sequence On-line: Must respond in real-time - but have
richer set of features - acceleration, velocity, pressure
• Use best-guess pattern matching, including digram, trigram probabilities and word lists to remove ambiguity and to correct errors 1 I l On-line: show choices or best guess
UI Design - Georgia Tech 20
![Page 21: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/21.jpg)
More Issues• Boxed vs. Free-Form input
Sometimes encounter boxes on forms• Printed vs. Cursive
Cursive is much more difficult to impossible• Letters vs. Words
Cursive is easier to do in words vs individual letters, as words create more context
Similar to typo correction on phones and tablets
UI Design - Georgia Tech 21
![Page 22: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/22.jpg)
Recognizing Printed Characters
UI Design - Georgia Tech 22
![Page 23: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/23.jpg)
Beyond Text: Gestures – single point
UI Design - Georgia Tech 23
-Might mean delete
-Insert
-Paragraph
Define a series of (hopefully) simple drawing gesturesthat mean different commands in a system
![Page 24: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/24.jpg)
Brown’s Line-o-grammer
UI Design - Georgia Tech 24
![Page 25: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/25.jpg)
Beyond Text: Gestures – Multiple Points
UI Design - Georgia Tech 25
Mac Trackpad: up to four points of input
For power users
![Page 26: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/26.jpg)
Pen Use Modes• Inputting text vs. selecting vs. action
On iPhone/iPadEnter text mode by selecting a text entry box
Single touch => selectSingle touch + depress => move actionMulti touch => other actions
• Other ways to switch modes? Mode icon(s) – for instance to draw lines
vs. print text Others with which you are familiar??
UI Design - Georgia Tech 26
![Page 27: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/27.jpg)
Creative Uses of Multi-touch Surfaces
• Liquid Text – Craig Tashman On my Mac
LiquidText | Multitouch Document Manipulation.flv
• Jeff Han http://www.ted.com/talks/
jeff_han_demos_his_breakthrough_touchscreen.html
• BrailleTouchUI Design - Georgia Tech 27
![Page 28: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/28.jpg)
Wall-size Single Touch => Single User
UI Design - Georgia Tech 28
Complete withpie menu
![Page 29: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/29.jpg)
Multi-touch => Multi-User• Multi-touch permits multi-users
Two users can simultaneously operate on two areas or display or on two objects
• But can’t determine which user is doing what
UI Design - Georgia Tech 29
![Page 30: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/30.jpg)
Wall-size, Multi-Touch => Multi-User
UI Design - Georgia Tech 30
![Page 31: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/31.jpg)
Table-size, Multi-Touch: MS Surface
UI Design - Georgia Tech 31
• Simultaneous user multi-touch
• No user ID• No pressure• Modest price
![Page 32: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/32.jpg)
Microsoft Surface•
UI Design - Georgia Tech 32
![Page 33: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/33.jpg)
Table-size, Multi-Touch: Mitsubishi Diamond Touch• Simultaneous
user multi-touch
• Pressure!!• User ID!!• Expensive!!!!
UI Design - Georgia Tech 33
![Page 34: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/34.jpg)
Diamond Touch Video• On my Mac
Diamond Touch CeBit 2011 – YouTube.flv
UI Design - Georgia Tech 34
![Page 35: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/35.jpg)
3D Gestures, Video Recognition
• Imaginary• On my Mac
interactive ui in minority report.flv
UI Design - Georgia Tech 35
![Page 36: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/36.jpg)
3D Gestures, Video Recognition• This one is real• On my Mac
Hands Free 3D_ Second Life Object Editing Demo.flv
UI Design - Georgia Tech 36
![Page 37: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/37.jpg)
A Different Application• Signature verification
• But not with a mouse :)
UI Design - Georgia Tech 37
![Page 38: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/38.jpg)
Pen-less Gestures• As in iPhone & iPad – 2D• Minority Report
http://www.youtube.com/watch?v=NwVBzx0LMNQ&feature=related
• Toshiba http://www.youtube.com/watch?v=MDUN01U--jE&feature=fvsr
• Multi-touch http://www.youtube.com/watch?v=ok1n0lwY1ZA&feature=related
• Put that there – gesture & speech http://www.youtube.com/watch?v=RyBEUyEtxQo
UI Design - Georgia Tech 38
![Page 39: Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory](https://reader036.vdocuments.us/reader036/viewer/2022062600/5a4d1b257f8b9ab05999700b/html5/thumbnails/39.jpg)
The End
UI Design - Georgia Tech 39