Context-aware / Multimodal UI Breakout Summary
James A. Landay, et. al.
HCC RetreatJuly 7, 2000
7/5/2000 2
Participants
James Landay Anoop Sinha Jimmy Lin Trevor Perring Greg Heinzinger Chris Long Ed Chi
Christine Halverson Gian Gonzaga Ken Fishkin John Lowe Adam Janin Russell Eames Elin Pedersen
7/5/2000 3
Applications
Alert management* sites beacon context
+“this is a quiet place, no interruptions please”+e.g., movie theater or restaurant
* devices use context to avoid interruptions* wearable t-shirt that jams local cell phones!
“Elvis has left the meeting”* easily share documents from meetings* beam tokens of documents to participants or* use shared context to find docs later
+e.g., I was in a meeting with Ken at Lake Tahoe, find docs
7/5/2000 4
Context Events
Signal changes* like a windowing system
Can use as triggers to cause other actions* change my phone forwarding when I change
locations
Can be immediate or logged for later tacit information mining
7/5/2000 5
Context Implementation Issues
Apps need to share context easily* build-in to apps like cut & paste* context dial tone or infrastructure
Global file system* easier to share context & not have to transfer it* just use pointers
How to search / browse* computers are good at searching large spaces* humans good at making associations
Why not search with Google instead of browser history?* Google easier to get at & seems to work well
7/5/2000 6
Context Toolkits / APIs / Refs
Bill Schilit’s Columbia / PARC Ph.D. GA Tech GVU (Anind Dey) IBM (Maria Ebling) MIT (?) ESPIRIT projects have looked at
context* German? project according to Elin Pedersen
7/5/2000 7
Interface Between Context & Multimodal UIs
“Context is just another kind of input”* different to the user, but similar to the system* user input caused by EXPLICIT user action* context is IMPLICT or DERRIVED
Context and multimodal UIs have similar privacy problems* natural inputs are human “readable”* may not want to share context or my input
7/5/2000 8
Interface Between Context & Multimodal UIs
Context to choose output modality* e.g., user is in a meeting, don’t use speech
Context to disambiguate input(s)* help fusion: there is noise, don’t rely on speech* “the clutching problem” – infer user intent
Modality used to help inter context* e.g., talking to device -> user is alone?
7/5/2000 9
Initial Design for Multimodal UI Design Tool
Create “rough cuts”* informal (sketching / “Wizard of Oz”)* iterative design (user testing/fast
mods)
Infer models from design* designer can augment model over time
Generate initial prototypes* UIs for multiple devices* designer adds detail / improve UI
+or even removes detail
Model
