Huddle and Information Triggers: Helping Users Automate Complex and Temporally-Distant Tasks

Jeffrey NicholsIBM Almaden Research Center

UC BerkeleyJune 14, 2007

2

Today – Two Mini Talks

Huddle• Automatic generation of

interfaces for systems of connected appliances

• Part of my dissertation work on the Personal Universal Controller

Information Triggers• Helps users recall important

information or tasks at the moment they are important

• New project at IBM Almaden Research Center

• In collaboration with Jake Wobbrock at the University of Washington

3

The Multi-Appliance Problem

4

The Multi-Appliance Problem, cont.

How can we provide improved interfaces for systems of connected appliances?

5

Control of Multiple Appliances

Built-in Interfaces

Universal Remote Controls

Custom Installations

6

Huddle

Improved multi-appliance interfaces based on content flow

Flow-Based Interface

Aggregate User Interfaces

7

Personal Universal Controller (PUC)

• Automatically generates interfaces for remotely controlling appliances [Nichols, UIST 2002]

• Specification language for describing appliance functionality

Dependency information that describes the availability of functions based on appliance state

• Automatically generate new interfaces to be consistent with previous interfaces [Nichols, CHI 2006]

Knowledge base that identifies and stores similarities between appliances

8

Related WorkRoadie [Lieberman, IUI 2006]

Goal-oriented natural language interface for appliancesUses planner to configure appliances based on common sense database (EventNet)

ICrafter [Ponnekanti, Ubicomp 2002]

Builds aggregate user interfaces based on “service interfaces”Hand-coded interface aggregatorsLimited automatic generation capabilities

Omojokun, et al. [Omojokun 2006]

Examined how remote control interfaces might be generated from recordings of actual usageRecorded remote control usage with IR receiver

General Work on Interface Generation• Task-based work: TERESA [Mori 2004], XIML [Puerta, IUI 2002], others…• SUPPLE [Gajos, IUI 2004]

9

Huddle Overview

• Introduction

• Content Flow Modeling

• Flow-Based Interface

• Aggregate User Interfaces

• Discussion and Future Work

10

Content Flow Model

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

Home Theater

11

Modeling Tasks

Home Theater: Watching Television

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

12

Modeling Tasks

Home Theater: Watching Television

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

13

Modeling Tasks

Home Theater: Watching Television

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

14

Modeling Tasks

Home Theater: Watch DVD

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

15

Modeling Tasks

Home Theater: Watch DVD + Copy Tape

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

16

Modeling Tasks

Presentation Room

VCR

PowerPoint

Laptop Task

Manager

External Video

Control

Projector

Media Player

17

Modeling Tasks

Presentation Room: Presenting PowerPoint

VCR

PowerPoint

Projector

Media Player

Laptop Task

Manager

External Video

Control

18

Where does the model come from?

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

19

Where does the model come from?

DVD

TVVCR #1

Receiver

VCR #2

20

Where does the model come from?

DVD TVVCR #1 Receiver

VCR #2

21

Flow-Based Interface

Flow-Based Interface (FBI)

• Allows user to manage high-level tasks

• Quick specification of flows

• AI Planner automatically configures appliances

• Question/Answer interface to address conflicts

22

Flow-Based Interface

23

Flow-Based Interface

24

Flow-Based Interface

25

Flow-Based Interface

26

Flow-Based Interface

27

Flow-Based Interface

28

Flow-Based Interface

29

Flow-Based Interface

30

Flow-Based Interface

31

Aggregate User Interfaces (AUIs)• Allows user to manage low-level

details

• Control or setup of active flows

• General setup of all appliances

• Merged functionsAggregate Controls for “Watch

DVD”

32

Aggregate User Interfaces

Three kinds• Control of Active Flows

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

33

Aggregate User Interfaces

Three kinds• Control of Active Flows

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

34

Aggregate User Interfaces

Three kinds• Control of Active Flows

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

35

Aggregate User Interfaces

Three kinds• Control of Active Flows

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

36

Aggregate User Interfaces

Three kinds• Control of Active Flows

DVD

TVVCR #1

BroadcastTelevision

Receiver

VCR #2

37

Aggregate User Interfaces

Three kinds• Control of Active Flows

“Copy Tape” “Watch TV using TV speakers”

“Watch TV using Receiver

speakers”

Volume for TV

Volume for Receiver

38

Aggregate User Interfaces

Three kinds• Control of Active Flows

• Setup of Active Flows

Two screens of “Watch DVD” setup

39

Aggregate User Interfaces

Three kinds• Control of Active Flows

• Setup of Active Flows

• General Setup of all appliances

Two screens of General Setup for a home theater config.

40

Aggregate User Interfaces

Three kinds• Control of Active Flows

• Setup of Active Flows

• General Setup of all appliances

Merged Functions• Most functions shouldn’t be

merged

• Some should: clock, language, sleep timer…

• Future work…Baudisch’s flat volume controlGeneral merging

Some merged functions

41

Discussion

How can our use of content flow be extended?• Lighting in a presentation room

• Video-conferencing rooms

• Outside the home: factories & offices

• Kitchen?

42

Future Work

Extending to more detailed content types• Home theater: support for HD signals, encoded audio, etc.

Setup and Troubleshooting• Tool for specifying initial wiring• Helping with Problems

More aggregate interfaces• Usage-based aggregate?

Usability evaluations

43

Information Triggers

44

Information Triggers

Motivation

It’s easy to find stuff on the web

It’s even easy to re-find something you’ve previously found...

...if you remember to re-find it when you need it

“Participants [in a re-finding study] complained that they often forgot to look for an information item until the period of its usefulness had

passed.” (Jones et al. 2005)

45

Information Triggers

What we’ve built so far

A Firefox extension that allows people to associate web pages with future events at which that page will be particularly relevant

When the event occurs, the web page is automatically opened for the user

Goal is to make triggers as easy and lightweight to configure as possible

Supports three different types of triggers:

•Date & time

•Arrival of an e-mail

•Visit to a web page

46

Beyond Current Reminder SystemsExisting reminder systems provide a short message about their topic, but info triggers provides the information itself

Context-based reminder systems (e.g. Cybreminder) have mainly used location for context

47

ScenariosWaiting to login to a website to configure account information (date & time trigger)

48

ScenariosResuming a task that can’t be completed in one sitting, e.g. waiting for manager approval (new e-mail trigger)

49

ScenariosOpening a note-taking site when visiting a page with specific keywords (page visit trigger)

information retrieval

web pages

50

ScenariosDelaying a task until a leisure period (page visit trigger)

51

ScenariosOpening a set of pages all at once, e.g. opening one bill payment site opens others (page visit trigger)

52

ScenariosAutomating routine browsing tasks (date & time trigger)

53

Demo

54

User FeedbackDistributed the extension within IBM, currently have several regular users

Needed features• Means for configuring mail triggers within mail reader• Ability to share triggers with others

Uses include:• Reminders of information related to a current event

• Automatically display the web page/blog of a person you are about to meet with

• Regular reminders of a task prior to a deadline• Make the conference registration page appear every morning on the week

before the early registration deadline• Automating phone call forwarding

• One user has a set of triggers that are able to automatically start and stop call forwarding from the office on days that he regularly works from home

• Automating regular browsing tasks• Weekly entry of status reports

55

Discussion

Programming triggers can be hard or require being “clever”

• Can’t always predict when some information will be useful• Current set of triggers is powerful, but making them do what

you want may not be straight-forward• Delaying until leisure time trigger• Triggering the next time you open your browser• Triggers that open pages that trigger other things

Where should triggers be set?• Triggered information vs. triggering information• Need usability studies to improve authoring of triggers

Deferring Triggered Information• Moving the window to the background can be sufficient• May need an additional explicit mechanism

56

Future Work

Other Trigger Events?• What other context information might be useful to trigger on?

Sharing Triggers• Can some of the overhead of sharing triggers be offset using

social mechanisms?

Distributed Triggers• No one is always at their browser• Trigger on mobile devices or across a user’s machines

Trigger Queues• A trigger that fires regularly associated with a queue of pages• Each time the trigger fires the next item from the queue is

displayed• “Interesting new articles from today”, “Quick online to-do items”,

“Papers to read”, etc.

57

Future Work, cont.

Usability and Usage Studies• Improve the interaction• Is it useful? When?

Deeper Questions• How well can people anticipate when they will need

information?• How well can people anticipate when they will next be

interruptable?

58

IBM Almaden Research Center

400+ research employees; 100+ students and postdocs

Research in Computer Science, Storage Systems, Science and Technology, Services Science

HCI-related Departments:• User Focused Systems group in CS• Services Sciences research

59

USER Group at Almaden

Four main research thrusts:Activities

• Unified Activity Management• Koala

Web 2.0 and Social Computing• BlueMail• Fringe

Content Management• AACS, etc.

Interaction in Healthcare• Data entry for EMR• Communication management (awareness & context)• Patient-centered healthcare

Thanks for listening!

For more information…http://www.pebbles.hcii.cmu.edu/puc/http://www.almaden.ibm.com/http://www.jeffreynichols.com/

Huddle and Information Triggers: Helping Users Automate Complex and Temporally-Distant Tasks

Jeffrey NicholsIBM Almaden Research Center

UC BerkeleyJune 14, 2007

62

63

64

Info Triggers Demo

65

Info Triggers Demo

66

Info Triggers Demo

67

Info Triggers Demo

68

Info Triggers Demo

69

Info Triggers Demo

70

Info Triggers Demo

71

Info Triggers Demo

72

Info Triggers Demo