automatically generating high-quality user interfaces for appliances jeffrey nichols doctoral...

Automatically Generating High-Quality User Interfaces for Appliances

Jeffrey NicholsDoctoral Colloquium Presentation

Second International Conference on Pervasive Computing April 18, 2004

2

ProblemAppliances are complex and their user interfaces are often hard to use!

3

Thesis Statement

“ A system can automatically generate user interfaces on a wide variety of platforms for remotely controlling appliances where the user’s performance is better than with the manufacturers’ interfaces for the

appliances.”

4

Approach: Use Mobile Devices

Common, cheap, and often have ability to communicate10 million smart phones sold 11.2 million handheld units shipped in 2003120 million mobile phone subscribers

5

Approach, cont.

Specifications

Control

Feedback

Appliances Mobile Devices

Use mobile devices to control all appliances in the environment

Key FeaturesTwo-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation

6

Automatic Generation of UIs

Benefits• Multiple modalities (GUI + Speech UI)• All interfaces consistent for a user

• With conventions of the handheld• Even from multiple manufacturers

• Create user interfaces that control multiple connected appliances

e.g. a home theater

7

Important Work By Others

INCITS/V2 Standardization EffortAlternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002]

Controlling Appliances• Xweb [Olsen Jr., UIST 2000]• Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001]• Speakeasy [Newman, UIST 2002]

Model-based User Interface Work• UIDE [Sukaviriya, CHI 1993], HUMANOID [Szekely, CHI 1993]• MASTERMIND [Szekely 1995], TRIDENT [Vanderdonckt 1995]• eXtensible Interface Markup Language (XIML) [Puerta, IUI 2002]• User Interface Modeling Language (UIML) [Abrams, W3C 1999]

None of these systems have been validated to generate high-quality interfaces without designer intervention.

8

Research Approach

1. Hand-design remote control interfaces

2. Determine functional information needed from appliances to design user interfaces

3. Design language for describing appliance functions

4. Build interface generators for multiple platforms

5. Perform user studies to evaluate the interface generators

9

Outline

• Introduction

• Personal Universal Controller (PUC) System

• Handling High-Level Conventions

• Interface Consistency

• Generating Interfaces for the “Experience”

• Validation

• Conclusion

10

Architecture

PROTOCOL(two-way communicationof specification & state)

COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)

PUC DEVICES(automatic interface generation)



APPLIANCES(Stereo, Alarm Clock, etc.)

ADAPTOR(publishes description +

appliance state + controls appliance)

control

device specification & state feedback

11

Architecture









control


12

Architecture









control


13

Architecture









control


14

Architecture









control


15

Language Design

Informed by hand-designed interfaces

• What functional information was needed to create interfaces?

Additional Requirements• Support complete functionality of

appliance• No specific layout information• Only one way to specify anything

Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/

16

Language Elements

Elements• State variables & commands• Labels• Group tree• Dependency information

Example media player specification

• Play, stop, pause, next track, previous track

• Play list

17

Language Elements, cont.

State Variables and Commands• Represent functions of

appliance

• State variables have types• Boolean, Enumeration,

Integer, String, etc.

• Variables sufficient for most functions but not all

• e.g. “seek” button on a Radio

18


Label InformationOne label not suitable

everywhere• The optimal label length

changes with screen size• Speech interfaces may benefit

from pronunciation and text-to-speech information

“Label Dictionary”• A group of semantically similar

labels• Different lengths• Information for different

modalities

19


Label InformationOne label not suitable

everywhere• The optimal label length

changes with screen size• Speech interfaces may benefit

from pronunciation and text-to-speech information

“Label Dictionary”• A group of semantically similar

labels• Different lengths• Information for different

modalities

20


Group Tree• Specify organization

of functions• We use n-ary tree

with variables or commands at leaves

• Also used for specifying complex types

ListsUnions

21


Group Tree• Specify organization

of functions• We use n-ary tree

with variables or commands at leaves

• Also used for specifying complex types

ListsUnions

22


Dependency Information• Formulas that specify

when a variable or command is active in terms of other state variables

Equals, Greater Than, Less Than, Is DefinedLinked with logical operators (AND, OR)

• Allows feedback to user when a function is not available

23

Interface Generators

Generators for Two Modalities

Graphical• Desktop, PocketPC, and

Microsoft Smartphone

Speech• Collaboration with others

at Carnegie Mellon• Built on top of the PUC

framework• Implemented using

Universal Speech Interface (USI) techniques [Rosenfeld 2001]

24

Controlling AppliancesWe have built adaptors for many actual appliances

• Sony Digital Camcorder• Windows Media Player• Axis UPnP Pan & Tilt Camera• Lutron Lighting• X10 Lighting• Audiophase Shelf Stereo• AudioReQuest MP3 player• GM Vehicle Information System• GM Vehicle Climate Control

Written specifications for others• Elevator• Telephone/Answering Machine• GM Navigation System• Several Alarm Clocks

25

Outline

• Introduction





• Validation

• Conclusion

26

High-Level Conventions

Problem• Human

designers rely partly on conventions when making an interface

• Users expect their appliances to use conventions they know about

27

Smart Templates

Need a way to specify high-level information to interface generators

Solution• Mark groups with tags that identify

high-level informationmedia-controls, phone-dialpad, time, date, etc.

• Restrict the contents of groups so that interface generators can interpret the high-level meaning

• Standardize the tags and restrictions in advance, so that designers know what interface generators expect

28

Smart Templates, cont.

Features• Parameterized

• Specified using primitive elements of specification language

Renderable by any interface generator

29

Interfaces with Smart TemplatesPreliminary Implementation

A few templates: image, image-list, media-controls, time-duration

30

Continuing Work

Define and implement Smart Templates• date, mute, power, time-absolute, volume, etc.• Develop more as more appliances are specified

Combinations of templates• Less implementation cost than a new template• e.g. date and time-absolute

Use of templates with data already on controller device

• e.g. calendar and address information• Might allow user to enter address from contact list

into navigation system

31

Outline

• Introduction





• Validation

• Conclusion

32

Interface Consistency

PUC devices have a unique opportunity to provide consistency for the user

• Personal device• Used for interacting with most appliances

Two ways that PUC UIs can be made consistent• With other applications on the same device• With past interfaces for similar appliances

33

Consistency with Past Interfaces

Two sub-problems to address:

• SimilarityWhich functions of a new appliance are similar to the functions of interfaces generated in the past?

• ConsistencyFor similar functions, what rules from previous interfaces can be applied to ensure consistency?

34

Similarity Problem

Difficult to conclusively know whether two functions from different appliances are the same

• Very little semantic information in the specification language

Can estimate similarity based on properties of state variables

• Smart Template• Name• Group Name• Labels• Type

Improve estimate by looking at relationships between multiple similar variables

35

Similarity & Consistency Problems

new previous

new previous

new previous

sparse similarity

branch similarity significant similarity

36

Outline

• Introduction





• Validation

• Conclusion

37

The “Experience”

38

The “Experience”, cont. (MIT dorm)

How can a PUC device provide improved interfaces for connected systems?

39

Improving Interfaces with PUC

Generate an interface that aggregates all functions into one set of screens

Organized by task instead of appliance

Automatically create macros for frequently used functions

e.g. “Play DVD” would: 1. turn on television, DVD player, stereo2. turn off VCR3. set the TV and stereo sources to the DVD player4. instruct the user to insert a DVD (if necessary)5. play the DVD

How can appliance descriptions support these features?

40

Distributed Task Language

My plan:• Store task information within each

appliance description Use an existing task language

• Combine sub-tasks from each appliance in a connected system to create complete tasks

41

Outline

• Introduction





• Validation

• Conclusion

42

Two Goals for PUC System

BreadthThe appliance specification language is capable of describing a wide variety of appliances

QualityInterfaces generated for specifications across that range beat the usability of the manufacturers’ interfaces for the same appliances

How do I validate that these goals are met?

43

Evaluation

Evaluation Method1. Develop a list of appliances that are

interesting• Complexity• Unique feature• Representative of a class of appliances

2. Specify each appliance3. Perform a comparative user study on

several appliances to test quality

44

Appliance List (so far…comments?)

Already Specified & Generated Propose to Specify

Axis Pan & Tilt Camera GMC Denali Navigation System

Audiophase Stereo Phone/Answering Machine

GMC Denali Driver Information Center

Windows Media Player with playlists

GMC Denali Climate Control System

Automated Teller Machine (ATM)

Sony Camcorder Microsoft Windows XP Media Center PC

Elevator Simulation Photocopier

Lutron Home Lighting System TV Tuner

Windows Media Player without playlists

Personal Video Recorder (e.g. TiVO)

X10 Lighting Powerpoint

Alarm Clock

Projector

Microwave oven

45

Outline

• Introduction





• Validation

• Conclusion

46

Conclusion

Problem• Appliances are increasingly complex

• Appliance user interfaces are often hard to use

Solution• Move appliance interfaces to a mobile device the user is

already carrying

• Automatically generate interfaces so that:Interfaces are customized for the device and modality that the user prefersInterfaces for similar appliances can be made consistent

Interfaces for multiple appliances can be combined into a single interface

• Validate generated interfaces to prove high quality claim

47

AcknowledgementsThesis Committee

• Brad A. Myers (chair)• Scott Hudson• John Zimmerman• Dan Olsen Jr.

Funding• National Science

Foundation• Microsoft• General Motors• Intel• Pittsburgh Digital

Greenhouse

Equipment Grants• Mitsubishi (MERL)• VividLogic• Lucent• Lutron• Lantronix• Nokia

PUC Project Members• Kevin Litwack• Thomas K. Harris• Michael Higgins• Joseph Hughes • Roni Rosenfeld• Rajesh Seenichamy• Pegeen Shen• Htet Htet Aung• Mathilde Pignol• Suporn Pongnumkul• Stefanie Shriver• Jeffrey Stylos• Peter Lucas

Collaborators & Friends• Naomi Ramos• Desney Tan• Daniel Avrahami• Gaetano Borriello• Laura Dabbish• Andrew Faulring• James Fogarty• Krzysztof Gajos• Darren Gergle• Andy Ko• Amy Nichols• Mick Nichols• Sally Nichols• Trevor Pering• Fleming Seay• Irina Shklovski• Roy Want• Jake Wobbrock• and many others…

Thanks for listening!

For more information…http://www.cs.cmu.edu/~pebbles/puc/http://www.cs.cmu.edu/~jeffreyn/

Automatically Generating High-Quality User Interfaces for Appliances

Jeffrey NicholsDoctoral Colloquium Presentation

Second International Conference on Pervasive Computing April 18, 2004

51

Problem, cont.

52

Problem, cont.

April 29, 1991

CNN – Jan 2004

May 2003

53

Outside the Scope

• Help systems for generated interfaces• Automated trouble-shooting for complex

systems• Service Discovery• Macros and End-User Programming• Security• Inter-operability with INCITS/V2

54

Outline

• Introduction





• Validation

• Conclusion

55

Initial Approach

What information is needed about the appliance to automatically generate remote control interfaces?

Investigate via a design process

Create interfaces by hand• AIWA Shelf Stereo• AT&T Telephone/Answering Machine

Improve quality with heuristic analysis and think-aloud studies with several users

Compare interfaces with actual appliance interfaces to validate PUC concept

Analyze interfaces for functional information

56

Palm Interfaces

Initially designed paper-prototype interfaces for Palm

telephone stereo

57

Hand-Designed Interfaces

Interfaces for Microsoft’s PocketPC (simulated remote control)

telephone stereo

Using our interfaces, users were twice as fast and made half as many errors

58

Comparison StudyCompared performance of first-time users (not experts)

Procedure • Each subject worked two sets of tasks

both stereo and phonecontrolled for order and interface

Performance Metrics• Time to complete all tasks

• Number of times a user manual was needed

• Number of missteps

Appliance Hand-design

Phone

Stereo

59

Comparison Study Results

Using our interfaces, users were twice as fastand made half as many errors

All differences are significant (p < 0.05)

60

Extra Group Tree

61

Graphical Interface Generator

Rule-based approach• Multiple phases that

iteratively transform a specification into a user interface

Focuses on panel structure of user interface

• Small groups of controls have basic layouts

• Complexity comes from structure of groups

• Structure can be inferred from dependency info!

62

Generation Process

1. Determine conceptual layout• Infer panel structure from

dependencies using “mutual exclusion” property

• Choose controls (decision tree)

• Choose row layout (one column, two column, etc.)

2. Allocate space• Examine panel contents and

choose sizes

3. Instantiate and place controls

4. Fix layout problems

63

Choosing Panel Structure

a)

b)

c)

full screen

tabbed

partial screen

64

Generation Process






choose sizes



65

Generation Process






choose sizes



66

Generation Process






choose sizes



67

Generation Process






choose sizes



68

Generation Process






choose sizes



Without layout fixing rules

With layout fixing rules

69

Smart Templates Example

70

Smart Templates, cont.

One template can be used across multiple appliances and can be generated on multiple platforms

71

Consistency Problem

Apply different consistency techniques depending on similarity

• Sparse similarityUse similar controls for similar functionse.g. use a scroll bar for new volume, even if this would not be the normal representation (might be a combo box)

• Branch similarityUse the previous interface structure of branch in the new interfacee.g. branch of previous interface was shown in a tabbed panel. In the new interface, separate functions of new interface with tabs and have similar functions in a tabbed panel.

• Significant similarityUse the group tree from the previously generated interfaceInterpolate extra functions into the tree

72

The “Experience”, cont.

How can a PUC device provide improved interfaces for these systems?

Users often think of multiple connected appliances as one system

• Home theater• CMU lecture room system• others…

73

Proposed Schedule

March 2004

September

1

2

3

4

5

6

7

8

1 Interface consistency

2 Smart Templates

3 Preliminary user studies with GM vehicle interfaces

4 Improve robustness and quality of interface generators on all platforms

5 Develop distributed task modeling language and multi-appliance user interface generation

6 Implement appliances for final user study

7 Conduct final user study and iterate UI generation rules, if necessary

8 Write dissertation

JuneJune December March2005

September

December2005

74

Proposed Contributions

• An abstract appliance modeling language for describing the functionality of appliances

• Algorithms for automatically generating high quality interfaces from that language

• The Smart Templates technique for incorporating high-level conventions

• Algorithms for ensuring consistency across generated interfaces

and…

75

Proposed Contributions, cont.

• A distributed task-modeling language for describing the sub-tasks specific to an appliance in a multi-appliance system

• Algorithms for automatically generating a single user interface for multiple appliances using distributed task information

• Interface generation software on multiple platforms that is shown by user testing to be better than manufacturers’ interfaces for the same appliances

automatically generating high-quality user interfaces for appliances jeffrey nichols doctoral...

Documents

appliances stereo

highquality user interfaces

appliance functions

problem appliances

highquality interfaces

manufacturers interfaces

multiple connected appliances

home theater slide