Lecture 13:

Continuing Work in Model-Based User Interfaces

Brad Myers

Slides originally authored by Jeffrey Nichols, 2004

05-830: Advanced User Interface Software

Last time…

Model-based User Interfaces Automatic generation of the user interface so the

programmer won’t do a bad job. Dialog boxes are relatively easy to generate The full application interface is hard to generate Abstract descriptions of the interface can be

longer and harder to generate than implementing the interface itself.

Interface builders turned out to be easier…

Research continued…

Multiple models were integrated Relational models were developed to explicitly

link other kinds of models Data models became more detailed Task models became very important

Research continued…

Fragmented into two different approaches Software engineering approach (early 90’s-)

Very detailed models to improve overall design process Intelligent design assistants instead of automatic generation Significant use of task models

Ubiquitous computing approach (2000-) Tons of “invisible” processors that perform tasks for us UIs for these processors are presented on other devices

(mobile phone, PDA, speech, etc.) Impossible to manually build user interfaces for every

combination

What are task models, anyway?

Key part of many current HCI approaches Description of the process a user takes to reach a

goal in a specific domain Typically have hierarchical structure

Introduced by GOMS

Number of different task modeling languages GOMS UAN ConcurTaskTrees

ConcurTaskTrees Developed by Fabio Paterno et al.

for the design of user interfaces

Goals Graphical for easy interpretation Concurrent model for representing UI

tasks Different task types

Represent all tasks, including those performed by the system

Used almost universally by new model-based systems

Task Building Process Three phases

Hierarchically decompose the tasks

Identify the temporal relationships among tasks at same level

Identify what objects are manipulated and what actions can be performed on them, and assign these to the tasks as appropriate.

Temporal Relationships T1 [] T2 - Choice T1 ||| T2 - Interleaving T1 |[]| T2 - Synchronization T1 >> T2 - Enabling T1 []>> T2 - Enabling with

Information Passing T1 [> T2 - Deactivation T1* - Iteration T1(n) - Finite Iteration [T1] - Optional T – Recursion

Example Note: First example is ambiguous

T1 [] T2 - Choice T1 ||| T2 - Interleaving

Another Example

T1 [] T2 – Choice T1 >> T2 - Enabling T1 []>> T2 - Enabling with

Information Passing T1 [> T2 – Deactivation

Building/Editing Task Models Tools are available

ConcurTaskTrees Environment

http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”

Software Engineering Approach

Lots and lots of systems! Commercial work MasterMind Angel Puerta’s work at Stanford

Mecano, Mobi-D, XIML UIML Cameleon Project

Teresa USIXML etc…

Software Engineering Approach Commercial work “Model-based design” Example: BPMN “business

process modeling notation” Business experts should be

able to author models Converted into code to

support the process (requires people) Keynote at ICSE’08: Herbert Hanselmann: Challenges in

Automotive Software Engineering “Model-based design (MBD) of functional behaviour has been a

big help in the recent past”

MasterMindNeches, et.al., IUI’93One of the first system to integrate multiple models

together

Mobi-D Angel Puerta, IUI’97 Set of tools to support

a clearly defined development cycle

Uses a series of different models

Explicit relationships that specify how the models are related to each other

Explicit interaction between end users and developers

Mobi-D Models User-task Model

Describes tasks the user performs and what interaction capabilities are needed

Domain Model Models of the entities that are manipulated in an interface and their

properties Dialog Model

Describes the human-computer conversation at a low level Presentation Model

Specifies how the interaction objects appear in all of the different states of the interface.

Relations Describes how each of the models relate to each other Tasks/Domain, Dialog/Presentation, Tasks/Dialog, etc.

Mobi-D Process1. Elicit user tasks

Start with informal description and then convert to outline (basis for task models in Mobi-D)

More formal task analysis methods could probably be used

2. User-task and domain modeling Skeleton domain model is built from task outline Developer refines model Explicit methods for generalizing pieces of model for reuse in other interface designs

3. Presentation and Dialog design Decision support tools provide recommendations from model and interface guidelines

(if available) System steps through task model and helps designer build final interface

By carrying the task model through the process, Mobi-D’s designers believe users can provide more useful feedback

Mobi-D Discussion

Provides assistance rather than automating design

Recommendations do not limit flexibility, but organize the design process

For usability engineers, not everyday users Benefits come from reuse among small projects or for

managing interaction data from a large project Models can be large and appear to require significant effort

to develop

Spawned a profitable company http://www.redwhale.com/ that does UI work

XIMLeXtensible Interface Markup Language XIML.org Based on Mobi-D work Supports full development lifecycle Used by RedWhale Software to drive

their interface consultant business They have developed many tools

move interaction data to/from XIML Leverage data in XIML to better

understand various interfaces Automate parts of the interface design

process

Example Use for XIML

Multi-platform interface development

Other Systems UIML (http://www.harmonia.com/)

Originally a research project at Virginia Tech, now being developed commercially by Harmonia

Goal is platform independent language for describing UI Early versions were not very platform independent Recent versions using task models to automatically generate parts of the old

language that were not platform independent

Teresa (http://giove.cnuce.cnr.it/teresa.html)

Transformation Environment for inteRacti Tool for taking ConcurTaskTrees models, building an abstract interface, and then

building a concrete interface on multiple platforms.

USIXML (http://www.usixml.org)

Many of the same features of XIML Novel aspect is the use of graph structure for modeling relations (seems very

complex)

Ubiquitous Computing Approach

“Pervasive computing cannot succeed if every device must be accompanied by its own interactive software and hardware…What is needed is a universal interactive service protocol to which any compliant interactive client can connect and access any service.”

-Dan Olsen (Xweb paper)

The web comes close to solving this problem, but is interactively insufficient.

Ubiquitous Computing Approach

There are two problems here: Infrastructure issues

How do devices communicate? How do devices discover each other?

User Interfaces issues Are devices described sufficiently to build a good UI? How are interfaces generated? How can one interface be created for controlling

combinations of related devices?

Infrastructure Issues

Possible to investigate these issues without automatically generating UIs

Being addressed by lots of systems Commercially

UPnP, JINI, Salutation, HAVi Research

Speakeasy (PARC), many others…

Most systems that address the UI issues also have some infrastructure component

Systems addressing UI issues

XWeb Now known as ICE – Interactive Computing Everywhere

ICrafter A system for integrating user interfaces from multiple

devices Supple

A system for automatically generating interfaces with a focus on customization/personalization.

Personal Universal Controller Jeff Nichol’s research…

XWeb

Work by Dan Olsen and group at BYU E.g. UIST’2000, pp.191 - 200  

Premise: Apply the web metaphor to services in general Support higher levels of interactivity

XWeb Protocols

Based upon the architecture of the web XTP Interaction Protocol Server-side data has a tree structure Structured Data in XML URLs for location of objects

xweb://my.site/games/chess/3/@winner xweb://automate.home/lights/livingroom/ xweb://automate.home/lights/familyroom/-1

XWeb & XTP

CHANGE message (similar to GET in HTTP) Sequence of editing operations to apply to a sub-tree

Set an attribute’s value Delete an attribute Change some child object to a new value Insert a new child object Move a subtree to a new location Copy a subtree to a new location

Platform Independent Interfaces

Two models are specified DataView – The attributes of the service XView – A mapping of the attributes into high-level “interactors”

Atomic Numeric Time Date Enumeration Text Links

Aggregation Group List

XWeb ExampleDataView

XwebExample

XView

XWeb ExampleInterface

Other XWeb Details Has simple approach for

adjusting to different screen sizes Shrink portions of the

interface Add additional columns of

widgets Also capable of generating

speech interfaces Based on a tree traversal

approach like Universal Speech Interfaces

ICrafter

Part of the Interactive Workspaces research project at Stanford

Ponnekanti, et. al. Ubicomp’2001 Main objective:

“to allow users of interactive workspaces to flexibly interact with services”

Contribution An intelligent infrastructure to find services, aggregate

them into a single interface, and generate an interface for the aggregate service.

In practice, much of the interface generation is done by hand though automatic generation is supported.

ICrafter Architecture

How is aggregation accomplished?

High-level service interfaces (programmatic) Data Producer Data Consumer

The Interface Manager has pattern generators Recognize patterns in the services used Generate interfaces for these patterns

This means that unique functionality will not be available in the aggregate interface

Automatic Generation in ICrafter

Manual Generation in ICrafter

Supple

Eventual goal is to support automatic personalization of user interfaces

Treats generation of interfaces as an optimization problem

Can take into account usage patterns in generation

Krzysztof Gajos and Daniel S. Weld, “SUPPLE: Automatically Generating User Interfaces” in Proceedings of Intelligent User Interfaces 2004, Funchal, Portugal.

Modeling Users with Traces

Supple uses traces to keep a usage model Sequences of events:

<interface element, old value, new value>

Interfaces are rendered taking the traces into account (though traces are not required)

Trails are segmented at interface close or reset

Generating with Optimization Uses a branch-and-

bound search to explore space of alternatives Guaranteed to find

an optimal solution

Optimization EquationCost of navigation between subsequent controls in a trace

Device-specific measure of how appropriate a control is for manipulating a variable of a given type

Total cost for one user traceTotal cost of all traces

Screenshots

Personal Universal Controller

Jeff Nichol’s PhD work Problem:

Appliance interfaces are too complex and too idiosyncratic.

Solution: Separate the interface from the appliance and use

a device with a richer interface to control the appliance: PDA, mobile phone, etc.

Control

Feedback

Approach

Specifications

Appliances Mobile Devices

Use mobile devices to control all appliances in the environment

Key FeaturesTwo-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation

The PUC System Architecture

PROTOCOL(two-way communicationof specification & state)

COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)

PUC DEVICES(automatic interface generation)

PROTOCOL(two-way communicationof specification & state)

COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)

APPLIANCES(Stereo, Alarm Clock, etc.)

ADAPTOR(publishes description +

appliance state + controls appliance)

control

device specification & state feedback

Automatic Generation of UIsBenefits

All interfaces consistent for a userWith conventions of the handheldEven from multiple manufacturers

Addresses hotel alarm clock problem Can take into account user preferences Multiple modalities (GUI + Speech UI)

A Hard Problem Previous automatic systems have not

generated high quality interfaces

PUC Specification Language XML Full documentation for the

specification language and protocol:

http://www.pebbles.hcii.cmu.edu/puc/

Contains sample specification for a stereo

Properties of PUC Language State variables & commands

Each can have multiple labels Useful when not enough room

Typed variables Base types: Boolean, string,

enumerated, integers,fixed-point, floating-point, etc.

Optional labels for values Hierarchical Structure

Groups

Dependency Information Crucial for high-quality interfaces Expressed as <active-if> clauses

Operations: Equals, Less-Than,

Greater-Than Combined Logically

AND, OR Used for:

Dynamic graying out Layout Widget selection

Specifications Have working specifications for:

Audiophase stereo X-10 lights control Sony CamCorder Windows Media Player Audio ReQuest hardware MP3 player WinAmp Media Player Elevator Parts of GMC Yukon Denali SUV Etc.

Controller Generators

iPaq PocketPC

SmartPhone No touchscreen

Desktop (TabletPC)

Speech

Generating Speech Interfaces “Universal Speech Interface” (USI) project

Prof. Roni Rosenfeld of CMU http://www.cs.cmu.edu/~usi

Creates grammar, language model and pronunciation dictionary from PUC specification Pronunciation from labels using phonetic rules Can provide other pronunciations as labels for fine-tuning

Will use dependency information to help with disambiguation and explanation

Supports queries and spoken feedback Paraphrases as confirmation

Adaptors “Adaptors” provide the interface to existing

(and future) appliances If do not support specification language directly

Custom hardware Custom software

Lutron Windows Media Player

X-10 Light switches, etc.

AV/C (standard protocol) Sony CamCorder

HAVi UPnP

Axis Camera

Generating Consistent UIs

Personally consistent Reduce learning time Add unique functions

Generating Combined UIs

For multiple appliances, such as home theaters

Specify content flow Combined controls

Summative Study Compared PUC to manufacturer’s

interfaces for HP and Canon printer/fax/copiers

PUC twice as fast, 1/3 the errors Consistent: another factor of 2 faster

0

200

400

600

800

1000

1200

1400

HP HP-PUC HP-Consistent

Canon Canon-PUC Canon-Consistent

Ave

rag

e T

ime

(sec

)

0

2

4

6

8

10

12

14

16

18

HP HP-PUC HP-Consistent

Canon Canon-PUC Canon-Consistent

Num

ber

of F

ailu

res

Details of the Language Design

Informed by hand-designed interfaces What functional information was

needed to create interfaces?

Additional Requirements Support complete functionality of

appliance No specific layout information Only one way to specify anything

Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/

Language ElementsElements

State variables & commands Labels Group tree Dependency information

Example media player specification

Play, stop, pause, next track, previous track

Play list

Language Elements, cont.

State Variables and Commands

Represent functions of appliance

State variables have types Boolean, Enumeration,

Integer, String, etc.

Variables sufficient for most functions but not all

e.g. “seek” button on a Radio

Language Elements, cont.Label Information

One label not suitable everywhere The optimal label length

changes with screen size Speech interfaces may benefit

from pronunciation and text-to-speech information

“Label Dictionary” A group of semantically similar

labels Different lengths Information for different

modalities

Language Elements,cont.Label Information

One label not suitable everywhere The optimal label length

changes with screen size Speech interfaces may benefit

from pronunciation and text-to-speech information

“Label Dictionary” A group of semantically similar

labels Different lengths Information for different

modalities

Language Elements, cont.

Group Tree Specify organization

of functions We use n-ary tree

with variables or commands at leaves

Also used for specifying complex types

Lists

Unions

Language Elements,cont.Group Tree

Specify organization of functions

We use n-ary tree with variables or commands at leaves

Also used for specifying complex types

Lists

Unions

Language Elements,cont.Dependency Information

Formulas that specify when a variable or command is active in terms of other state variables

Equals, Greater Than, Less Than, Is Defined

Linked with logical operators (AND, OR)

Allows feedback to user when a function is not available

Graphical Interface GeneratorRule-based approach

Multiple phases that iteratively transform a specification into a user interface

Focuses on panel structure of user interface

Small groups of controls have basic layouts

Complexity comes from structure of groups

Structure can be inferred from dependency info!

Generation Process1. Determine conceptual layout

Infer panel structure from dependencies using “mutual exclusion” property

Choose controls (decision tree) Choose row layout

(one column, two column, etc.)

2. Allocate space Examine panel contents and

choose sizes

3. Instantiate and place controls

4. Fix layout problems

Generation Process1. Determine conceptual layout

Infer panel structure from dependencies using “mutual exclusion” property

Choose controls (decision tree) Choose row layout

(one column, two column, etc.)

2. Allocate space Examine panel contents and

choose sizes

3. Instantiate and place controls

4. Fix layout problems

Without layout fixing rules

With layout fixing rules