Recent Work in Model-Based User Interfaces

Jeffrey Nichols

Lecture #13

05-830: Advanced User Interface Software

Last time…

Model-based User Interfaces Automatic generation of the user interface so the

programmer won’t do a bad job. Dialog boxes are relatively easy to generate The full application interface is hard to generate Abstract descriptions of the interface can be

longer and harder to generate than implementing the interface itself.

Interface builders turned out to be easier…

But work continued…

Focus Changed Task models were leveraged more Design assistant aspect emphasized

A Couple Projects of Interest: Trident Mecano & Mobi-D FUSE AIDE

TRIDENT

Vanderdonckt, J., Knowledge-Based Systems for Automated User Interface Generation: the TRIDENT Experience, Technical Report RP-95-010, Facultes Universitaires Notre-Dame de la Paix, Institut d’Informatique, Namur, 1995.

An interface design assistant Interesting features:

Knowledge-based approach (i.e. expert system) Choosing Widgets Doing Layout

Use of Task Models Decides where separate windows are needed

Choosing Widgets

Used a decision tree Chose abstract

interaction objects (AIO)

Similar to Brad’s Interactor Model

Lots of parameters Continuous? Capacity Etc.

Choosing Layout

Uses Right/Bottom Strategy Next component is placed to the right or below

the current component Decision made by heuristics or designer

Right/Bottom Strategy

Windows from Task Models

Basically used for constructing wizard-like interfaces What information should be on the first screen, etc.

What are task models, anyway?

Description of the process a user takes to reach a goal in a specific domain

Typically have hierarchical structure Introduced by GOMS

Number of different task modeling languages GOMS UAN ConcurTaskTrees

ConcurTaskTrees

Developed by Fabio Paterno et al. for the design of user interfaces

Goals Graphical for easy interpretation Concurrent model for

representing UI tasks Different task types

Represent all tasks, including those performed by the system

Task Building Process

Three phases Hierarchically decompose

the tasks Identify the temporal

relationships among tasks at same level

Identify what objects are manipulated and what actions can be performed on them, and assign these to the tasks as appropriate.

Temporal Relationships T1 [] T2 - Choice T1 ||| T2 - Interleaving T1 |[]| T2 - Synchronization T1 >> T2 - Enabling T1 []>> T2 - Enabling with

Information Passing T1 [> T2 - Deactivation T1* - Iteration T1(n) - Finite Iteration [T1] - Optional T – Recursion

Example

Note: First example is ambiguous

Another Example

Building/Editing Task Models

Tools are available ConcurTaskTrees

Environment

http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”

Recent Systems

XIML – eXtensible Interface Markup Language Developed by the makers of Mecano/Mobi-D and Trident Kitchen-sink language for modeling any part of the

interface design process XWeb

Now known as ICE – Interactive Computing Everywhere ICrafter

A system for integrating user interfaces from multiple devices

Personal Universal Controller My research…

XIML

eXtensible Interface Markup Language Designed by RedWhale Software Intended to support the

full lifecycle of interfacebuilding

XIML Requirements

Central Repository of Data For one user interface or many

Comprehensive Lifecycle Support Abstract and Concrete Elements Relational Support Underlying Technology

XIML must be independent of particular tools

Models in XIML

An XIML document can contain any type of model Task Domain User Presentation Dialog

Example Use for XIML

Multi-platform interface development

Status of XIML

Used by RedWhale Software to drive their interface consultant business They have developed many tools

move interaction data to/from XIML Leverage data in XIML to better understand various

interfaces Automate parts of the interface design process

Model-Based Interfaces for Control XWeb ICrafter PUC

XWeb

Work by Dan Olsen and group at BYU Premise:

“Pervasive computing cannot succeed if every device must be accompanied by its own interactive software and hardware…What is needed is a universal interactive service protocol to which any compliant interactive client can connect and access any service.”

The web comes close to solving this problem, but is interactively insufficient.

XWeb Protocols

Based upon the architecture of the web XTP Interaction Protocol Server-side data has a tree structure Structured Data in XML URLs for location of objects

xweb://my.site/games/chess/3/@winner xweb://automate.home/lights/livingroom/ xweb://automate.home/lights/familyroom/-1

XWeb & XTP

CHANGE message (similar to GET in HTTP) Sequence of editing operations to apply to a sub-tree

Set an attribute’s value Delete an attribute Change some child object to a new value Insert a new child object Move a subtree to a new location Copy a subtree to a new location

Platform Independent Interfaces

Two models are specified DataView – The attributes of the service XView – A mapping of the attributes into high-level “interactors” Interactors are somewhat like abstract interaction objects

Atomic Numeric Time Date Enumeration Text Links

Aggregation Group List

XWeb Example

DataView

XWeb Example

XView

XWeb Example

Interface

Other XWeb Details

Has simple approach for adjusting to different screen sizes Shrink portions of the

interface Add additional columns of

widgets Also capable of generating

speech interfaces Based on a tree traversal

approach like Universal Speech Interfaces

ICrafter

Part of the Interactive Workspaces research project at Stanford

Main objective: “to allow users of interactive workspaces to flexibly interact

with services” Contribution

An intelligent infrastructure to find services, aggregate them into a single interface, and generate an interface for the aggregate service.

In practice, much of the interface generation is done by hand though automatic generation is supported.

ICrafter Architecture

How is aggregation accomplished? High-level service interfaces (programmatic)

Data Producer Data Consumer

The Interface Manager has pattern generators Recognize patterns in the services used Generate interfaces for these patterns

This means that unique functionality will not be available in the aggregate interface

Automatic Generation in ICrafter

Manual Generation in ICrafter

Personal Universal Controller

My work with Brad Problem:

Appliance interfaces are too complex and too idiosyncratic.

Solution: Separate the interface from the appliance and use

a device with a richer interface to control the appliance: PDA, mobile phone, etc.

Idea

Specifications

Control

Feedback

Control existing appliances Generate multi-modal interfaces

- Comm. Protocol- Interface Generators- Specification Lang.Architecture

XML-based

- Appliance Adaptors

Language Design

Approach Create reference interfaces

AIWA Shelf Stereo AT&T Telephone/Answering

Machine

Test interfaces with subjects Users twice as fast and made half

the errors with reference interfaces as compared to manufacturers’ interfaces

Analyze interfaces for functional information

Language Elements

State Variables and Commands Represent functions of appliance State variables have types

Boolean, Enumeration, Integer, String, etc. Variables sufficient for most functions but not all

“seek” button on a Radio

Label Information One label not suitable everywhere

The optimal label length changes with screen size Speech interfaces may benefit from pronunciation and

text-to-speech information

Language Elements, cont.

Group Tree Specify organization

of functions We use n-ary tree

with variables or commands at leaves

Language Elements, cont.

Dependency Information Formulas that specify when a variable or

command is active in terms of other state variables

Equals, Greater Than, Less Than Linked with logical operators (AND, OR)

For example, <and>

<equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals></and>

Interface Generators

Generators for Two Modalities Graphical

Implemented for PocketPC in Java 1.1 Uses dependency information to generate panel structure of

interface

Speech Implemented using Universal Speech Interface (USI)

techniques [Rosenfeld 2001] Uses dependency information to disambiguate shortcut

words (e.g. “play”) and resolve pre-conditions for a requested function (e.g. “play CD”)

Graphical Interface Generator

Focuses on panel structure of user interface

Small groups of controls have basic layouts

Complexity comes from structure of groups

Structure can be inferred from dependency info!

Inferring Structure

Find sets of variables that are “mutually exclusive” Every variable in a set will

never be active at the same time as a variable in another set

Create structure with sets, using overlapping panels

Choosing Panel Types

a)

b)

c)

full screen

tabbed

partial screen

Making the Interface ConcreteFinish conceptual layout

Choose controls (decision tree)

Choose row layouts(one column, two column, etc.)

Allocate space Examine panel contents and

choose sizes

Instantiate and place controls

Generating Speech InterfacesAutomatically build USI tree from dependencies

Allows verbal navigation of functional groups

Automatically generate grammar for parser Phrases for query and control

“What is playmode?”“Set playmode to play”“play”

Automatically generate language model and pronunciation for recognizer