flexible user interfaces for mobile devices · seminar ubiquitous human computer interaction summer...

Oliver Bende

Seminar Ubiquitous Human Computer InteractionSummer term 2007

Institute Of Operating Systems And Computer Networks Technische Universität Braunschweig

Advisor: Matthias Kranz

Flexible User Interfaces For Mobile Devices

2/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Flexible User Interfaces For Mobile Devices

User interfaces are one of the most important partsDifficult to develop because of the wide range of devicesFlexible user interfaces cope with different capabilities and circumstancesAre developed for many years

3/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Flexible User Interfaces For Mobile Devices

Different possibilities for output of information (Output)Audible, Displays

Different possibilities for input of information (Input)Keyboards, pointing devices, speech, specific devices

Internal conditions (calculating power, battery lifetime,…)Possibility to show a web interface instead of running application (Web)

Context-awareness (Context) Adaptation of display to changing circumstances

Short statement about actual situation

4/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Output methods

Different requests to interfaceHMD has other requests than other “usual” displays

– wGUI for a HMD (Head Mounted Display)

5/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Output methods: Usual displays

One user interface for all devices inappropriateTiny on desktop monitor or scrolling on mobile display

Adaptation to according display: TasksAdapt to many different devices and circumstances Never confuse the user (should look like the used one)

– Decide what is importantModel-based and rule-based toolkits

Model-based: abstract modelsRule-based: strict rules how interface should look

6/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

SUPPLE

Model-basedFunctional interface specificationDevice-model describes capabilitiesUser-model (typical activities)

Special algorithm to calculate user effortCost for every possible interfaceBranch and boundChooses the best possible interface

Adapts itself to user actions at runtime

7/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

SUPPLE

The same application rendered on different devices

8/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

SUPPLE

Calculation very elaborate for processorRemote rendering possible

Automatic usage of “solver server”Remote rendering service accelerates processNetwork connection essential

No support for context eventsEnhancement: Factor in the cost to learn new interface

Same application with new device

9/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Enhanced SUPPLE

SUPPLE makes trade-off between optimization and similarity

10/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Enhanced SUPPLE

SUPPLE makes trade-off between optimization and similarity

11/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Enhanced SUPPLE

SUPPLE makes trade-off between optimization and similarity

12/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

TERESA

Model-based toolDifferent abstraction layers

Top-down methodXML-Based

Possibility to import/export definitionsOnly flexible at generation, not at runtimeOne task-model yields many interfaces

13/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

TERESA

The „one model, many interfaces approach“

14/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

The Ubiquitous Interactor

Creates device-independent user interfacesGives the ability to control presentation of interface

Important for branding purposesPresentation information separated from user-service informationUses mappings and media resources

Media resources: pictures or soundsMappings: link interactions to templates

Interface generation by server: Network connection required

15/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Personal Universal Controller

Rule-based interface generatorSpecific knowledge about target-device is required

Developed to access functions of appliancesArchitecture has four parts

Appliance adaptorSpecification languageCommunication protocolInterface generator

16/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Flexible User Interfaces For Mobile Devices

Different possibilities for output of information (Output)Audible, Displays

Different possibilities for input of information (Input)Keyboards, pointing devices, speech, specific devices

Internal conditions (calculation power, battery lifetime,…)Possibility to show a web interface instead of running application (Web)

Context-awareness (Context) Adaptation of display to changing circumstances

Short statement about actual situation

17/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Input Methods

Users familiar with WIMP (Windows, Icons, Menus and Pointer)

Mostly not possible for mobile devicesTwo popular input methods

Four-way cursorTouch screen

Translations / adaptationsSoft keyboardInput Adaptor Tool

Completely new methods

18/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Input Adapter Tool

Two problems to solve:User must be able to select any interactorUser must be able to control the selected interactor

IAT adapts the running software to given input methodManipulates the GUI

Has to have control over the GUIVery general and applicable to all devices

19/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Input Adapter Tool

Four input methodsPointer (no keyboard)

– Suitable for most navigationSwitchSpeech

– Cursor moves with regulation of voice pitch– Spatial: speak out grid regions– Jumping or scanning

Keyboard (no pointer)– Tab-stop or direct mapping (jumping)

20/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Input Adapter Tool

Automatic addition of shortcutsReplacement of combo-boxes with text fields

21/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

“Paperweight Method”

Refine of touch screen pointing device (stylus)Supports the recognition of wrist pressure

Sensor under the bottom right screen of a PDAUser lays down the hand for editingUser lifts hand for navigationVery specific – not for use with other methods / devices

22/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Flexible User Interfaces For Mobile Devices

Different possibilities for output of information (Output)Audible, Displays

Different possibilities for input of information (Input)Keyboards, pointing devices, speech, specific devices

Internal conditions (calculation power, battery lifetime,…)Possibility to show a web interface instead of running application (Web)

Context-awareness (Context) Adaptation of display to changing circumstances

Short statement about actual situation

23/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Web Technology

Displaying web pages on mobile devicesNo need to install additional software

Common languages are used (Java, XML,…)Many people can handle toolkits

Adaptive User Interface Toolkit (AUIT)Includes Java Swing based design toolCombination of XML and Java Server Pages (JSP)

– Other languages are possible (prototype)Specifications stored in XML format

Drawback: Server connection essential

24/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Flexible User Interfaces For Mobile Devices

Different possibilities for output of information (Output)Audible, Displays

Different possibilities for input of information (Input)Keyboards, pointing devices, speech, specific devices

Internal conditions (calculation power, battery lifetime,…)Possibility to show a web interface instead of running application (Web)

Context-awareness (Context) Adaptation of display to changing circumstances

Short statement about actual situation

25/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

SULAWESI

User interface which adapts the whole interfaceE.g. Sulawesi is able to adapt input and output

Detects movement:– switches from pointing device to speech input and from

display to audio outputAdapts on commands:

– “tell me the time” or “show me the time”

26/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

FLUSI

User interface which only adapts its displayDeveloped for Symbian mobile phonesAdapts operating system GUI to different devicesLocation-based context-awareness (not yet implemented)

Locate device (Cell ID or GPS)Adapting interface to the users task

– work / homeCopes with local context (shop / cinema poster)

– Active / passiveDisplay-adaptation already implemented

27/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Flexible User Interfaces For Mobile Devices

Different possibilities for output of information (Output)Audible, Displays

Different possibilities for input of information (Input)Keyboards, pointing devices, speech, specific devices

Internal conditions (calculation power, battery lifetime,…)Possibility to show a web interface instead of running application (Web)

Context-awareness (Context) Adaptation of display to changing circumstances

Short statement about actual situation

28/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Statement

Server-based systems only applicable for concerns with intranet

Online-services too expensive and unstableSome features already implementedSeparated projects, prototypes and researches“Designing flexible and dynamically configurable user interfaces (UIs) is difficult and it is unlikely that the developer will come up with a solution of all problems for all users. “(Bogdan Dorohonceanu, Boi Sletterink, Ivan Marsic: A Novel User Interface for Group Collaboration)

29/29Flexible User Interfaces For Mobile DevicesOliver Bende

Introduction Output Input Web Context Statement

Flexible User Interfaces For Mobile Devices

Questions ?