George VellisGeorge Vellisaa

Dimitrios KotsalisDimitrios Kotsalisaa

Demosthenes AkoumianakisDemosthenes Akoumianakisaa

Jean VanderdoncktJean Vanderdoncktbb

aaDepartment of Applied Informatics & Multimedia, Department of Applied Informatics & Multimedia, Technological Education Institution of Crete, GreeceTechnological Education Institution of Crete, GreecebbUniversité catholique de Louvain,Université catholique de Louvain,Louvain Interaction Laboratory (www.lilab.eu)Louvain Interaction Laboratory (www.lilab.eu)Louvain School of Management, BelgiumLouvain School of Management, Belgium

Introduction

• User interface challenges for software engineers– Programming intensive– Broad range of interaction components and widgets– Platform specificity (GUI objects, Web widgets, etc) – Lack of interoperability due to platform specificity

• (Some) Model-based engineering claims – UI engineering = Model specification– Focus on design rather than implementation– Linking (rather than direct calls) to platform-specific

libraries

Introduction

• User interface challenges for software engineers– UI development time takes a large portion of time

Software plans and requirements

Validation

Validation

Product design

Verification

Detailed design

Verification

CodeUnit test

IntegrationProduct verification

Operations and maintenance

Revalidation

Implementation

System test

System feasibility

UI Design occupied 45% of the totalamount of time devoted to each activity

50%

37%

Mean = 44%

Retrospective assessment

• Claims proven useful– Model specifications can provide an engineering

basis for UI engineering– Run-time linking alleviates the complexity of

programming

• Claims proven possible– Multi-platform execution– Relatively straightforward Uis (e.g., form-based UI)

Retrospective assessment (Cont.)

• Several initiatives – One of them is UsiXML which is used in this work

• Reference Architectures– CAMELEON Reference framework

• Recommended by W3C Working Group on Model-based User Interface Design

• UsiXML is compliant with this framework

• Plethora of methods with some focus on collaboration– FlowiXML, AMENITES, CIAM, TOUCHE

www.usixml.euwww.usixml.org

Retrospective assessment (cont.)

• Challenges still pending attention – non-standard and customized widgets?– synchronous and collaborative UIs?– new affordances such as social awareness in

distributed collaborative settings?– radically different interaction vocabularies?– design and run-time support tools for all the above?

• Theory development– Build abstraction to capture interaction components as

assemblages of capabilities (or affordances)– Substantiate such capabilities (or affordances) to polymporphic

instantiation schemes– Specify polymorphic properties in a model-based fashion

• Engineering framework– Extend UsiXML as needed to support the new specification– Provide dedicated design tools and run-time components– Demonstrate the concept’s validity through pilot scenarios

Research design

• Consider various generations of ‘buttons’– GUI buttons (in the 90s)

• The (primitive) assemblage of button = {two-state device to exercise control over functionality}

Why abstraction?

• A continuum of abstractions (Cameleon Ref. Framework)

Why abstraction?

Final UserInterface (FUI)

Concrete UserInterface (CUI)

Abstract UserInterface (AUI)

Task &Concepts

Rendering

Code

Platform-independentCIO

Modality-independentAIO type

TaskClasses

Download

<input type=submit value=“Download" name=btnG>

HTML pushbutton

Graphical 2D push button

Software control AIO

Control AIO

Method triggered: download fileObject: computer file

OSF/MotifXmButton

Windowspush button

Download Down

Load

VRML97/X3Dbutton

Softwarekey

Functionkey

Graphical 3D push button Physical push buttonPlatform-independentCIO type

Physical control AIOModality-dependentAIO

Final UserInterface (FUI)

Concrete UserInterface (CUI)

Abstract UserInterface (AUI)

Task &Concepts

Rendering

Code

Platform-independentCIO

Modality-independentAIO type

TaskClasses

DownloadDownload

<input type=submit value=“Download" name=btnG>

HTML pushbutton

Graphical 2D push button

Software control AIO

Control AIO

Method triggered: download fileObject: computer file

OSF/MotifXmButton

Windowspush button

DownloadDownload Down

LoadDown

Load

VRML97/X3Dbutton

Softwarekey

Functionkey

Graphical 3D push button Physical push buttonPlatform-independentCIO type

Physical control AIOModality-dependentAIO

• Scented button (cf. Willett et al., 2007)

• The (augmented) assemblage of ‘scented’ button ={ Two-state device to exercise control }

+{ Collectivity (i.e., who else is in the same state of mind)}

‘Adding’ social scent

• ‘Like’ buttons (in the Web 2.0)

• The (new) assemblage of the ‘like’ button ={Two-state devices to exercise control}

+{Expression of opinion (i.e. I like this)}

‘Adding’ expression of intention

• ‘Like’ buttons with social scent

• The (combined) assemblage ={Two-state device to exercise control}

+{Expression of opinion (i.e. I like this)}

+ {Collectivity (i.e., who else is in the same state of mind)}

Putting them all together

• Why limit our creativity to conventional widgets?– elastic buttons and their interactive behaviour

Generalizing the concept

Note the embedded affordances

• What happens when an elastic button is to be instantiated across devices and platforms

Adding to complexity

• How can we …– model digital assemblages in interaction components?– instantiate them as the need arises and depending on

context / platform capabilities– integrate them into specifications as first class elements /

properties– compile specs that are general enough

Motivating question

• A scheme that relies on implementation agnostic (i.e., abstract) specifications of UIs– i.e. an abstract button can be instantiated as

• a conventional GUI button when implemented with the ‘primitive’ assemblage

• a scented button when implemented with the ‘augmented’ assemblage

• a ‘like’ button when implemented with the ‘combined’ assemblage

• At run-time and once user and usage context parameters are discovered, the implementation agnostic spec is translated to context-specific interaction vocabularies using dedicated tools

(One) Answer: Polymorphic instantiation

• Introduce new widgets and their properties as first-class design elements

Polymorphic Widget SpecificationsBasic concept

WSL LibrariesResources

Widget Archive

• Specify complex capabilities & affordances in a model-based fashion– Devise required properties– Define model elements– Create specification– Mapping models

The role of MBUI engineering

The role of MBUI engineering the case of a socially aware button

• Extend UsiXML as needed to support the new specification– New set of models

• Widget Resource Model• Behavior model

– Additional models for collaborative work• Squad model• Consistency model• Abstraction model• Session model

A new set of models

A new design tool

A new run-time environment

Design Environment

Platform Server

Abstraction Classes

Collaboration Plugin

Social Proxy

Polling

Social Proxy

Social Proxy

Abstraction Classes

Abstraction Classes Abstraction

Classes

Collaboration Plugin

Social Proxy

Polling

Social Proxy

Social Proxy

Abstraction Classes

Abstraction Classes

Runtime Environment

Server Side Framework (SSF)

UI-1

Axis2-Framework

Web-Services Layer

Session Manager

Widget Archives Repository

Notification Queue

UI-Model Repository2.join

3.getUIModel

4.getArchives

5.buildUI6.attachStates

1.initPS

trackChanges

applyChangesdistributeChanges

Client-1 Client-N

UI-N

P.S

de

plo

yU

I UI Design Tool

Widget Spec Tool

UIDL’2011 (simple) desktop scenario

MDSD’12 multiplatform scenario (in the paper)

Player-1:Desktop/PC Player-2: Android/mobile Viewer-1: Desktop/PC

• Collaborative assembly of vacation packages• User roles

– Business partner (desktop)– Administrator (desktop or mobile– Platforms (desktop/swing, mobile/Android)– Context (synchronous and collaborative)

• Complex and custom widgets (i.e., elastic buttons)• Synchronous collaboration

Post MDSD’12 scenario

The desktop case administrator’s view of vacation package / swing

The desktop casebusiness partners’ view of vacation package / swing

The mobile caseadministrator’s view of vacation package - Android

Due to space constraints certain functions become automatically adapted to vocal interaction

The designer’s role?Designing for certain affordances

3

2

456

1

The run-time instantiation the specified affordances enabled

3 4 5 6

2

2

1

• Current and on-going work– Advanced toolkits (i.e., visualization)– Run-time adaptivity and UI plasticity in distributed and

ubiquitous settings– ‘Social’ games

• Acknowledgements – The work is supported by ARCHIMEDES III– Thanks to Université catholique de Louvain (UCL) for

supporting the first two authors’ PhD dissertations• Due to be submitted in 2013

Future work & acknowledgements