Dimitrios Kotsalisa

George Vellisa

Demosthenes Akoumianakisa

Jean Vanderdoncktb

aDepartment of Informatics Engineering, Technological Education Institution of Crete, GreecebUniversité catholique de Louvain, Louvain School of Management, Belgium

‘Implementation-agnostic’ instantiation schemes for ubiquitous, synchronous

multi-user interfaces

4th Mobile Device and Web software Development - Small-systems development (MDSD) 2014"

Athens, GreeceOct. 2-3, 2014

• Research on advanced, creativity-based User Interface engineering, meaning– non-trivial case scenarios– multi-user interfaces – synchronous collaboration – distributed users– ubiquitous settings

• Research protocol– Joint supervision of two researchers of the istlab

(Department of Informatics Engineering of TEI Crete) in PhD program at the Université catholique de Louvain, Louvain School of Management, Belgium

Context of the research

• Case study problem descriptions and overview of solutions (i.e., current version of the implementation)

• Brief and non-technical discussion of the engineering challenges and the technicalities of the work– Design-oriented inspiration– Development platforms and tools– On-going work

Plan of the presentation

• Two human collaborators / players– A single shared digital representation

• The soccer field

– Two types of agents• Human (collaborators) and non-human (soccer players)• Human collaborators virtualized and represented by soccer players

– Affordances of soccer players (inspired from semiotic engineering)• Each human collaborator operates with the currently selected soccer player• Each soccer player operates on a shared space inhabited by other players• Each human collaborator operates through different devices (Desktop/Swing and

Mobile/Android)• The combined experience (i.e., game co-play) entails operations within different

representations

• The system in its current operational setting

Scenario in MDSD’14 - The virtual setting

• How can we design and generate non-trivial multi-user interfaces that support synchronous collaboration between distributed users in ubiquitous settings?

• Existing approaches– Toolkit-based programming– Model-based UI engineering– Both exhibit limitations which do not allow effective

solution to the kind of problem described

Research question & related works

• Theory-based insights– Interaction devices versus affordances– Semiotic engineering for qualifying virtualities by type of agents

and kind of operation

• Engineering approach– Development of an abstraction-based model anchoring

interaction in to human intentions and capabilities– Specify capabilities in a model-based fashion– Extend UsiXML as needed to support the new specification

• Provisions for extensible interaction vocabularies

– Build dedicated • design tools • run-time environment components

Proposed approach

• Current implementation of platform-agnostic UI instantiation schemes

An illustration of the approach

• A scheme that relies on implementation agnostic (i.e., abstract) specifications of UIs

• At run-time and once user and usage context parameters are discovered, the implementation agnostic spec is translated to context-specific interaction vocabularies using dedicated tools

Platform-agnostic instantiation

Widget gallery

Polymorphic classification scheme for the ‘abstract button’ widget

Download

Download

• Introduce new widgets as first-class design objects

Widget Specification – Basic concept

WSL LibrariesResources

Widget Archive

Polymorphism & extensible interaction vocabularies

Widget Archive

• Note that polymorphism at the UI-level is a much more demanding notion than polymorphism (i.e., polymorphic method invocation) as implemented in popular Object-Oriented languages.

An example of polymorphic specification of the ‘abstract SoccerField’ widget

• Note that polymorphism at the UI-level is a much more demanding notion than polymorphism (i.e., polymorphic method invocation) as implemented in popular Object-Oriented languages.

An example of polymorphic specification of the ‘abstract SoccerField’ widget

Run-time scenario (MDSD’14 paper)

Some technical challenges resolvedInput/output techniques and event models

Model

View View

Window Window

selected tapped activated

View

Window

Breakdowns Different semantics Implementation

language/toolkitdependent

Some technical challenges resolvedStates and state transitions

onMouseDown

onMouseRelease

onSelectCommand

onChangePlayerCommand

onTap

onUntap

moving still movingstill movingstill

Sharing schemes for Synchronous collaboration: “Common States” sharing

Not abstract enough since there may be no common states or states at all

More recent developments(THALES-MusiNet, 2012-2015)

Score’ interactive music metaphor: PC, expert users

Tablature’ interactive music metaphor: PC, novice users

The ‘circular’ interactive music metaphor: Web, novice users

• Distributed Collaborative music learning• User roles

– Teacher (Expert)– Student (Novice)

• User stereotypes– Sighted– Blind

• Platforms– PC: Java/swing– PC: JNVT2– Web: HTML5 – Prototype

• Interactive metaphors– Score– Tablature– Circular

ScenariosMulti-user Music Notation Lessons

JNVT2

JNVT2

HTML5/Prototype

• Advanced collaborative features– Interim-feedback, Group awareness support (Multi-

user selection, Multi-user highlighting, Radar view, Film-view), Social awareness

Novel features

• YouTube channel (“MusicNet istLab - Tei Crete”)

Polymorphic specification

• Current and on-going work– Support for web2 UIs– Generic support for group awareness– Run-time adaptivity and UI plasticity in distributed and

ubiquitous settings– Distributed music learning

• Acknowledgements – The work is supported by ARCHIMEDES III, THALES– KUL for PhD dissertations (first two authors)

Future work & acknowledgements

Final remark

• All screenshots presented were taken from running prototypes.

• For those interested some of them are already available in our channels on YouTube (“MusicNet istLab - Tei Crete”, iSTLab TeiCrete).