direct manipulation of user interfaces for migration j. p. molina 1,2, j. vanderdonckt 1 and p....

Direct Manipulation Direct Manipulation of User Interfaces for of User Interfaces for

MigrationMigrationJ. P. MolinaJ. P. Molina1,21,2, J. Vanderdonckt, J. Vanderdonckt11 and P. González and P. González22

11 Université catholique de Louvain Université catholique de Louvain22 University of Castilla-La Mancha University of Castilla-La Mancha

www.usixml.org

Intelligent User Intelligent User Interfaces 2006Interfaces 2006Jan 29 - Feb 1 2005, Sydney, Jan 29 - Feb 1 2005, Sydney, AustraliaAustralia

January 29 - February 1 2Intelligent User Interfaces 2006, Sydney, Australia

ContentsContents

IntroductionIntroduction MigriXMLMigriXML Case studyCase study Migration process in detailMigration process in detail ConclusionConclusion


ContentsContents



IntroductionIntroduction End users of modern interactive systems face End users of modern interactive systems face

a wide variety of computing platforms to a wide variety of computing platforms to support their interactive tasks, from mobile support their interactive tasks, from mobile phones to wall screensphones to wall screens

The specific needs of the user’s environment The specific needs of the user’s environment at a given moment, or its evolution, can make at a given moment, or its evolution, can make necessary to remove one application from one necessary to remove one application from one platform and make it run in another one, that platform and make it run in another one, that is, to is, to migratemigrate the application the application


IntroductionIntroduction

Migration can be:Migration can be: TotalTotal, when the whole application is migrated, when the whole application is migrated PartialPartial, only the UI is migrated; if it is , only the UI is migrated; if it is

decomposed into control and presentation, then decomposed into control and presentation, then migration can be:migration can be: Control-orientedControl-oriented, migrates only the control component, migrates only the control component Presentation-orientedPresentation-oriented, the control remains, the control remains MixedMixed, parts of both control and presentation are , parts of both control and presentation are

migratedmigrated

A user interface (UI) is said to be migratable A user interface (UI) is said to be migratable if it holds the migration abilityif it holds the migration ability


IntroductionIntroduction A special UI is required for controlling A special UI is required for controlling

the run-time migration of the UI of the the run-time migration of the UI of the interactive systemsinteractive systems

This UI is referred to as the meta-user This UI is referred to as the meta-user interface, implemented in very different interface, implemented in very different ways:ways: Sytem-initiatedSytem-initiated, the system initiates the , the system initiates the

migrationmigration User-initiatedUser-initiated, the user initiates the migration, the user initiates the migration Mixed-initiatedMixed-initiated, user and system collaborate , user and system collaborate

to perform the migrationto perform the migration



Related workRelated work Migration Initiated by

appuser interface

system usercontrol presentation

Bharat and Cardelli, 95 Song, Chu and Kurakake, 02 MIGRATION ProjectBandelloni et al., 04 CamNoteBalme et al., 04 Drag&Drop, Drag&PickBaudish et al., 03 Aurade Sousa and Garlan, 02



Elements Elements represented in in represented in in the meta-UIthe meta-UI

Representation of

source platform

target platform

space between

Detachable InterfacesGrolaux et al., 2004 Pick&Drop Rekimoto, 97

Augmented Surfaces Rekimoto and Masanori, 99 physical

Proximal InteractionsRekimoto et al., 03

screenshot

ARISBiehl and Baily, 04

icon icon 2D map



Tasks are usually distributed among Tasks are usually distributed among platforms, but without any consistent platforms, but without any consistent way to distribute them, the space way to distribute them, the space between platforms introduces a between platforms introduces a discontinuity that may disrupt the discontinuity that may disrupt the fluency of the interactive taskfluency of the interactive task

Direct manipulation is proposed to Direct manipulation is proposed to tackle that discontinuity, and thus tackle that discontinuity, and thus avoid possible usability problemsavoid possible usability problems


ContentsContents



MigriXMLMigriXML

In order to make user interface In order to make user interface migration as natural as direct migration as natural as direct manipulation, a meta-UI in the form of manipulation, a meta-UI in the form of a virtual environment has been a virtual environment has been developeddeveloped

MigriXMLMigriXML enables the user to carry enables the user to carry out that migration process without the out that migration process without the physical presence of the involved physical presence of the involved platforms, in a virtual and interactive platforms, in a virtual and interactive way, relaying on way, relaying on UsiXMLUsiXML language and language and a set of modelsa set of models


MigriXMLMigriXML

To generate the virtual reality scene To generate the virtual reality scene representing the migration representing the migration environment, a suite of models is used environment, a suite of models is used describing relevant aspects of the describing relevant aspects of the problem in terms of UsiXML:problem in terms of UsiXML: The Concrete User InterfaceThe Concrete User Interface The User ModelThe User Model The Platform ModelThe Platform Model The Environment ModelThe Environment Model

Describes properties of interest, which can be Describes properties of interest, which can be physical, psychological and organizationalphysical, psychological and organizational


MigriXML: Environment MigriXML: Environment ModelModel

The physical part consists of:The physical part consists of: a scene model based on VRML97/X3D, a scene model based on VRML97/X3D, the multi-surface interaction ontology, the multi-surface interaction ontology, and the Stanford topology model.and the Stanford topology model.

Each Each scenescene is composed of physical resources is composed of physical resources and a series of and a series of planesplanes, which are decomposed , which are decomposed into into areasareas

Each area may be an interaction surface: a Each area may be an interaction surface: a monitor on a table, a wall screen, or any area monitor on a table, a wall screen, or any area where the UI is projected and/or recognized where the UI is projected and/or recognized with computer visionwith computer vision

Each area is described by attributes like Each area is described by attributes like dimensions, relative position, relationships with dimensions, relative position, relationships with other objects, etcother objects, etc


MigriXML architectureMigriXML architecture

Interactive system Interactive system architecturearchitecture GUI applicationGUI application

Arch meta-model or Arch meta-model or ‘Slinky’ meta-model‘Slinky’ meta-model

PlatformPlatform Layered structureLayered structure


MigriXML architectureMigriXML architecture General MigriXML General MigriXML

architecturearchitecture User’s environmentUser’s environment GUI applicationGUI application

Arch meta-model fits Arch meta-model fits the UsiXML modelsthe UsiXML models

PlatformPlatform Virtual Hardware Virtual Hardware

(VHW)(VHW) Virtual Machine (VM)Virtual Machine (VM) Virtual Window Virtual Window

Manager (VWM)Manager (VWM) Migration Manager Migration Manager

(MM)(MM)


ImplementationImplementation

This work focuses on the original part of This work focuses on the original part of providing a virtual control environment providing a virtual control environment for migrating parts or whole of a UI from for migrating parts or whole of a UI from one platform to anotherone platform to another

To address its implementation, the To address its implementation, the VRML97/X3D languagesVRML97/X3D languages have been used have been used

User Interfaces and the Arch meta-model:User Interfaces and the Arch meta-model: Interaction toolkit component: Interaction toolkit component: VUIToolkitVUIToolkit Domain-specific component: Domain-specific component: JavaJava and and

JavascriptJavascript



VUIToolkitVUIToolkit A set of VRML97/X3D A set of VRML97/X3D

PROTOs that transforms PROTOs that transforms the standard plain the standard plain widgets into a truly 3D widgets into a truly 3D representation representation ((Virtualized User Virtualized User InterfaceInterface))

This toolkit was developed This toolkit was developed starting from the object starting from the object classes described in the classes described in the Concrete Model of Concrete Model of UsiXML languageUsiXML language



Java and JavascriptJava and Javascript Most VRML97/X3D browsers can interpret Most VRML97/X3D browsers can interpret

Javascript code and execute Java codeJavascript code and execute Java code VRML97/X3D specifications describe two VRML97/X3D specifications describe two

programming interfaces to access the scene programming interfaces to access the scene graph:graph: SAISAI (Scene Authoring Interface) (Scene Authoring Interface) EAIEAI (External Authoring Interface) (External Authoring Interface)

Allowed us to leave aside, for this first Allowed us to leave aside, for this first implementation of MigriXML, the complex implementation of MigriXML, the complex integration of emulators for each specific integration of emulators for each specific platformplatform



Interactive Interactive Java/Javascript Java/Javascript application, application, virtualized user virtualized user interface rendered interface rendered in the 3D window in the 3D window of VRML97/X3D of VRML97/X3D browserbrowser



Interactive Interactive application, application, virtualized user virtualized user interface and interface and virtual platform in virtual platform in MigriXMLMigriXML


ContentsContents



Case studyCase study

The user’s environment is a small The user’s environment is a small office, with five different platforms office, with five different platforms



Display characteristics of each Display characteristics of each platform platform



UsiXML was used to describe the UsiXML was used to describe the environmentenvironment

The virtual world was the result of a The virtual world was the result of a structured process of creation:structured process of creation: RequirementsRequirements: from UsiXML specifications, : from UsiXML specifications,

designers knew the details of the environmentdesigners knew the details of the environment PreparationPreparation: compilation of audio-visual : compilation of audio-visual

material; sketches and mock-ups were createdmaterial; sketches and mock-ups were created Design, test and optimizeDesign, test and optimize: model creation, edit : model creation, edit

appearance, enhance world appearance, enhance world



User interfaces were specified in User interfaces were specified in UsiXML tooUsiXML too

Different tools have been developed Different tools have been developed to help the UI designer to visually to help the UI designer to visually create this modelscreate this models IdealXMLIdealXML [Montero [Montero et al.et al., 05], 05]



User interfaces were specified in User interfaces were specified in UsiXML tooUsiXML too

Different tools have been developed Different tools have been developed to help the UI designer to visually to help the UI designer to visually create this modelscreate this models IdealXMLIdealXML GrafiXML GrafiXML

[B. Michotte][B. Michotte]



Two different user interfacesTwo different user interfaces



Migrating from one laptop to the Migrating from one laptop to the otherother

12

3 4

Aspire 200015”, 16:10 1280x800

PT-LB10SU4:3

800x600

Latitude15”, 4:3

1024x768



Migrating from a laptop to the Migrating from a laptop to the Pocket PCPocket PC

1

2

3

Aspire 200015”, 16:10 1280x800

e7503,8”, 3:4240x320


ContentsContents



Migration process in Migration process in detaildetail

The migration process is divided in The migration process is divided in four stages (A, B, C and D), each one four stages (A, B, C and D), each one having a finite set of steps, having a finite set of steps, representing a total amount of 14 representing a total amount of 14 stepssteps



Stage A: Select an interactive Stage A: Select an interactive applicationapplication




The user presses button (M) to grab the window, which follows the

user wherever she points at




The action is transmitted to the corresponding Virtual Window Manager (VWM) by an event




The VWM forwards the information to the Migration Manager (MM)



Stage B: Select target platformStage B: Select target platform



Stage B: Select target platformStage B: Select target platformThe MM broadcasts a message

to all platforms




Platforms change their state to ‘wait-for-selection’, a

TouchSensor is activated




The user selects the target platform by

clicking on its screen




The Virtual Hardware (VHW) casts an event to inform the VWM




The VWM forwards the message to

the MM



Stage C: Migrate to target platformStage C: Migrate to target platform




The MM commands to detach the application from source platform




The MM broadcast amessage to the rest of

platforms, which changeto normal execution

10




The MM takes the UsiXML specification and re-generates the app UI



Stage D: Adapt to target platformStage D: Adapt to target platform



Stage D: Adapt to target platformStage D: Adapt to target platformThe MM asks the target platform to host the re-generated application




The application is launched, and its virtualized UI added to the scene graph of the

target VHW




The virtualized UI is adapted to the target screen and

windowing system


ContentsContents



ConclusionConclusion

MigriXML andMigriXML andprevious workprevious work

Migration Initiated by

appuser interface

system usercontrol presentation

Bharat and Cardelli, 95 Song, Chu and Kurakake, 02 MIGRATION ProjectBandelloni et al., 04 CamNoteBalme et al., 04 Drag&Drop, Drag&PickBaudish et al., 03 Aurade Sousa and Garlan, 02 MIGRIXML



MigriXML and MigriXML and previous previous representationsrepresentations

Representation of

source platform

target platform

space between

Detachable InterfacesGrolaux et al., 2004 Pick&Drop Rekimoto, 97

Augmented Surfaces Rekimoto and Masanori, 99 physical

Proximal InteractionsRekimoto et al., 03

screenshot

ARISBiehl and Baily, 04

icon icon 2D map

MIGRIXML 3D3Dvirtualvirtual

3D3Dvirtualvirtual

3D3Dvirtualvirtual



MigriXML highlights:MigriXML highlights: As an environment for simulating migration, there is As an environment for simulating migration, there is

no need to have the physical device to carry on the no need to have the physical device to carry on the process of adapting a previous UI to a new platformprocess of adapting a previous UI to a new platform

As a meta-UI for migration, it could also render As a meta-UI for migration, it could also render remote spaces, moving us forward tele-presence remote spaces, moving us forward tele-presence applicationsapplications

In both cases, all involved elements are In both cases, all involved elements are graphically rendered and the migration is graphically rendered and the migration is continuously depicted during the process, continuously depicted during the process, avoiding disruption in the user’s task, and thus avoiding disruption in the user’s task, and thus improving usabilityimproving usability


ConclusionConclusion As future work, evaluations are As future work, evaluations are

desirable:desirable: On the one hand, expert and user evaluation On the one hand, expert and user evaluation

in order to assess the influence of the in order to assess the influence of the meta-meta-display problemdisplay problem, that is, how the resolution , that is, how the resolution of the display that renders the meta-UI can of the display that renders the meta-UI can affect utility and usability of this approachaffect utility and usability of this approach

On the other hand, summative evaluation On the other hand, summative evaluation with other meta-UI representations, such as with other meta-UI representations, such as the iconic 2D map of ARIS, in order to the iconic 2D map of ARIS, in order to assess which approach is better for a given assess which approach is better for a given scenario scenario

Thank youThank youMercíMercíGraciasGracias

More info:More info:www.usixml.orgwww.usixml.org

Contact:Contact:[email protected]@info-ab.uclm.es

direct manipulation of user interfaces for migration j. p. molina 1,2, j. vanderdonckt 1 and p....

Documents

uiintelligent user interfaces

user interface migration

migrateda user interface

migration environment

interactive systemsthis

virtual environment

interactive way

users environment