multi-paradigm modeling of hmi applications
DESCRIPTION
This presentation about the use of multi-paradigm modeling for human-machine interaction was given at the Universiteit Antwerpen as part of a research seminar of the Belgian PAI research project MoVES (Modeling, Verification and Evolution of Software). It presents an overview of the planned research in the context of the PhD of Romuald Deshayes, under supervision by Tom Mens at the Software Engineering Lab of the University of Mons, Belgium.TRANSCRIPT
Challenges in using MDE for HMI
Multi-Paradigm Modelingof HMI Applications
Tom Mens Romuald Deshayes
Software Engineering Lab
MoVES event, Universiteit Antwerpen, 20 December 2011Tom Mens – UMONS 1 / 23
Challenges in using MDE for HMI
HMI modeling
Outline : HMI modeling
HMI modeling
Tom Mens – UMONS 2 / 23
Challenges in using MDE for HMI
HMI modeling
Example of a HMI application
3D visual drawing toolHaptic (gestural) user interface(using hands) based on KinectCreation and manipulation of 3Dobjects, using OpenGLComplex UI behaviour representedusing statecharts www.youtube.com/
watch?v=lVcqzWTnpMY
Tom Mens – UMONS 3 / 23
Challenges in using MDE for HMI
HMI modeling
Multi-Paradigm Modeling for HMI
GoalUse MDE techniques to advance the state-of-the-art in HMI (HCI)
Multi-paradigm modeling. Existing approaches focus on :Task modeling : understand user tasks to improve UIs (cf.Jean Vanderdonckt - UCL)Presentation modeling : focus on visual appearance of theapplication (e.g., OpenGL)User modeling : focus on the user characteristicsBehavioural modeling : model how the application reacts tousers actions (e.g. event-based)Physical modeling : physical behaviour of real or virtual objects. . .
Tom Mens – UMONS 4 / 23
Challenges in using MDE for HMI
HMI modeling
Task modeling
Example : ConcurTaskTrees task model for a digital home
Tom Mens – UMONS 5 / 23
Challenges in using MDE for HMI
HMI modeling
Behavioural modeling
Example : Statecharts model of hand behaviour [DM11]
Tom Mens – UMONS 6 / 23
Challenges in using MDE for HMI
HMI modeling
Behavioural modeling
Example : Statecharts mode of UI component behaviour [DM11]
Tom Mens – UMONS 7 / 23
Challenges in using MDE for HMI
HMI modeling
Behavioural modeling
Example : Statecharts model of graphical scene behaviour [DM11]
Tom Mens – UMONS 8 / 23
Challenges in using MDE for HMI
HMI modeling
Behavioural modeling
Example : Petri net model of double click behaviour(Palanque - Toulouse) [NPLB09]
Tom Mens – UMONS 9 / 23
Challenges in using MDE for HMI
HMI modeling
Physical modeling
GoalModeling physical behaviour or interaction of virtual objects
E.g., falling, rolling, colliding, bouncing, breaking, . . .Requires continuous models (e.g., differential equations)
ChallengeCombining discrete and continuous models
Possible solution : hybrid automataDiscrete automata containing differential equations in the states
hybridAutomaton.png
Tom Mens – UMONS 10 / 23
Challenges in using MDE for HMI
Research Challenges
Outline : Research Challenges
Research Challenges
Tom Mens – UMONS 11 / 23
Challenges in using MDE for HMI
Research Challenges
Research Challenges
Top-down
Come up with domain-specific modeling languages (DSML) thatcombine multiple views on HMI modeling.
Bottom-upEnable formal reasoning over HMI models
Detect unintended behavioural problems (nondeterminism,non-termination, ...)Detect HMI vulnerabilities, inconsistencies, incompletenessFacilitate code generation. . .
Tom Mens – UMONS 12 / 23
Challenges in using MDE for HMI
Research Challenges
Formal reasoning - Example
Work by Combéfis and Pecheur to analyse HMI vulnerabilities[CP09]Using labeled-transition systems (LTS) semantics to generate andanalyse users’ mental modelsExample : LTS of user’s mental model of a lamp, compared to thereal system model.
Tom Mens – UMONS 13 / 23
Challenges in using MDE for HMI
Research Challenges
Formal reasoning - Example
Work by Combéfis and Pecheur to analyse HMI vulnerabilitiesExample : accident with Therac-25 medical system due to aincorrect operator-machine interaction
Fig. 7. Statechart model of the Therac-25 medical system [16]. The enter command is represented with
�
and the up command is represented with ↑.
X-ray beam (selectX), validating it (enter) and administeringthe treatment (fire), the X-ray command may or may notbe available depending on the execution followed in thesystem. This means that the system may end up either inthe X-ray or in the E-beam mode, non-deterministically andwith no observable difference. That behaviour is due to aninternal transition which occurs when the treatment has beenadministered, which represents the fact that the system is resetto its initial state. The revealed controllability issue indicatesthat there should be an observation informing the user when thesystem is reset. Adding a reset observation makes the systemfull-controllable and a minimal full-control mental model forit can be generated, with 24 states.
The other issue with that system, the well-known one,can also be detected with our framework, after fixing theother issues in the model. If the operator is not aware ofthe 8-seconds timer (or does not track the countdown), theissue described in [2], [16] can also be found. It suffices toturn the observation 8seconds into an internal transition τ andto make the system being reset when the operator pressesenter after the treatment has been administered. The lastactions of the returned counterexample (the most relevantpart) is: �. . . , selectE, up, E-beam, selectX, E-beam�. Thatcorresponds to the user selecting the electron beam mode, thenchanging his mind by pushing on the up button and selectingthe X-ray mode. After that, the system may either be in E-beamor X-ray mode.
B. The Video Cassette Recorder example
The second example is a model of a video-cassette recorder(VCR). That model has been obtained from an ADEPT modelwhich has been translated into a JPF statechart Java program,manually. The system has six main operating modes: play,stopped, fast forward, rewind, pause and record. There is onecommand to activate each of those modes and one additional
command to turn the system on or off. Moreover, there areseveral speeds for the fast forward and rewind modes. Thosespeeds are automatically chosen by the system according to theremaining tape length. Finally, the system enters automaticallyinto the rewind mode when reaching the end of the tape. Thesystem model has 1088 states and 3740 transitions. There areseven commands (one to activate each mode and a power button)and two observations (tape moving forward and backward).
Analyzing the system against the full-control relation high-lights some bad interaction that may happen and lead toconfusion by the operator. The issue comes from the [ON,STOP, 0.01] state which corresponds to the VCR being turnedon, the tape being stopped and the remaining length of tapebeing 0.01. In this state, when the user presses the playcommand, the system can go to one of the two followingstates: [ON, PLAY, 0.01] or [ON, REWIND FULL, 0.00]. Inthe first case, the system just switches in the play mode. In thesecond case, after the user presses the play button, the tapehas just moved forward until it remains zero, in which case thesystem automatically goes to rewind mode. The controllabilityissue pointed by the framework is that the pause commandis not available in the two states that are reachable from [ON,STOP, 0.01] after executing the action play.
Such kinds of error can be harmful. For the VCR example,suppose that the operator wants to press the play button andthen, someone calls him and he presses the pause button as,for him, it is an authorized operation. When he comes backto his VCR, he observes that the tape was rewinded up tothe beginning. That situation occurs because after pressing theplay button, the system can either transition into a state wherethe pause button is enabled or in another state where it isnot. For critical systems, such an error can have more seriousconsequences.
Tom Mens – UMONS 14 / 23
Challenges in using MDE for HMI
Research Challenges
Domain-Specific Modeling for HMI
DSMLHow to define a DSML that combines the multiple modelinglanguages/paradigms used for HMI ?
Tom Mens – UMONS 15 / 23
Challenges in using MDE for HMI
Research Challenges
Domain-Specific Modeling for HMI
Domain-specific model : specific to application domainE.g., robotics, gaming, home automation, virtualreality
Platform model : describes platform specificitiesmodality (vocal, tactile, WIMP)input devices (hardware sensors, keyboard,mouse, Kinect)output (screen, VR goggles, voice). . .
Transformation model : weaves both models into platform-specificmodel, using model transformation technology
Tom Mens – UMONS 16 / 23
Challenges in using MDE for HMI
Research Challenges
Visual formalismsAre used for formal reasoning and analysis over the models
Questions :Which formalisms to use for which purpose ?How to combine results from different formalisms ?How to feed back results into DSML ?
Remark : model transformations are models too, amenable toformal analysis
Tom Mens – UMONS 17 / 23
Challenges in using MDE for HMI
Research Challenges
Visual Formalisms
Use most appropriate formalism, at most suitable abstraction level,for a given purpose
Possible candidatesStatecharts : for event-driven, deterministic, reactive systemsPetri nets : for analysing concurrency-related issuesLabelled Transition SystemsHybrid automata (combining discrete and continuous models)Logic-based or constraint-based formalismsGraph transformation (e.g., for model transformation)
Tom Mens – UMONS 18 / 23
Challenges in using MDE for HMI
Research Challenges
Visual Formalisms
Which semantics to associate to each formalism ?Which tools to use for formal reasoning ?
Graph transformation tools (e.g., AGG, Fujaba, GrGen)Model manipulation tools (ATL, EMF, Epsilon, AtoMPM,TCore, . . . )HyTech tool (for hybrid automata)Model checkersSAT solvers. . .
Tom Mens – UMONS 19 / 23
Challenges in using MDE for HMI
Research Challenges
More challenges
How to facilitate evolution ?How to generate code from models ?How to develop generic framework (independent of particularDSM or platform model) ?Where and how to use graph transformation ?Killer app ? Augmented reality, gaming, home automation,robotics ?
Tom Mens – UMONS 20 / 23
Challenges in using MDE for HMI
Research Challenges
Thank you
Thank you for your attention !
Questions, suggestions ?
Tom Mens – UMONS 21 / 23
Challenges in using MDE for HMI
Research Challenges
Bibliography I
Sébastien Combéfis and Charles Pecheur.A bisimulation-based approach to the analysis ofhuman-computer interaction.In EICS, pages 101–110, 2009.
Romuald Deshayes and Tom Mens.Statechart modelling of interactive gesture-based applications.In CoDeisMoto Workshop, co-located with Interact 2011, 2011.
Tom Mens – UMONS 22 / 23
Challenges in using MDE for HMI
Research Challenges
Bibliography II
David Navarre, Philippe A. Palanque, Jean-François Ladry, andEric Barboni.Icos : A model-based user interface description techniquededicated to interactive systems addressing usability, reliabilityand scalability.ACM Trans. Comput.-Hum. Interact., 16(4) :56, 2009.
Tom Mens – UMONS 23 / 23