a model-driven development methodology for 3d user interface for information systems in a principle...

Unuse

d

Sectio

n

Space

1

A Model-Driven Approach for Developing 3D User Interfaces

of Information Systems in a Principle–Based WayJuan Manuel González Calleros

PhD Public Defense

February 4, 2010 - LLN

1

February 4, 2010 - LLN PHD Public Defense

Virtual Reality

Augmented Reality

2


Windows Vista

3


Locate your sit in a stadium

4

Outline

1. Introduction2. State of the Art3. Methodology

1. Ontology2. The Method3. Towards a UIDL for 3DUI4. Software for Supporting the Methodology

4. Validation5. Conclusion

February 4, 2010 - LLN PHD Public Defense 5

What is a 3D User Interfaces?


Augmented Reality Virtual Reality

3D User Interface

3D desktop environments

6

Why 3D User Interfaces?

Increase user satisfaction.

Improve cognitive perception

Indices sense of (tele)presence

3DUIs are not automatically superior or inferior to

2DUIs.


Why NOT 3D User Interfaces?

Decrease user performance

Increase manipulation complexity for an average user

Are not appropriate for any task

Are hard to evaluate for their usability


Focus of the dissertation

Information systems (data, process, resources)

Administrative tasks (routine, repetition)

Interaction styles (form filling, multi-windowing, direct manipulation, iconic interaction, graphic interaction, multimedia interaction, and 3DUIs)


Thesis Statement

In this thesis we argue that developing 3DUIs for Information Systems is an activity that would benefit from the application of a methodology which is typically composed of:

1. a set of models gathered in an ontology, 2. a method manipulating the involved models

based on guidelines, 3. a language that express models in the

method.Define a model-driven approach for structuring

the development life cycle of a Three-Dimensional User Interface of an Information System in a principle based way.


Comparative analysis


• 3DUI development methodologies

• Language• Transformation

Engines

• 3D Toolkits

• Models

• Language

• Methods

• Software tools

11

Requirements

Methodological (7)

Ontological (3)

Language (1)

Software tool support (1)

Methodological support for the development life-cycle of 3DUI for Information Systems


Outline






• 3D UI as an specialization of 2D

• Benefits:

• Relying on existing 2D UI by specializing

• Factoring out 3D vs 2D UIs

VS.

14


3D Rendering of a 2D GUI

VUITOOLKIT [Moli08]

15


• Shortcomings:• Adding another modality breaks the factoring

out• Hard to provide multiple representations of

3D objects• No full separation of concerns

16


• Shortcomings:• Adding another modality breaks the factoring

out• Hard to provide multiple representations of

3D objects• No full separation of concerns

17


Support for different

representations

Support for basic Haptic Interaction

3DUIs

18

Outline





Method Outline


Task and Domain Model

Model to Model

Abstract UI Model

Model to Model

Concrete UI Model

Code Generatio

n

Final UI

Control


Physical Control Software Control

Physical interaction object 2D 3D

20



Model to Model

Abstract UI Model

Model to Model

Concrete UI Model

Code Generatio

n

Final UI

• A structured catalog of transformation rules that form a body of design knowledge that can be reused in any 3D method

21


Code Generatio

n

Final UI


Based onGuidelines

Model to model

Abstract UI Model

Based onGuidelines

Model to model

Concrete UI Model

Usability Advisor

AutomaticEvaluation

Refined Concrete UI

Model

Code Generatio

n

3D User Interface

A set of Principles were added to the method

•Guidelines•Task patterns•Canonical list of task types

22


Canonical list of task types

23


Based onGuidelines

Model to model

Abstract UI Model

Based onGuidelines

Model to Model

Concrete UI Model

Usability Advisor

AutomaticEvaluation

Refined Concrete UI

Model

Code Generatio

n

3D User Interface


Facet Selection

24


Based onGuidelines

Model to model

Abstract UI Model

Based onGuidelines

Model to Model

Concrete UI Model

Usability Advisor

AutomaticEvaluation

Refined Concrete UI

Model

Code Generatio

n

3D User Interface


Select Element

Input

Slider

AIO Selection

25


Based onGuidelines

Model to model

Abstract UI Model

Based onGuidelines

Model to Model

Concrete UI Model

Usability Advisor

AutomaticEvaluation

Refined Concrete UI

Model

Code Generatio

n

3D User Interface


Graphical representation selection

26


Based onGuidelines

Model to model

Abstract UI Model

Based onGuidelines

Model to Model

Concrete UI Model

Usability Advisor

AutomaticEvaluation

Refined Concrete UI

Model

Code Generatio

n

3D User Interface


Automatic guidelines evaluation

27


Based onGuidelines

Model to model

Abstract UI Model

Based onGuidelines

Model to Model

Concrete UI Model

Usability Advisor

AutomaticEvaluation

Refined Concrete UI

Model

Code Generatio

n

3D User Interface

Outline


1. Ontology2. Method3. Towards a UIDL for 3DUI4. Software for Supporting the Methodology



Language Engineering Approach

Semantics – Meta Models, UML Class diagrams

SyntaxAbstract – XML SchemaConcrete – XML

Stylistics – Different graphical representations of the concepts


UsiXMLStructured accordingly to the Model Driven

paradigmUsiXML relies on a transformational

approach UsiXML allows the modification of the

developments stepsUsiXML allows reusing parts of previously

specified UsiXML is openFollows a Language Engineering Approach


Outline


1. Ontology2. Method 3. Towards a UIDL for 3DUI4. Software for Supporting the Methodology





Based onGuidelines

Model to model

Abstract UI Model

Based onGuidelines

Model to Model

Concrete UI Model

Usability Advisor

AutomaticEvaluation

Refined Concrete UI

Model

Code Generatio

n

3D User Interface

IdealXML



Based onGuidelines

Model to model

Abstract UI Model

Based onGuidelines

Model to Model

Concrete UI Model

Usability Advisor

AutomaticEvaluation

Refined Concrete UI

Model

Code Generatio

n

3D User Interface

YATE



Based onGuidelines

Model to model

Abstract UI Model

Based onGuidelines

Model to Model

Concrete UI Model

Usability Advisor

AutomaticEvaluation

Refined Concrete UI

Model

Code Generatio

n

3D User Interface

TransformiXML



Based onGuidelines

Model to model

Abstract UI Model

Based onGuidelines

Model to Model

Concrete UI Model

Usability Advisor

AutomaticEvaluation

Refined Concrete UI

Model

Code Generatio

n

3D User Interface

UsabilityAdvisor



Based onGuidelines

Model to model

Abstract UI Model

Based onGuidelines

Model to Model

Concrete UI Model

Usability Advisor

AutomaticEvaluation

Refined Concrete UI

Model

Code Generatio

n

3D User Interface

Outline

1. Introduction2. State of the Art 3. Methodology

1. Ontology2. Method 3. Towards a UIDL for 3DUI 4. Software for Supporting the Methodology




Case Study

Complexity

Man/Month

LOC Context

Polling System

+ 6 6, 000VRML

Information Systems

Student TrainerSystem

+ 6 3, 000X3D

Information Systems

Haptic Browser

++ 4 (collaboration)

0 Information SystemsVisual Impair users

AHMI +++ 48 2, 000X3D10, 000 +Open GL

Information SystemsAeronautics

41


• Polling System. Devoted to the development of an opinion polling system.

42


• Hapget = 3D widget + haptic feedback

43


• Haptic Browser- Google web site. Rendering of web site in the haptic web browser.

44


• The AFMS is a text-only system that calculates the exact route of the aircraft along a given list of waypoints considering possible altitude, time or speed constraints.

• After a trajectory is generated and successfully negotiated with ATC the AFMS can generate input commands for the autopilot and thus control the aircraft in space and in time.

45


• The AFMS application is controlled via the Advanced Human Machine Interface (AHMI), the main application of the target system to be investigated in the HUMAN project

46


• Problems:• Discontinuity with

the rest of the environment

• Difficult to modify due to its Complexity

• Usability guidelines• Widgets have a 3D

nature in the context of a cockpit

47


• Our goal is to provide a 3D Advanced Human Machine Interface.• New set of CUI models• Consolidation of

existing models• New application

domain• …

48

External evaluation

0,00

1,00

2,00

3,00

4,00

5,00

6,00

7,00

SYSUSE INFOQUAL INTERQUAL OVERALL

Val

ue

Metric

CSUQ Perceived Methodology

High

Low

Mean


External evaluation


• Final rendering – Feedback from the haptic browser• Difficult recognition of some hapgets.• Lack of user preference adaptation• Limited exploitation of the haptic characteristics• Long sentences to provide a value• Size of hapgets• Speech recognition problems• Sound become annoying• Slow loading

Outline

1. Introduction2. State of the Art 3. Methodology

1. Ontology2. Method 3. Towards a UIDL for 3DUI 4. Software for Supporting the Methodology

4. Validation 5. Conclusion


We introduced a 3DUI Development Methodology articulated on three axes:

Models

Method

Language


Models (More than 200 attributes, 90 classes, 100 relationships)

Task: Canonical list of task types and guidelines for task

modeling.

Domain Model: Improving existing Domain Model that will be

considered for the NexOF-RA the European standardization

process

AUI: Improving existing AUI Model that will be considered for

the NexOF-RA The European standardization process

CUI: Enriching CUI Model by adding haptic support,

separation of concerns 2D vs 3D by specialization. 3D support.

Hapgets (set of 3D widgets enhanced with haptic feedback)



DomainModel

DomainSubModel

+subModelId

DomainElement

*

DomainItem

+itemId+itemType

DomainCollection

+collectionId+collectionType

Constraint

+condition

0..*

0..*

RelevanceExistence

DomainRelationship

FunctionalDependency

+semanticFunction

ValueConstraint

ReplicationConstraintCardinalityConstraint

Domain Model

AUI Model

Integrates 3DUI in the context of MBUI development Incubator group

Integrates new concepts to their ontology based on UsiXML

http://www.w3.org/2005/Incubator/model-based-ui/wiki/Main_Page

http://forge.morfeo-project.org/wiki_en/index.php/Interactive_Application_Models

54

Methodological aspects adheres to the MDE paradigm. Models and transformations are explicitly defined and used (Around 85 transformations rules) Step 1: Task and concepts (T&C) definition

Task and Concepts Consolidation Selection of task types, items and user categories Guidelines of best-practices for structuring task models

Step 2: From T & C to Abstract UI (AUI) Model Identification of abstract UI structure Selection of AIC Spatio-Temporal arrangement of abstract interaction objects Definition of abstract dialog control Derivation of AUI to domain mappings

Step 3: From AUI to CUI Model Reification of AC into CC Selection of CICs Selection of the graphical representation of the CIC

Step 4: From CUI to FUI Model


All aspects are stored in UsiXML files that can be exchanged, shared, and communicated between stakeholders (designers, developers, and end users).

Advantages of this language were discussed Modifiability Complexity Rigorous. Reasoning. Processable.

Adopt a Language Engineering Approach Semantics – Meta Models Syntax

Abstract – XML Schema Concrete – XML

Stylistics – Different graphical representations of the concepts Our contribution is independent of UsiXML


Five Software modules are used to support the methodology. Conceptual definition of the tools. Supervising of their developmentTesting with our case studies the performance

of the toolsFinal rendering (48 men/month, more than

10, 000 LOC)


Benefits from our methodology


MDA Expected Benefits

Reducing the gap between requirements and implementation

Developer coordination. Dividing the system into several parts and defining interfaces between them

Well-structured systems. This facilitates implementation itself as well as maintenance.

Planning on adequate level of abstraction

Improved communication by visual models

Validation: (Semi-)Formal modeling languages enable automatic validation of the design

Documentation: Models can be used as documentation when maintaining the system.

Platform-independence. At least serve as starting point when implementing the system for a different platform

Enhanced productivity.

Expert knowledge can be put into the code generator

Reduction of errors

Knowledge about creation of modeling languages

Frameworks and tools

Maintenance of modeling language and transformations

Reuse of meta-models and transformations

59

Future WorkExternal evaluation should be targeted to three:

designers, developers and users:User testing with control experiments in labsDesigner and developers

Extend the set of 3DUIs components. Integrated development Environment Explore Other dimensions of the Cameleon

frameworkExplore other context of use

Learning ObjectsAugmented reality


Acknowledgements


2004 20092007 20082005 2006

•UsiXML Consortium

•The Information Systems Research Unit

•The ALBAN program

•The CONACYT program

•The SIMILAR project

•The Human project

•The NexOF project

•The Itea 2 UsiXML project

•The Promep project

_____________PhD_____________________DEA_______

2010

61

Q&A


a model-driven development methodology for 3d user interface for information systems in a principle...

Education

domain model model

abstract ui model model

llnphd public defense

llnphd public defense

d method

modeldriven approach

d objects

d rendering