model based development for 3 d user interfaces

A model-based development A model-based development for 3D User Interfacesfor 3D User Interfaces

Juan Manuel Gonzalez CallerosJuan Manuel Gonzalez Calleros

Université catholique de Louvain, School of Management (IAG)Information Systems Unit (ISYS)

Louvain-la-Neuve, 13th June, 2006

June 13th June 13th DEA Louvain-la-Neuve, Belgium DEA Louvain-la-Neuve, Belgium 22

ContentsContents

IntroductionIntroduction Related WorkRelated Work Model-based development for 3D User Model-based development for 3D User

InterfacesInterfaces Case StudyCase Study ConclusionConclusion


Introduction - ProblemIntroduction - Problem

Current methods:Current methods: rarely provide the design knowledgerarely provide the design knowledge are more focusing on the implementationare more focusing on the implementation

Available tools for 3D UIs are: Toolkits, Available tools for 3D UIs are: Toolkits, interface builders, rendering engines interface builders, rendering engines


Introduction – proposalIntroduction – proposal

We propose:We propose: a user-centered approacha user-centered approach model-driven architecturemodel-driven architecture separation of concernsseparation of concerns

Our goal:Our goal: ““to demonstrate the feasibility of a MDE-to demonstrate the feasibility of a MDE-

compliant method that is user-centered as compliant method that is user-centered as opposed to contents-centric for developing opposed to contents-centric for developing 3D UIs”3D UIs”


ContentsContents




Related work – 3D UIs generationRelated work – 3D UIs generation

Programmatic approach:Programmatic approach: OpenGL, OpenGL, Direct3D, C++, MCNL, VRML, X3D, … Direct3D, C++, MCNL, VRML, X3D, …

Toolkit programming:Toolkit programming: Maya, Max3D, Maya, Max3D, Crazy Eddie's GUI System, Anark, AliceCrazy Eddie's GUI System, Anark, Alice

Render engines:Render engines: Microsoft task gallery, Microsoft task gallery, infinite 3D Cube, Sphere XP, looking infinite 3D Cube, Sphere XP, looking glass, Clara browser, 3DNAglass, Clara browser, 3DNA

Model based: Model based: ontoWEb (VR-Wise), ontoWEb (VR-Wise), ConGenIVE (VRIXML), InTML, ContigraConGenIVE (VRIXML), InTML, Contigra


Related work – Programmatic Related work – Programmatic Approach, VRMLApproach, VRML


Related work – Toolkit Approach, AnarkRelated work – Toolkit Approach, Anark


Related work – Render Engines, 3DNARelated work – Render Engines, 3DNA


Related work – Model-based approach, Related work – Model-based approach, CoGenIVE-VRIXMLCoGenIVE-VRIXML

<<GroupItemGroupItem >> <<UISliderUISlider paramID paramID ="="1010">"> <<ValueValue min min ="="00"" max max="="255255"/>"/> <<TickstyleTickstyle orientation orientation ="=" horizontal horizontal“ “ position position ="="

both both "" frequency frequency ="="1616"/>"/> </</ UISlider UISlider >> <<PositionPosition >> <<XX>>1.0 1.0 </</XX><><YY>>1.5 1.5 </</YY>> </</ Position Position >> </</ GroupItem GroupItem >>


Related work – ObservationsRelated work – Observations

Methodological diversity Methodological diversity Not user centered approachesNot user centered approaches Single entry and single output Single entry and single output Transformations are hiddenTransformations are hidden


Related work – ShortcomingsRelated work – Shortcomings

Lack of a methodology for developing 3D Lack of a methodology for developing 3D UIsUIs

Lack of user task modelsLack of user task models Lack of models independent to the final Lack of models independent to the final

implementationsimplementations Lack of a toolkit for such workLack of a toolkit for such work Lack of genuine 3D UIsLack of genuine 3D UIs


ContentsContents

IntroductionIntroduction Related workRelated work Model-based development for 3D User Model-based development for 3D User



Model-based development of 3DUIs Model-based development of 3DUIs requirementsrequirements

ExpressivityExpressivity Machine processableMachine processable Human readableHuman readable Methodological explicitness,Methodological explicitness, flexibility, flexibility,

separation of concernsseparation of concerns


We rely on…We rely on…

Cameleon Cameleon FrameworkFramework

UsiXMLUsiXML UML class diagramsUML class diagrams Graph Graph

transformationstransformations

Node typeNode

(Attribute,value)

Edge typeEdge

(Attribute,value)

Node

Edge

Node typeNode

(Attribute,value)

Edge typeEdge

(Attribute,value)

Node

Edge


Method OutlineMethod Outline


Development pathsDevelopment paths


Graph transformation tool support AGGGraph transformation tool support AGG


Method Step1 – Task and concepts Method Step1 – Task and concepts description description


Method Step1 - User Action Task Method Step1 - User Action Task ExtensionExtension

actionType Definition

Trigger Specifies the triggers of an action

Terminate Specifies the end to an action

Indicate Specifies some sort of indication

Create Specifies the creation an item

Modify An action of modifying an item

Toggle The existence of two different states of an item

Communicate The action to communicate the user to the system or vice versa

Mediate The action of mediate user actions

Perceive The action of identifying any user action


Method Step1– 3D Task PatternsMethod Step1– 3D Task Patterns

A pattern is not a finished design that can A pattern is not a finished design that can be transformed directly into code be transformed directly into code

It is a description or template for how to It is a description or template for how to solve a problem that can be used in many solve a problem that can be used in many different situations different situations

We propose some patterns for common We propose some patterns for common 3D user tasks3D user tasks


Method Step1 – 3D Task PatternsMethod Step1 – 3D Task Patterns

Navigation patternNavigation pattern TravelTravel Wayfinding Wayfinding


Method Step1– 3D Task PatternsMethod Step1– 3D Task Patterns

Navigation patternNavigation pattern Travel patternTravel pattern Wayfinding Wayfinding


Method Step1– 3D Task PatternsMethod Step1– 3D Task Patterns Navigation patternNavigation pattern

TravelTravel Wayfinding patternWayfinding pattern


Method Step1– Domain ModelMethod Step1– Domain Model


Method Step1 – Software toolMethod Step1 – Software tool


Method Step2 – Abstract ModelMethod Step2 – Abstract Model

An An Abstract Individual ComponentAbstract Individual Component ((AICAIC) ) is an abstraction that allows the is an abstraction that allows the description of interaction objects in a way description of interaction objects in a way that is independent of the modality that is independent of the modality

An An Abstract ContainerAbstract Container (AC) is an entity (AC) is an entity allowing a logical grouping of other allowing a logical grouping of other abstract containers or abstract individual abstract containers or abstract individual components. components.


Method Step2 – Tool supportMethod Step2 – Tool support


Method Step3 –To obtain a Concrete Method Step3 –To obtain a Concrete specificationspecification


Method Step3 –Taxonomy of 3D UIsMethod Step3 –Taxonomy of 3D UIs


Method Step3 -Taxonomy of 3D UIsMethod Step3 -Taxonomy of 3D UIs

Pure Reality Augmented reality Augmented

Virtuality Virtual 3D GUI Digital 3D GUI 3D rendering of 2D

GUI 2D GUI


Method Step3 -Method Step3 - Lesson learnt from Lesson learnt from TaxonomyTaxonomy

Source of inspirationSource of inspiration Abstract characteristics Abstract characteristics Design optionsDesign options Gaps to fulfillGaps to fulfill Concrete User Interface ModelConcrete User Interface Model


Method Step3 -Method Step3 - Tool support - Tool support - AliceAlice


Method Step3 -Method Step3 - Tool support - Maya Tool support - Maya


Method Step3 -Method Step3 - Tool support - Tool support - VUIToolkitVUIToolkit


Method Step4 – Final User InterfaceMethod Step4 – Final User Interface

VRMLVRML X3DX3D Java 3DJava 3D


ContentsContents




Case Study – Task and domain Case Study – Task and domain definitiondefinition


Case Study – Concrete DefinitionCase Study – Concrete Definition


Case Study – Concrete Model BCase Study – Concrete Model B


Case Study – From Abstract to concrete Case Study – From Abstract to concrete ModelModel

NAC NAC LHS :=

RHS

x>2


Case Study – From Abstract to concrete ModelCase Study – From Abstract to concrete Model

“Show Questionnaire”

Output (value unknown)

Attribute, data type, domain characteristics

An output text

“Select Answer” Select attribute value + repetitive (selection values not known)

Data type, domain characteristics

A dropdown list, a group of option buttons

“Send Questionnaire”

Control Feedback A trigger

“Navigation” Navigation Feedback A trigger


Mock-up of the Concrete User Mock-up of the Concrete User InterfaceInterface


Case Study – From Abstract to Case Study – From Abstract to concrete Modelconcrete Model

<<GroupGroup>><<CubeFaceCubeFace id id="="C1C1">">

<<SphereTriggerSphereTrigger defaultContentdefaultContent="="StartStart"" radiousradious="="1.51.5"" solid solid="="TrueTrue"" isVisibleisVisible="="truetrue""

isEnabledisEnabled="="truetrue">"><<TransformTransform scale scale="="8.23 8.23 8.238.23 8.23 8.23""

translationtranslation="="0.27 12.14 0.27 12.14 18.3018.30"/>"/><<TouchSensorTouchSensor id id="="TS1TS1""

enabledenabled="="TrueTrue"/>"/><<AppearanceAppearance name name="="ButtonAppeButtonAppe""

id id="="App1App1">"><<MaterialMaterial diffuseColordiffuseColor="="0.8 0.8 0.00.8 0.8 0.0""

specularColorspecularColor="="0.11 0.11 0.11 0.11 0.110.11""

emissiveColoremissiveColor="="0.0 0.0 0.00.0 0.0 0.0"" shininessshininess="="0.30.3"/>"/></</AppearanceAppearance>></</SphereTriggerSphereTrigger>>

</</GroupGroup>>


Case Study – Editing the FUI in MayaCase Study – Editing the FUI in Maya


Case Study – Editing the FUI in AliceCase Study – Editing the FUI in Alice


Case Study – FUI generated with Case Study – FUI generated with AliceAlice


Case Study – FUI generated with Case Study – FUI generated with the VUIToolkitthe VUIToolkit


Case Study – FUI generated with Case Study – FUI generated with the Mayathe Maya


Use of the virtual polling system Use of the virtual polling system Case Study in a virtual worldCase Study in a virtual world


ContentsContents




Conclusion –Model-based approachesConclusion –Model-based approaches

Modelst= task, Do = Domain Di = dialog AUI=abstract presentation CUI=concrete user interface U = user, C = context.

Inter Model Transformation Bidirectional derivation→ Derivation link≈→ Manual Derivation≈ Manual Bidirectional der.FUI = Final User Interface

FUI target languages*Indicates that it is done manually

VR-Wise CUI CUI → FUI VRML, X3D

CoGenIVE T, Di, CUI ( T, Di, CUI ) FUI C++

InTML Di, CUI T ≈→ (Di, CUI), (Di, CUI) ≈→ FUI

Contigra CUI, Di (CUI, Di) → FUI X3D, Behavior3D, Audio3D

Our methodology T, Do, C, AUI,

CUI

T Do, T AUI, AUI CUI, CUI AUI, T CUI, (T, Do,AUI,CUI) C, CUI≈→ FUI

Java, XHTML, Flash, HTML, Voice XML* Java 3D*, X3D*, VRML*.


ConclusionConclusion

The method based on graph transformation The method based on graph transformation has been introduced, defined and illustrated has been introduced, defined and illustrated

We proposed a method for 3D UI development We proposed a method for 3D UI development whose life cycle was divided into four steps whose life cycle was divided into four steps ranging from the most abstract to the most ranging from the most abstract to the most concretes according to the principles of Model-concretes according to the principles of Model-Driven Engineering.Driven Engineering.

The feasibility of the approach is much The feasibility of the approach is much depending on the amount and the quality of the depending on the amount and the quality of the design rules that are encoded in UsiXML. design rules that are encoded in UsiXML.


ConclusionConclusion

Future work.Future work. Exploring more design Exploring more design options and encode them in UsiXML options and encode them in UsiXML ModelsModels Transformational rulesTransformational rules Software toolsSoftware tools More FUI More FUI

Consolidate conceptsConsolidate concepts


Thank youThank you

Juan Manuel Gonzalez CallerosJuan Manuel Gonzalez Calleros

model based development for 3 d user interfaces

Business