CUIML:A language for generating multimodal human-computer interfaceChristian Sandor

Thomas Reicher

報告人try

Presentation Agenda DWARF簡介 Introduction UIML簡介 Requirement Analysis Related Work Design Example Future Work Value to our research

Reference [1]Christian Sandor,Thomas Reicher(2001),“CUIML: A lang

uage for generating multimodal human-computer interfaces”, Proceedings of the European UIML.

[2]Marc Abrams,et al.,”UIML:An Appliance-Independent XML User Interface Language”,Garvin Innovation Center,VA.

[3]Micah Dubinko(2001),”What Are XForms” ,http://www.xml.com/pub/a/2001/09/05/xforms.html.

[4]Just van den Broecke(2000),”Pushlets: Send events from servlets to DHTML client browsers”,http://www.javaworld.com/javaworld/jw-03-2000/jw-03-pushlet_p.html.

[5]Martin Bauer, Bernd Bruegge, Gudrun Klinker, Asa MacWilliams, Thomas Reicher, Stefan Riß, Christian Sandor, Martin Wagner (2001),”Design of a Component-Based Augmented Reality Framework”, ISAR 2001

DWARF簡介

DWARF: A Wearable Computer

Augmented Reality

虛擬物件與現實世界的結合

DWARF 架構圖

CUIML

Introduction CUIML:Cooperative User Interfaces Markup

Language. CUIML is part of DWARF project The DWARF is to develop a framework for

applications on wearable computers which consists numerous devices.

The interaction with the user should be multimodal in this situation.

Multimodal System Multimodal System supports communication with

the user through different modalities such as voice,gesture and typing

Modal means “Modality” and “Mode” Modality

Communication channel used to convey information.

Mode State that determines the way information is interpreted to

extract meaning.

Multimodal System(2) In the multimodal HCI,system should support user in

put and system output by various ways. Examples

Input:speech,gesture,keyboard Output:voice(VoiceXML),3D graphics(VRML),2D graphi

cs,Text(HTML).

In UIML,we want to describe HCI using an abstract language,so this language should allow us to define the behaviors of various input / output components.

UIML 簡介 Create natural separation of UI design from program

ming Job. Allow many to one relationship between the two.

Transform to native markup language by XSL A measure of the power of UIML is that it can replac

e hand-coding of Java AWT/SWING user interfaces. (和 SALT有何不同 ?)

Where UIML Fits InApplication database

Content (XML, audio,…)

Device Adaption

UIML

WML VoiceXML XHTML cHTML

UIML Perspective

Machine language

Assembly language

“High-level” language (C++, Java)

Scripting languages

Device-dependent markup

Device-independent markup

Features of UIML UIML can be rendered to many device-dependent la

nguage like VoiceXML,HTML or WML. Web-based Interfaces generated with UIML can send events but

can not receive events.(每次改變時，都要重新送一次文件 )

UIML only generate views that do not interact with each other.

CUIML Overview CUIML use UIML as a starting point. CUIML is a unified user interface description

language to describe multimodal user interface in an abstract way.

CUIML suggests a MVC(Model-View-Controller) framework to solve the synchronization problems.

CUIML Overview(2) By XSL transformations,these descriptions are

converted into various markup language for various devices.

The controller’s internal structure is a DFA(deterministic finite automaton),it map CUIML to various input and output device

Requirements analysis Modeling information at a high level of abstraction Mapping generic information to device-dependent

presentations Synchronization of the different channels Controller configuration

Related Work UIML Petri Net S.Riss Work Flow Engine XForms Mozquito

Design of CUIML

Runtime behavior of the components The client request a HCI by sending a HTTP request. Server renders the components and send back to

client. The rendering is done by applying XSL

transformations.

Initializing the Client

Controller Config /

Information for Manipulator

Generation of HCI components

The generated Components(1) Controller

Synchronize the view components Keep track of current state of HCI Use WFE implementations in prototype

The generated Components(2) View

Display markup languages Send event to controller The concept of describing and rendering views is adopted

form UIML.

The generated Components(3) Manipulator

An enhancement over UIML. Changes to the view is done by manipulator instead of ren

dering new view. View is accessed by manipulator by the Document Object

Model(DOM).(這代表 View不能太大 )

The generated Components

Structure of CUIML

Create Multi User Interface with CUIML

Example:StickySelected

Edit this sticky

Edit here紅色代表目前被選擇的利貼

DFA of the Controller

Define the interface

Define the events

Future Work To develop a CUIML IDE Security concerns of the CUIML component sent to

client side Internal structures of Controller using DFA is not

sufficient for more complex situations.One possibility will be Petri Nets or even Colored Petri Nets.

The value to our research The idea of synchronize VRML 、 VoiceXML and

HTML with MVC Model.(Instead of time tag) To change views at runtime by Manipulator. The flow control language may be substitue by XLA

NG or Petri Net.

Problems may solved using CUIML Voice /Animation synchronization. Protocol integration. ASR/TTS integration and its performance issues. Virtual user autonomy. The “Voice propagation range” issues.

System Design Prototype(Before)

System Design Prototype(After)

Q & A