how to survive multi-device user interface design with uiml
DESCRIPTION
Talk given at the DSP Valley seminar "Exploring GUI design for embedded systems" together with Jan Meskens on June 9, 2009. See also: http://www.dspvalley.com/upload/event/site/5270/index.html Abstract: The User Interface Markup Language has been around for 10 years now and it is still evolving. The language did not reach a large audience yet and there are few publicly known cases where the language is used. Nevertheless, UIML is probably one of the most generic and flexible user interface markup languages available, mainly because it is in fact a meta-language that is not limited to a particular domain, device, platform or widget set. The design of the language makes it an ideal candidate to support future evolutions in user interface techniques and guarantees the survival of user interface designs over time. At the dawn of the 4.0 version of the UIML specification—which will be released soon—this talk provides an overview of the language, the novelties of the 4.0 specification and some tools that we have built and are available today. We also provide an overview of our work to integrate UIML in a multi-device user interface design process that allows designers to create user interfaces for multiple devices without having to deal with an abstract user interface specification. Finally, we will highlight our experiences with designing tailored user interfaces for embedded platforms with UIML, and discuss the pros and cons of using a high-level user interface description language on devices with limited resources.TRANSCRIPT
How to Survive Multi-Device User Interface Development with UIML
Jo Vermeulen and Jan Meskens
Hasselt University – tUL – IBBTExpertise Centre for Digital Media
Belgium
{jo.vermeulen,jan.meskens}@uhasselt.be
A little bit about us …
A little bit about us …
About 80 researchers …
Specializing in …
A little bit about us …
About 80 researchers …
Specializing in …
The UIML Crew
Kris Luyten Jo Vermeulen Jan Meskens
Professor PhD student PhD student
• PhD about XML UIDLs• developed Uiml.net• several research projects• > 25 related publications• numerous students
• extended Uiml.net• MSc thesis• 2 applied research proj.• 6 related publications• 6 students
• extended Uiml.net• developed Gummy• BSc & MSc thesis• basic research project• 5 related publications• 1 student
Since 2002 Since 2004 Since 2006
What is UIML?
A description of a user interface
UIDLs in practice
UIML is differentUIML is different
UIML is different
• High level
• Generic
– No specific tags (e.g. <button>)
• Meta: build your own User Interface Language
– cfr. XML
What does UIML look like?
What does UIML look like?
<uiml><interface><structure><part>...</part>
</structure><style><property>...</property>
</style><behavior><rule>...</rule>
</behavior></interface><peers>...</peers>
</uiml>
What does UIML look like?
A bit of history
UIML: a pioneering UIDL
Professor Marc Abrams (Virginia Tech, USA)
• 20 years experience with building GUIs
• Frustrated with then-current methods
• UIML 1.0 spec and Java renderer
• Founded Harmonia to provide tools & services for UIML
1997: UIML is born.
UIML: a pioneering UIDL
• 1998
– PhD dissertation by Constantinos Phanouriou.
– Corresponding UIML 2.0 spec
• 1999:
– Windows CE renderer
– Demo at WWW6 conference / W3C DI workshop
• 2000:
– HTML, WML, VoiceXML renderers
1998-2000: updated specification & tool support.
UIML: a pioneering UIDL
• 2001
– UIML 3.0 spec
– LiquidUI tool (Harmonia)
– UIML Europe 2001 conference (Paris, France)
• 2002:
– UIML Technical Committee starts
• 2003:
– Kris Luyten creates open source Uiml.net renderer
• 2004:
– UIML 3.1 specification
2001-2004: UIML comes of age
The company behind UIML
Harmonia, Inc.
• contracts with US DoD
• e.g. UIs for missile control
UIML as abstraction
UIML as abstraction
Hardware
OS
UI toolkits Logic
UIML
Same device, different toolkit
Device A
…
…
Toolkit 1 …
UIML
Device A
…
…
Toolkit 2 …
UIML
Different device/OS/toolkit
…
…
… …
UIML
Device A Device B
…
…
… …
UIML
Multi-device/toolkit examples
Extra component necessary
Hardware
OS
UI toolkits Logic
UIML
UIML renderer
UIML Renderer
• Runtime environment
• Allows for:
– rapid prototyping
– dynamic changes at runtime
Our UIML Renderers
• .NET
– Works on most .NET-enabled platforms:
• Mono (*NIX)
• Microsoft .NET
• Compact .NET
• Java– Runs on various Java VMs and APIs, e.g.
• Java ME
• IBM J9
• MHP Xlet
UIML research/teaching
Dynamic mobile guide UIs
Context-aware UIs
Adapt according to the context-of-use (e.g. location, user profile)
Dynamic changes at runtime
OSGi + UIML
UIML Activator Bundlecontrols rendering core and connects with other OSGI components
Distributed Applications
• MSc thesis Neal Robben
• Distributed user interfaces with Uiml.net
• Synchronization between different UI instances
UIML and distributed Applications
UIML and X + V
• MSc thesis Rob Van Roey
• X+V backend for UIML
• multimodal user interfaces
– GUI interaction (XHTML)
– speech (VoiceXML)
UIML and X + V
UIML on CarMelody platform
• Thesis Pieter Gevers (industrial engineering)
• Prototyping UI with Uiml.net that invokes hardware APIs of Philips car radio
• Philips PNX0106 running
Montavista Linux
UIML on CarMelody platform
Disadvantages of UIML
• Lacks mature/available tools
• Lacks community
• Lacks documentation
• Lacks proof that it “just works” in industrial/commercial settings
What about performance?
Smart optimization
• Rendering certainly slower
• But on the other hand, …
• The UI description provides more data, allowing for smarter optimizations
Case study
• iDTV application, several linked UIML “pages”
• Painfully slow (7 seconds to render a page)
• Serious performance boost after optimization using pre-caching and MPEG-2 frames
Performance comparison
7
0,35
0
1
2
3
4
5
6
7
8
Rendering time per page (seconds)
Unoptimized
Pre-caching
Intermediate UIML
Motivation
• UIML language is platform independent
• Renderers have to be created
– For different computing platforms
– Different programming languages
Java Swing
Swing Backend
UIML for Java
Hardware
OS
Java AWT Logic
UIML
UIML Java renderer
AWT Backend
• Containment
• UI type conversion
UIML for .NET
Hardware
OS
GTK# Logic
UIML
UIML .NET renderer
GTK# Backend
SWF
SWF Backend
High maintenance and development cost
Hardware
OS
Toolkit Logic
UIML
UIML .NET renderer
Toolkit Backend
Hardware
OS
ToolkitLogic
UIML
UIML Java renderer
Toolkit Backend
Flexible Intermediate rendering
Hardware
OS
Toolkit Logic
UIML
UIML .NET renderer
Toolkit Backend
Hardware
OS
Toolkit Logic
UIML
UIML Java renderer
Toolkit Backend
iRenderer iRenderer
Intermediate Renderer
Structure
Style
Behavior
Vocabulary
UIML iUIML
Concrete Widget
PropertiesCallbacks
Concrete Widget
PropertiesCallbacks
Concrete Widget
PropertiesCallbacks
Concrete Widget
PropertiesCallbacks
Concrete Widget
PropertiesCallbacks
IntermediateRenderer
Example UIML
<d-class id=« frame » maps-to=« System.Windows.Forms.GroupBox »>…
</d-class>
Corresponding iUIML
Intermediate UIML
• JSON language (http://json.org)
• All UIML abstractions are resolved
– Concrete widget names
– Concrete property names
– Concrete event names
• iUIML is easy to translate into a running UI
• We built an Adobe Flex renderer in 3 days…
Adobe Flex Renderer
Design Tools
Motivation
• Learning UIML takes time
– Especially for non-technical designers
• Polishing a UI in UIML is difficult
– Requires switching between edit and run mode
Motivation
• Creating one UI for a range of devices is challenging
• Automatic transformations often not aesthetically pleasing
• Manual adjustment is needed
Design Tool: Gummy
Gummy
• Platform independent GUI builder
• Features
– Design GUIs graphically
– Export GUIs as UIML
– Transform GUIs from one platform to another (based on UIML)
– Load a platform-specific design workspace dynamically
Dynamic Design Workspace Loading
Gummy Architecture
DVB-MHP renderer
PDA renderer
Desktop renderer
Desktop Proxy
PDA proxy
DVB-MHP proxy
<uiml></uiml>
Desktop
W2P
Problem in Mobile UI Design
Designer Device
Target Device
Spacing and Positioning
• Foto
Touch Screen Characteristics
Touch Screen Sensitivity?
Occluded Areas?
Interaction Techniques and Emulators
Our Solution: Gummy-Live
“Mirror design changes to the mobile target device in real time”
Gummy-Live
Conclusion
Renderer Tool Support Designer Support