cross-media information systems - quo vadis?
DESCRIPTION
Seminar presentation about cross-media information spaces given at EPFL.TRANSCRIPT
2 December 2005
Cross-Media Information Systems - Quo Vadis?
Prof. Beat Signer
http://www.beatsigner.com
Web & Information Systems Engineering Lab (WISE)
Department of Computer Science
Vrije Universiteit Brussel
01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 00101000100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110
Fluid Cross-Media Information Spaces
Prof. Dr. Beat Signer Interactive Paper, Cross-Media
Information Architectures
Dr. Bruno Dumas Multimodal Interaction, Cross-
Media Information Architectures
Lode Hoste Multimodal Interaction, Midas
and Mudra Framework
Brecht De Rooms Implicit Context-aware Human-
Computer Interaction, HCII
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
WEB & INFORMATION SYSTEMS ENGINEERING
CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)
Sandra Trullemans Personal Cross-Media
Information Management
Reinout Roels MindXpres Next Generation
Presentation Tool
Ahmed A.O. Tayeh Open Cross-Media Authoring,
Fluid Document Formats
WEB & INFORMATION SYSTEMS ENGINEERING
CISA
Human-Machine &
Human-Information
Interaction
Information
Systems &
Management
Information
Visualisation
& Navigation
CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
PARC User Interface (Mid 70s)
Desktop metaphor
WIMP interaction
"filing cabinets" with
hierarchical folders
Paper paradigm
"What You See Is What
You Get" (WYSIWYG)
what about richer
document formats that
existed at that time?
Xerox Star 8010 [http://www.digibarn.com/collections/systems/xerox-8010/]
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Digital Documents as a Paper Simulator?
Vannevar Bush
Ted Nelson
Most people don't understand the logic of the concept: "What You See Is What You Get" is based on printing the document out ("get" means "get WHEN YOU PRINT IT OUT"). And that means a metaphysical shift: a document can only consist of what can be printed! [...] No overlays [...] – PAPER UNDER GLASS.
When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used [...] The human mind does not work that way. It operates by association.
As We May Think, Atlantic Monthly, July 1945
Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
What is Wrong with Digital Documents?
Existing document formats are based on the simulation
of paper affordances on desktop computers
How to manage mixed-media "documents" in open and
fluid cross-media information spaces? on the data level
- context-sensitive adaptation
- cross-media transclusion
on the visualisation and navigation level
- zoomable user interfaces
on the cross-media interaction level
- fluid multimodal cross-media interfaces
Remediation of the "paper simulation" approach WYSIWYG is only one out of many options!
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
RSL Metamodel and iServer
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Links
Users
Layers Plug-ins
Structure
Active
Content
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Bridging the Paper-Digital Divide
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Interactive Paper Applications
Enhanced Reading
interactive paper maps
interactive festival brochures
educational materials
auction brochures
scientific publications
…
Enhanced Writing
photo album
researcher's notebook
mammography annotation
query by sketching
…
Paper-based Interfaces
PaperPoint
document proof-editing
digital libraries
video analysis tool
educational games
Art Installations
Lost Cosmonaut
Generosa Enterprise
Interactive Tabletops
photo browser
collaborative sketch tool
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Interactive Festival Guide (EdFest)
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
The Lost Cosmonaut
Writing as a collaborative
act of memory and story-
telling
Reading, writing and
narrative as an act of
making sense
Handwriting as a tool for
human-computer
interaction
In collaboration with Axel Vogelsang,
Artists in Labs Programme
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
ArtVis ...
Advanced visualisation
techniques in combination
with a TUI
explore Web Gallery of Art
faceted browsing
Phidgets-based TUI
RFID-tagged physical
objects
Three main components
to explore, analyse and
browse the information
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
RSL-based Associative File System
signer
My Pictures
Vancouver ER 2010
Granville Island
Vancouver Harbour
Public Market
Keynote J. Thorp
Olga and Lamia
Dinner ER 2010
multiple classification
structural link
navigational link
My Videos
Vancouver Aquarium
ER 2010 Keynote
cross-media association
My Presentations
ER 2010 WISE 2009
cross-media
transclusion
single slide
transclusion
(via OOXML
RSL selector)
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
RSL-based Associative File System ...
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
What is Wrong with Slideware?
Simulation of physical slides limited space due to the slide concept
linear navigation from slide to slide
difficult to reuse content and embed rich media types
MindXpres addresses these issues unlimited canvas with zoomable user interface
non-linear navigation and associative linking
content-based approach with automatic visualisation
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
MindXpres Presentation Tool
HTML5-based extensible
core with three types of
JavaScript plug-ins
components
- source code, graph, video, ...
containers
- slide, image gallery, ...
structures
- grid, graph, clustered, ...
Presentation as RSL-
based overlay structure
enables navigation across
different presentations
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
MindXpres Presentation Tool ...
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
MindXpres Presentation Tool ...
Functional MindXpres prototype
Ongoing work visual authoring tool
- alternative to XML-based authoring
rich media plug-ins
- videos with subtitles, annotations etc.
- statistical data with alternative visualisations
- live sensor data
- ...
audience-driven presentations
collaborative annotations and information sharing
multimodal interaction (e.g. integration of PaperPoint concepts)
import of existing slideware formats
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Fluid Multimodal Cross-Media Interfaces
Investigation of mobile and distributed document formats Mobile Cross-Media Applications (MobiCraNT)
Passive or active information units (crosslets) processing and storage of sensor data
binding of specific multimodal interactions to resources
Implicit context-aware human-computer interaction (HCII)
Personal Cross-Media Information Management
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Midas/Mudra Framework declarative definition of multimodal
and multi-touch interactions
rule-based language approach
rapid protoyping and application development
iGesture Workbench create and test gesture sets and algorithms
different modalities: digital pen, Wii remote, …
open source (www.igesture.org)
SpeeG multimodal speech- and body gesture-based
text input
real-time correction of recognised voice input
Multimodal and Multi-Touch Interaction
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Open Cross-Media Linking
Four existing classes of cross-media link systems
A: single source and single target type
B: single source and multiple target types
C: multiple source and single target type
D: multiple source and multiple target types
Our proposed open cross-media link approach covers the whole space
# source types1 2 3 4 5 ...
1
2
3
4
5
...
# target types
B1
B2
D2
open cross-media
linking
D1
A1... An C1 C2 C3
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Open Cross-Media Authoring
Cross-Media
Link Service
Client Application
Cross-Media
Browser & Editor
Visual Plug-ins
Resource Plug-in Repository
Data Plug-ins
Visual Plug-ins Data Plug-insVisual Plug-ins
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Open Cross-Media Authoring ...
Integration of a new resource type includes development of data and visual plug-ins
registration with the resource plug-in repository
Cross-media browser loads plug-ins on demand internal or external (third party application) visualisation
Third party applications delegate the visualisation of a
link target to the cross-media browser similar to "integration" of third-party applications in web browsers
Non-monolithic link model and authoring tool resource-specific plug-ins
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Open Cross-Media Browser and Editor
Cross-media browser & editor Client application
showEntity(), getSelectedEntity()
showEntity()
C
R
U
D
R
S
L
Menu
Source Window Target Window
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Visual Plug-ins
Cross-media browser deals only with RSL-specific data
but not with any media-specific details create/delete links (CRUD), user management, …
Definition of link source and target anchors (resources or
selectors) handled by visual plug-ins separation of concerns
Two basic purposes rendering of specific resource types (resources and selectors)
creation and deletion of resources and selectors
Well-defined visual plug-in API
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
Conclusions
Time to question existing document formats which are
based on the simulation of paper on desktop computers
RSL-based representation of open and fluid cross-media
information spaces interactive paper solutions
associative file system
MindXpres presentation tool
crosslet-based fluid multimodal interfaces
open cross-media information browser and editor
Interlinked structural components rather than a
document-centric representation of information data no longer "owned" by applications
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
References
Beat Signer, Fundamental Concepts for
Interactive Paper and Cross-Media Information
Spaces, ISBN 978-3-8370-2713-6, May 2008
Vannevar Bush, As We May Think, Atlanic Monthly,
July 1945
Beat Signer and Moira C. Norrie, As We May Link: A
General Metamodel for Hypermedia Systems,
Proceedings of ER 2007, Auckland, New Zealand,
November 2007
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
References ...
Beat Signer, What is Wrong with Digital
Documents? A Conceptual Model for Structural Cross-
Media Content Composition and Reuse, Proceedings of
ER 2010, Vancouver, Canada, November 2010
Moira C. Norrie, Beat Signer, Michael Grossniklaus, Rudi
Belotti, Corsin Decurtins and Nadir Weibel, Context-
Aware Platform for Mobile Data Management, Journal on
Wireless Networks (WINET), Vol. 13, No. 6, Springer,
December 2007
ArtVis http://wise.vub.ac.be/content/artvis-exploring-information-through-advanced-visualisation-techniques
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
References ...
Axel Vogelsang and Beat Signer, The Lost
Cosmonaut: An Interactive Narrative Environment on
Basis of Digitally Enhanced Paper, Proceedings of
International Conference on Virtual Storytelling,
Strasbourg, France, December 2005
Christophe Scholiers, Lode Hoste, Beat Signer and
Wolfgang De Meuter, Midas: A Declarative Multi-Touch
Interaction Framework, Proceedings of TEI 2011,
Funchal, Portugal, January 2011
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
References ...
Lode Hoste, Bruno Dumas and Beat Signer,
Mudra: A Unified Multimodal Interaction Framework,
Proceedings of ICMI 2011, Alicante, Spain, November
2011
Beat Signer, Ueli Kurmann and Moira C. Norrie,
iGesture: A General Gesture Recognition Framework,
Proceedings of ICDAR 2007, Curitiba, Brazil, September
2007
Lode Hoste, Bruno Dumas and Beat Signer, SpeeG: A
Multimodal Speech- and Gesture-based Text Input
Solution, Proceedings of AVI 2012, Capri Island, Italy,
May 2012
Beat Signer - Department of Computer Science - [email protected]
November 2, 2012
References ...
Beat Signer and Moira C. Norrie, A Model and
Architecture for Open Cross-Media Annotation and Link
Services, Information Systems 36(3), Elsevier, May 2011