Display-agnostic HypermediaDisplay-agnostic Hypermedia

Unmil P. Karadkar, Unmil P. Karadkar, Richard FurutaRichard Furuta, Selen Ustun, YoungJoo Park, , Selen Ustun, YoungJoo Park, Jin-Cheon Na*, Vivek Gupta, Tolga Ciftci, Yungah ParkJin-Cheon Na*, Vivek Gupta, Tolga Ciftci, Yungah Park

Center for the Study of Digital LibrariesCenter for the Study of Digital LibrariesDepartment of Computer ScienceDepartment of Computer Science

Texas A&M UniversityTexas A&M University

*Division of Information Studies*Division of Information StudiesSchool of Communication and InformationSchool of Communication and Information

Nanyang Technological UniversityNanyang Technological University

Talk OverviewTalk Overview• Trellis and caT Trellis and caT (context-aware Trellis)(context-aware Trellis)

• MIDASMIDAS– Multi-device Integrated Dynamic Activity Spaces– Objectives– Architecture– Browser enhancements

• Display-agnostic hypermediaDisplay-agnostic hypermedia

• Resource ManagementResource Management

• Future workFuture work

TrellisTrellis

• Hypertext specificationHypertext specification– Based on a formal Petri net model– Includes the browsing semantics– Separates structure from its information content

• The browsing experience depends uponThe browsing experience depends upon– User identity– Initial state of the hypertext– Actions of other users browsing the hypertext

context-aware Trellis (caT)context-aware Trellis (caT)

• Extends TrellisExtends Trellis– Inherits its characteristics– Incorporates context-awareness– Includes support for hierarchical structures

• User Context affects the browsing User Context affects the browsing experienceexperience– Location– Time– Characteristics and preferences

caT and TrelliscaT and Trellis• Server-centricServer-centric

– Server maintains the state of browsing– Directs browsers to display information content for the

current state

• BrowsersBrowsers– Receive pointers to the content to be displayed– If possible, display the content– Type-specific browsers, multiple browsers may be active

simultaneously

• Information RenderingInformation Rendering– Onus for invoking appropriate browsers is on the reader– Information that cannot be presented by active browsers

may not be rendered at all

MIDASMIDAS• Multi-device Integrated Dynamic Activity SpacesMulti-device Integrated Dynamic Activity Spaces

• Extends caT - Inherits its characteristicsExtends caT - Inherits its characteristics

• ObjectivesObjectives– Ensure that all information content associated with the

current browsing state is presented (completeness)– Ensure consistency of the presented information

(coherence)– Coordinate the information rendered over all available

browsers– Support browsing from multiple browsers on a set of

diverse client devices– Allow clients and browsers to dynamically join or leave a

browsing session – implications for maintaining coherence

Display-agnosticismDisplay-agnosticism• MIDAS users may browse hypertexts from MIDAS users may browse hypertexts from

– Various devices• Desktop, notebook computers• PDAs, cell phones, palm top computers

– Locations with different characteristics• Homes, offices (private, protected)• Airport terminals, hotel lobbies (public, insecure)• Cars (semi-private)

• MIDAS must cope with diversity in MIDAS must cope with diversity in – device characteristics– Environmental constraints– User preferences

• Separation of information content from its renderingSeparation of information content from its rendering

Display-agnostic hypermediaDisplay-agnostic hypermedia• Lend themselves to multiple forms of presentationLend themselves to multiple forms of presentation

– Single structural representation– Alter the presentation for best rendering on individual

devices– Conform to user preferences and characteristics– Within constraints of characteristics of available

information

• SeparateSeparate– Structure from information content– Information content from presentation

• Address diversity of the information access Address diversity of the information access environmentenvironment

Architecture – Trellis and caTArchitecture – Trellis and caT

Users

Hypertext Authoring

Tool

Hypertext Specification

Information Service

Device 3Device 2

Device 1

Browser 4Browser 3

Browser 2

Authors

Browser 1

Device 3Device 2

Device 1

Browser 4Browser 3

Browser 2

MIDAS

MIDAS - ArchitectureMIDAS - Architecture

Authors

Users

Device Manager

user and device profiles, current device load

resource handles,author preferences

user actions

Browser 1

select device(s)

status,user actions

Browser Coordinator

Browser registry

user actionsinformation content

Hypertext Authoring

Tool

Hypertext Specification

Information Service

resource handles, constraints, preferences

information content, resource properties

Resource Realizer

Resource Repository

resource instance(s)

resource handle, constraints

Resource Author

Authors

Attributes That Affect BrowsingAttributes That Affect Browsing• DeviceDevice

– Display (resolution, colors)

– Storage space– Processor speed– Bandwidth

• UserUser– Expertise– Media, device and

other preferences– Access privileges

• EnvironmentEnvironment– Privacy– Security– Interference

• Information elementsInformation elements– Media (mime) type– File size– Version– Creator– Creation timestamp– Location– Textual description– Required display space– Display colors – Required bandwidth– Language– Priority– Privacy

High-level High-level propertiesproperties– Optimization

• quality

• time

• Cost

– Interaction vs. autonomy

Browser-driven Information Rendering Browser-driven Information Rendering

Different browsers render a caT specification Different browsers render a caT specification differently, displaying only the elements they can differently, displaying only the elements they can renderrender

Text browsing Text browsing

Spatial BrowserSpatial Browser

• Composite browserComposite browser– Combines various information elements for

display on a single canvas– Authors may

• Specify the display location of individual elements• Leave the organization of elements to the browser

• Reinterprets hypertexts for an alternate Reinterprets hypertexts for an alternate presentationpresentation

Spatial BrowsingSpatial Browsing

Audio-video BrowserAudio-video Browser• Audio-only feedback to usersAudio-only feedback to users

• Users interact via KeyboardUsers interact via Keyboard

• Serves asServes as– The primary browsing mechanism for visually impaired

users– Alternate or supporting browsing mode for normal sighted

users

• Dual mode browserDual mode browser– Hypermedia browsing– Access to individual information elements

• Extends caT’s browsing support to audio-video Extends caT’s browsing support to audio-video mediamedia– Converts text to audio for rendering

Audio BrowsingAudio Browsing

Multi-device BrowsingMulti-device Browsing• Support for variation in device characteristicsSupport for variation in device characteristics

– Display size, network bandwidth– Availability of browsers for each device

• Information parallelismInformation parallelism– Media types (text, images, audio, video)– Variation in resolution

• Image scaling• Text summarization• Audio/video quality

• Authoring hypertexts and creating multiple Authoring hypertexts and creating multiple instances of each resource is a tedious task!instances of each resource is a tedious task!

Resource ManagementResource Management• Provide feedback to support a Provide feedback to support a

variety of devicesvariety of devices– Feedback regarding device

categories currently supported– Suggest media formats needed to

support additional device types

• Aid resource creatorsAid resource creators– Automatic conversion of formats

• Audio to text, text to audio

• Image scaling and color-depth adjustments

• Extraction of text from formatted documents

• Text summarization

Future WorkFuture Work• Unified hypertext authoring and resource Unified hypertext authoring and resource

management interfacemanagement interface

• What is the closest match for a 640X480 image?What is the closest match for a 640X480 image?– 80X60 thumbnail or a textual description of the image

contents?– Similarity metrics for delivering the most appropriate

instances of a hypertext’s information content

• How do users deal with multiple devices?How do users deal with multiple devices?– Implications for partitioning information across devices

• Integration of similarity metrics and partitioning Integration of similarity metrics and partitioning algorithms into the Device manager and Browser algorithms into the Device manager and Browser CoordinatorsCoordinators

For more informationFor more information

• The caT project homepageThe caT project homepage

http://www.csdl.tamu.edu/caT

• EmailEmail

furuta@csdl.tamu.edu

Composition of Hypertext SystemsComposition of Hypertext Systems• ModelModel• SpecificationSpecification• ContentContent• StructureStructure• PresentationPresentation• BehaviorBehavior• ServerServer• BrowsersBrowsers• ExperienceExperience

Hypertext SystemsHypertext Systems

SpecificationSpecification

ModelModel

ContentStructureBehavior

PresentationPresentation

ServerServer BrowserBrowser

ExperienceExperience

AuthorAuthor ReaderReader

Diversity in Information Access DevicesDiversity in Information Access Devices• The information environment is diversifyingThe information environment is diversifying

– Information devices• Mobile devices, Desktop computers, and Large screen devices• Form factor, network bandwidth, storage, processing power

– Properties of the environment• Private (homes, offices), semi-private (cars), and public (malls)

• Most hypertexts are authored for a single class of Most hypertexts are authored for a single class of devices – desktop computers or mobile devicesdevices – desktop computers or mobile devices– Separate structure as well as services for each class

• The Weather Channel, CNN, Yahoo!

– Do not account for diversity of device characteristics within their class

Architectural EnhancementsArchitectural Enhancements

Multi-device Integrated Dynamic Activity SpacesMulti-device Integrated Dynamic Activity Spaces

Trellis and caT

ArchitectureArchitecture

Users

Hypertext Authoring

Tool

Hypertext Specification

Information Service

Device 3Device 2

Device 1

Browser 4Browser 3

Browser 2

AuthorsThe Web

Browser 1

context-aware Trelliscontext-aware Trellis• Petri net-based hypertext systemPetri net-based hypertext system

• Stateful serverStateful server– Maintains state (unlike HTTP servers)– Users may browse simultaneously from

• multiple browsers• More than one computer

– Browsers mirror the state of browsing– State is maintained in the absence of any clients

• Type-specific browsers support variousType-specific browsers support various– Media types (text, images, audio, video)– Browsing metaphors (combine media types – Web

browser, spatial browser)

context-aware Trelliscontext-aware Trellis

• Browsing experience depends onBrowsing experience depends on– User identity (privileges)– User environment– Actions of other users

John

Desktop

Browser A Browser B

Notebook 1

Browser C

Bob

Notebook 2

Browser D Browser E

caT Server(hypertext specification, state)

context-aware Trelliscontext-aware Trellis

• caT servercaT server– Separates structure from information content – Directs browsers to render this content

• BrowsersBrowsers– Render the content to the best of their abilities

• Image browser will not display textual information at all

– Accept user actions and forward these to the server

• Link activation

Enhancements to caTEnhancements to caT

• Extend caT’s browser-baseExtend caT’s browser-base

• Architectural enhancementsArchitectural enhancements– MIDAS

Browser multiplicityBrowser multiplicity

• Earlier caT’s browsersEarlier caT’s browsers– Text, image, and Web browsers

• New browsers supportNew browsers support– Additional media types (audio-video browser)– Different presentation metaphors (spatial

browser)