personalized interface agents for virtual webpages

© W. Wahlster, DFKI

Third International Conference on Autonomous AgentsAgents 99

Seattle, May 3. 1999

German Research Center for Artificial Intelligence, DFKI GmbH

Stuhlsatzenhausweg 366123 Saarbruecken, Germany

phone: (+49 681) 302-5252/4162fax: (+49 681) 302-5341e-mail: [email protected]

WWW:http://www.dfki.de/~wahlster

Wolfgang Wahlster

Personalized Interface Agentsfor Virtual Webpages


What is a Virtual Webpage?

What are Personalized Interface Agents?

Using Life-Like Characters for the Personalization of User Interfaces

The Combination of Retrieved and Generated Media Objects for the Generation of Virtual Webpages

PET: A Persona-Enabling Toolkit

Commercial Applications in Advanced WebCommerce

Psychological Effects of Personas

Our Research Agenda for Agent-based User Interfaces

Multiple Interface Agents

Conclusion

Outline


First Generation Second Generation Third Generation

Static Web Sites

Fossils cast in HTML

Interactive Web Sites

JavaScripts and Applets

Database Access andTemplate-based Generation

Dynamic Web Sites

Virtual Web Sites

Netbots,Information Extraction,Presentation Planners

Adaptive Web Sites

User Modeling,Machine Learning,

Online Layout

Three Generations of Web Sites


A Virtual Web Page

is generated on the fly as a combination of various media objects frommultiple web sites or as a transformation of a real web page.

looks like a real web page, but is not persistently stored.

integrates generated and retrieved material in a coordinated way.

can be tailored to a particular user profile and adapted to a particularinteraction context.

has an underlying representation of the presentation context so that an Interface Agent can comment, point to and explain its components.

Virtual Memory, Virtual Relation, Virtual Reality...

What is a Virtual Web Page?


Hotel Guide

Yahoo News Server

Yahoo Weather Server

PAN Travel Agent Andi Car Route Planner

Gault Millau Restaurant

Guide

AiA: Information Integration for Virtual Webpages


Virtual Webpage Retrieved from 5 Different Servers


Virtual Webpage Augmented by Persona


respond immediatelyto interruptions

criticism and clarification questions

direct manipulation

INTERFACEAGENTS

RE-ACTIVE

ACTIVE

appear as life-like characters plan interactive behavior

autonomously can initiate interaction

PROACTIVE

anticipate the user's needs adopt the user's goals provide unsolicited comments

INTER-ACTIVE

understand the user‘s requests

answer clarification questions

allow mixed initiativedialogs

What are Agent-based User Interfaces?


Consumer Provider

sells InformationGoodsServices

buys InformationGoodsServices

Web Sites

Knowledge about: Usage PatternsUser ModelsConsumer Profiles

Netbot

Intelligent Parallel Retrieval Information Extraction and

SummarizationPersonalized Presentation Matchmaking Teleshopping Assistance Telemarketing Assistance Translation Services Data Mining Services

Intelligent Web Services


System is able to flexibly tailor presentations to the individual user and the current situation.

Enhancement of User Interfaces through Personalization

An animated characterserves as “Alter Ego”

of the presentation system.

Personalized Presenters at DFKI


Persona as a Personal Travel Consultant


The Personal Picture Finder


Portrait Photo

Netbot: Personal Picture Finder

Parallel Meta-Search of Webpages for<Name>

<Name>

Parallel Searchin Picture ArchivesHome Pages

Extraction of Images

Filtering ofLogos, Graphics,...

WWW Knowledge Sources

Applications: Journalism, Contact Preparation, Tracing Criminals

http://finder.dfki.de:7000

A Netbot for Portrait Photos: The Personal Picture Finder


Personalization = adaption of system behavior

according to a user model

Personalization =the “agent” appears as

animated character which presents, explains and

comments an offer, and to which the user can talk

to.

Personalized Package Presentation by an Animated Agent


Video Character Presents the Interior of a Boeing 757


Reactive Behavior of the Persona Agent


PersonaServer

Behaviors

Presentation GesturesReactive BehaviorsIdle-time actionsNavigation actions

Auditory Characteristics

Sound effects, auditory iconsVoice: male, female

Visual Appearances

Hand-drawnCartoonBitmaps

GeneratedBitmaps from 3D-Models

Video Bitmaps

PPP’s Persona Server implements a generic Presentation Agent that can be easily adapted to various applications


More than 200 cartoon frames were drawn by a professional artist.

A real persona was filmed with a video-camera and the pictures weredigitized with a frame grabber.

The frames of the visual appearance of persona can be cartoon-like images or video-frames which show real persons


The Persona Editor


take-position (t1 t2 ) point-to (t3 t4 )

move-to (t1 t2 ) r-stick-pointing (t3 t4 )

High-LevelPersona Actions

Context-SensitiveExpansion(including Navigation Actions)

Decomposition into Uninterruptable Basic Postures

r-turn (t1 t21 )

r-step (t21 t22 )

f-turn (t22 t2 )

r-hand-lift (t3 t31)

r-stick-expose (t31 t4 )

Bitmaps... ... ... ...

Context-Sensitive Decomposition of Persona Actions


MultimediaGeneration

TextGenerator

GraphicsGenerator

GestureGenerator

AnimationGenerator

PresentationPlanner

(PREPLAN)

KR & Reasoning

(RAT)

ProductionActs

GeneratedMaterial

PresentationActs

Signals +Events

MultimediaServer

LayoutManager

PersonaServer

MusicGenerator

PPP System Architecture


Production ActPresentation Act

Introduce

Create-Graphics

S-Show

S-Wait

S-Position Elaborate-Parts

S-Create-Window

S-Depict

Label Label

S-Point S-Speak S-SpeakS-Point

Qualitative constraints: Create-Graphics meets S-Show, ...

Metric constraints: 1 <= Duration S-Wait <= 1, ...

Distinction between production and presentation acts

(i.e. Persona- or display acts)

Explicit representation of qualitative and quantitative constraints

Extensions of the Representation Formalism


(define-plan-operator:HEADER (A0 (INTRODUCE P A ?object ?window)):INFERIORS ((A1 (CREATE-GRAPHICS P A ?window ?object))

(A2 (S-SHOW P A ?window ?object))(A3 (S-POSITION P A ?window))(A4 (S-WAIT P A))(A5 (ELABORATE-PARTS P A ?object ?window)))

:QUALITATIVE((A1 (m) A2) (A3 (s) A2) (A3 (m) A5) (A5 (m) A4) (A4 (f) A2)):METRIC((10 <= DUR A2) (2 <= DUR A4 <= 2)):START A1:FINISH A2)

(cf. [André/Rist 97])

Presentation Strategies in PPP contain qualitative and metric constraints


Multi-DomainProblem Specs

NETBOT

RetrievedResults

Information Structures Relations, Lists KR Terms

Media Objects Texts, Sounds, Videos Pictures, Maps,

Animations

DistributedInformation

MultipleData Sources

The Combination of Retrieved and Generated Media Objects for Virtual Webpages


RetrievedResults

Select & Design

Select CannedMedia Objects

Design NewMedia Objects

Information Structures Relations, Lists KR Terms

Graphics,Animation

Text, Speech,Mimic

Icons, Clip Art

Frames, Sounds

Reuse & Transform

CoordinateMedia Objects

TransformMedia Objects

Temporal Synchroni-zation

SpatialLayout

Clip, Convert,Abstract

Zoom, Pan,TransitionEffects

Media Objects Texts, Sounds, Videos Pictures, Maps,

Animations

The Combination of Retrieved and Generated Media Objects for Virtual Webpages


Persona Presents an Automatically DesignedBusiness Chart


Virtual Webpage with Animation Effects Based on a Single GIF Image


Transition Effects in a Series of Retrieved Pictures


NetbotPAN

Trip Data

Pictures andGraphics

Pieces of Text

Coordinates forPointing Gestures

Input for SpeechSynthesis

Icons forHyperlinks

HotelAgent

MapAgent

Address

WeatherAgent

Train & FlightScheduling

Agent

Major EventAgent

VirtualWeb

Presentation

Constraint-basedOnlineLayout

PresentationPlanner

PersonaServer

Componentsof virtual

Webpages

AiA

The Generation of Virtual Webpages with PAN and AiA


Objective:

Enable non-professional computer users to populate their web pages with lifelike characters

PET comes with:

a set of characters and basic gestures

an easy-to-learn Persona markup language

Developer’s PET will include:

a character design tool which enables users to build their own characters

Technical Realization:

Based on standard XML, JavaScript, Java and DHTML

PET: Persona-Enabling Toolkit


Specification of the character to be used

Specification of Persona actions

<html><head><title> Persona Test </title></head><body><persona bitmap=“cartoon” ...><uselib url= .../><do name=“greet”/><do name =“speak” args=“hello”/></persona></body></html>

Features:– XML-based

– easy to learn

The Persona Markup Language


URL of Webpage with Persona Tag

Persona Engine

Behavior Monitor

Character Composer

Event Handler

<html><head><title> Persona Test </title></head><body><persona bitmap=“cartoon” ...><uselib url= .../><do name=“greet”/><do name=“standard”/><do name =“speak” args=“hello”/></persona></body></html>

Persona Scriptswaitscreen 4gesture greet 0 0 nullgesture laugh 0 0 null...

Audio Data

Bitmaps

PET Application Server

PET Parser

PET Generator

Webpage with Reference to Java Applet

<html> ...<APPLET archive=“personaplayer.jar”...</APPLET>...</html>

Functional View of PET


PET-ParserPreprocessing (required for non-standard HTML pages) Identification of Persona Tags

PET-Generator

Replacement of Persona Tag with Java Applet

Determination of parameters for Java Applet

Realization of Persona active elements as JavaScript

Generation of DHTML layers (transparency of Applet box)

Generation of data, e.g. audio files for speech output

Generation of scripts

Tasks of PET


Text Input

Speech Input

Menu Input

Direct Manipulation

Input

Web Persona

Triggers

actions of the Persona

Triggers

operations on elements

of the webpage

Mouse Clicks

Mouse Movements

The Bidirectional Control Flow onPersona-Enabled Webpages


<PDIV DIVNAME=“name of the element” DVFRAME=“frame name” TOP=“anchor-y” LEFT=“anchor-x”>some HTML elements</PDIV>

<ACTIVEIMAGE SRC=“image” HREF=“url” NAME=“image name” STATUS=“status message” ALT=“tooltip” CACTION=“persona action onClick” OACTION=“persona action on MouseOver” ...>

Active Images An active image starts a persona action when clicked.

Addressable ObjectsAn addressable object is an object which can be addressed and manipulated by Persona via its name and its position.

Persona Active Elements (PAE)


Use of a Life-like Character for Electronic Commerce


0.5

0.6

0.7

0.8

1.0

Experimental evidence for effects of modality on the user's trust (van Mulken, 1999)

The system gives recommendations, which turn out to be wrong in some cases.How much does a user trust the system's advice depending on the modality of a presentation?

Self-animated Persona,Speech, Gesture, Facial

Expression, Pointing

Speech,Graphical

Highlighting

Text,Graphical

Highlighting

Personified Agents Increase the User's Trust in the System's Presentation


Result: Persona > Speech > Text

Conclusion: If the presentation is more human-like, recommendations are more readily followed

For decision support systems tutoring systems

recommendation systems virtual sales agents

personified interface agents have a clear advantage:They increase the user's trust in the information presented by the system

Impact of the modality of a Presentation on the User's Trustfulness


Plug-Ins Applets

Add features (character players) to browser Download triggered by user Requires disk space on client Unrestricted access to client Less appropriate for WebCommerce,

Guides Agents integrated in 3D environments Appropriate for Entertainment

Examples:

Extempo's Jennifer James (Hayes-Roth et al. 98)

PFMagic's virtual petz

Java animation code sent over the net Automatic loading Requires no disk space on client Restricted access to client Appropriate for WebCommerce, Guides Agents integrated in 2D environments Less appropriate for Entertainment

Examples:

DFKI's Web Persona(Müller et al. 98)

ISI's Adele (Johnson et al 98)

New in AiA/PAN: Balanced combination of Applets and ServeletsEfficient distribution of client-side Java and server-side Java for driving theInterface Agent

Sending Interface Agents to Clients: Plug-Ins or Applets?


MediaInterfaceEuropean Media LabUinv. Of

MunichUniv. ofStuttgart

Saarbrücken

Aachen

Dresden Berkeley

Stuttgart

MunichUniv. ofErlangen

Heidelberg

Main ContractorProject Management

TestbedSoftware Integration

DFKISaarbrücken

The SmartKom Consortium:

Project Budget: $ 34 MProject Duration: 4 years

DAIMLERCHRYSLERUlm

SmartKom: Intuitive Multimodal Interaction


Smartcard/ Credit Cardfor authentication and billing

Docking stationfor PDA/Notebook/Camcorderhigh speed and broadbandwidth Internet connectivity

High-resolution scanner

Loudspeaker

Room microphone

Face-tracking camera

Virtual touchscreenprotected against vandalism

Multipoint video conferencing

SmartKom-Public:A Multimodal Communication Booth


MOBILE

Camera

GPS

Microphone

Loudspeaker

Stylus-Activated Sketch Pad

WearableComputeServer

Docking Stationfor Car PC

Biosensorfor Authentication

& Emotional Feedback

GSM for Telephone,Fax, Internet Connectivity

SmartKom-Mobile: A Handheld Communication Assistant


SpeechMikeVirtual Touchscreen

Natural Gesture Recognition

SmartKom-Home/Office: A Versatile Agent-based Interface


Personalized Presentation Agents (limited user interaction)

WIP: 1989-1993

1

Personalized Interface Agents (full user interaction)

2PPP:1994-1996

Multiple Interface Agents (agent-agent and user interaction)3

Multiple Presentation Agents in one scene (eg. pros and cons)

Multiple Role-Taking (eg. Travel Assistance vs. Comparison Shopper)

Multiple Interface Agents (eg. human-computer, human-human interaction)

AiA: 1997-2000

Our Research Agenda for Agent-based Interfaces(Wahlster, André, Rist, Müller, Graf etc; www.dfki.de/imedia)


Use of a Life-like Character for Electronic Commerce

Digital Assistant Selector


But, it’s fast!

Multiple Decision Support Agents

... ...weighted propositions

User-Adaptive Search Planning

Spare parts for this car are rather

expensive!

Current and Future Work: Multiple Interface Agents for User-adaptive Decision Support


Multiple Presentation Agents Convey different attitudes, beliefs and biases (e.g. proponent vs.

opponent)

Generate argumentatives dialogs among a group of agents

Multiple Interface Agents human and virtual agents in collaborative browsing environments

different dialog roles (e.g. moderator, advisor, sales agent)

Multiple Role-Taking of One Interface Agent e.g. travel assistant vs. comparison shopper

(use of a different outfit of the character to signal a role change)

Multiple Interface Agents


Multi-User Interaction with Multiple interface

Agents for Ecommerce on a Single WebTV


Advanced WebCommerce

Advanced WebCommerce

Virtual WebPages

One-to-OneMarketing

One-to-OneMarketing

Intuitive,Multilingual

Access

Dialogue withVirtual Sales

Agents

Shopbots forAutomated

ComparisonShopping

Text Analysis and Generation

User Modeling and Language GenerationCoordinated Text & Graphics Planning

Robust Dialogue Understanding

Advanced SpeechSynthesis

Information Extraction from HTML/XML Documents

MachineTranslation

MultimodalInterfaces

MultimediaPresentation

Planning

Intelligent Interface Technology is a Prerequisitefor Advanced WebCommerce


IntelligentWeb

Services

Personalized Interface Agents

UserModeling

Planning

Natural LanguageUnderstanding

KnowledgeRepresentation

ImageUnderstanding

MachineLearning

PlanRecognition

InformationRetrievalMultimodal

UserInterfaces

Research on Personalized Interface Agents brings disparate subfields in the area of intelligent systems

together


ECommerce projects of DFKI have shown that research on personalizedinterface agents can be transferred to real world applications:

Dekra (largest European organization of used car dealers):FairCar as an ECommerce platform with NL access and a comparison shopping agent for used cars

DaimlerChrysler: IKP for online user modelling in a one-to-one marketing system for Mercedes cars

Otto/Shopping24/Eddie Bauer (largest European mail order company):virtual sales agents for one-to-one marketing of fashion and computer hardware

Lufthansa/Condor: direct marketing of charter flights

Conclusion


The generation of virtual webpages with personalized interface agents leads to innovative applications in:

Electronic Commerce, Electronic TV Guides (EPG)Telelearning environments, Call Centers and Help Desks

Two Research Challenges:

Making the Interface agents sensitive to temporary limitations of the user‘s time and working memory capacity (cf. our READY project, Jameson et al., p. 79-85 in IUI99 Proceedings)

Making the agents instructible, so that they can learn from the user in a dialog, how to extract information in a new domain (cf. Mathias Bauer, Dietmar Dengler TrIAs: Trainable Information Assistants for Cooperative Problem Solving in Agents'99, on Tuesday)

Conclusion


Computer Science:

Elisabeth AndréMathias BauerDietmar DenglerAlexander Kröner Marcus MeyerJochen MüllerStephan NeurohrGaby PaulThomas RistWolfgang Wahlster

Graphics Design:

Bernhard KirschRenato OrsiniPeter Rist

Cognitive Psychology:

Susanne van Mulken

The WebPersona and Virtual WebpagesTeam at DFKI