responsive media

16
Responsive Media Bo Begole James Glasnapp Strategic review March 2009 parc confidential

Upload: bo-begole

Post on 18-Dec-2014

780 views

Category:

Technology


1 download

DESCRIPTION

Presentation of concepts and research around the idea of Responsive Media presented at the 2009 Workshop on Pervasive Advertising in Nara Japan.

TRANSCRIPT

Page 1: Responsive Media

Responsive Media

Bo BegoleJames Glasnapp

Strategic review March 2009 parc confidential

Page 2: Responsive Media

Mixed-Initiative Interaction

Conventional systems: User initiates interaction and commands the system

Mixed-initiative: System sometimes initiates interaction with the user– You have mail.– Can I help you find that?– Here is something useful to you.

ResponsiveMirror

Responsive Technologies InformationRecommendation

ClothingRecognition

Psychographic Profiling

[ICDSC 2008][IUI 2008, HCII 2009][CHI 2008]Magitti

Related PARC Research:also:Human-Robot Interaction

Multi-party conversations

Camera-based tracking

MIT Media Lab

Microsoft Research

Page 3: Responsive Media

3

Business Marketplace

In-store signage– Traditional: Point-of-Purchase displays, shelf positioning,

packaging, store-handouts (coupons), specials (e.g., Kmart blue light), aisle coupons, loyalty programs (lower price)

– Emerging: digital kiosks, digital signage, directed audio Companies

– NewsAmerica leases store space and sells ad spaces to consumer packaged goods

Search Engine Marketing $13B to $26B in 2013

Advertisers pay more for personalization

Reactrix charged higher rates than static digital signage.Reactrix is just the tip of the iceberg.

Page 4: Responsive Media

4

Today we are just at the tip of the iceberg in

conversational

interaction

Today we are just at the tip of the iceberg in

conversational

interaction

Voice Systems

Robots

Media

Avatars

In the future we

will interact with all types of

technology as if they were social entities

In the future we

will interact with all types of

technology as if they were social entities

Service Agents

Marketing

Sales

Education

Therapy

Performance Coaching

Page 5: Responsive Media

In-Store Product Recommendations

PersonalProfile

What productIs she looking

at now?

Eye-contactsensors

TrackingSensors

Previouspurchases

Is she searchingor just browsing?

FloorSensors

Is this a groupor individual?

MotionSensors

Is sherushing ina hurry?

Items: x, y, z, ….

A blue blazer

browsing

Data TypeSensor

PerceptionPersonalized

Recommendation

Group

Question

DisplayImpulse-buy

itemsRushing

Similar items

Matchingskirts in the

store

Highlight gift items

Highlight new trendy fashions

Inferred User Goal

Interest Profile:Style, colors, price

range, etc.

Looking forBusiness clothes

Shopping for gifts

Wants to show“Fashion sense”

Needs to decidequickly

Web Shopping

today

Responsive Personalized Sales Promotions

Page 6: Responsive Media

Existing Research: Many indicators of a person’s engagement with media

Component technologies exist, but not integrated, not directed by behavioral models:- What are sequential structures of engaged interactions?- Which indicators are most predictive of engagement?- Can we predict disengagement before it happens?

[Vogel & Balakrishnan ‘04]

[Grauman et al. ‘01]

[Cohen et al. ‘03]

[Yu, Aoki, Woodruff, PARC ‘04]

vocal affect

proximity, orientation of head & body

[Daugman ‘94]

pupil dilation

eye blinksfacial affect

skin temperature

[Haro, Flickner, Essa 2000]

eye gaze

Page 7: Responsive Media

Engineering approach (Reactrix) currently achieves Phase 1 using disruptive techniques Phase 4 is the real value – requires recognizing human micro-behaviors Conversation and interaction analysis bring clarity to vague notions like “engagement”

– Detect, describe and model the structured organization of natural interaction– Create systems that interact and respond to individuals

Responsive and Personalized Public Information Display

[HCII 2009]

Interaction Structure of a Marketing Engagement1

– Approach (hook)– Assess– Relax– Describe– Benefit– How to buy– Reduce resistance– Incentive to act

Monitor &Re-engageas needed

[1Robert Prus, Making Sales]

Attract and maintain audience engagement Content follows interaction model toward

an objective: Marketing, Entertainment, Education, …

Page 8: Responsive Media

8

Improving social capability and interactive personalization

Making systems socially interactive Conversation analysis (CA) can build a more personalized, smooth interaction between technological

systems and humans

Interaction Analysis provides Technology designed using frameworks inspired by conversational structures

Previous research: Sotto Voce, Responsive Mirror, Human-Robot Interaction

Broad Applications of Conversational Responsiveness Any field with interactive features with customers: call centers and interactive voice responses to

improve voice interactions; games – making characters more interactive; mobile phone manufactures can make more use of conversational data (i.e., providing analysis of conversations to provide feedback); and automobile - design better audio-based interfaces

Linking research in human behavior to technology

design.

Page 9: Responsive Media

Sales Interaction ModelRepresenting Elements of Sellers’ Goals

Representation of dependencies and degree to which each sales goal has been achieved

[adapted from Making Sales, Robert Prus]

Engage Assess

Offer Service

Present Products

Generate Trust

Show Customer Need

Neutralize Reservations

ObtainCommitment

Maintain Trust

engaged engaged

low

engaged engaged

Not engaged

Maximize Trust

Appears uninterested

Page 10: Responsive Media

Psychographic Profiling through Clothes Recognition

Mens shirts: multiple features– Collar vs. crew neck– Short vs. long sleeve– Color, texture– Pattern, emblems

[Zhang, et al. IUI 2008]

What you wear says more about your tastes than demographics

Page 11: Responsive Media

Similarity: Example shirt matches

Page 12: Responsive Media

Shirt style classification Classes

SVM results

Class Collar Sleeve Button

T-shirt No Short No

Polo shirt Yes Short Half

Casual shirt Collar Short Full

Business shirt Collar Long Full

Classified as ->

T-shirt Polo Casual Business

T-shirt 80.8% 3.9% 15.4% 0%

Polo 16.7% 41.7% 8.3% 33.3%

Casual 0% 12.5% 50% 37.5%

Business 0% 5% 5% 90%

Overall accuracy: 72.7%

Sellers would approach someone wearing a T-shirt differently than someone wearing a Business shirt

Page 13: Responsive Media

Research Opportunities

Decision Engine & Objective Model• Select best abstract

response toward objective

Perception• Detect external cues that

indicate internal mental state

Composable Content• Content organized according to

abstract actions

Computer Vision• Robust algorithms to detect

specific behaviors• Measures of inaccuracy• Other Sensors

• Audio, thermal, pupil, etc.

Multimedia Data Structures• Efficient data structures for

realtime program re-composition

Ethnography• Internal user mental states• External behavioral cues• Abstract actions toward objectiveInteraction Engine• Develop realtime decision engine

Page 14: Responsive Media

Interdependencies Between Perception, Decision and Action Components

Decision Engine & Objective Model• Select best abstract

response toward objective

Perception• Detect external cues that

indicate internal mental state

Composable Content• Content organized according to

abstract actions

Computer Vision• Robust algorithms to

detect specific behaviors

• Measures of inaccuracy

Multimedia Data Structures• Efficient data

structures for realtime program composition

Ethnography• Identify user mental states• Identify external cues of

mental state• Identify abstract actions

leading to an objective

1. Decision engine and objective model depend on reliability of computer vision techniques.

2. Required computer vision depends on needs of decision engine and object model.

1. Structure of composable content framework depends on output of decision engine and object model.

2. Output of decision engine and object model should allow for realtime composition of content.

Page 15: Responsive Media

Responsive Interaction PlatformSensing of Environment

Image/VideoAnalyzer

AudioAnalyzer

Perception of Environment

Person ModelPerson Model

Person ModelModel of internal state

eye gazehand/body gesturesfacial expression

non-vocal soundsspeech

Emotional stateEnergy levelPatienceMental activity – thinking, confusionInterest levelAttitude toward informationHome position…

Interaction among peoplePositions and postures…

Decision EngineSelect “best” abstract action based on abstract state of environment and the objective. Use the framework of Partially Observable Markov Decision Processes (POMDP).

state of environment

Objective ModelThis is the objective function in the POMDP framework that defines what the “best” action is. Example Objectives: Increase brand awareness, Introduce new product, Direct sales to mobile device, Provide navigation information, …

Content Actuation EngineConvert abstract action to content segments.

Abstract action

Display Sound Ambient Motion, Lights

Promote Interest

Fast Animation Catchy music Movement, light flash

Gain trust Scenes of family life with product

Smooth music Non-distracting

… … … …

abstract action – e.g., Promote Interest, Gain Trust, Present

Product, make joke, …

actuator control

Sensor Analyzer

Interaction Engine

sensor features

Content Actuation

Interaction ModelThis is the sequencing structure in the POMDP framework that defines what stages the interaction should follow. E.g., Sales*:

• Approach• Assess• Relax• Pitch• Benefit• Reduce Resistance• Incentive to act

Interaction stages

Objective metrics

Page 16: Responsive Media

Summary

Mixed-Initiative Interaction generates new business opportunities Mixed-Initiative Interaction Engine

– Inference models to measure audience engagement» Identify the most predictive set of sensors and the cost tradeoffs

– Precise assessment metrics of content effectiveness– Engagement Detection

» Convert raw data to human-meaningful cues of engagement Dynamic content framework

– Maps abstract actions to content segments to achieve the objective– Tailorable to structure of engagements across multiple target domains

» Education, Training, Service, Sales, etc.

Far-reaching research and invention of next-generation interaction paradigm for media technologies– Displays, mobile device, speech conversation, etc.