activity-based serendipitous recommendations with the magitti mobile leisure guide

System Codename: MagittiDesigned and Prototyped by PARC forDai Nippon Printing Co. Ltd.

Presenters• Victoria Bellotti• Bo Begole• Ellen Isaacs

The Other Co-authorsEd H. Chi, Nicolas Ducheneaut, Ji Fang,Tracy King, Mark W. Newman, Kurt Partridge, Bob Price, Paul Rasmussen, Michael Roberts, Diane J. Schiano, Alan Walendowski

Activity-Based Serendipitous Recommendations Activity-Based Serendipitous Recommendations with the Magitti Mobile Leisure Guidewith the Magitti Mobile Leisure Guide

2

Overview

•Background and motivating fieldwork

•System design

•Evaluation

Recommendation Server

ConsumerLocal Area

Context: Time, Location, etc. Restaurants, stores,

events, etc.

Mobile Device

Preferences:Sushi, Bookstores,

etc.

Filter and Rank Database Items

Infer Activity

FeedbackFeedback

Model Preferences

3

About Dai Nippon Printing Co. Ltd.

• DNP is a world leader in printing technology and solutions

• Affected by the shift from paper to digital media

The Past: People carried magazines The Present: Most Japanese use a mobile

phone to browse the Web and read/write E-mail

• DNP asked PARC to develop core technology for new, consumer-friendly digital media

• All design to be driven by real need motivated a lot of work to identify:

• Best target users• Best solution for their needs

Traditional Publishing

Modern Publishing

4

Fieldwork 2 Confirm

and Refine

Fieldwork 2 Confirm

and RefineEvaluate design mock-up in situ

Refine design based on user feedback

Fieldwork 1 Choose

Best Idea

Fieldwork 1 Choose

Best IdeaInterviews,

observations, and scenario

feedback

Analyze results

Refine concept design

Future technology

analysis

Finalized Concept Proposal

Finalized Concept Proposal

Leisure guide concept

proposal,“Magitti”

Contextual Publishing Concept Development

Technology BrainstormTechnology Brainstorm

Personas bring customer to life

Share background domain info

Brainstorm design ideas

Discover Target Users

Discover Target Users

Assess many markets

Develop scenarios and obtain feedback

Choose the best

Young Adults at Leisure

Activity-Aware Leisure Guide

What to Build

5

Many User Studies During Concept Development and Early System Development

Observation Focus Groups

Activity Sampling

In-depth interviews

Mobile-phone Diaries

NotesDiary entriesLocation

Survey responses

1000’s of Photos 40 Transcripts 10 Transcripts

Time Time

Fashion

Identity

Technology use

Transportation

Leisure activity type frequency

Leisure activity venue types popularity

Leisure activity type timing & probability

3000 activity & time reports

370 activity, time & location reports

Planning

Media use

Information sources

Information desired

Social factors in leisure

Knowledge of locale

Observation reminders Practices Needs Priorities

Surveys

670 Responses

Problems

Classifying CountingCodingCorrelating

Leisure activity type locations

Coordination

Activity type prediction

Leisure activity types

Form-factorFeatures

FunctionsInteraction style

Venue database classification

Content

Study MethodsStudy Methods

Informing Design ofInforming Design of

DataData

Analysis: Analysis: increasing abstractionincreasing abstraction

6

From Fieldwork: Who Are the Users

• Japanese youth are especially receptive to new technology

• 19-25 year-olds spend 1.5 times more time in leisure activities than 16-19 year-olds or 26-33 year-olds

• Less school and work pressure• Ideal target for our design

• Still very, very busy• School, jobs and little sleep• Relaxation is a priority• The system should do the work

• Want to know what others think• Value opinions of real people• Include end-user content

7

From Fieldwork: What do they Do?

• Outings often involve meeting friends• Often at “halfway point” far from

homes

• Eager for local and localized info• Unfamiliar with locations they visit

• Open to suggestions• May not plan the main activity• May not plan follow-on activities

• Motivation for Magitti• A city-guide that assists in

exploration

0

10

20

30

40

50

60

1 2 3 4 5 6 7

1 = Not at All 7 = Extremely Well

Ratings of “How well I know this neighborhood”

given by 170 young people stopped on the streets in diverse neighborhoods in Tokyo

8

Overview


•System design

•Evaluation


ConsumerLocal Area


events, etc.

Mobile Device


etc.


Infer Activity

Model Preferences

9

User Interface

Map

Pie Menu Details

10

Demo Videohttp://www2.parc.com/csl/groups/ubicomp/videos/magitti_project_demonstration.wmv

http://www2.parc.com/csl/groups/ubicomp/videos/magitti_project_demonstration.wmv

Recommendable Items

Restaurant ReviewsStore DescriptionsParks Descriptions

Movie ListingsMuseum Events

Magazine Articles…

Recommendable Items




EAT Straits Cafe 0.77

EAT Fuki Sushi 0.64

SEE J. Gallery 0.60

EAT Tamarine 0.57

DO Sam’s Salsa 0.39

EAT Bistro Elan 0.38

BUY Apple Store 0.33

EAT Spalti 0.31

Filteringand

Ranking

Filteringand

Ranking

Activity UtilityInformation

Recommendable Items




Recommendable Items





EAT Fuki Sushi 0.64

SEE J. Gallery 0.60

EAT Tamarine 0.57




EAT Spalti 0.31

Filteringand

Ranking

Filteringand

Ranking


ContextContext• Time Time • LocationLocation• Email analysisEmail analysis• Calendar analysisCalendar analysis

HistoryHistory• Prior population Prior population

patternspatterns• User QueriesUser Queries• User Locations User Locations

Eat 35%Buy 20%See 25%Do 10%Read 10%

What you are doing now

Recommendable Items




Recommendable Items





EAT Fuki Sushi 0.64

SEE J. Gallery 0.60

EAT Tamarine 0.57




EAT Spalti 0.31

Filteringand

Ranking

Filteringand

Ranking


What you likeWhat

you like

Personal PreferencesPersonal Preferences• Explicit preferencesExplicit preferences• Ratings of placesRatings of places• Topics of documents readTopics of documents read• Behavior; where/when/whatBehavior; where/when/what

ContextContext• Time Time • LocationLocation• Email analysisEmail analysis• Calendar analysisCalendar analysis

HistoryHistory• Prior population Prior population

patternspatterns• User QueriesUser Queries• User Locations User Locations

Eat 35%Buy 20%See 25%Do 10%Read 10%

What you are doing now

15

Predicting Activities from Population Priors

Mobile-phone Diaries

Hourly activity report:• Who• Where• When• What• Info used & desired

Code each respondent’s activities over 7-day week

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Time of Day

0

5

10

15

20

Sa

mp

le C

ou

nt

(To

tal)

NOTSEEDOEAT OUTSHOP

Friday

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Time of Day

0

5

10

15

20

Sa

mp

le C

ou

nt

(To

tal)

NOTSEEDOEAT OUTSHOP

Sunday

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Time of Day

0

5

10

15

20

25

Sa

mp

le C

ou

nt

(To

tal)

NOTSEEDOEAT OUTSHOP

Saturday

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Time of Day

0

10

20

30

40

50

60

70

80

Sa

mp

le C

ou

nt

(To

tal)

NOTSEEDOEAT OUTSHOP

Mon-Thu

Predict probability of each activity type

Aggregate all dataWhen there is no user-specific When there is no user-specific

data, prior population data is useddata, prior population data is used

16

Predicting Activities from Email/SMS• How well do messages suggest activity?

• We examined a public set of 10,000 SMS messages from National University of Singapore students, similar to the Magitti target demographic

• Approximately 11% of the messages contain information related to leisure activities

tomorrow what time you be in school? think me and shuhui meeting in school around 4. then duno still can see movie or not because duno if a rest want meet for dinner.

• Keywords and linguistic structures are identified and sent to the activity inference mechanism

ACTCAT=SEE, EAT :: ACTTIME=2007/05/26 16:00 :: UNCERTAINTY=10 minutes :: TENSE=FUTURE

17

Learning Individual Patterns

Date/Time Location Address Venue Name

Venue Type

Activity Class

Sun, 27 Jan 200811:57- 12:45

37°26’39”-122°9’38”

389 Ramona Evvia Restaurant EAT

Tue, 29 Jan 20081:22 - 1:31

37°23’11”-122°9’02”

545 Hamilton,

Brickworks Cafe EAT

Wed, 30 Jan 200811:57- 12:45

37°26’39”-122°9’18”

143 Quarry Road

Walgreens Store SHOP

Fri, 1 Feb 200813:11 - 13:37

37°24’11”-122°9’00”

854 University

Restoration Hardware

Store SHOP

… … … … …

0 2 4 6 8 10 12 14 16 18 20 22 24

Shopping Center

0 2 4 6 8 10 12 14 16 18 20 22 24

Downtown

EAT MostLikely

SHOP MostLikely

Undetermined

Time Individualized

pattern by region

18

Activity Inference Evaluation

Magitti Accuracy on Palo Alto Field Evaluation Data

62%

77%82%

0%

20%

40%

60%

80%

100%

Baseline (EAT) Time and Place Priors Priors + Learning

* Time and Place Priors is significantly different than Baseline (Chi Square p=0.014, McNemar p=0.048).† Priors + Learning is significantly different than Baseline (Chi Square p=0.0027, McNemar p=0.008).

* †

19

Overview


•System design

•Evaluation


ConsumerLocal Area


events, etc.

Mobile Device


etc.


Infer Activity

Model Preferences

20

Preliminary Field Evaluation

• 11 people, 32 outings (2.9 per person)• Shadowed one outing per participant

• 60 places visited (1.9 per outing)• 30 restaurants, 27 shops, 3 parks

• 16 outings accompanied by companion(s)

Using Magitti in a demo

21

Overall Usefulness

• Usefulness• Average of 35.0 recommendation list pages viewed per outing• People rated “helpfulness” 4.1 on 5-point scale (5 high)

• "Cool! I like that. I would never have found that place if it wasn't for this.”

• "It makes life more interesting. It allows you to get out of your daily routine, almost as if you’re going to a different city.”

• Serendipitous Discovery• 53% of places visited were new to the participants• On 67% of outings they went to at least one new place • On 69% of outings, they noticed another new place to visit later

22

User Response

• Predicting User Activity• People changed activity 5.1 times per outing • “It’s very nice that it recommends things without you

having to do anything, but sometimes you want to ask for specific things.”

• Even when Magitti got it right, they still sometimes switched, apparently because they wanted all the recommendations to be for that activity

• Social Use• Five of eight users reported difficulty in sharing

experience with another person• Magitti user seen as disconnected from others and/or

controlling the outing

23

Quality of Recommendations

• Recommendations rated 3.8 on 1-5 scale of "relevant and of interest“

• "Most of the time, the list contained a mix of useful and not so useful recommendations“

• Biggest factors to reduce confidence in recommendations

• Not seeing a nearby place in the list• Getting recommendations for places too far away• Lack of transparency of reasons for recommendations

24

• Information and suggestions based on

• Situation• Past behavior• Personal preferences

Replace Tedious Mobile Searching with Personalized Recommendations

Stop searching!

Let information find you!

Victoria Bellotti, Bo Begole, Ed H. Chi, Nicolas Ducheneaut, Ji Fang, Ellen Isaacs, Tracy King, Mark W. Newman, Kurt Partridge, Bob Price, Paul Rasmussen, Michael Roberts, Diane J. Schiano, Alan Walendowski

Thanks also to: Ame Elliott and Dai Nippon Printing

25

Supplemental Slides

50%

50%

Venue Likelihood:

1:00

Monday Tuesda

…

12:00 to 1:00

1:00 to

Hector’s CafeAstrid’s Grocery

12:00

Time Location Visit

11:57- 12:45 37°26’39”-122°9’38”

1:22 - 1:31 37°23’11”-122°9’02”

… … …

Context HistoryContext HistoryWeekly Behavior PatternsWeekly Behavior Patterns

$$$

GroceryCafe

…

…

…

$$$

GroceryCafe

…

…

Predicting Activities fromLearned User Patterns

BUYEAT

activity-based serendipitous recommendations with the magitti mobile leisure guide

Technology

leisure leisure guide

main activity

user feedback design

design of form

leisure activities

printing technology

fieldwork restaurants

user studies