session, focus and engagement
DESCRIPTION
ECIR 2012 workshop "Information Retrieval Over Query Sessions" KeynoteTRANSCRIPT
A bit about myself
1999-2008: Lecturer (assistant professor) to Professor at Queen Mary, University of London
2008-2010 Microsoft Research/RAEng Research Professor at the University of Glasgow (and lived outside London)
2011- Visiting Principal Scientist at Yahoo! Research Barcelona
Research topics XML retrieval and evaluation (INEX) Quantum theory to model interactive information retrieval Aggregated search Bridging the digital divide Models and measures of user engagement
Message and Outline
Interaction and search Beyond result relevance Beyond search session
Towards “engagement”
1. Motivations2. Engagement3. Future directions
1. Outline
1. Motivations• Relevance in multimedia search• Relevance in focused retrieval• Online multi-tasking
2. Engagement
1. Future directions
Information Retrieval Over Query Sessions
Retrieval Models & Ranking: How to analyze/model/predict user interactions and use these findings to improve retrieval performance? How can we adapt ranking/retrieval models and IR theory in the light of a sequence of user interactions.
Evaluation & Test Collections: How can we evaluate retrieval system performance over entire query sessions? How can we build reusable test collections to study this IR task? How can we model/simulate user interactions over a session?
User Interaction & Interfaces: How can we model user interactions so we can predict and improve the user experience over sessions? How can we design and perform user studies that reveal new information about users? How can we make use of implicit feedback from users?
Multimedia search activities often driven by entertainment needs, not by information needs
Relevance in multimedia search
M. Slaney, Precision-Recall Is Wrong for Multimedia, IEEE Multimedia Magazine, 2011
Relevance in focused retrievalRelevance in context
Table of Content Focused retrieval is about putting results (element, fact, passage) in context, to understand and trust them
Courtesy Jaap Kamps, Zoltan Szlavik, Norbert Goevert
Beyond search session
On month browsing data, sample of Yahoo! sites
On month browsing data, sample of sites
(INT=Yahoo site,EXT=non Yahoo site)
Courtesy of Janette Lehmann
users spend more and more of their online session multi-tasking, e.g. emailing, reading news, searching for information ONLINE MULTI-TASKING navigating between sites, using browser tabs, etc seamless integration of social networks platforms into many services
Interactive IR …
P Ingwersen, Human Aspects in IR, ESSIR 2011.
2. Outline
1. Motivations
2. (User) Engagement• Definition• Characteristics• Measuring• Models
1. Future directions
User Engagement – connecting three sides User engagement is a quality of user experience that emphasizes the positive aspects of
interaction – in particular the fact of being captivated by the technology.
Successful technologies are not just used, they are engaged with.
user feelings: happy, sad,excited, bored, …
The emotional, cognitive and/or behavioural connection that exists, at any point in time and over time, between a user and a technological resource
user interactions: click, readcomment, recommend, buy, …
user mental states: concentrated,challenged, lost, interested …
Characteristics of user engagement (I)
S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.
Characteristics of user engagement (II)
S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.
The four I’s
Measuring Engagement, Forrester Research, June 2008
Measuring user engagement
Objective measures – Online activities
Proxy of user engagement
Models of user engagementOnline sites differ concerning their engagement!
GamesUsers spend much time per visit
SearchUsers come frequently and do not stay long
Social mediaUsers come frequently and stay long
SpecialUsers come on average once per time considered
NewsUsers comeperiodically
ServiceUsers visit site, when needed
Is it possible to model these differences?
Data and Metrics
Interaction data, 2M users, July 2011, 80 US sites
Popularity #Users Number of distinct users
#Visits Number of visits
#Clicks Number of clicks
Activity ClickDepth Average number of page views per visit.
DwellTimeA Average time per visit
Loyalty ActiveDays Number of days a user visited the site
ReturnRate Number of times a user visited the site
DwellTimeL Average time a user spend on the site.
Diversity in user engagement
Users and Loyalty Sites have different user groups Proportion of user groups is site-
dependent
Time and Popularity Site engagement can be periodic
or contains peaks
Engagement of a site depends on users and time
mail, social media
shopping, entertainment
media(special events)
daily activity,navigation
media,entertainment
Methodology
General models User-based models Time-based modelsDimensions
8 metrics5 user groups8 metrics per user group
weekdays, weekend8 metrics per time span
#Dimensions 8 40 16
Kernel k-means with Kendall tau rank correlation kernel
Nb of clusters based on eigenvalue distribution of kernel matrixSignificant metric values with Kruskal-Wallis/Bonferonni
#Clusters (Models) 6 7 5
Analysing cluster centroids = models
Models of user engagement
• 6 general models
• Popularity, activity and loyalty are independent from each other
• Popularity and loyalty are influenced by external and internal factors e.g. frequency of publishing new
information, events, personal interests
• Activity depends on the structure of the site
Models based on engagement metrics
interest-specific
e-commerce,configuration
periodicmedia
Models of user engagement
User-based [7 models] Models based on engagement per
user group
Time-based [5 models] Models based on engagement
over weekdays and weekend
Models based on engagement metrics, user and time
navigation game, sporthobbies,interest-specific
daily news
Sites of the same type (e.g. mainstream media) do not necessarily belong to the same model
The groups of models describe different aspects of engagement, i.e. they are independent from each other
Recap & NextUser engagement is complex and standard
metrics capture only a part of itFirst step towards a taxonomy of models of user
engagement … and associated metrics
NextInteraction between modelsInteraction between sites (multi-tasking)User demographics, time of the day, geo-location, etc
J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.
3. Outline
1. Motivations
2. Engagement
1. Future directions1. The three sides of user engagement2. Interactive IR3. Towards engagement
+ layout +links+ saliency + content
user engagement within and across site Measurements and methodologies
+ online analytics metrics (dwell time, CTR, …) + complex networks metrics
+ questionnaires, surveys, … + crowd-sourcing
+ biometrics (eye tracking, mouse tracking, …)
Goals + Models of user engagement + Metrics of user engagement
The three sides
+ emotional
+ cognitive
+ behavioral
Let us revisit … connecting three sides
Let us revisit … Interactive IR
P Ingwersen, Human Aspects in IR, ESSIR 2011.
session, interaction, multi-tasking, network, search, relevance, …
•I Aapakis, K Athanasakos, J Jose, A comparison of general vs personalised affective models for the prediction of topical relevance, SIGIR 2010.•J Huang, R White, S Dumais, No clicks, no problem: using cursor movements to understand and improve search, CHI 2011.• P Ingwersen & K Järvelin, The turn: integration of information seeking and retrieval in context, 2005.
TOWARDS ENGAGEMENT
Information Retrieval Over Query Sessions
Retrieval Models & Ranking: How to analyze/model/predict user interactions and use these findings to improve retrieval performance? How can we adapt ranking/retrieval models and IR theory in the light of a sequence of user interactions.
Evaluation & Test Collections: How can we evaluate retrieval system performance over entire query sessions? How can we build reusable test collections to study this IR task? How can we model/simulate user interactions over a session?
User Interaction & Interfaces: How can we model user interactions so we can predict and improve the user experience over sessions? How can we design and perform user studies that reveal new information about users? How can we make use of implicit feedback from users?
TOWARDS ENGAGEMENT
beyond session and relevance
Thank you
www.dcs.gla.ac.uk/~mounia
TOWARDS ENGAGEMENT
beyond session and relevance