measuring web user engagement: a cauldron of many things
DESCRIPTION
In the online world, user engagement refers to the quality of the user experience that emphasizes the positive aspects of the interaction with a web application and, in particular, the phenomena associated with wanting to use that application longer and frequently. User engagement is a multifaceted, complex phenomenon; this gives rise to a number of potential approaches for its measurement. Common ways of measuring user engagement include: self-reporting (e.g., questionnaires); observational methods (e.g., facial expression analysis, speech analysis, desktop actions); and web analytics using online behavior metrics that assess users’ depth of engagement with a site. These methods represent various tradeoffs between the scale of data analyzed and the depth of understanding. For instance, surveys are small-scale but deep, whereas clicks can be collected on a large-scale but provide shallow understanding. However, little is known in validating and relating these types of measurement. This talk will present various efforts aiming at combining techniques from web analytics (in particular clicks) and existing works on user engagement coming from the domains of information science, multimodal human computer interaction and cognitive psychology. This is a revised presentation of a keynote given at TPDL 2012. New work include online multi-tasking and exploring mouse movement.TRANSCRIPT
Measuring Web User Engagement: a cauldron of web analytics, focus attention,
positive affect, user interest, saliency, mouse movement & multi-tasking
Mounia Lalmas
Yahoo! Labs
Barcelona
Click-through rate as proxy of user engagement!
Multimedia search activities often driven by entertainment needs, not by information needs
Click-through rate as proxy of relevance!
M. Slaney, Precision-Recall Is Wrong for Multimedia, IEEE Multimedia Magazine, 2011.
Click-through rate as proxy of user user satisfaction!
I just wanted the phone number … I am totally satisfied
In this talk – results, messages & questions
1. Big data and in-depth focused user studies “a must”!
2. Users “multi-task” online, what does this mean?
3. Mouse movement hard to “experiment with” and/or “interpret”.
4. Using crowd-sourcing “I think” worked fine.
This talk is not about aesthetics … but see later
http://www.lowpriceskates.com/ (e-commerce – skating)
Source: http://www.webpagesthatsuck.com/
This talk is not about usability
http://chiptune.com/ (music repository)
Source: http://www.webpagesthatsuck.com/
User Engagement – connecting three sides User engagement is a quality of the user experience that emphasizes the positive aspects of
interaction – in particular the fact of being captivated by the technology.
Successful technologies are not just used, they are engaged with.
user feelings: happy, sad,excited, …
The emotional, cognitive and behavioural connection that exists, at any point in time and over time, between a user and a technological resource
user interactions: click, readcomment, recommend, buy, …
user mental states: concentrated,lost, involved, …
S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.
Characteristics of user engagement
Positive Affect
Focused Attention
Motivation, Interests, Incentives
& Benefits
Novelty
Aesthetics
Richness & Control
Reputation, Trust & Expectation
Endurability
H.L. O'Brien & E.G. Toms. JASIST 2008, JASIST 2010.
H.L. O'Brien. Defining and Measuring Engagement in User Experiences with Technology. PhD Thesis, 2008.
Measuring user engagement
cognitive engagement
self-reported engagement
interaction engagement
Connecting three measurement approaches
USER ENGAGEMENT
cognitive engagement
self-reported engagement
interaction engagement
Models of user engagement … towards a taxonomy?
USER ENGAGEMENT
Models of user engagementOnline sites differ concerning their engagement!
GamesUsers spend much time per visit
SearchUsers come frequently and do not stay long
Social mediaUsers come frequently and stay long
SpecialUsers come on average once
NewsUsers comeperiodically
ServiceUsers visit site, when needed
Data and Metrics
Interaction data, 2M users, July 2011, 80 US sites
Popularity #Users Number of distinct users
#Visits Number of visits
#Clicks Number of clicks
Activity ClickDepth Average number of page views per visit.
DwellTimeA Average time per visit
Loyalty ActiveDays Number of days a user visited the site
ReturnRate Number of times a user visited the site
DwellTimeL Average time a user spend on the site.
Methodology
General models Time-based modelsDimensions
8 metricsweekdays, weekend
8 metrics per time span#Dimensions 8 16
Kernel k-means with Kendall tau rank correlation kernel
Nb of clusters based on eigenvalue distribution of kernel matrixSignificant metric values with Kruskal-Wallis/Bonferonni
#Clusters (Models) 6 5
Analysing cluster centroids = models
Models of user engagement [6 general]
• Popularity, activity and loyalty are independent from each other• Popularity and loyalty are influenced by external and internal factors
e.g. frequency of publishing new information, events, personal interests
• Activity depends on the structure of the site
interest-specific
media (daily)search
periodicmedia
e-commerce
models based on engagement metrics only
Time-based [5 models]Models based on engagement over weekdays and weekend
hobbies,interest-specificweather
daily newswork-related
time-based models ≠ general models
next put all and more together! let machine learning tell you more!
Models of user engagement – Recap & NextUser engagement is complex and standard
metrics capture only a part of itUser engagement depends on time (and users)
First step towards a taxonomy of models of user engagement … and associated metrics
NextMore sites, more modelsUser demographics, time of the day, geo-location,
etc.Online multi-tasking
J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.
cognitive engagement
self-reported engagement
interaction engagement
Online multi-tasking
USER ENGAGEMENT
Online multi-tasking
users spend more and more of their online session multi-tasking, e.g. emailing, reading news, searching for information ONLINE MULTI-TASKING navigating between sites, using browser tabs, bookmarks, etc seamless integration of social networks platforms into many services
leaving a site isnot a “bad thing!”
(fictitious navigation between sites within an online session)
181K users, 2 months browser data, 600 sites, 4.8M sessions
•only 40% of the sessions have no site revisitation
•hyperlinking, backpaging and teleporting
Navigating between sites – hyperliking, backpaging and teleporting
Number of backpaging actions is an under-estimate!
Revisitation and navigation patterns
Online multi-tasking – Some results48% sites visited at least 9 timesRevisitation “level” depends on site
10% users accessed a site 9+ times (23% for search
sites); 28% at least four times (44% for search sites)
Activity on site decreases with each revisit but activity on many search and adult sites increases
Backpaging usually increases with each revisit but hyperlinking remains important means to navigate between sites
Online multi-tasking – Recap & Next
J. Lehmann, M. Lalmas & G. Dupret. Online Multi-Tasking and User Engagement. Submitted for publication, 2013.
cognitive engagement
self-reported engagement
interaction engagement
Focus attention, positive affect & saliency
USER ENGAGEMENT
Saliency, attention and positive affect
How the visual catchiness (saliency) of “relevant” information impacts user engagement metrics such as focused attention and emotion (affect)focused attention refers to the exclusion of
other thingsaffect relates to the emotions experienced
during the interaction
Saliency model of visual attention developed by Itti and Koch L. Itti and C. Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40, 2000.
Manipulating saliency
Web page screenshot Saliency maps
salie
nt c
ondi
tion
non-
salie
nt c
ondi
tion
Study design8 tasks = finding latest news or headline on celebrity or
entertainment topicAffect measured pre- and post- task using the Positive
e.g. “determined”, “attentive” and Negative e.g. “hostile”, “afraid” Affect Schedule (PANAS)
Focused attention measured with 7-item focused attention subscale e.g. “I was so involved in my news tasks that I lost track of time”, “I blocked things out around me when I was
completing the news tasks” and perceived time Interest level in topics (pre-task) and questionnaire
(post-task) e.g. “I was interested in the content of the web pages”, “I wanted to find out more about the topics that I encountered on the web pages”
189 (90+99) participants from Amazon Mechanical Turk
Saliency and positive affect
When headlines are visually non-salient users are slow at finding them, report more
distraction due to web page features, and show a drop in affect
When headlines are visually catchy or salient user find them faster, report that it is easy to focus,
and maintain positive affect
Saliency is helpful in task performance, focusing/avoiding distraction and in maintaining positive affect
Saliency and focused attentionAdapted focused attention subscale from the online
shopping domain to entertainment news domain
Users reported “easier to focus in the salient condition” BUT no significant improvement in the focused attention subscale or differences in perceived time spent on tasks
User interest in web page content is a good predictor of focused attention, which in turn is a good predictor of positive affect
Saliency and user engagement – Recap & NextInteraction of saliency, focused attention, and
affect, together with user interest, is complexNext:
include web page content as a quality of user engagement in focused attention scale
more “realistic” user (interactive) reading experiencebio-metrics (mouse-tracking, eye-tracking, facial
expression, etc)
L. McCay-Peet, M. Lalmas, V. Navalpakkam. On saliency, affect and focused attention, CHI 2012
cognitive engagement
self-reported engagement
interaction engagement
Mouse tracking, positive effect, attention
USER ENGAGEMENT
Mouse tracking … and user engagement
324 users from Amazon Mechanical Turk (between subject design)
Two domains (BBC and Wikipedia) Two tasks (reading and quiz) “Normal vs Ugly” interface Questionnaires (qualitative data)
focus attention, positive effect, novelty, interest, usability, aesthetics
+ demographics, handeness & hardware
Mouse tracking (quantitative data) movement speed, movement rate, click rate, pause length,
percentage of time still
“Ugly” vs “Normal” Interface (BBC News)
“Ugly” vs “Normal” (Wikipedia)
Mouse tracking can tell about
Age
HardwareMouseTrackpad
Task Searching: There are many different types of phobia. What is
Gephyrophobia a fear of? Reading: (Wikipedia) Archimedes, Section 1: Biography
Mouse tracking could not tell much on
Mouse tracking and user engagement —Recap & Next
High level of ecological validityAge, task, and hardwareDo we have a Hawthorne Effect???“Usability” vs engagement
“Even uglier” interface? I don’t think so
Within- vs between-subject design?Next
Sequence of movementsAutomatic clustering
D. Warnock and M. Lalmas. An Exploration of Cursor tracking Data. Submitted for publication, 2013.
cognitive
self-reported
interaction
Connecting three measurement approaches
The value of a click?
Thank you
Ioannis ArapakisRicardo Baeza-YatesGeorges DupretJanette LehmannLori McCay-Peet (Dalhousie University)Vidhya NavalpakkamDavid Warnock (Glasgow University)Elad Yom-Tov
and many others at Yahoo! Labs
Collaborators:
Contact: [email protected]