Understanding and Measuring User Engagement and Attention

in Online News Reading Dmitry Lagun and Mounia Lalmas

1Thanks to Yahoo Faculty Research and Engagement Program for supporting this work.

User Engagement in Online News Reading

2

User engagement:“emotional, cognitive and behavioral connection that exists between a user and a resource” (Attfield et al., 2011)

Stickiness:concerned with users spending time on a news site.

User Attention in Online News Reading

Challenge II:identifying which aspects of the online interaction influence user engagement the most.

Challenge I:attract large shares of online attention by keeping users engaged.

Measuring user engagement with news content

3

Method PROS CONS

Dwell time (click duration)(Agichtein et al., 2006)

scalable; captures engagement at coarse level

cannot distinguish time spent on parts of the article

Eye tracking (Arapakis et al., 2014)

very detailed small scale; very expensive

Mouse cursor movement(Huang et al., 2011)

scalable; more fine grained than dwell time

cursor is often kept still during article reading, when no pointing action is required

coarse but more robust instrument to measure user attention at large scale during news reading

VIEWPORT TRACKING

Our Method: Viewport Tracking

4

viewport

time spent at i-th scroll position

i-th viewport defined by a rectangle (left, top, width, height)

viewport

Research questions

● Where do users spend their time during news article viewing?

● Does media image and video content affect time spent at a vertical position?

● What are typical patterns of news article reading?

● Can we accurately predict user engagement from textual content?

5

● 1,971 Yahoo news articles● 267,210 page views on desktopDATASET

6

Overall Pattern of Viewport Time (proxy for user attention)

Many users spend significantly smaller amount of time at lower scroll positions.

Some users find the article interesting enough to spend significant amount of time at the lower part of the article.

Some articles entice users to deeply engage with their content.

Image and Video do matter … for the first screen

7

Video Image

How do users browse through the article?

8

comment

header

top

middle

bottomartic

le b

odystart

top

middle

bottom

comment

leave

Markov States

beginning of a page view

top area occupies most of the viewport

middle area occupies most of the viewport

bottom area occupies most of the viewport

comment area occupies most of the viewport

user leaves the page

V1 V2 ... Vn

Mixture of Markov Chains Model

9

Single markov model:

Mixture of K markov models:

probability of starting at state v1

probability of transition from state Vi to V(i-1)

Markov States:{Start, Top, Middle, Bottom,

Comment, Leave}

weight of k-th mixture component

K=6 is optimal

Patterns of Attention in News Reading

10

Engagement Depth

most probable sequence

Engagement depth: Four User Engagement Classes

11

Engagem

ent Depth

12

Engagement depth: Four User Engagement Classes

Engagem

ent Depth

13

Engagement depth: Four User Engagement Classes

Engagem

ent Depth

14

Engagement depth: Four User Engagement Classes

Engagem

ent Depth

Distribution of Attention is Different across Engagement Classes

15

Modeling of User Engagement from Article Content

16

?news article

%Bounce

%Shallow

%Deep

%Complete

user engagement profile

TUNE: Topics of User Engagement with News

17

TUNEnews article

%Bounce

%Shallow

%Deep

%Complete

user engagement profile

Unlike LDA, in TUNE topic is a combination of word co-occurrence and similarity of user

engagement profile.

Distribution of user engagement level

Experimental Setting

● Task○ Predict User Engagement Level Profile

● Model○ Linear regression

● Features○ Number of words in the article○ Presence of media content (e.g., image and video)○ Distribution of article topics with LDA○ Distribution of article topics with TUNE (our model)

● Evaluation Metric○ Pearson’s correlation between ground truth and predicted value○ Ten fold cross-validation

18

%Bounce

%Shallow

%Deep

%Complete

Results: Baselines

19

Feature Set %Bounce %Shallow %Deep %CompleteNumWords 0.063 0.494 0.370 0.017

NumWords + Media (M) 0.071 0.571 0.410 0.185

Results: Baselines

20

Feature Set %Bounce %Shallow %Deep %CompleteNumWords 0.063 0.494 0.370 0.017

NumWords + Media (M) 0.071 0.571 0.410 0.185NumWords + M + LDA (T=5) 0.119 0.597 0.466 0.328

NumWords + M + LDA (T=10) 0.110 0.606 0.497 0.379NumWords + M + LDA (T=20) 0.150 0.626 0.531 0.402NumWords + M + LDA (T=50) 0.143 0.629 0.538 0.405

Results: Baselines vs. TUNE

21

Feature Set %Bounce %Shallow %Deep %CompleteNumWords 0.063 0.494 0.370 0.017

NumWords + Media (M) 0.071 0.571 0.410 0.185NumWords + M + LDA (T=5) 0.119 0.597 0.466 0.328

NumWords + M + LDA (T=10) 0.110 0.606 0.497 0.379NumWords + M + LDA (T=20) 0.150 0.626 0.531 0.402NumWords + M + LDA (T=50) 0.143 0.629 0.538 0.405NumWords + M + TUNE (T=5) 0.079 0.648 0.544 0.282

NumWords + M + TUNE (T=10) 0.311 0.713 0.660 0.400NumWords + M + TUNE (T=20) 0.349 0.724 0.682 0.409

NumWords + M + TUNE (T=50) 0.333(+132%)

0.742(+18%)

0.697(+29%)

0.428(+6%)

NumWords + M + LDA + TUNE 0.334 0.730 0.696 0.442Dwell 0.392 0.203 0.128 0.351

Conclusions● Unlike in search, user attention in news reading is not constantly decaying

with vertical position (e.g., can be bi-modal)

● Engagement with a news article can be categorized by depth of examination (Bounce, Shallow, Deep & Complete)

● The proposed engagement metrics go beyond “dwell time” as they capture user attention and engagement at sub-document level

● We can obtain accurate prediction of article engagement profile purely from its textual content

22

Summary: Viewport time attention as proxy of user engagement

23

Effect of position and content on viewport time at vertical position

V1 V2 ... VnArticle examination can be categorized by depth of examination

Four engagement classes: Bounce, Shallow, Deep and Complete

Joint model of article topics and user engagement classes improves prediction accuracy:

● Bounce (+140%)● Shallow (+18%) ● Deep (29%) ● Complete (+9%)

Appendix

24

User Attention vs. Engagement Classes

25

MetricBounce

(N=26542)Shallow

(N=63982)Deep

(N=164197)Complete (N=12489)

dwell 6.17 (0.02) 63.75 (0.37) 99.02 (0.22) 228.35 (1.48)header time 2.99 (0.03) 15.39 (0.14) 18.48 (0.08) 17.41 (0.25)body time 5.06 (0.02) 35.13 (0.21) 86.24 (0.20) 85.00 (0.70)

comment time 0.56 (0.01) 17.27 (0.23) 9.72 (0.07) 110.90 (0.89)% header time 0.31 (0.00) 0.23 (0.00) 0.17 (0.00) 0.09 (0.00)% body time 0.62 (0.00) 0.58 (0.00) 0.76 (0.00) 0.40 (0.00)

% comment time 0.07 (0.00) 0.20 (0.00) 0.07 (0.00) 0.51 (0.00)% article read 0.12 (0.00) 0.23 (0.00) 0.83 (0.00) 0.84 (0.00)

# comment clicks 0.01 (0.00) 0.43 (0.01) 0.00 (0.00 3.14 (0.03)

User Engagement Classes and User Attention

26

Dwell time and viewport time on head, body and comment increase from Bounce to Complete.

Viewport time on head steadily decreases from Bounce to Complete: users spend an increasing amount of time reading content deeper in article.

Percentage of article read steadily increases from Bounce to Complete, as expected.

Deep and Complete correspond to the situations when the majority (83%) of the article was read.

Number of comment clicks is highest for Complete and then Shallow: users may engage with comments even if they do not read a large proportion of the article.

comment

headertop

middle

bottomartic

le b

ody

header