understanding user engagement with digital interventions

22
Understanding user engagement with digital interventions Dr Jason Rentfrow, Dr Leanne Morrison, & Dr Sharon Lin On behalf of the UBhave, Emotion sense, LifeGuide, and POWeR teams

Upload: ginger

Post on 23-Feb-2016

41 views

Category:

Documents


0 download

DESCRIPTION

Understanding user engagement with digital interventions. Dr Jason Rentfrow , Dr Leanne Morrison, & Dr Sharon Lin On behalf of the UBhave , Emotion sense, LifeGuide , and POWeR teams. Understanding user engagement with power: digital intervention for weight management. Leanne Morrison - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Understanding user engagement with digital interventions

Understanding user engagement with digital interventions

Dr Jason Rentfrow, Dr Leanne Morrison, & Dr Sharon Lin

On behalf of the UBhave, Emotion sense, LifeGuide, and POWeR teams

Page 2: Understanding user engagement with digital interventions

UNDERSTANDING USER ENGAGEMENT WITH POWER: DIGITAL INTERVENTION

FOR WEIGHT MANAGEMENT

Leanne MorrisonUniversity of Southampton

Page 3: Understanding user engagement with digital interventions

How can we encourage better engagement with online interventions?

STUDY 1: Role of supplemental human support (“coaching”)

– RCT of online POWeR programme + supplemental telephone coaching across communities in North East England (n = 786)

STUDY 2: Role of supplemental mobile support (“apps”)– Series of in depth mixed methods ‘N-of-1’ case studies to

explore engagement with and impact of online POWeR programme + POWeR Tracker app (N = 13)

How do users feel about these forms of supplemental support?

What impact do they have on engagement and health-related outcomes?

Page 4: Understanding user engagement with digital interventions

• Theory and evidence-based online programme to support users to adopt a sustainable and positive approach to weight management

• Developed using LifeGuide

• POWeR Tracker: Android smartphone application to accompany the online programme

– Maintain awareness of

personal POWeR goals – Monitor progress

Page 5: Understanding user engagement with digital interventions

Study 1: Community based RCT

Consent and RegistrationN=1131

CoachN=247 (31.4%)

Web onlyN=264 (33.6%)

ControlN=275 (35%)

RandomisationN=786 (69.5%)

Responded to f/upN=162 (58.9%)

Responded to f/upN=40 (15.2%)

Responded to f/upN=53 (21.5%)

Qualitative interviews (n=19, purposively

sampled)

Coaching Protocol:

2 short phone calls from a ‘POWeR coach’ in week 1 and week 4

Mohr et al. (‘Supportive Accountability’)

In collaboration with public health teams (Scott Lloyd, NHS Tees, NHS Durham and Darlington)

Is usage enhanced by the addition of brief human support in the form of telephone coaching?

Page 6: Understanding user engagement with digital interventions

Did telephone coaching encourage usage?

Participants in the coach arm significantly more likely to complete the core POWeR sessions

Web only Web+ Coach Significance testN (% )completing 3 core sessions

47 (17.8%) 64 (25.9%) ᵪ2(1,n=511)=4.93, p=.026

1 2 3 4 5 6 7 80

10

20

30

40

50

60

70

80

% of sample still using POWeR at each session

webcoach

Page 7: Understanding user engagement with digital interventions

Perceptions of coaching

• Low uptake– only 23.5% had one phone call, 18.6% had both

• How did coaching help? – Praise and positivity – Accountability

DEMOGRAPHICS: Older, lower health literacy, higher BMI, hypertension, previous referral to weight loss scheme

USAGE AND ENGAGEMENT: More sessions completed, more log ins, more time spent online, Satisfaction with POWeR, fewer doubts about how to use POWeR, autonomous motivation

PROGRESS: Greater weight loss

Page 8: Understanding user engagement with digital interventions

• Variety of measures:– Daily questionnaire measures – Step counts (via pedometer)– Weekly telephone interviews – Objective data on web and app usage

Q1. Does an app improve goal perceptions/progress?

Study 2: POWeR Tracker• 13 participants followed over 4 weeks in series

of n-of-1 case studies (ABAB vs. BABA) • Compare web-based POWeR with and without

POWeR Tracker app

Q2. When, why and how do people engage with a web + mobile intervention?

A: Week 1 B: Week 2 A: Week 3B: Week 4

Page 9: Understanding user engagement with digital interventions

Did the POWeR Tracker app improve goal perceptions?

• No clear effect of app availability on daily goal perceptions or step count

• Measurement effect Goal Effort Goal

awarenessGoal

motivationGoal self-efficacy

Goal achievement

Step count

1 2 3 4 5 6 Diet PA Diet PA Diet PA Diet PA

Alex 0.50 -0.16 -0.30 -0.01 -0.22 -0.13 -0.74 -0.16 -0.55 -0.61 -0.14 -0.61 0.28 -0.76 0.26

Susan 0.14 0.68 0.34 -0.22 -0.67 -0.26 0.21 0.05 0.60 -0.44 0.54 -0.32 -0.07 0.14 0.26

Hannah 0.05 -0.33 -0.20 -0.24 -0.28 -1.19 -0.15 -0.12 0.23 0.12 -0.98 -0.73 -0.78 -0.67 -0.17

Dan 0.11 0.25 0.12 -0.04 -0.56 0.00 0.19 0.73 0.10 -0.15 -0.30 0.44 -0.09 0.55 0.01

Natalie 0.77 -0.08 0.40 0.05 -0.33 0.57 -0.51 -0.21 0.37 0.02 -0.76 -0.22 -0.10 -0.87 0.12

Lucy 0.28 0.37 -0.20 0.24 0.34 0.26 0.66 0.76 0.57 0.64 0.40 0.62 0.81 0.60 0.74

Rachel -0.04 0.22 -0.25 0.42 0.95 0.85 0.25 0.55 0.61 0.64 0.71 0.68 0.34 0.47 -0.11

Lisha 0.34 -0.01 0.09 0.00 -0.16 -0.53 0.05 -0.58 0.16 -0.78 0.10 -0.51 -0.26 -0.19 0.43

Marcus 0.18 0.18 -0.32 -0.43 -0.22 -0.41 0.03 -0.39 -0.29 -0.45 0.18 -0.66 -0.17 -0.52 -0.08

Ian -0.68 -0.72 -0.34 -0.64 0.65 -0.92 0.96 0.69 1.09 1.09 0.91 0.20 0.38 0.42 0.00

Laura - - - - - - - - - - - - - - -

Chris - - - - - - - - - - - - - - -

Andrew - - - - - - - - - - - - - - -

“I guess if you weren’t like…with the questionnaires every day, um…if you weren’t doing the questionnaires every day then I think you would miss the app more…..Because…I guess the questionnaires every day were making you think about how well have I done today, or kept you motivated and yeah...it was just a period of evaluation. Whereas if you didn’t have those then you…yeah I think you would miss the app or miss the website more.” (Susan)

Page 10: Understanding user engagement with digital interventions

• Short bursts of on-the-go access or time-relevant use

• Notifications prompted app use (when used)• Use app primarily for a reminder of key

information (e.g. food lists, goals) • Variation in approach to using hybrid web +

app intervention

Engagement with POWeR Tracker: When, where, why?

Example: Dan

• Use at key times (e.g. lunchtime – food choices, spare time in between lectures)

• Short bursts of use ~ up to 10 minutes at a time

• Response to app notifications

1 2 3 4 5 6 7 8 9 10 11 12 130

5

10

15

20

25

30

35

40

Diary completionsGoal updateInformation viewsGoal checks

Participant

Page 11: Understanding user engagement with digital interventions

Engagement with POWeR: Summary

• Telephone coaching appears to improve engagement with online interventions and offer benefits to particular groups of user

• Offering mobile tools or apps appears to improve the convenience and accessibility of health behaviour change interventions– Individual variation in tool preferences and

patterns of use

Page 12: Understanding user engagement with digital interventions

ANALYSING COMPLEX DATA SETS

Sharon LinUniversity of Southampton

Page 13: Understanding user engagement with digital interventions

Data analyses

• Visualization• N-of-1 studies

Page 14: Understanding user engagement with digital interventions

Time spent on groups of pages

Page 15: Understanding user engagement with digital interventions

Re-ordered time spent on groups of pages – clustered time

Page 16: Understanding user engagement with digital interventions

Analyses of N-of-1 studies

• To draw in the regression model:,

)

Page 17: Understanding user engagement with digital interventions

A sample of Ubhave dataDay

Dan_Total step Inhibitor

1 NA 02 11471 03 9760 04 3558 05 4739 06 3662 07 NA 08 5729 19 2794 1

10 7636 111 3996 112 7467 113 10587 114 3863 115 1649 016 5659 017 8390 018 2221 019 8980 020 3566 021 3457 022 NA 123 5038 124 4798 125 5414 126 5380 127 3678 128 6335 1

Effect size = 0.006α = 5592, β = 17.27, ρ(rho) = 0.06, σ = 2698

1 3 5 7 9 11 13 15 17 19 21 23 25 270

2000

4000

6000

8000

10000

12000

14000

Dan’s total Steps

p242M15

Page 18: Understanding user engagement with digital interventions

Statistical challenges of N-of-1 studies

• Challenges– Small sample size – Autocorrelated errors in repeated measures

arising from the individual under study– Non-normality of the responses

• Remedy under investigations– Parametric bootstrap tests

Page 19: Understanding user engagement with digital interventions

N-of-1 Power Function Off (7D)-On(7D)-Off (7D)-On(7D) or ABAB

0 0.5 1 1.5 2 2.5 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Bootstrapped Tests, Nint=1000, B=1000

ρ=0, Tsta,OLSρ=0, Tsta,GLSρ=0.2, Tstaρ=0.5, Tstaρ=0.7, Tsta

Measure

Individual

Effect size

Ρ (rho) Power

M15 Dan 0.01 0.1 0.05

M12 Lucy 0.61 0.2 0.25

M14 Dan 0.55 0.5 0.13

M2 Chris 0.59 0.7 0.15

Page 20: Understanding user engagement with digital interventions

Tools will be available in the future

• Functions for visualisation• Analysis tool for N-of-1 studies

– correcting small sample and correlated data problems

• Power functions for N-of-1 studies– giving guidance for N-of-1 study design

Page 21: Understanding user engagement with digital interventions
Page 22: Understanding user engagement with digital interventions

Acknowledgements Funded by the EPSRC under the UBhave projectFor more information please visit: http://ubhave.org

22

Behavioural/Social Scientists:

Professor Lucy Yardley

Professor Susan Michie

Professor Peter Smith

Dr Jason Rentfrow

Dr Leanne Morrison

Dr Laura Dennison

Dr Sharon Lin

Computer Scientists:

Dr Cecilia Mascolo

Dr Mark Weal

Dr Mirco Musolesi

Dr Danius Michaelides

Dr Charlie Hargood

Dr Neal Lathia

Dr Veljko Pejovic