"using data science to design effective precision preventative behavioral medicine" - ryan...

1

Using Data Science to Design Effective Precision Preventative Behavioral Medicine

Ryan QuanData ScientistOmada Health

Outline Omada Health

• Context! Who are we? What do we do? Data science?

Building a Data Science Culture from the Ground Up• Winning hearts & minds with data

The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works

The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way

• Impactful experimentation: beyond the sub-group analysis

3

Outline Omada Health

• Context! Who are we? What do we do? Data science?

Building a Data Science Culture from the Ground Up• Winning hearts & minds with data

The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works

The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way

• Impactful experimentation: beyond the sub-group analysis

4

OMADA HEALTH 5OMADA HEALTH

THE PROBLEM

6

7

8

THE SOLUTION

9

Lifestyle Change

10

58% Reduction

11

Large Body of Clinical Evidence for DPP

THE PROBLEMTHE SOLUTION

WHAT’S THE DEAL?

12

13

Behavior Change is Hard

EDUCATION

TRACKERS

CALORIE COUNTING

HEALTH COACHING

SOCIAL SUPPORT

One Dimension is Not Enough

19

20

INTRODUCINGThe Omada Program

OMADA HEALTH

PARTICIPANT PEERGROUP

Participants are matched into peer groups for

encouragement and healthy competition with the entire group progressing along a

shared timeline.

EASY-TO-USETECHNOLOGY

A wireless scale, pedometer, mobile apps, and interactive

portal are used to track weight loss, activity, meals,

and program accomplishments.

ONLINE INTERACTIVE LESSONS

Participants learn how to eat healthier, increase activity

levels, and overcome challenges through fun lessons

and games accessed via an online portal or mobile app.

FULL-TIME PROFESSIONALPREVENT COACH

A dedicated coach provides participants with support,

motivation, and personalized recommendations along their

Prevent journey.

22OMADA HEALTH

OVER 19 POINTS OF ENGAGEMENT PER WEEK

Group DiscussionPrevent Coach ConversationSkill ChallengeSuccess Updates

Weigh-InsExercise LogMeal TrackingLesson Completions

23

OMADA HEALTH 24

OMADA

25

26Data Science

Data Science @ Omada

27

• Use analytics, machine learning and experimentation to parameterize what works (and what doesn’t) for behavior change

• Deploy the right intervention, at the right time, for the right patient

Precision Preventative Medicine

The Data (one participant out of 100k+)

28

5

0

Weight Loss (%

)

15

10

5

0

Weight Loss (%

)

15

10

29

Private Messages• Private messages between

HC and participants• Over 1.2M messages• Data:

- Emotionally charged raw text (NLP)

The Data (one participant out of 100k+)

5

0

Weight Loss (%

)

15

10

30

Lessons / Curriculum• Based on original DPP

curriculum• Basic nutrition/activity

information• Interactive w/ games, etc.• Data:

- Completion rates- Reading times- Comprehension

The Data (one participant out of 100k+)

5

0

Weight Loss (%

)

15

10

31

Physical Activity Tracking• Personalized adaptive daily step goals• MyFitnessPal and auto-step tracking

integrations• Over 3.6M activities recorded• Data:

- Steps entered- Minutes of activity- Location (physical proximity triggers)

The Data (one participant out of 100k+)

5

0

Weight Loss (%

)

15

10

32

Group Messaging / Social Support• Group communication • participant <> participant• participant <> health coach• Data:

- Emotionally charged text communication (NLP)

The Data (one participant out of 100k+)

5

0

Weight Loss (%

)

15

10

33

Meal Tracking

• Daily food tracking• 6.2M meals tracked• Active feedback from coach• Data:

- Raw text (food)- Healthiness- Portion size- Time of day

The Data (one participant out of 100k+)

5

0

Weight Loss (%

)

15

10

34

Weighing-in / Progress • Low-power 3G scale• Automatically uploads to

profile on progress page• Weighing in as habit

formation• 6M weigh-ins• Data:

- Weight value- Time of day

The Data (one participant out of 100k+)

The Data

Outline Omada Health

• Context! Who are we? What do we do? Data science?

Building a Data Science Culture from the Ground Up

• Winning hearts & minds with data

The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works

The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way

• Impactful experimentation: beyond the sub-group analysis

36

Building a data science cultureScenario:

You’re the first/only data scientist at your company. You have amazing data with a lot of potential, but your company has no data science

team or experience with data driven product development.

What do you do? Two choices:

Building a data science cultureScenario:

You’re the first/only data scientist at your company. You have amazing data with a lot of potential, but your company has no data science

team or experience with data driven product development.

What do you do? Two choices:Choice 1: Data Police

Building a data science cultureScenario:

You’re the first/only data scientist at your company. You have amazing data with a lot of potential, but your company has no data science

team or experience with data driven product development.

What do you do? Two choices:Choice 2: Build a data

science culture

Building a data science culture

Lots of ways to kick this off:• Brown bag lunch presentations / journal club• Deploy visualization tools• Data education/proselytization• Pro-active analytics as a service• Start a data blog…

Building a data science culture“Plot of the Week (PotW)”

• Internal weekly email data blog • ‘Buzz-feedy’ stories using our data in interesting ways• Exposed company to opportunities with data• Mobilized support and excitement for data science

Building a data science culture“Plot of the Week (PotW)” examples:

1. All in the Family Provides view into raw data

2. Minds over Matter Illustrates potential uses of our data

3. Just Breathe Educates subject matter with data

Building a data science culture“Plot of the Week (PotW)” examples:

1. All in the Family Provides view into raw data

2. Minds over Matter Illustrates potential uses of our data

3. Just Breathe Educates subject matter with data

Our scales become ‘part of the family’ sending weight data of anyone (anything) that steps

on it

PotW: All in the Family

44

Family of two, losing weight together

PotW: All in the Family

45

Another family of two, losing weight together, a few trips to Sandals in there too?

PotW: All in the Family

46

Family of three, growing child

PotW: All in the Family

47

Family of three, neighborhood party

PotW: All in the Family

48

???

Outline Omada Health

• Context! Who are we? What do we do? Data science?

Building a Data Science Culture from the Ground Up• Winning hearts & minds with data

The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works

The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way

• Impactful experimentation: beyond the sub-group analysis

49

The Data Science & Product Relationship

Product

50

Sub-optimal - data science as ‘advisor’

Data-Science

• Exploration/data mining

• Modeling• Proto-typing• Lean-experimentation

• Prioritizes engineering work

• Creates data-models• Implements experiments• Requests analysis

Recommendations/insights

Requests

The Data Science & Product Relationship

Product

51

Sub-optimal - data science as ‘advisor’

Data-Science

Recommendations/insights

Requests

The Data Science & Product Relationship

Data Science

52

ProductOptimal - data science as ‘driver’

• Has own backlog/engineers• Drives data-model design• Experimentation built into

Product• Data scientists can launch

production experiments

Hypothesis

Generation

Experimentation

IterationOptimization

Outline Omada Health

• Context! Who are we? What do we do? Data science?

Building a Data Science Culture from the Ground Up• Winning hearts & minds with data

The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works

The Paths to Personalized (Precision) Prevention

• How to target the right patient, at the right time, in the right way• Impactful experimentation: beyond the sub-group analysis

53

Outline Omada Health

• Context! Who are we? What do we do? Data science?

Building a Data Science Culture from the Ground Up• Winning hearts & minds with data

The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works

The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way

• Impactful experimentation: beyond the sub-group analysis

54

Impactful Experimentation

55

Pros Cons• Causality !=

Correlation• Common language of

scientific discovery

• Low N• Labor intensive, expensive• Slow• Bias in patient populations• Results often not generalizable

For over a century, the Randomized Control Trial (RCT) has been the gold standard for scientific discovery

Impactful Experimentation

56

Largest meta-analysisof DPP ~5.5k participants

>60k Omada participants

vs

At Omada, we have built Randomized Control Experiments into the product to take advantage of the measurement power of experimentation, while the digital nature of our program mitigates the cons (e.g. high N, quick iteration, inexpensive, generalized population, deeper data).

Real-time, Randomized Controlled Experiments (RCEs)

35 in-product experiments & growing quicklyThis enables:

• Rapid exploration of product hypotheses• Structured innovation and evolution of

Prevent• Massive personalization of sub-specifics of

the intervention • Centralized intervention delivery allows

coordinated optimization across all demographics and sectors

• Asking fundamental about what works, and what doesn’t, in behavior change

Personalization through ExperimentationSubgroup Analysis

Answers questions like “Do older people respond differently to Tx than younger?”, or “Do males respond better than females?”

Broad cuts lose resolution to subtle changes

Guided by heuristics (e.g. demographics) and variables available – can lead to biases ITT

Age > 65

Uplift Modeling• Statistical model, based on experimental

data• Predicts who the intervention is likely to

be most impactful for• Don’t need heuristics (other than

choosing inputs) – let data do the personalization for you

59

Subgroup Analysis: Physical Activity Engagement

• Physical activity is a large part of a healthy lifestyle• The second phase of the Omada Program focuses on

increasing patient’s physical activity:1. Providing pedometers to patients to collect physical activity (“steps”)

data2. Educational components and health-coach interaction about exercise3. Setting daily “step goals” for participants

60

Subgroup Analysis: Personalization of Step Goals

Challenge: can we provide personalized step goals to increase physical activity behaviors?

Patients are challenged with daily ‘step goals’ to increase physical activity

61

What does the data tell us:

Subgroup Analysis: Personalization of Step Goals

62

Historical steps recorded, segmented by age/BMI

Assign each participant a personalized step goal based on: similar age/BMI historical mean + 20%

Subgroup Analysis: Personalization of Step Goals

63

Experiment: 50% receive ‘personalized steps’, 50% receive static (7500/day)

Personalized

Static (7500)

Step

s Tra

cked

/ W

eek

Program Week Program Week

Subgroup Analysis: Personalization of Step Goals

64

Experiment: 50% receive ‘personalized steps’, 50% receive static (7500/day)

Adaptive goals may be more impactful for younger [18-30] males

PersonalizedStatic (7500)

Step

s Tra

cked

/ W

eek

Program Week Program Week

Subgroup Analysis: Personalization of Step Goals

65

PersonalizedStatic (7500)

Step

s Tra

cked

/ W

eek

Program Week Program Week

No apparent effect in middle-aged adults

Experiment: 50% receive ‘personalized steps’, 50% receive static (7500/day)Subgroup Analysis: Personalization of Step Goals

Personalization through ExperimentationSubgroup Analysis

Answers questions like “Do older people respond differently to Tx than younger?”, or “Do males respond better than females?”

Broad cuts lose resolution to subtle changes

Guided by heuristics (e.g. demographics) and variables available – can lead to biases ITT

Age > 65

Uplift Modeling• Statistical model, based on experimental

data• Predicts who the intervention is likely to

be most impactful for• Don’t need heuristics (other than

choosing inputs) – let data do the personalization for you

Personalization through ExperimentationSubgroup Analysis

Answers questions like “Do older people respond differently to Tx than younger?”, or “Do males respond better than females?”

Broad cuts lose resolution to subtle changes

Guided by heuristics (e.g. demographics) and variables available – limiting and can lead to biases ITT

Age > 65

Uplift Modeling• Statistical model, based on experimental

data• Predicts who the intervention is likely to

be most impactful for• Don’t need heuristics (other than

choosing inputs) – let data do personalization for you

Experiment: Health coach feedback on participant food tracking

Context:• Building awareness of nutrition through food tracking is a

core part of the Prevent program• Ran experiment to understand the effects of coach feedback

on participant food tracking• Participants in the experiment received feedback about their

food choices from their coaches, those in the control arm did not.

Experiment: Providing impactful food feedback

Experimental design

RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weigh gain – please provide feedback

Experiment: Providing impactful food feedback

1. Candidate Selection• Participants who track a meal & predicted to gain weight

2. Randomization• 50% are eligible to receive feedback

3. Prescription Generation• Coaches are notified of tracking behavior, with suggested

feedback4. Intervention Delivery

• Coach reaches out to participant with feedback on the flagged meal

5. Monitor outcomes

RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weigh gain – please provide feedback

Experiment: Providing impactful food feedback

1. Candidate Selection• Participants who track a meal & predicted to gain weight

2. Randomization• 50% are eligible to receive feedback

3. Prescription Generation• Coaches are notified of tracking behavior, with suggested

feedback4. Intervention Delivery

• Coach reaches out to participant with feedback on the flagged meal

5. Monitor outcomes

Experimental design

RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weight gain – please provide feedback

Experiment: Providing impactful food feedback

1. Candidate Selection• Participants who track a meal & predicted to gain weight

2. Randomization• 50% are eligible to receive feedback (other 50% in ‘control’)

3. Prescription Generation• Coaches are notified of tracking behavior, with suggested

feedback4. Intervention Delivery

• Coach reaches out to participant with feedback on the flagged meal

5. Monitor outcomes

Experimental design

1. Candidate Selection• Participants who track a meal & predicted to gain weight

2. Randomization• 50% are eligible to receive feedback

3. Prescription Generation• Coaches are notified of tracking behavior, with suggested

feedback4. Intervention Delivery

• Coach reaches out to participant with feedback on the flagged meal

5. Monitor outcomes

Experiment: Providing impactful food feedback

RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weight gain – please provide feedback

Experimental design

RxParticipant X, has tracked protein shake for 5 days in a row and has shown recent weight gain – please provide feedback

Experiment: Providing impactful food feedback

1. Candidate Selection• Participants who track a meal & predicted to gain weight

2. Randomization• 50% are eligible to receive feedback

3. Prescription Generation• Coaches are notified of tracking behavior, with suggested

feedback4. Intervention Delivery

• Coach reaches out to participant with feedback on the flagged meal

5. Monitor outcomes

Experimental design

5. Monitor outcomes 10-15% increase in days with meals tracked

Experiment: Providing impactful food feedback

8-12% increase in meals with healthiness tracked5. Monitor outcomes

Experiment: Providing impactful food feedback

+8% in relative week-16 weight loss

5. Monitor outcomes

Experiment: Providing impactful food feedback

Using experimental data for personalizationUplift modeling, built on experimental data, can tell you probability of incremental response for an individual, given a hypothetical intervention

When delivering interventions:

• The Sure Things: response regardless of intervention• The Lost Causes: no response regardless of

intervention• The Sleeping Dogs: less likely to respond with

intervention• The Persuadables: only respond to because of

intervention

Uplift modeling targets The Persuadables and leaves the Sleeping Dogs alone.

Uplift modeling Example: Food Feedback

• Random forest uplift model trained on likelihood of participant continuing to track meals after receiving food feedback

• Model accounts for participant demographics, program behavior and response to previous feedback

• Preliminary results: targeting ~80% of eligible population with uplift model increases response ~2% (from 5% to 7%)

Outline

79

Omada Health• Context! Who are we? What do we do? Data science?

Building a Data Science Culture from the Ground Up• Winning hearts & minds with data

The Data Science / Product Relationship• What we’ve learned doesn’t work, and what we think works

The Paths to Personalized (Precision) Prevention• How to target the right patient, at the right time, in the right way

• Impactful experimentation: beyond the sub-group analysis

WHAT’S NEXT?

80

THANKS!@omadahealth

@sciencethedata@ryancquan

82