369 diane bartoli

Analytics to Drive Effective Product Development

Diane Bartoli

Vice President, Product Development

SSP Boston, June 1, 2011

About Me

Vice President of Product Management

Elsevier Health Sciences

Clinical Reference Group

Analytics EXPERT

NOT*

*(Just a product manager desperate for data to inform good product development...)

2

With seemingly endless data, is the world not your...?

3

A few of the tools at your disposal...

4

WHICH TO USE???

The trick is not capturing the data, it’s asking the right questions...

Case Study: Elsevier Evergreen

A new Online Medical Reference product in development intended for Doctors in

the Hospital setting

6

Where is the need? Understanding the user

• Ethnographic research brings end-users to life• Highlights pain points• Understanding motivations and ambitions

What is needed and WHEN? Mapping workflow…

1Diagnosis

2

Create Care Plan

Medical treatment Surgical treatment

Procedures

After treatment care plan

Patient Education & Compliance

Sharing information

•Patient specific information•Case presentation (within dept)•Presentation to other bodies (e.g. Treatment committees, drug committees etc)•External presentations (Grand rounds, association presentations etc).

2a 2b

3

4

Non-surgical Surgical

Keeping current

•Conferences / society meetings•Reading current literature

Treatment

5

•Patient care management workflows take over 45% of the average physician's time

Patient care management workflows Collateral workflows

•Collateral workflows take 24% of the average physician’s time

Source: Qualitative research (76 in person respondents)

99

Key Findings

Quantifying qualitative findings....

Daily activity

Weekly activity

Weekly/monthly

Monthly activity

Annual activity

% of time medical reference products are requested

To

tal

tim

e s

pen

t (h

rs)

• Validation of critical information needs for physicians: trustworthy, current content

• Validation of critical pain points: lack of comprehensive access to information, clinical search relevance not optimal today

• Quantification of physician workflow: Understanding what they do, how often they do it, how important those activities are, when they need information and what information they need

• Business justification: Sizing market opportunity and potential growth

No current offering delivers on all three components doctors need in a medical information product

Zeroing in on the goal…

Speed to AnswerSpeed to Answer

TrustedTrustedComprehensiveComprehensive

Wikis

Aggregators

Clinical Reviews

Online Journals

ElsevierEvergreen

Online Books

The Physician’s Dilemma

• Residents report significant use of Google (64%), Wikipedia (27%) and WebMD (36%) given need for fast and easy reference

• Regardless of where they start their online search, 40-53% of doctors use more than one source

• Despite generally preferring E over P, physicians find PRINT resources more trustworthy (75% vs 59%) than current E offerings

Building products with analytics at the core: Alpha, Beta....Zeta

11

Volunteered comments :

Capturing the voice of the user

Measuring the Right things: Composite metrics intended to get to “workflow”

Quantifying results: Quantitative

Survey at close of Alpha Period

Elsevier Evergreen

Understanding Motivation: Why are you here? Are

you satisfied?

Lots of Data.... What to do with it?

System Data:• What are they looking for? Search logs help tune

search relevance and recall

Usage Data:• What was used? What was not used?• Are users coming back? • Are we inspiring loyalty and engaging our users?

Satisfaction:• Why are they coming to our site?• Did they really find what they were looking for?• Touchy-feely: visual design? Cultural differences?

Quantifying results:• Prioritization of development• Directing go-to-market planning• Return on investment?

System Data:• What are they looking for? Search logs help tune

search relevance and recall

Usage Data:• What was used? What was not used?• Are users coming back? • Are we inspiring loyalty and engaging our users?

Satisfaction:• Why are they coming to our site?• Did they really find what they were looking for?• Touchy-feely: visual design? Cultural differences?

Quantifying results:• Prioritization of development• Directing go-to-market planning• Return on investment?

Some lessons

• Setting appropriate expectations• Imperfect data yields imperfect results

• Apples, oranges, and some pears too: matching mis-matched data

• The Holy Grail: marrying customer demographics to usage and conversion

13

More questions?

Diane Bartoli,

VP, Product Development

D.bartoli@elsevier.com

215-239-3602

14