supporting innovation in insurance with randomized experimentation

Supporting Innovation in Insurance with Randomized Experimentation

Matt Best

Senior Data ScientistAllstate Insurance Company

DOMINO DATA SCIENCE POP-UP CHICAGO

Examples from Ronny Kohavi’s “Introduction to A/B Testing” at KDD ’17 and Blake and Coey 2014: Why Marketplace Experimentation is Harder than it Seems

But, it’s difficult to accurately forecast the impact of an innovation on customer experience

Randomized experiments are an effective method to learn the impact of innovation


Random sampling


• Randomized experiments are also sometimes referred to as randomized controlled trials, A/B/n tests, bucket tests, and field experiments depending upon discipline

Treatment Group

Control Group

Random sampling

Random assignment

CUSTOMERS CUSTOMERS

Web Mobile Service Rep AgencyWeb Mobile

Before an experiment, consider the importance of statistical power

How much data is needed to assess an innovation’s impact?

How large does the impact need to be for it to be detectable with a fixed quantity of data?

Ideally,

we’d ask …

In practice,

operational constraints often

shift the question to …

Note: All axes units are arbitrary to keep proprietary information confidential

Implied MDE

Fixed sample size


Optimistic estimateof impact


Implied MDE

Fixed sample size


New sample size


Implied MDE

Fixed sample size

A power analysis saved us from running a test with almost no chance of success!


New sample size


Implied MDE

Fixed sample size

Key takeaways before the experiment begins

• Lessons learned:• Need to be able to rapidly iterate on power/sample analysis and experimental

design as operational constraints are identified

• Observations are rarely independent and identically distributed; be explicit about sources of variability

• Technological solutions:• Using a knowledge management platform has enabled us to track the

evolution of assumptions through the design process

• Developed python package to verbosely describe and simulate progress through process flows

After an experiment, consider how cognitive biases influence decision making

Treatment Group

Control Group

Treatment Group

Control Group

Confirmation bias We look for and more strongly weigh information that confirms what we already believe

Look again…my hypothesis must

be true!

Treatment Group

Control Group

Hindsight bias After we see results, we tend to overestimate how well we would have predicted (or did predict) those results all along

That result was obvious! Why

run a test?

How to benefit from hindsight, prospectively?

• Pre-mortem: “Imagine your experiment has spectacularly failed –write the story of that failure.”

• Pre-register: “What would you do if we observe a {positive|negative|null} result?”

• Good decision ‘hygiene’ helps reveal critical risks, assumptions, and disagreements early on… while we can still do something about it!

Summary and Closing Thoughts

Randomized experimentation is a powerful tool data scientists may leverage to create value.

Though challenging, insurance firms may benefit from wider adoption of the methodology, even in situations where it’s operationally challenging.

Data scientists can enable experimentation by driving forward both technological and cultural solutions.

Thanks for your attention!

XD Team

• Anthony Pham

• Andrew Mehrmann

• Matthew McAuley

• Melissa Alvarado

• Nicholas Syring (intern)

BehavioralSight

• Linnea Gandhi

Allstate - D3

• Xiaoyan Anderson

• Neal Coleman

• Tony Eberle

• Florent Buisson

• Jason Khan

Domino

• Anna Anisin

• Jeremy Mason

Data and Analytics at Allstate: Our Centralized Organization

Managing and governing

data

Developing analytics

solutions

Effectively delivering solutions

through technology

250 data and analytics

experts

Who We Are

We have experts across five locations:

Silicon Valley, CA; Seattle, WA; Northbrook, IL;

Chicago, IL; Belfast, Northern Ireland

Data and analytics is embedded in

everything we do. Each day, Allstate uses

analytic models to create millions of

targeted digital media impressions, process

tens of thousands of claims, produce tens

of thousands of quotes, and predict

thousands of decision making actions

across the corporation.

Where We Work What We Do

Join Us for a Tour of the Allstate Office

Sign up before noon at:

Registration desk or Allstate booth

Tuesday, November 14

3:30 - 4:00 pm

Meet at the Allstate booth

supporting innovation in insurance with randomized experimentation

Data & Analytics