#evidence4impact

Associate Professor Moira Inkelas

Centre for Healthier Children, Families and Communities

University of California, Los Angeles

Moira Inkelas, PhD, MPH Associate Professor

UCLA Fielding School of Public Health, Department of Health Policy and Management

A learning system for improving family and community outcomes

1

Symposium: Evidence for Impact June 2017

3

Percent of children with cystic fibrosis who are below 5th percentile for weight and are receiving supplemental feedings

Source: Schechter MS & Margolis P. 2005. Improving subspecialty healthcare: Lessons from cystic fibrosis. Journal of Pediatrics.

100%

80%

60%

40%

20%

0%

Rates for 120 Centers of Excellence (ranked low to high)

Guideline/goal

Actual

Overall rate for 120 Centers of Excellence

Median Predicted Survival Age, 1994-2006

25

30

35

40

'94 '95 '96 '97 '98 '99 '00 '01 '02 '03 '04 '05 '06

Year

Predicted survival

improves from 29

years to 37 years

First reports reveal

significant

variability

Quality Improvement

starts

Predicted survival

improves from 28

years to 29 years

741 Lives

Source: GT O’Connor/Cystic Fibrosis Foundation

The Bell Curve

published

To achieve an outcome for a population, we are seeking solutions that…

…work at scale (do not break down when we

try it for everyone)

…will spread to others (all organizations

implement the change, not just the most “enlightened” organization)

…are sustained over time (do not degrade as

attention turns to other topics)

Source: Parry, Carson-Stevens, Luff, McPherson, Goldmann. Recommendations for evaluation of health care improvement initiatives. Academic Pediatrics. 2013;13:S23-S30.

AchievingEnduringImprovementbyWorkingasaSystem

2

I’msuregladtheholeisnotinour

end!

Every system is perfectly designed to achieve exactly the results it gets.

How is improving a system different from improving a program?

• Programs can be planned, implemented and evaluated.

• It is not possible to plan and specify each of the detailed actions necessary for a system to produce better results.

• Optimizing one part of a system does not optimize the overall system.

• Meddling with one part of a system often sets off other problems

• Community systems are complex and are never permanently “fixed”.

To change outcomes for a population, we need an approach that sets a heading but allows for adaptation and adjustment,

using testing to learn its way forward.

Separation - avoid crowding neighbors

Alignment - steer towards average heading of neighbors

Cohesion - steer towards average position of neighbors

“A learning system is designed to generate and apply the best evidence for the collaborative choices of each person and provider; to drive the process of discovery as a natural outgrowth of care; and to ensure innovation, quality, safety, and value in care.”

Institute of Medicine (IOM). Best Care at Lower Cost: The Path to Continuously Learning

Health Care in America. September 2012.

What are we trying to accomplish? By when?

How will we know that a change is an improvement?

What change can we make that will result in improvement?

Model for Improvement

Act Plan

Study Do

The Model for Improvement

Aim

Measures

Changes

Source: Provost L. Model for improvement: Aims, measures, changes. Associates in Process Improvement.

Some is not a number, soon is not a time, hope is not a plan.

Serve as trusted partners for

families

Identify social goals, needs,

priorities

Primary Drivers

Work as a single system to

achieve goals for a population

Population Care for Child Wellness: Collaborative Key Driver Diagram

Measures

Families reach outcome goals

Up-to-date risk assignment

Families achieve their personal goal

Families receive bundled care

By October 2017, each cross-partner team (site)

achieves family health and social goals for a shared population of at least 25

families

Learn and improve to

solve complex problems

Secondary Drivers Changes

GLOBAL AIM

Effective and accountable partnerships optimize

children’s health potential and family opportunity

SMART AIM

Revision date: 3/16/17

Elicit risks, assets, priorities

Inviting to families

Intra & inter agency workflows

Registry functionality

Use family well-being measure

Family agency; succeed w/ goal

Trusted relationships with families

Offer value, match services to families’ needs and priorities

Evolve services and resources to be effective and user-friendly

Frequent and transparent data for learning

Proactive, capable, accessible team

Services and staff preserve dignity

Learn families’ assets, risks and needs

Co-manage care between organizations

Segment population to reduce disparity, inequity

Bundle care for accountability

Person-centered process design (co-design)

Use the Model for Improvement

Sustained resources, investment & key stakeholders (ROI)

Mass customize & risk stratify

Communication workflows

Measures dashboard

Find and fix barriers

Same day access to team

Future-casting to anticipate & address needs, goals

Plan new services needed: build or partner

Pareto of care needs, & of desired experiences with care

Innovate with peers Network innovation groups

Common terms (care/wellness)

Personas to aid design

Single point of contact

Use small tests of change

“Hot oil! We need hot oil!.... Forget the water balloons!”

Changes that result in

improvement

Theories, hunches,

& best practices

A P

S D

A P

S D

A P

S D

A P

S D

Small Scale

More Testing

Test new conditions

Implement the change

Using plan-do-study-act (PDSA) cycles for sequential building of knowledge

Include a range of conditions in the sequence of tests, before implementing the change

Source: Associates in Process Improvement

Current Situation Not Ready Indifferent Ready

Low Confidence that current change idea will lead to Improvement

Cost of failure is

large

Cost of failure is

small

High Confidence that current change idea will lead to Improvement

Cost of failure is

large

Cost of failure is

small

Implement

Deciding the Scale of Testing

Readiness to Make the Change

Source: The Improvement Guide, Langley et al. 2009

Test small at the outset, when we know less, which make it easier to see cause and effect

“Houston, we’ve had a problem”

Carbon dioxide buildup

Power-up protocol INTEROPERABILITY

”We’re on number 8. You’re talking about number 692.”

”We gotta find a way to make this fit into a hole for that.”

REDUCE COMPONENTS

”Don’t give me anything they don’t have up there.”

MANAGE UNCERTAINTY, NOT TASKS

Apollo 13 flight path

Ambiguity

Mann Gulch, Montana 1947

24

How We Collaborate to Innovate

Extreme family orientation Put families at the center of care

Clarity of purpose Produce a coherent vision out of many problems

Solutions that scale Create solutions that customize to work for all, spread, and sustain

Bias toward action More “creating and doing”, than “meeting and planning”

Embrace experimentation and use of data for learning Build to think and learn

Embrace ambiguity Expect fog and take small steps to get unstuck

All contribute and take ownership Bring together partners with diverse roles and viewpoints

Sources: StartStrong Co-Creation Session, February 25, 2014 (Business Innovation Factory), and IDEO 24

Lucas B & Nacer H. The habits of an improver. The Health Foundation. 2015.

The goal of measurement is to drive a change

• Focus diverse partners on shared outcomes

• Establish shared accountability for reaching goal targets

• Shape an understanding of what matters and how to

influence it

• Build and maintain enthusiasm for improvement

• Enable partners to think and work as a system

• Support improvement with frequent, real-time

information

“Measure only what matters, and mainly for learning.”

Purpose of Measurement

Accountability Improvement Research

Key question “Are we better or worse than…?”

“Are we getting better?”

“What is the truth?”

Penalty for being wrong

Misdirected reward, penalty, resources

Misdirection for an initiative

Misdirection for the profession

Requirements and characteristics

Risk adjusted, with denominators,

validity

Real time, raw counts, consistent definitions,

utility

Complete, accurate, controlled, glacial pace, expensive

Typical displays Performance relative to benchmarks and

standards

Run charts, control charts, time between

events

Comparison of control and

experimental populations

Social conditions for use of measures

Neutrality; leaders and managers are the primary users

Data shared in low-stakes, safe

environment that is conducive to change

Meets scientific standards of

discipline; utility to participants is

usually secondary

Adapted from Solberg, Mosser, McDonald Jt Comm J Qual Improv. 1997 Mar;23(3):135-47.

!

29%

Health 7 31 9 35 24 15 15 22 19 31 9 8 5 - - - - - - - - - - -

Child care 20 - - - 33 39 40 40 - 40 - - - - - 23 30 - 20 37 8 17 11

Family support 51 50 57 68 70 44 60 59 33 51 22 65 19 19 28 23 28 14 25 14 37 25

Linkage orgs - - - - - - - - - - - - - 14 - 16 - - - - 16 9

Community 127 - 106 - - - 96 - 97 - - 86 - - - 158 - - 99 - - - - - 63

J A S O N D J F M A M J J A S O N D J F M A M J J

2011

Number of questionnaires per month

2012 2013

0%! 100%!

0%!

20%!

40%!

60%!

80%!

100%!

J! A! S!O!N!D! J! F!M!A!M!J! J! A! S!O!N!D! J! F!M!A!M!J! J!

2011! 2012! 2013!

J! A! S! O!N!D! J! F!M!A!M! J! J! A! S! O!N!D! J! F!M!A!M! J! J!

2011! 2012! 2013!

0%!

20%!

40%!

60%!

80%!

100%!

J! A! S!O!N!D! J! F!M!A!M!J! J! A! S!O!N!D! J! F!M!A!M!J! J!

2011! 2012! 2013!

J! A! S!O!N!D! J! F!M!A!M!J! J! A! S!O!N!D! J! F!M!A!M!J! J!

2011! 2012! 2013!

J! A! S! O!N!D! J! F!M!A!M! J! J! A! S! O!N!D! J! F!M!A!M! J! J!

2011! 2012! 2013!

0%!

20%!

40%!

60%!

80%!

100%!

J! A! S!O!N!D! J! F!M!A!M!J! J! A! S!O!N!D! J! F!M!A!M!J! J!

2011! 2012! 2013!

J! A! S! O!N!D! J! F!M!A!M! J! J! A! S! O!N!D! J! F!M!A!M! J! J!

2011! 2012! 2013!

J! A! S! O!N! D! J! F! M!A!M! J! J! A! S! O!N! D! J! F! M!A!M! J!

2011! 2012! 2013!

0%!

20%!

40%!

60%!

80%!

100%!

J! A! S! O!N!D! J! F!M!A!M! J! J! A! S! O!N!D! J! F!M!A!M! J! J!

2011! 2012! 2013!

J! A! S! O!N!D! J! F!M!A!M! J! J! A! S! O!N!D! J! F!M!A!M! J! J!

2011! 2012! 2013!

% receiving care in this system

% reached by network child care

% reached by network doctor

0%!

20%!

40%!

60%!

80%!

100%!

At l

east

1 n

eig

hbo

r w

ith w

hom

yo

u co

uld

dis

cuss

a

pers

ona

l pro

blem

Ca

n g

et

me

dic

al c

are

w

he

n n

ee

ded

Fle

xibl

e w

hen

life

does

n’t g

o as

pl

anne

d

0%!

20%!

40%!

60%!

80%!

100%!

Not$dep

ressed

$

Safe%places%for%

children%to%play%

Caregivers)see)

child)regularly)

Food$has$

not$run$out$

EDSI . EARLY

DEVELOPMENTAL

SCREENING

AND INTERVENTION

INITIATIVE

Early Developmental Screening

and Intervention InitiativeEDSI.

In doctor offices

In child care

In the community overall

In family support programs

Goal

In community hubs

16%! 12%! 13%! 15%! 12%! 20%!9%!

20%!16%! 20%! 20%! 19%!

65%! 72%! 68%! 65%! 70%!

Vulnerable At risk On track

Developmental progress at school entry

Conditions of families

Home routines and health behaviors

Measures of real-time improvement in services and supports

Family and community conditions

Reading proficiency, third grade

Potential and actual reach to children in the community

0%

20%

40%

60%

80%

100%

Health Childcare TotalFamilySupport

FamilySupport

FamilySupport

FamilySupport

FamilySupport

Discussedresourcesforsocialsupport

0%

20%

40%

60%

80%

100%

Health Childcare TotalFamilySupport

FamilySupport

FamilySupport

FamilySupport

FamilySupport

DiscussedresourcesforsocialsupportGoal

target

Family support

Opportunities for learning within sectors

“The provider/staff shared with me local resources for social support”

Opportunities for learning across sectors

Measurement for Learning

0%

20%

40%

60%

80%

100%

Health Childcare TotalFamilySupport

FamilySupport

FamilySupport

FamilySupport

FamilySupport

Discussedresourcesforsocialsupport

Family support

Partner A

Family support

Partner B

Family support

Partner C

Family support

Partner D

1

2

3

4

5

6

CBSC

CNI

Expo

FamSourceStJ

DCFS

FamSourceCB

StJohns

CSSD

DPSS

WIC

Echo

211

Rightway

Esperanza

FEP

CrystalStairs

CFRC

PubCounsel

1736

BestStart

PACE

PanAm

LACCEO

StThom

as

Cam

ino

LIFT

Pathways

MagElem

HopeSt

Leopoli

Lanterm

an

Hoover

NAC

WelcomeBaby

USC_FM

Redshield

Allpeople

SAJE

Norm

andie

Toberman

Angelica

Jumpstart

KYCC

Kidw

atch

CII

DMH

GrandasParents

LAPerinatalM

entalH

ealth

MPFamilyCtr

Redeemer

YPI

Familiaritywithservices/supportsofferedbypartnersKnowquite

abit

Knowli le

20132014

1

2

3

4

5

6

StJohns

FEP

KYCC

SAJE

Cam

ino

Rightway

Kidw

atch

DPSS

LACCEO

NAC

Redshield

Hoover

PanA

m

PACE

WelcomeBaby

Lanterman

FamSourceStJ

211

Jumpstart

Angelica

PubCounsel

1736

CII

GrandasParents

MPFamilyCtr

YPI

Easeoflinkage

20132014

It’sveryeasy

It’snoteasy

1

2

3

4

5

6

StJohns

KYCC

CSSD

Esperanza

FEP

SAJE

StThom

as

LIFT

Cam

ino

Rightway

Kidw

atch

WIC

Redshield

CBSC

USC_FM

Echo

211

NAC

Normandie

WelcomeBaby

CNI

PanA

m

DPSS

Hoover

PACE

Lanterm

an

HopeSt

LACCEO

CrystalStairs

DCFS

Pathways

Jumpstart

Allpeople

MagElem

FamSourceStJ

CFRC

Expo

FamSourceCB

Leopoli

Angelica

PubCounsel

Toberm

an

1736

BestStart

CII

DMH

GrandasParents

LAPerinatalM

entalH

ealth

MPFamilyCtr

Redeemer

YPI

FrequencyoflinkageRou nely

Never

20132014

Know quite a bit

Know little

It’s very easy

It’s not easy

Routinely

Never

Measuring Experiences in a Process

60% 50% 80% 40%

1% Provider asks if

parent has any

concerns

Parent

shares their

concern

Provider probes

and evaluates

concern based

on history and

home context

Provider & parent

plan together how

to address the

concern

What proportion of parents with a concern about their child leave a visit with an written idea about how they can address

the concern?

Provider reviews

information about the

child before the visit

Provider refers

to case manager

to develop plan

100%

50%

Provider

checks if parent

understands

the plan

25%

Plan is written

down for the

parent and

documented in

the record

__%

“Once this organization implements a change, the change tends to stick.” 53% “Most people in this organization are willing to change how they do things in response to feedback from others.”

55%

“When people in this organization experience a problem, they make a serious effort to figure out what’s really going on.”

70%

36

Average of 12 data points before and after a change

0

10

20

30

40

50

60

70

80

Avg Before

Change

Avg After

Change

Cy

cle

Tim

e (m

in.)

0

10

20

30

40

50

60

70

80

90

100

da

te

Jan

Feb

Ma

r

Ap

r

Ma

y

Ju

n

Ju

l

Au

g

Sep

Oct

Nov

Dec

Change

Made

Cycle

Tim

e (

min

.)

0

10

20

30

40

50

60

70

80

90

100

date

Ja

n

Feb

Mar

Ap

r

May

Ju

n

Ju

l

Au

g

Sep

Oct

Nov

Dec

Change

Made

Cy

cle

Tim

e (

min

.)

0

10

20

30

40

50

60

70

80

90

100

da

te

Ja

n

Feb

Ma

r

Ap

r

Ma

y

Ju

n

Ju

l

Au

g

Sep

Oct

No

v

Dec

Change

Made

Cy

cle

Tim

e (m

in.)

What is our confidence that the change led to an improvement?

Displaying Measures for Learning

Source: The Improvement Guide, Langley et al. 2009

% of clients reporting being asked about depression

% who discussed local resources for social support

0%

20%

40%

60%

80%

100%

1234567891011121314151617181920212223242526272829303132333435363738394041

0%

20%

40%

60%

80%

100%

1234567891011121314151617181920212223242526272829303132333435363738394041

0%

20%

40%

60%

80%

100%

1234567891011121314151617181920212223242526272829303132333435363738394041

0%

20%

40%

60%

80%

100%

1234567891011121314151617181920212223242526272829303132333435363738394041

“If we are not the best, we can certainly be the best at getting better, and then we will be the best.”

Summary

• A learning system enables people to:

– Build and maintain enthusiasm for changes;

– Think and work as a system;

– Plan collective actions around shared outcomes;

– Learn how to design and implement small tests of

change, to be more successful with improvements;

– Use co-design, testing and prototyping to learn how to

scale, spread and sustain what works