Making Sense of Scorecard and Indicator Chaos

November 26, 2012

11:30 am – 1:00 pm

Paula Blackstien-Hirsch

2

Overview of the Series

• Establish a quality committee, to report to the board on quality-related issues

• Develop an annual quality improvement plan:

• Overview of the quality landscape and components of a comprehensive Quality Plan

• Scorecards and Indicators

• High level aims, Action Plans, and Targets

• Risk Management

Overview for Today

• Indicators and scorecards: Where do these fit within the Quality Agenda/Landscape?

• What does ECFAA require of organizations?

• Completing the template

• HQO guidelines and advice

• Indicators for community support agencies

• Using measurement for improvement

• Guidance for developing an organizational Quality Scorecard

• Methods for making measurement meaningful

3

What role do you fulfill for your community agency?

1. Board Member

2. Executive Director/CEO

3. Senior Manager

4. Staff Member

5. Other

4

5

Organizational Quality Agenda

5

v

Webinar 2

Webinar 2

Webinar 3

Webinar 3

Webinar 3

Is my organization’s Balanced Scorecard the same as a Quality and Safety Scorecard? (“maybe but not necessarily…”)

6

Corporate Balanced Scorecard

Quality & Safety

Scorecard

What should be the dimensions of the Q &S Scorecard?

Frequently divided into 4 dimensions defined by Kaplan & Norton: Internal Processes, Learning & Growth, Finance, Customer Satisfaction

. _____

. _____

. _____

. _____

. _____

. _____

. _____

. _____

What should be the indicators and targets? Subset from BSC

. _____

. _____

. _____

. _____

. _____

. _____

. _____

. _____

Plus Other Indicators

What type of indicators might be in a Corporate BSC that don’t necessarily fit in a Q & S?

• Human Resource

• Research

• Communications or IT unrelated to quality of care

• Financial indicators that relate to the entire organization (eg balanced budget, accounts receivable)

• Learning and growth (forging of partnerships, investments in training, etc)

7

Quality Dimensions Used Across Canada and the US

8

BC

Health

Quality

Matrix

Alberta

Quality

Matrix

Health

Quality

Ontario

Quality

Framework

Excellent

Care for

All Act –

Ontario

New

Brunswick

Quality

Framework*

Nova Scotia

Quality &

Safety

Framework

for Capital

Health

Triple

Aim –

IHI

Cancer

Care

Ontario

Institute

of

Medicine

Accreditation

Canada

Safe x x x X x x x X x

Effective x x x x x x x X x

Accessible x x x x x x x X x

Appropriate x x x

Efficient x x x x x X x

Patient-

centred/

Acceptable/

Responsive

x x x x x x x X x

Equitable x x x X

Acceptable

Integrated/

Continuity

x x x x x

Appropriately

Resourced

x

Population

Focused

x x x x

Work Life x x

Cost x

9/10

3/10

7/10

9/10

4/10

0/10

5/10

1/10

4/10

1/10

2/10

9/10

9/10

9/10

What does ECFAA require of organizations?

9

10

ECFAA Scorecard for Quality and Safety

Quality

dimension Objective Measure/Indicator

Current

performan

ce

Target for

2012/13

Target

justificati

on

Priorit

y level

Safety

Effectiveness

Access

Patient-centred

Integrated

Indicators Directional Statement re: Indicator

Baseline Performance

Outcome Indicator

Target

Target Rationale

Priority Dimensions

Reduce or increase ____

Full operational definition

Relevant Sections of ECFAA Legislation: Section 8, Quality Improvement Plans

• In every fiscal year, every health care organization shall develop a quality improvement plan for the next fiscal year and make the quality improvement plan available to the public

• The QIP must be developed having regard to at least the following:

• The results of surveys

• Data relating to the patient relations process

• Aggregated critical incident data as compiled based on disclosures of critical incidents pursuant to the Public Hospitals Act

• Any factors provided for in the regulations

11

Client Experience

Safety

Relevant Sections of ECFAA Legislation: Section 8, Quality Improvement Plans (cont)

• Content: The QIP must contain, at a minimum:

• Annual performance improvement targets and the justification for those targets

• Information concerning the manner in and extent to which health care organization executive compensation is linked to achievement of those targets, and

• Anything else provided for in the regulations.

12

So….what might some appropriate indicators be…. and

are there any guidelines for developing targets?

Relevant Sections of HQO Guidelines

In completing the QIP, organizations should be guided by the following principles:

• Select at least one core indicator from each dimension (as applicable) (Note that this is a recommendation only)

• Select any other core indicators as applicable to the organization

• Add additional indicators as relevant to the organization

• Prioritize indicators and select which ones will have improvement initiatives (i.e. Priority level 1)

• Priority 1 or 2: performance is below organizational goal; if priority 1, should be aligned with strategic priorities, and in areas where quality problems are most frequent and for which the consequences are most serious

• Priority 3: consistent with organizational goals and/or at or near theoretical best

13

First let’s define different types of indicators:

• Outcome Measures

• Answer “so what?”

• What patients and payers care about

• Some will be organization-specific (within the control of a single organization); others will be paired (contributed to by more than one sector) • Example: Average/Median # days on a wait list

• A “balancing measure” is an outcome measure intended to ensure that there are no unintended consequences (example: if you decrease length of stay on service, is there an increase in visits by these clients to the ED?)

• Process Measures

• Provide information about the extent to which a practice/intervention has been implemented • Example: % clients screened within ___ days for risk of _______________

14

Current MSAA Indicators relative to ECFAA dimensions: General indicators • Balanced Budget

• Proportion of Budget Spent on Administration

• Variance Forecast to Actual Expenditures

• Variance Forecast to Actual Units of Service

15

Effective?

No fit; financial/utilization focus

Human Resource; Effective? No fit with current dimensions; future Value dimension?

Integrated

• % Total Margin

• Cost per Unit of Service

• Cost per Individual Served

• Turnover Rate

• % ALC Days

• Repeat Unplanned ED Visits within 30 Days (MH Conditions)

• Repeat Unplanned ED Visits within 30 days (Substance Abuse)

CHC:

• Cervical Cancer Screening

• Colorectal Cancer Screening

• Inter-professional Diabetic Care

• Influenza Vaccination Rate

• Breast Cancer Screening Rate

• Periodic Health Exam

• Clients Hospitalized for ACSC

• Vacancy Rate for NPs and Physicians

16

Current MSAA Indicators relative to ECFAA dimensions: Agency-specific Indicators

Home Health:

• Average # Days on Wait List

• # People Waiting for Service

Effective

Access

Residential:

• # Days from Referral to Assessment

• % Clients Satisfied with Services Received

Access

Client Satis.

17

Where might you look for indicators?

And….other sector-relevant reports/scorecards….provincial, national, international

18

Quality

dimension Objective Measure/Indicator

Current

performan

ce

Target for

2012/13

Target

justificati

on

Priorit

y level

Safety

Effectiveness

Access

Client-centred

Integrated

Space for additional

ECFAA INDICATORS for COMMUNITY

19

Safe Effective Accessible Client Experience Integrated

# Adverse Events (incidents resulting in harm) (Falls, Pressure Ulcers)

Link to ROPs for CSS

[# Staff Injuries Resulting in Absenteeism]

Cervical Cancer Screening

Colorectal Cancer Screening

Inter-professional Diabetic Care

Influenza Vaccination Rate

Breast Cancer Screening Rate

Periodic Health Exam

Clients Hospitalized for ACSC

% Unplanned Hospitalizations for Clients on Service

% Clients with High/Very High MAPLe Scores Supported by CSS in the Community

% Clients with Stable or Improved Functional Ability

# Days from Referral to Assessment

Average # Days on Wait List

# People Waiting for Service

90th Percentile Wait Time from Hospital to CSS Service Initiation

Time from Referral to CSS First Visit

% Clients Satisfied with Services Received

Questions related to:

• Provider Continuity

• Engagement of Client/Family in Goal Setting

• Overall Satisfaction

• Satisfaction with Transition Planning

% ALC Days

Repeat Unplanned ED Visits within 30 Days (MH Conditions)

Repeat Unplanned ED Visits within 30 days (Substance Abuse)

% Unplanned Hospitalizations for Clients on Service

Readmissions to Hospital for CSS Clients within 30 Days of Discharge

Repeat ED Visits for Clients Classified as CTAS 4, 5

Additional indicators that might be considered for the sector…

Are there 1-2 additional indicators that you use for one of the dimensions (below) that you could share as potential future indicators?

20

Safe Effective Accessible Client Experience Integrated

Please type your dimension and the indicator into the chat box…..

Potentially New Dimension….Value

• May be future indicators linked to Funding Reform

• Indicators:

• Current: Average cost of service per client episode (stratified by client type)

• ? Percent client reduction in need for CSS beyond ____ wks/mos

• ? Tied specifically to clinical quality groups and/or specific initiatives/guidelines

21

22

• Based on: • Organization’s own experience • Other organizations in the top 10th percentile • Best in class • Theoretical best (e.g. “0 defects”) • “half life” – 50% increases/decreases over a few

years • Benchmarks in the literature

Setting Indicator Targets…. as much art as science

• Key is finding a balance between an inspirational “stretch goal” and ensuring staff are not demoralized by a target that is not within reach

• Tie to executive compensation has featured into the targets set by hospitals to date

• Targets linked to compensation have tended to be attached to process measures which are easier to impact

• Key HQO comments after the first year related to the very conservative targets set by hospitals

• Rule of Thumb for targets for Process Measures (85-100%)

Second analysis by HQO reflecting on QIPs submitted on April, 2012

• Analysis highlights the following areas:

• Progress achieved – section included on each of the indicators

• Stronger overall than prior year in terms of completeness and robustness of change plans

• Priority setting - number of priorities selected and by type

• On average: 4-5 Priority 1; 4 Priority 2; 3 Priority 3 (Note: reflects project level and not high level aims)

• Target setting – extent to which organizations set stretch targets

• Better than last year, but still considered an issue

• Change plans – types of change ideas and whether they are sufficiently strong/detailed

• Stronger than last year, but HQO desires more detail to make sharing meaningful

23

24

Example of Indicator Level Information in the HQO QIP Analysis

November 2012, QIP Analysis, HQO Website

Potentially useful information once the indicators are more relevant to the community sector

25

Using measurement for improvement:

Guidance for an Organizational Quality Scorecard Methods for Making Measurement Meaningful

26

Principles for Selecting Indicators

• Rule of thumb….ideally no more than 20-25….maximum should be 30 (Niven, 2002)

• Include indicators where there is a gap between current and target

• Focus on outcomes for Scorecard….we’ll talk about Process Indicators next week

• Actionable

• Balance across dimensions (don’t want to optimize in one area to the detriment of other areas)

• Align with:

• Strategy

• High level aims

• Provincial/LHIN priorities

• Other key external accountabilities

27

Keep these to a minimum; Include if outside of acceptable/desired range, But otherwise ensure someone is reviewing…include only if outside range

28

Remove the denominator!!!

vs

ED Visits for Identified Conditions (ACSC): 14.6/1000 population

OR

25% clients with falls in last 90 days

300 ED Visits last month for Identified ACSC’s

OR

10 clients fell last month, 3 resulting in harm

29

Key questions for Board members to ask…

• Are our services getting better?

• Are we on track to achieve our key quality

and safety objectives?

• If not, why not?

• Is the strategy wrong?

• Is the strategy insufficient?

• Is it not being executed effectively?

• Is it too early in the process?

• How much variation is there among our providers?

• How good are our services?

• How do we compare to others like us?

• How much of a gap is there between our

current and desired performance?

• Why does the gap exist and what is our

plan to close the gap?

• What can we learn from others?

30 30

The standard for measurement in quality improvement is run charts or control charts…

“Before” and “After” data points demonstrate 110% improvement, from a satisfaction score of 33% satisfied to 70% satisfied!

Bed turns

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Before After

Client Satisfaction Would you say that:

a) These data demonstrate significant improvement b) There is not enough change to be sure the change is significant

c) Not sure

31 31

LHIN Name Site Name, Ward Name

Bed Turns

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1 2 3 4 1 2 3 4 1 2 3 4 5 1 2 3 4 1 2 3 4 1 2 3 4 5 1 2 3 4 1 2 3 4 1 2 3 4 5 1 2 3 4 1 2 3 4 1 2 3 4 5 1 2 3 4 1 2 3 4 1 2 3

Apr

2008

May June July Aug Sept Oct Nov Dec Jan

2009

Feb Mar Apr May June

Month

Be

d T

urn

s (#

se

par

ati

on

s/#

be

ds)

Bed Turns Baseline Median median

Test

#1

The standard for measurement in quality improvement is run charts or control charts…

Bed turns

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Before After

“Before” and “After” data points demonstrate 110% improvement, from a satisfaction score of 33% satisfied to 70% satisfied!

Client Satisfaction Client Satisfaction

Here is a run chart. Would you say that this shows: a) No change b) Significant change c) Not sure

32

LHIN Name Site Name, Ward Name

Bed Turns

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1 2 3 4 1 2 3 4 1 2 3 4 5 1 2 3 4 1 2 3 4 1 2 3 4 5 1 2 3 4 1 2 3 4 1 2 3 4 5 1 2 3 4 1 2 3 4 1 2 3 4 5 1 2 3 4 1 2 3 4 1 2 3

Apr

2008

May June July Aug Sept Oct Nov Dec Jan

2009

Feb Mar Apr May June

Month

Be

d T

urn

s (#

se

par

ati

on

s/#

be

ds)

Bed Turns Baseline Median median

Test

#1

The standard for measurement in quality improvement is run charts or control charts…

This run chart demonstrates that these two data points are part of normal random variation.

Bed turns

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Before After

“Before” and “After” data points demonstrate 110% improvement, from a satisfaction score of 33% satisfied to 70% satisfied!

Client Satisfaction Client Satisfaction

This is the standard format for most Board scorecards

33

If you only look at data in tabular format, you will miss important information…

34

35

A scorecard /dashboard is a key tool for the Board

Big Dot Drivers

Strategic dashboard for initiative review

at the Quality Committee

36

37

Getting from here…… to here

Quality

dimension Objective Measure/Indicator

Current

performan

ce

Target for

2012/13

Target

justificati

on

Priorit

y level

Safety

Space for additional Effectiveness

Space for additional Access

Space for additional

Patient-centred Please choose the question that is relevant to your hospital:

From NRC Picker / HCAPHS: "Would you recommend this hospital to your friends and family?"

From NRC Picker: "Overall, how would you rate the care and services you received at the hospital?"

In-house survey (if available): provide the percent response to a summary question such as the

"Willingness of patients to recommend the hospital to friends or family" (Please list the question and

the range of possible responses when you return the QIP)

Space for additional Integrated

Space for additional

VAP rate per 1,000 ventilator days: the total number of newly diagnosed VAP cases in the ICU after

at least 48 hours of mechanical ventilation, divided by the number of ventilator days in that reporting

period, multiplied by 1,000 - Average for Jan-Dec. 2

Reduce clostridium

difficile associated

diseases (CDI)

CDI rate per 1,000 patient days: Number of patients newly diagnosed with hospital-acquired CDI,

divided by the number of patient days in that month, multiplied by 1,000 - Average for Jan-Dec. 2011,

consistent with publicly reportable patient safety data

Reduce incidence of

Ventilator Associated

Pnemonia (VAP)

HSMR: number of observed deaths/number of expected deaths x 100 - FY 2010/11, as of December

2011, CIHI

Reduce unecessary

time spent in acute

care

Percentage ALC days: Total number of inpatient days designated as ALC, divided by the total

number of inpatient days. Q2 2011/12, DAD, CIHI

Reduce wait times in

the ED

ER Wait times: 90th Percentile ER length of stay for Admitted patients. Q3 2011/12, NACRS, CIHI

Improve patient

satisfaction

Improve

organizational

financial health

Total Margin (consolidated): Percent by which total corporate (consolidated) revenues exceed or

fall short of total corporate (consolidated) expense, excluding the impact of facility amortization, in a

given year. Q3 2011/12, OHRS

Reduce unnecessary

deaths in hospitals

CHANGE

Planned improvement

initiatives (Change Ideas)

Methods and

process measure

s

Goal for change ideas

(2012/13)

Comments

1) 2) … N) 1) 2) … N) 1) 2) … N) 1) 2) … N) 1) 2) … N) 1) 2) … N) 1) 2) … N)