change starts here. the one about outcomes and indicators icpc national coordinating center this...

Change Starts Here.The One about Outcomes and

IndicatorsICPC National Coordinating Center

This material was prepared by CFMC (PM-4010-080 CO 2011), the Medicare Quality Improvement Organization for Colorado under contract with the Centers for Medicare &

Medicaid Services (CMS), an agency of the U.S. Department of Health and Human Services. The contents presented do not necessarily reflect CMS policy.

Recap: measurement for IC-4

1. Time series outcomes– Effect on root cause/driver– Success of the intervention

• Rates; scores; rating scales• Best-fit line or other signal indicating improvement• What to do about outcomes not well portrayed as time-series

2. Intervention implementation– Reach/dosage of an intervention– Who was affected?

• Counts• Rates among eligible population (offered, refused, completed)

Recap: suggested approach

1. Map out a detailed, community-level logic model of the intervention strategy.

2. Select and operationalize outcomes and processes from the logic model.

3. Develop and enforce the system for tracking implementation and outcome.

4. Effectively report time series data.

Who should participate in outcomes selection and measurement?

• Considerations– Who will implement the interventions?– Who will be collecting data?– Community stakeholders

• To whom will the observed changes will be meaningful?

• Resources brought by the QIO– Building capacity for outcome measurement

• Logic models• Data collection

– Technical assistance; analytic support• Data management and reporting• Detecting improvement (tests of trend, run/control charts)

Recap: logic model outcomes

Expected short-, medium-, and long-term changes and improvements• Short-term

– Specific improvements in the targeted driver or root cause

• Medium-term– Related outcomes along the causal path

• Long-term– Improved care transitions– Avoided readmission– Improved health care utilization

• Implications of potential negative changes or non-changes

Short-/medium-term outcomes are measured for IC-4.

Recap: outcomes selection

SMART objectives guide outcome selection.

SpecificConcrete; represents what, or who, is expected to change

Measureable Can be seen, heard, counted, etc.

Attainable Is likely to be achieved

Results-orientedGenerates meaningful, valued results

TimedHas an acceptable target date

Going further: attainability, moveabilty

Short-term outcomes are more likely to show movement…

…but consider downstream (medium-term) outcomes if short-term outcomes are not feasibly measured.

InterventionOutcome A

(short-term; proximal)

Outcome B Outcome CAvoided

readmission(long-term; distal)

Transitions coaching

Improved patient

activation

Improved health self-

management

Improved symptom control

Avoided readmission

Discharge process standardization

Improved patient knowledge, resources;

coordination of follow-up care

Fewer unmet needs and gaps in

care

Improved symptom control

Avoided readmission

Outputs Short-term outcomes --------------- Long-term outcomes

Indicators

What does the outcome look like? How would we know it improved?• Create operational definitions.

– Specifics of how the outcome is measured• Make it a number. Quantitative data can be plotted over time or compared

across groups.– Rate, percentage

• Numerator and denominator

– Score (continuous)• e.g., 0 to 100 points

– Rating scale (ordinal)• e.g., [0 - ‘never’] -- [1 - ‘sometimes’] -- [2 - ‘always’]

Simple counts are not very informative on their own. Find a way to use them as the numerator of a rate or percentage.

Timing and duration

When will improvement be detected?• Considerations

– How long should it take to observe an effect?– What should the effect look like?

• Abrupt, sustained improvement• Rises and falls, with gradual trend towards improvement

• IC-4: ≥4 quarters of data within 18 months of community engagement– Ensure that the measurement period includes pre-intervention baseline data.

• Measure frequently– The more data points, the better. Monthly indicators lend themselves to run/control

charts.

Level of analysis

Where will improvement be seen?• Provider-/initiative-level

– A change occurs across several events, people, or organizations.• Process improvements (re: standardization; information transfer)

– e.g., inpatient satisfaction with discharge information

• Utilization outcomes– e.g, 7-day readmission rates; 30-day primary care follow-up rates

• Patient-level– Something happens to an individual person related to his/her patient experience.

• Patient activation (e,g., change in PAM score)• Utilization (e.g. , prevented readmission)

• Patient-level data should be aggregated to provider-/initiative-level for reporting.

Collecting data

• Considerations– What method?

• Sampling• Survey, case review, etc.

– Who collects the data? Who analyzes it?– How will data quality be ensured?

• Make it explicit.– Standardization, data specifications– Accountability

• Don’t forget to track implementation (process) measures, as well.– Counts, rates among eligible population (offered, refused,

completed)

More to come

• Detecting and reporting improvement• Context and reasons for success/failure

Resources

• Toolkit – measurementhttp://www.cfmc.org/caretransitions/toolkit_measure.htm

• Measuring Program Outcomes: A Practical Approach

http://www.unitedwaystore.com/product/measuring_program_outcomes_a_practical_approach/program_film

Excerpts: http://www.unitedwayslo.org/ComImpacFund/10/Excerpts_Outcomes.pdf

• ICPCA NCC contact: Tom Venturatventura@coqio.sdps.org303-784-5766

Questions?

CO-ICPCTechnical@coqio.sdps.org

The ICPC National Coordinating Center – www.cfmc.org/caretransitions

Change Starts Here.