Providing Treatment, Restoring Hope

Program Evaluation and ImprovementUsing Small Tests Of Change

Kristen A. Stafford, MPH

Pat Bass, RN, MA

Track 1.0 MeetingSeptember 25, 2007

Slide 2

“Any road will do if you don’t know where you are going”

- Lewis G. Carroll Alice in Wonderland

Slide 3

Slide 4

Slide 5

Objective

So they’ve collected the data…

…what now?

A look at how our local partners are using data to inform and improve

care and treatment

Slide 6

Components of our Process

Patient Outcomes Data QualityCare DeliverySystem

S

T

O

C

Slide 7

What is the STOC Process

SeeWhat is your data telling

youWhat is your goal

Try/T

rackW

hat will you try?

Who w

ill do it?

When

How

will yo

u track?

ObserveWhat happened?

Did it work?What were the balancing

results?

Con

tinue

Do

it on

a la

rge

r sc

ale

or t

ry s

omet

hing

el

se.

Slide 8

What is it

Intended to speed up system improvements

Evidenced based management Site driven Incorporated into daily routine Consistent and repeated reviews of

information already captured

Slide 9

Why do it

To build the monitoring and evaluation capacity of in-country teams and treatment sites to provide a sustainable system of quality care and treatment

To work ourselves out of a job

Slide 10

Steps to STOC

Leadership buy in Engage all levels of care delivery and clinic

teams Incorporate STOC activities into day to day

duties – not extra work Try something small If it works, keep doing it…if it doesn’t try

something else Keep track of what you try and what

happened Share, share, share now

Slide 11

How to tell if its an improvement

% w

ith u

ndet

ecta

ble

Vira

l Loa

d

Months

Started adherence support group on clinic days

Goal/target

Slide 12

Patient Outcomes

On treatment cross-sectional review Randomized population of focus

Started on tx 9 – 15 months before review Chart abstraction Patient Survey Viral Load Aggregated and by treatment site analysis Group and site by site feedback Selection of indicators most related to viral

suppression and failure STOC development with sites Bi-annual to annual

Slide 13

Patient Outcome Tools

Slide 14

Example of Findings

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Disclosed Not Disclosed

Ad

here

nce

Statistically Significant p<.05

Slide 15

Small Test of Change Example

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Baseline Week 1 Week 2 Week 3 Week 4

% o

f C

hild

ren

Lo

cate

d D

uri

ng

H

om

e V

isit

s

Attached Unique Identifier to Match Children to their Guardians

Slide 16

Building Local Capacity: ARV Pickup

Tracking ARV pickup in Kenya Goal: To improve patient adherence to picking up ARVs Strategy: Automated reports were created in IQTools:

1) Identify patients that are supposed to come in during a certain time period (i.e. this week) to pick up ARVs

2) Identify patients that have missed picking up their ARVs after x days

Reports are run and analyzed by LPTF LPTFs are required to report (monthly) the number of

patients that did not pick up their ARVs within x days and feedback is shared

Slide 17

Pharmacy Visits

Average Number of Patients Without ARVs for 30/20 days, August 2007

3

12

18

24

19 17

0

5

10

15

20

25

30

Ma

r 2

00

7(3

0 d

ays

)

Ap

r 2

00

7(3

0 d

ays

)

Ma

y 2

00

7(3

0 d

ays

)

Jun

20

07

(20

da

ys)

Jul 2

00

7(2

0 d

ays

)

Au

g 2

00

7(2

0 d

ays

)

Month (Number of Days Missed)

Ave

rag

e N

um

ber

of

Pat

ien

ts W

ith

ou

t A

RV

s fo

r x

day

s

Slide 18

How can LPTF use the data?

Based on the data, the local Clinical, SI, and Management Team work with LPTFs to develop a small test of change process to improve patient adherence to picking up ARVs

Since this process was implemented, LPTFs have had positive outcomes in the number of patients picking up their ARVs within x days

Best practices shared across project

Slide 19

STOC Plan from LPTFs

The monthly analysis of missed ARV appointments can point to problems in various areas: Data entry backlog/data entry errors Poor communication True defaulters

Slide 20

Data entry backlog/data entry errors

Problem Defined: Computerizing patient data is long and data entry errors leading to patients erroneously appearing as defaulters.

One STOC Plan (Mombasa): 76 patients late in May but only 9 by June; patients in May

were not true defaulters. The problem was due to an accumulation of backlog

Put a plan in place to use additional temporary data clerks to clear their backlog. [Temp staff must be trained in PMM system]

Eliminated backlog and true default rate determined Implemented as solution when/if backlog arises

Slide 21

Data Quality

Scalable, robust, flexible and sustainable platforms for data collection and retrieval

Routine data cleaning and monitoring

Treatment site driven data use and analysis for adaptive management

Slide 22

Patient Monitoring and Management Systems: IQ Solutions

Strategy: To offer a library of tools and solutions built

around adaptive management, quality, and sustainability

Requirements developed through practical field experience and lessons learned

Collaborative approach using local experts throughout the development process

Current areas of focus PMM, ART Registers, Data Quality Tools

Slide 23

Summary

Quality evaluation and improvement activities are a vital function of program management

The ability to scale ARV treatment programs ultimately will be dependent on efficient and sustainable care.

Sustainable care is intimately tied to achieving consistent high levels of medical care.

Data is for more than data reporting

Changing the culture of data capture and use will ensure the most effective and sustainable use of this funding

Slide 24

Thank You