using patient feedback for creating and improving innovative programs

Using Patient Feedback for Creating and Improving Innovative Programs

by Doris Quinn, PkD Director, Quality Education and Management Vandevbilt University Clinic Nashville, TN

There are two goals that should be of paramount concern to clinicians today. With the current emphasis on cost control in healthcare, i t has become necessary to search for ways to (a) decrease the total burden of illness for costly conditions and (b) compile outcomes data to measure effectiveness and efficiency. This article describes an experimental work therapy program that seeks to address the needs of early-stage Alzheimer's disease and other dementia patients. Feedback from patients and caregivers was used to validate the benefits of the program activities in reducing the total burden of illness. These data also were used as a basis for improving program components. Once the program was well defined, patient and caregiver feedback measures were used to ensure that the program continued to meet their needs.

Introduction There is a general perception that lit-

tle can be done to relieve symptoms and improve quality of life when patients have a debilitating disease, such as Alzheimer's disease (AD). Once diag- nosed, such patients often feel alone in dealing with the depression, the anger, and the many losses associated with AD. Increasing these patients' quality of life by maximizing their remaining abilities is a new concept. The goal of any program for AD patients should be to defer the need for total nursing care, thus reducing financial and emotional costs.

Financial impact: The financial cost of this disease in 1988 was estimated at $24-40 billion (Cook-Desan, 1988). By the year 2000, Blass (1990) forecasts total costs (including indirect costs) to be more than $88 billion annually, which will make it the third most ex- pensive health condition in the Unit- ed States. Therefore, delaying institu- tionalization could have a significant impact on the families of AD patients and on society.

Adapted Work Services Program The Adapted Work Services (AWS)

program was begun in 1993 and was based on the concept of the sheltered workshop. The parent company, Rochelle Center of Nashville, TN, had a similar program for mentally retard- ed clients, and its program director thought that the concept could be used to help early-stage AD patients. He knew that these patients experienced significant, debilitating emotional problems, which lead to difficult behaviors

that added significantly to the burden of their devastating illness.

Maximizing remaining skills: The director hypothesized that if AD patients' many remaining skdls were max- imized, they would be able to remain productive and increase their feeling of self-worth, which would in turn de-

The goal of any program for AD patients should be to defer the need for total nursing care.

crease the stress on their families. The concept of the clients coming to "work" seemed a positive alternative to day care, especially in the disease's early stages. The director sought the input of experts on AD (including a member of the Nashville Council on Aging, a geropsychiatric nurse, a professor of psychology and human development specializing in the support of caregivers, an assistant professor of psychiatry specializing in geriatrics, and a social worker specializing in memory disorders), which resulted in the formation of an advisory group. This group was con- vinced that early-stage AD patients would benefit substantially from being in a work atmosphere in which they would be paid for performing repetitive but meaningful jobs. Staff would be available to support and train or re- train patients in a safe environment. The patient work group also would provide

26 JHQ . July/August 1996. VO~. 18, No. 4

a \‘enue for socialization and would Serve as a support to its members.

Determining a need for the program: Going from idea to reality requires planning, but even more importantly, it requires data. The first task was to veri- fy that the need and desire to work existed for this patient population. Fo- cus groups were conducted to evaluate whether the needs of the key stakeholders matched the perceived needs articulated by the program developers. (The key stakeholders in this endeav- or were the patients and their caregivers, as the burden falls most heavily on caregivers as the disease progresses.) The need seemed sufficient. A proposal to the Robert Wood Johnson Foundation (RWJF) was successful; funding was granted in November 1993 for 3 years, and a new concept became a program.

Determining the program’s goals: The program began with 8 patients, who were very positive about the program. But after nearly 2 years, referrals were very sparse, and the program director was having difficulty recruiting more participants. At this juncture, I was asked to conduct a formative evaluation study of the program. The study’s in- tent was to ensure that the program was indeed meeting the needs of the target population and to provide data for the referral sources, who still seemed skep- tical of the value of AWS. I began the study by formulating the following questions: What are the goals of the program? Are they meeting the needs of the patient and caregiver? What are the program activities? Do they relate to the goals, and do they need to be improved? Is there a program theory to guide the

choice of activities? Is the theory capable of linking activities to the program goals? What performance measures could be used to monitor the ongoing functioning of the program and the effect of the program on the participants? How could performance measures predict the likely success of a program that had never been tried before?

Using feedback data: Innovative programs can have difficulty in securing additional funding if there are no data to prove their effectiveness. The difficulty AWS faced was that there were too few patients to conduct an impact analysis, which would have been the best way to provide this evidence. The alternative was to conduct a formative evaluation of the program; that is, to an- alyze data from program activities and assess their validity with respect to the

Figure 1. Adapted Work Services, as Originally Conceived

Outcomes Structure Process

Workload: Marketing to: Participants: Self-esteem Need for help at Rochelle Center Plan for aging clients and parents

Referral sources (physicians) Families, churches, etc. Early-stage memory loss individuals

Decreased isolation, depression, anger, and anxiety

Community need: Work contracts Use of present abilities Increasing elderly population means increasing Increased quality of life (acceptance

incidence of dementia No services for early-stage dementia

Marketing plan: Referral sources Participants Caregivers

Existing model of sheltered workshop: Rochelle and VA Personnel with experience in sheltered workshops

Other community resources to support AWS

Facility: Building/space

Transportation

4 Work therapy tasks: Simple, repetitive Meaningful Paid

Socialization/support: Identification with peers Interaction with staff

Case Manaeement/Counselinr: Support groups for participank and caregivers Counseling for legal and financial planning Link to other agencies Information about and referrals to community

resources

and dignity) . Usefulness of doing meaningful work Maintained or increased self%are

Caregivers: Long-term caregiving plan Completion of legal and financial

Respite; reduction of stress More normal lifestyle Ability to keep employment

plan

Referral sources: Option to present to families Referrals from AWS

Society: Decreased total cost, because patients

Development of a program that can live at home longer

be replicated

Business customers: Quality of work at an affordable price

JHQ 1 July/August 1996 1 VO~. 18, NO. 4.27

Patient Feedback

patients (Scheirer, 1994). Formative evaluation relies heavily on feedback from the patient, caregivers, and staff to cre- ate an operational model of the program (a hgh-level flowchart), as well as a conceptual or theoretical model. These models then become blueprints for se- lecting measures that will provide data on the functioning of the program, as well as on the program’s effect on the patients and their caregivers.

Reviewing the program as originally conceived: We began to address the questions by first mapping out the program as it was intended to operate, using a framework of structure, process, and outcome. Agency documents such as the RWJF proposal, various talks giv- en by the director, and initial meeting notes were reviewed to put together a diagram of the program as originally conceived by the advisory group (Fig- ure 1). Interviews with the staff, the director, and the advisory board also were helpful in fleshing out this diagram.

Conducting interviews: The next step was to be certain that the identified outcomes or goals of the program were important to the patients and the caregvers. To obtain these data, I conducted interviews with patients. Although they sometimes had difficulty expressing themselves, the patients were articulate enough to be good sources of information. This was a significant lesson for the interviewers. Early-stage AD patients are stdl capable of expressing their needs and wants, yet often we treat them as if they are totally demented. The interviews were videotaped (with the patients’ consent) so that content analysis could be done. Focus groups with caregivers also were conducted. We believed that using focus groups would allow caregivers to support one another as they discussed the difficul- ties of dealing with a loved one afflict- ed with AD. A semistructured format was used in which specific questions were asked, although the caregivers were also allowed to speak freely about certain issues important to them. A professional facilitator was used for two reasons: (a) because the facilitator was not part of the program, he could help elicit complaints as well as praises for the program, and (b) the reliability and

validity of the information were enhanced by using a trained professional. These sessions also were videotaped.

Analyzing the data: The analysis of the focus group and interview data was accomplished using qualitative data analysis techniques (Miles & Huber- man, 1994). The focus group data and the interview data were compiled in a format that allowed codes to be assigned to various words or ideas (see Figure 2). Themes began to emerge from the data. Qualitative data analysis provided a mechanism for distilling what was im-

~~

The concept of the clients coming t o ”work” seemed a positive alternative to day care.

portant from many pages of narration. Patients’ needs that emerged were as follows: to feel productive, to continue to be part of society by going to work rather than to an agency that felt like baby-sitting, and to be in a safe environment where there was no need to cover up their forgetfulness. Important themes for caregivers included issues of support and respite from constant care, and the need to have their loved one feel useful and to be prepared for transitions.

Determining response to the program: The patients’ and caregivers’ overall response to the existing program was very positive, as evidenced from their willingness to be quoted in news releases or promotional information, as well as their comments during focus groups. As a result, the program’s de- signers felt the program was indeed meeting important needs of the AD patients and their caregivers. They needed to be sure, however, that the activities the patients engaged in were appropriate. The value of some tasks appeared unclear to the staff. The advisory group members also had differ- ing opinions as to what tasks the work group members should perform.

Clarifying the program: To clanfy the program, I facilitated, with staff mem-

bers’ help, the creation of a high-level flowchart, called the operational model (see Figure 3 ) . Staff members were asked to identify all the activities in the program. These activities were then clustered into program components, which were identified as intake, assessment, program, and transition/exit. This categorization was useful in that it helped identify ”holes” in the program. Four problem areas we discovered were (a) the lack of clear definition of the program’s target population-AD patients only, or anyone with memory deficit? (b) the lack of attention to marketing and cultivating new referral sources; (c) the absence of a mechanism to system- atically collect patient and caregiver feedback; (4) the lack of a mechanism for reporting to families and private physicians about participants’ behavior and productivity in the program.

Identifying program theory: The next question to be addressed was the identification of program theory. If we are to learn which activities cause which outcomes, it is important that a theory be made explicit. (Theory sometimes is implicit for each staff member/design- er of a program, but unless it is made explicit, it is not shared by all.) Deming often said in his seminars that “With- out theory there is no knowledge.”

The need to identify the program’s theory also was driven by the need eventually to show evidence of its effectiveness. To answer the question “What in the program contributes to its success?” the program staff and I cre- ated a conceptual model (Figure 4). The patients and caregivers already had provided data concerning their needs and their reaction to the program. It now was necessary to put the program into the broader context of societal need. Staff members knew that decreasing the total burden of illness was the ultimate desired outcome. This burden includes cost to families, lost wages of caregvers, the need for support or sitters, the fre- quent caregiver illnesses resulting from stress, and the eventual institutional- ization of patients. It was hypothesized that if patients can be kept productive, many of their problematic behaviors will be lessened, and the emotional burnout of caregivers will therefore de-

28 1 JHQ 1 July/August 1996 1 Vol. 18, NO. 4

Figure 2. Qualitative Data Analysis of Caregiver Focus Group Comments

Comments by Caregivers

We are bombarded with so much negativity about AD. AWS is positive. They helped me take steps to change housing.

Fee of $20-25 is low for this kind of service; it’s cheaper than other programs

My mother‘s paycheck pays costs for one week of the month

They would want to keep this program even if the cost went up.

My sister would not come if she knew I was paying a fee. My father would also resist if he knew I paid for him to come.

Mother does not understand why she has to pay to work, but since she gets confused about it, it’s ok.

Day care is equated with babysitting. They are vegetables there. The patients are too far gone

The only alternative is day care.

This is very stressful, to put it mildly

I turn to old friends in Missouri for help

I talk to other staff members when I need help.

The church has a list of people who can help, but you really feel alone.

I don’t talk to anyone. This is my cross to bear and I do it. I learned fast what I needed to do.

The social worker at the doctor’s office has also been helpful

I cannot cry to my children, because they have lives of their own.

They need to reach out to me; I cannot burden them.

We don’t talk to each other much, because we don’t get together.

Location is not a problem. The real problem is distance from home. Those close by find it great.

He was very nervous about going, because he was afraid he couldn’t do the work. He didn’t remember, but it was OK. Lately he’s been more tired and doesn’t want to go.

When Ruth was moved, she really deteriorated quickly. Major events seem to trigger fast decline.

Mother is so different now than before the program. Sometimes she did not dress at all, but now she dresses herself and makes me hurry to get her here.

There is no other program like this in the country

Code 1

AD

COST

COST

COST

COST

COST

DC

DC

FAM

FAM

FAM

FAM

FAM

FAM

FAM

FAM

FAM

LOC

PART

PART

PART

PRG

Code 2

PRG

WORK

SUP

SUP

SUP

SUP

SUP

SUP

SUP

SUP

SUP

SUP

TRP

WORK

AD

BEH

PUB

Code Summary AD Alzheimer’s disease PRG Program SUP Support COST Cost WORK Work DC Day care FAM Family LOC Location TRP Transportation PART ParticipanUpatient BEH Behavior PUB Publicity/marketing

Note: Data were sorted by Code 1.

Code 3

SUP

PRG

JHQ 1 July/August 1996. Vol. 18, NO. 4.29

Patient Feedback

crease. This burnout has been shown to influence the decision to institutional- ize AD patients before they truly need it (Weddington, 1994).

Identifying intermediate outcomes: Once the program staff were clear about the ultimate outcome, they identified intermediate outcomes of maintaining cognitive abilities for as long as possi- ble and providing respite and support

for the caregivers. The logic was that if the intermediate outcomes were achieved, the ultimate outcomes also would be realized. To achieve the intermediate outcomes, the staff identified proximal outcomes for each of the four components identified in the operational model (intake, assessment, program, and transition/exit). This ex- ercise was important, because identify-

ing the outcomes for each component provided guidance to the design team and the program staff in identifying measures to monitor the program’s functioning and its effect on the patients.

Importance of Documentation Observing patients: It is particularly

important for the program staff to stay attuned to the patients’ and caregivers’

Marketing Community need

(data on AD) Creation

of image Education of public and referral sources

Advertising of AWS

Case FindingIntake

Referral sources: Family members Health providers Assisted living

facilities Hopital dis-

charge planners Church leaders Senior citizens’

centers

Referral sources

Admission criteria: Memory loss

diagnosis Ability to

produce No behavior

problems Motor

competence Transportation

Intake assessment form

*

Figure 3. Operational Model of Adapted Work Services

Inquiry Questions (from families, referral sources, and participants)

Information: Costs, meals,

hours, absences, meds, trial, transportation, payment, ongoing evaluation, visitors, discharge, atten- dance.

Set of AWS questions (for families, referral sources, and participants)

Referral form

Invitation to visit program

1 Contact

Visit to facility: Introduction to

staff/peers Observation of

work in progress

Staff observation of participant/ family

Information sheet

Willingness to apply

Application Process

Paperwork Application form Agreement form Billing Permission to

release information for RWJF grant

Trial period: Competence for

working Work samples

Assessment: Contact notes Memory and

behavior check list

Geriatric deores- sion scale RWJF

Testing: Mini Mental

Status Exam (MMSE)

Neuro-psychi- atric testing

Family dynamics: Support Need for respite

L

Decision/ Assignment

Analysis of Assessment Testing Family dynamics Trial period

Eligible to participate: Enrollment

Not eligible to participate: Referral to other services

Communication with: Referral sources Physician Family

Daily Program Transportation to AWS: Self, family,

car pool, other

Socialization upon arrival

Morning work

Training of participants: Traidretrain

for old job Train for new

job Ongoing

training/ support

Lunch (socialization)

Afternoon work (fatigue factor)

Ongoing training/support

Departure -

TransitiodExit Reasons for leaving: Self-selection Transportation Health Family/participant choice

Criteria for discharge : Decline in

productivity Decline in social functioning

Inappropriate behaviors

Family burden

Feedback to: Physician Referral sources Participants

Follow-up: Next level of

services . Case

Management Ongoing evaluation: Productivity/functional status Behavior/emotional status Physical health

Documentation: Progress notes Flow/check sheets Repeat MMSE (x intervals) Report of ongoing evaluation

Family Report/Communication: Creation of plan with caregivers

Community resources

-Intake - Assessment -+ + Program + +Transition/Exit+

30 JHQ July/August 1996 Vol. 18, No. 4

Inta

ke

Cha

ract

erist

ics

Prox

imal

out

com

es

prep

arat

ion

of ca

regi

vers

fo

r nex

t st

age

Faci

litat

ion

of c

ontin

u-

um of

car

e se

rvic

es

thro

ugh

tran

sitio

ns

Cle

arly

de

fine

d 1

prog

ram

Cle

arly

de

fine

d po

p-

Prom

pt a

nd

supp

ortiv

e at

tent

ion

ongo

ing

cont

act

1

mat

ch n

eed

to s

ervi

ces

Eas

y ac

cess

F-

Enh

ance

d cu

stom

er

satis

fact

ion

Figu

re 4

. C

once

ptua

l Mod

el o

f Ada

pted

Wor

k Se

rvic

es

Ass

essm

ent

Cha

ract

erist

ics

Prox

imal

out

com

es

Stan

dard

- A

ccur

ate

ized

pr

oced

ures

pria

te to

go

als o

f pr

ogra

m

Prog

ram

Cha

ract

erist

ics

Prox

imal

out

com

es

Wor

k co

mpo

nent

: Si

mpl

e R

epet

itive

M

eani

ngfu

l In

divi

dual

- iz

ed e

xpec

- ta

tions

Use

of

rem

aini

ng

skill

s Pr

oduc

tivity

Feel

ing

of se

lf-w

orth

Soci

aliz

atio

n co

mpo

nent

: Id

entif

icat

ion

with

pee

rs

acce

ptan

ce

by st

aff

Unc

ondi

tion-

al

acc

ep-

tanc

e O

wne

rshi

p of

prog

ram

by

par

tici-

supp

ort/

pant

s

Cas

e m

anag

emen

t:

eval

uatio

n Pr

epar

atio

n of

ca

regv

ers

for t

rans

i-

tiOnS

C

omm

unic

a-

tion

with

fa

mily

and

ca

re pr

o-

vide

rs

ongo

ing

I

Dec

reas

ed

isol

atio

n

Mai

ntai

ned

or in

crea

sed

self-

care

D

ecre

ased

ne

gativ

e em

otio

ns/

beha

vior

s

Out

com

es

Cha

ract

erist

ics

Prox

imal

out

com

es

Mai

nten

ance

of

func

tioni

ng

as lo

ng a

s po

ssib

le

Prov

isio

n of

resp

ite an

d su

ppor

t for

ca

regi

vers

Dec

reas

ed

tota

l bur

den

of i

llnes

s

Patient Feedback

perspectives. The progressive nature of AD makes assessing and using patient feedback more difficult. In the very early stages of ths disease, patients can verbalize how they feel, what they think and what they like or do not like about the program. But as the disease progresses, staff members have to get data by observing the patients. What behaviors do patients exhibit when they arrive in the morning? How do they in- teract with other patients and with the staff? How do they function during the day? These observations are clues to things that patients may not be able to verbalize.

Training aides: Such observations, however, are not useful unless the context of what is happening to the patient also is documented. A social worker was available to help with documentation, but she had many duties and could not perform all the documentation. Train- ing the staff aides to take notes was important for collecting good patient data. The aides are the staff members clos- est to the patients; they get to know the individual patients well.

Using documentation: Once the observations are recorded and notes writ- ten about the context of the behavior, they can become feedback to the families and the referral sources. Patient feedback, whether verbal or indirect through observation, helps the staff an- ticipate and prepare caregivers for the inevitable transitions. Staff members try to meet with families when they first bring the patients to AWS. There are pe- riodic potluck suppers, which bring the staff, patients, and families together. At the present time there is no way to get regular feedback from the caregivers; this need will be addressed soon.

Summary The three models that were developed

as part of the formative evaluation helped answer the question ”Can we predict the likely success of a program that has never been tried before?” The diagram of the program as it was originally conceived provided information on the desired outcomes, the process to achieve these outcomes, and the struc-

ture required. Creating a conceptual model of the program helped ensure that there was a theory behind the activities; the logic of the program was made explicit by the conceptual model. No research was available on the rela- tionship between work and AD patients, but we did find concepts that could be generalized to this program. For example, work therapy has been used in programs for occupational therapy, mental illness, mental retardation, rehabilitation, and geriatric patients in a nursing home. The theory behind these programs seemed also to apply to this program. We also researched the effectiveness of socialization for patients with AD. Little on the subject of socialization was found in the literature, but our own patients were very clear about its importance to them.

Conceptual model: In deciding whether the AWS program was likely to succeed, we looked at the literature and the questions posed for the formative evaluation study. Both patients and caregivers provided feedback that the program was meeting most of their needs. However, the staff found that there was room for improvement. By creating a conceptual model of the log- ical progression from proximal outcomes to intermediate and to ultimate outcomes, staff members were able to ensure that activities were clearly fo- cused on the needs of the patients, caregivers, and society. The conceptual model also provided the template for identifying measures of program performance and more importantly, effects of the program on participants and caregivers. With measures to monitor the functioning of the program and the effect on the target population, the staff are now able to show referral sources and potential funders important data about the program. The likelihood of securing additional funding and gaining the confidence of the professional community is greatly enhanced with these data. With the models in hand, the program’s staff members have been able to show what they did, why they did it, how they knew the program was doing well (operationally and from the pa-

tients‘ perspective), and what evidence from the literature helped ensure that the program was on solid ground. The program will continue to be solid if staff members continue to keep their fingers on the ”customer pulse” and learn to adapt as needed. Flexibility is the key to succeeding, and knowing exactly what the program is meant to be will help to keep it flexible.

References Blass, J.P. (1990). The report of the advi-

sory panel on Alzheimer‘s disease, 1988- 1989. Generations, 14(2), 68-69.

Cook-Desan, R.M. (1988). Losing a million minds: Confronting the tragedy of Alzheimer’s disease and other dernentias. Philadelphia: Office of Technology As- sessment Task Force, Science Informa- tion Resource Center.

Miles, M.B., & Huberman, A.M. (1994). Qualitative data analysis. Thousand Oaks, CA: Sage Publications.

Scheirer, M.A. (1994). Designing and using process evaluation. In J. Wholey, H. Hatry, & K. Newcomer (Eds.), Handbook of practical program evaluation. San Fran- cisco: Jossey-Bass Publications.

Weddington, D. (1994). Early-stage Alzheimer’s care. New York: Springer Publishing Co.

Suggested Reading Deming, W.E. (1982). Out qfthe crisis. Cam-

bridge, MA: MIT Press.

Doris Quinn is the director of quality education and management at Vanderbilt University Clinic in Nashville, TN. She is responsible for facilitating the integration of process improvement and outcomes research in the clinicS clinical practiceand op- erations. Quinn, who assisted W. Ed- wards Deming with consulting and teaching tours in the late 1980s and early 1990s, has 20 yeurs of experience in thefields of management and education.

32 1 JHQ 1 July/August 1996 1 Vol. 18, NO. 4

using patient feedback for creating and improving innovative programs

Documents