improving the quality of services through measurement and
Post on 16-Oct-2021
1 Views
Preview:
TRANSCRIPT
www.TheNationalCouncil.org
Improving the Quality of Services Through Measurement and Feedback
Supported by the National Institute of Mental Health grant (RO1 MH068589) and a grant from the Leon Lowenstein Foundation .
Leonard Bickman, Ph.D., Susan Douglas Kelley, Ph.D., and Susan Casey, MSW
www.CFSystemsOnline.com
1
Outline• The challenge and opportunity• The theory of change• Contextualized Feedback
Systems and its components• Preliminary results of evaluation• Benefits• Plans
“The greatest obstacle to discovery is not ignorance - it is the illusion of knowledge”Daniel Boorstin, 1984 Librarian of Congress
3
Little Research Support for the Effectiveness of Services in
Community Settings& Little Change in a Decade
• “The findings to date offer little support for the effectiveness of conventional clinical intervention for young people.” (Weisz & Jensen, 1999, p. 133)
• Some common elements of effective practice have been observed in treatment as usual but not in the depth and breadth needed for effective treatment. (Garland, Bickman, & Chorpita, 2010)
• But lack of evidence doesn’t necessarily mean lack of effectiveness 44
Little Accuracy About Client Progress & Focus of Treatment
• No correlation between standardized measures and clinicians’ perceptions of progress (Love et al., 2007)
• Difficulty predicting and detecting worsening of symptoms and functioning over the course of treatment (Hannan et al., 2005; Hatfield et al., 2009) – Yet, 89% of clinicians said they would know based on their
clinical judgment
• Little agreement on target problems by parents and therapists, and children and therapists (Hawley, KM & Weisz, JR. (2003).
• Chance Agreement on Level of Care Assignment (Bickman, L., Karver, M., & Schut, L. J. A. (1997).
5
Typical Community Based Care has Significant limitations
• “The findings to date offer little support for the effectiveness of conventional clinical intervention for young people.” (Weisz & Jensen, 1999, p. 133)
• Some common elements of effective practice have been observed in treatment as usual but not in the depth and breadth needed for effective treatment. (Garland, Bickman, & Chorpita, 2010)
• But lack of evidence doesn’t necessarily mean lack of effectiveness
66
Clinical Intuition and Regulations are Insufficient to Produce Good
Outcomes • Fifty years of research does not support
relying only on clinical judgment for effective practices
• Reliance on presumed standards of quality such as licensing and accreditation may also impede development of effective services
• Unmonitored “single shot” training on EBTs are not effective
7
9
“Thus, use of patient monitoring and feedback in routine practice is imperative.
The focus on the individual rather than the average patient empowers the patient and encourages dialogue about progress, the direction of treatment and achievement of treatment goals”.
Newnham & Page. (2010). Bridging the gap between best evidence and best practice in mental health. Clinical Psychology Review, 30, 127–142 (p.138).
Strong Support for Feedback Concept
• Measurement and feedback are the core of all management and learning theories.
• Thousands of studies outside of mental health show that improvement is minimal without measuring performance and providing feedback.
• Direct feedback occupations show improvement with experience. However, clinician experience alone is not a good predictor of client outcomes.
10
Providence Service Corporation delivers home and community based social services to government beneficiaries and privatized social service projects.
Providence Service Corporation
12
Home and Community
Based Services
Management Services
Foster Care Services
PRSC ServiceOffering
.
Providence Overview
• Providence Service Corporation was aware of the need for feedback and was committed to implementing a quality enhancement initiative
• In 2004 we began a collaboration with Vanderbilt University to combine their efforts to create an evidence-based feedback system
13
Providence Core Values
• Community-based and Multi-systemic Services
• Identifying and Building on Strengths• Local Viability that is Nationally
Supported• Respectful Organizational Culture• Best Practice Models • Cultural Diversity
Providence Service Corporation
Quality Enhancement Initiative
15
• 12 states, 38 sites, 287 clinicians
• 836 youth 11-18 years of age and their caregivers
• 10,431 client weeks of Providence provided services
CFS - A Concurrent, Systematic Monitoring and Formative Feedback
Practice Improvement Tool
Practice Without Feedback Does Not Lead To Improvement
16
CFS is a Practice Improvement Strategy
• is part of treatment – provides an opportunity for brief reflection at the close of a session
• is primarily designed to support clinicians – provides guidance for those moments when little seems to be working
• supports supervisors – helps supervisors identify areas where clinicians need extra guidance to ensure they feel confident
• is flexible – optional custom measures, schedules, reports can be tailored for office workflow and QI initiatives
• can be used for administrative and reporting functions –provides tools for leadership to become or remain successful at meeting the needs of their funders and clients
17
18
CFS Theory of Change is on Two Levels
• The individual or psychological level
• The group or organizational level
19
Key Domains of CFS
• System Development• Measurement• Feedback• Training in Common Factors• Comprehensive Support
Dashboard-driven System
20
Easy to navigateHighlights common actions by userDashboards are designed for clients and
caregivers, clinicians, supervisors, administrative assistants, program directors, etc.
Sample Clinical Alerts
21
New = most recent alerts; not yet viewed
Pending = old alert; not yet viewed
Link to the feedback report containing the alert; Archive alerts
Measurement: The Peabody Treatment Progress Battery (PTPB)
• Domains include processes and outcomes• Includes strength-based measures• Brief – each takes 15 seconds to 2 minutes• Reliable • Show convergent and divergent validity• Information from youth, clinician, and caregiver• Easy to score and interpret• Sensitive to change• Available in English and Spanish• Free for anyone to use – now in second edition• http:\\peabody.vanderbilt.edu/ptpb
26
CFS Measures Map onto the Relevant Questions
• How is the youth doing overall?– Severity of symptoms and functioning (y, cg, cl)
– Life satisfaction (y)
– Hope (y)
• How is treatment going with the youth?– Therapeutic alliance (y, cl)
– Motivation for treatment (y)
– Counseling impact (y)
– Service satisfaction (y)
• What are important caregiver issues?– Caregiver strain– Life satisfaction– Therapeutic alliance– Motivation for treatment– Service satisfaction
27
28
Organizational Measures• Services, staffing, any data already
collected; experience with other initiatives, etc.
• Culture, climate, adaptability, leadership, organizational learning, & initial perceptions of CFS
• Current attitudes toward CFS, barriers/supports, value of reports, self-efficacy, & goal commitment
CFS Consultation and Training Model
• CFS is a tool for transforming your agency into a learning organization
• Ongoing support to– Promote sustainability through local ownership– Contextualize CFS operations and feedback to
your agency’s needs and resources– Apply feedback to inform clinical sessions,
supervision, program planning, professional development, and more
– Empower practice improvement leadership at all levels of your agency
29
30
Analyses of Current Project• Randomized experiment in 33 sites -
Analyses ongoing• Randomly assigned sites to feedback and no
feedback conditions• Started with most direct question: Does
feedback appear to affect clinician behavior?• The answer to this question does not depend
upon the effectiveness of the treatment provided
CFS Feedback Influences Clinician Behavior in Sessions• Cases selected where clinician had not previously reported
a problem with youth symptoms and functioning in a specific domain (e.g., behavioral issues, emotional problems, friend/peer issues)
• When a client or caregiver alert was present for that domain– Viewing clinical feedback reports associated with addressing that
domain in a future session– More feedback viewed more times addressed– Not explained by more reporting of problems by clients or
caregivers
31
Feedback Reduces Symptoms and Improves Functioning
• Used HLM random effects with new clients, controlled for nesting within sites, and number of sessions.
• Youth respondent (ages 11-18): 340 clients• Caregiver respondent: 308 clients• Clinician respondent: 294 clients• There were no significant differences at baseline
between the two groups for any respondent• Results showed Feedback Group improved more
than the non-Feedback Group according to all three respondents.
32
Modeled Values of Youths’ SFSS p = 0.025
33
30
35
40
45
50
55
60
65
70
75
80
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Yout
hs’ S
FSS
in C
BC
L U
nits
Months in CFS
Feedback
No Feedback
Modeled Values of Caregivers’ SFSS p = 0.05
34
30
35
40
45
50
55
60
65
70
75
80
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Car
egiv
ers’
SFS
S in
CB
CL
Uni
ts
Months in CFS
Feedback
No Feedback
Modeled Values of Clinicians’ SFSS p = .0005
35
30
35
40
45
50
55
60
65
70
75
80
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Clin
icia
ns’
SFSS
in C
BC
L U
nits
Months in CFS
Feedback
No Feedback
What Can CFS DoFor Clients and Caregivers?
• Provide a way to raise issues they may not feel comfortable addressing aloud or in the presence of caregiver or youth
• Reassure them that the clinician is paying attention to their needs and is contextualizing services to their concerns
• Demonstrate that effective services are a priority
• CFS is a tool that takes the consumer’s voice beyond the session and into all aspects of care
36
What Can CFS DoFor Clinicians?
• Better identify thoughts, events, and feelings of clients and caregivers
• Identify successes and problem areas to focus sessions• Provide consistent and systematic feedback of clinically
relevant information• Inform treatment planning and goal setting• Focus sessions on clients’ issues to show that you are
attending to their concerns• Check on how well treatment is working• CFS is a tool that enhances the clinician’s ability to tailor
treatment as it progresses37
What Can CFS DoFor Clinical Supervisors?
• Provide clinical data on each case’s progress and areas that seem to be improving or declining
• Provide a tangible framework from which to guide clinical supervision that is independent of the clinician
• Provide the needed information for a continuous quality improvement effort that facilitates accountability
• Most of all…CFS is a tool that provides the supervisor with needed resources to promote evidence-based practice
38
What Can CFS DoFor an Agency?
• Serve as evidence to funding agencies that quality of services and effectiveness are a priority
• Manage clinical services on their impact as well as their cost
• Provide data on the effectiveness of services, how clients are improving, the typical problems being encountered, and where needs are not being met
• CFS is a tool that supports overall practice improvement
39
What CFS Can Do for the Funder?
• Determine who is being served and their progress
• Provide a Quality Assurance mechanism• Compare services and agencies on cost
effectiveness to optimize value• Reduce costs of some regulations and
documentation activities such as excessively detailed treatment planning and progress notes
40
Plans• Continued partnership between Vanderbilt and Providence in
training, development, and research• Continue software development• Use with adults as well as children and youths• Continue development of new measures• Proposal pending to test CFS with different types of
feedback with youth with substance abuse problems• Integrate CFS with other mental health EBTs (new NIMH
grant for Functional Family Therapy combined with CFS)• Develop web-based “treatment” intervention with CFS
(Australia Project)• Use CFS in primary care settings to bridge medical and
mental health fields
41
top related