Doing Effectiveness Research at the County Level

Robert Landry, Ph.D.

Yolo County Department of Alcohol, Drug and Mental Health

Contact: robert.landry@ccm.yolocounty.org

Problem• We are charged with helping the most difficult welfare

recipients move from a culture of welfare dependency to independence.

• We must not only address traditional substance abuse and mental health issues. We need to find new ways to facilitate what is essentially a cultural transition.

• There has been little research on how to do this, let alone whether it is even possible.

• This presentation covers Yolo county’s attempt to come to grips with this problem.

Questions• What percentage of CalWORKs participants are referred for

Alcohol Drug and Mental Health services?

• How many of these participants enter treatment?

• What barriers do they face?

• How are the barriers related to engagement in ADMH treatment and to success in employment services?

• Do clients get better in treatment?

• Does ADMH treatment help clients achieve greater financial independence?

What percentage of CalWORKs participants are referred for Alcohol Drug and Mental Health

services?

26%

How many of these participants enter treatment?

What barriers do clients face?

• In a program with a broader mandate than simply treating mental health problems, we found it necessary to summarize the clients’ employment barriers.

• We developed an Employment Barriers Checklist.

• We also measured the clients’ symptom distress with the BASIS 32 and their social functioning with the

CA-QOL.

Percentage of Clients with Barriers

Mental Health 85.3 Alcohol/Drug 60.6 Domestic Conflict 55.8 Attitude 35.5 Education 32.9 Children 30.4 Work 25.6 Medical Problems 22.4 Housing 19.6 Legal 16.8 Transportation 15.9

Client Reported Symptomatology and Quality of Life

• The Basis32 and the CA-QOL were administered during assessment (n = 288).

• The BASIS32 is a symptom inventory completed by the client.

• We compared the means of our sample to that of the normative female in-patient group at discharge .

• The CalWORKs group is slightly worse than the in-patient group at discharge, indicating CalWORKs ADMH clients are reporting serious pathology.

How are the barriers related to engagement in ADMH treatment and success in employment

services?

Method Tasks:

• Collapse the ADMH termination status into a treatment and no treatment group.

• Collapse the CWES disposition (what the client was does after termination) into a participation and no participation group.

• Decide which barriers, or independent variables, we will measure.

57%

43%

69%

31%

Frequency of Cancellation/No Shows For Clients Who Start Treatment

MISSEDAPPOINTMENTS

PERCENT

0 51.31 or 2 37.3

3+ 27.4

There was a significant relationship between the number of no shows and starting treatment.

(Pearson’s Chi Squared 17.599 df = 2 p < .001)

Frequency of Cancellation/No Shows For Clients Who Participate in CWES

MISSEDAPPOINTMENTS

PERCENT

0 46.11 or 2 47.3

3+ 24.6

There was no difference between the 0 missed session group and the 1or 2 missed session group. However, there

was a large difference starting with 3 missed sessions (Pearson’s Chi Squared 11.147 p = .004)

Client Reported Symptomatology and Quality of Life

• The Basis32 and the CA-QUAL were administered during assessment (n = 288).

• There was no relation found between client reported symptomatology or quality of life and participation in either treatment or employment services.

Percentage of Clients Who Participate in CWES By Number of Barriers

NUMBER OF BARRIERS

PERCENT

2-4 50%5-6 40%7+ 14%

Barriers associated with starting

treatment:

• Anxiety

• Depression

• Interpersonal/Axis II Problems

• No High School Diploma

• Diagnosed Chronic Physical Disability

Barriers associated with a lack of participating in CWES:

•AD abuse

•Poor Attendance

•Past Domestic Violence (inverse relationship)

•Unstable Housing

• Never Worked.

Conclusions about Barriers and Starting Treatment or Participating in CWES

• The presence of barriers increases the chances of starting treatment and decreases the chances of participating in CWES.

• Assessment no shows were associated with poor participation in both treatment and CWES.

Do clients get better in treatment?

• Tracking therapeutic and quality of life changes is essential for determining the impact of treatment on employment.

• The BASIS 32 and CA-QOL are readministered every 15 sessions and at termination.

• Significant symptom reduction was indicated by the BASIS 32 and significant improvements in quality of life were indicated on 3 of the 9 CA-QOL scales.

How Does Starting Treatment Affect the Chances That a Client Will Participate in CWES?

Started Treatment

Probability of CWES

Participation

NO 36%

YES 58%

(Pearson’s Chi Squared p < .001).

Is there a causal relationship between treatment and success in employment?

•Experimental designs provide the most powerful proof of causality, but can be ethically and legally problematic.

•We recommend A B designs and other time study designs.

•We tracked quarterly change in treatment completion and found the percentage of clients who completed treatment increased by 38% over 2 years.

•The percentage of treatment completers who participated in CalWORKs and found employment stayed constant at 56-58%.

•This indicates there is a causal relationship between treatment and employment.

Conclusions: Outcome Research

• The clients self report high levels of symptomatology.

• The degree of symptomatology within the group referred for treatment

is not a good predictor of participation in treatment or CWES.

• Therapy drop out rates are very high.

• Most clients report they improve in therapy.

• There is a relationship between starting treatment and participation in

employment services.

Conclusions: Program Implications

• More than 2 missed assessment appointments warrants flagging for alternate

interventions.

• Never employed participants need alternate interventions.

• There needs to be a high degree of integration of therapy and other components in

order to address interrelated issues.

• Therapy should have a significant focus on motivating clients to overcome their life

barriers.

• Intensive case management may be needed to engage the most intractable clients.

• Once the most intractable clients are engaged there needs to be graduated

pathways towards increasing responsibility, or an effective SSI advocacy program if

indicated.

The Prerequisites For Outcome Assessment

• A manager committed to outcome assessment.

• A user friendly database that makes life easier for staff by:

– Simplifying standard case management paperwork tasks.

– Collects most data as a bi-product of what they are doing anyway.

– Provides the staff with quick access to useful information.

• A commitment of resources to the start up process.

• Someone familiar with research to help with the design and number

crunching.