mhp final report - caleqrocaleqro.com/data/reports_and_presentations/fiscal year 2016-2017... ·...

FY 16-17

Medi-Cal Specialty Mental Health

External Quality Review

MHP Final Report

Prepared by:

Behavioral Health Concepts, Inc.

5901 Christie Avenue, Suite 502

Emeryville, CA 94608

www.caleqro.com

Colusa County Conducted on

August 4, 2016

http://www.caleqro.com/

TABLE OF CONTENTS

COLUSA MENTAL HEALTH PLAN SUMMARY FINDINGS ............................................................................................. 4

INTRODUCTION ..................................................................................................................................................................... 6

PRIOR YEAR REVIEW FINDINGS, FY15-16 ..................................................................................................................... 8

STATUS OF FY15-16 REVIEW RECOMMENDATIONS ................................................................................................................................ 8 Assignment of Ratings ............................................................................................................................................................................... 8 Key Recommendations from FY15-16 ................................................................................................................................................ 8

CHANGES IN THE MHP ENVIRONMENT AND WITHIN THE MHP—IMPACT AND IMPLICATIONS .................................................... 11

PERFORMANCE MEASUREMENT .................................................................................................................................... 13

TOTAL BENEFICIARIES SERVED................................................................................................................................................................... 13 PENETRATION RATES AND APPROVED CLAIM DOLLARS PER BENEFICIARY ....................................................................................... 14

PERFORMANCE IMPROVEMENT PROJECT VALIDATION ....................................................................................... 22

COLUSA MHP PIPS IDENTIFIED FOR VALIDATION .................................................................................................................................. 22 CLINICAL PIP—RECOVERY .......................................................................................................................................................................... 24 NON-CLINICAL PIP—PARENT INTERVENTION ........................................................................................................................................ 26 PERFORMANCE IMPROVEMENT PROJECT FINDINGS—IMPACT AND IMPLICATIONS .......................................................................... 27

PERFORMANCE & QUALITY MANAGEMENT KEY COMPONENTS ......................................................................... 28

Access to Care .............................................................................................................................................................................................. 28 Timeliness of Services ............................................................................................................................................................................... 29 Quality of Care ............................................................................................................................................................................................. 31

KEY COMPONENTS FINDINGS—IMPACT AND IMPLICATIONS ................................................................................................................ 35

CONSUMER AND FAMILY MEMBER FOCUS GROUP(S) ............................................................................................ 37

CONSUMER/FAMILY MEMBER FOCUS GROUP 1 ...................................................................................................................................... 37 CONSUMER/FAMILY MEMBER FOCUS GROUP FINDINGS—IMPLICATIONS ......................................................................................... 38

INFORMATION SYSTEMS REVIEW ................................................................................................................................. 39

KEY ISCA INFORMATION PROVIDED BY THE MHP ................................................................................................................................. 39 CURRENT OPERATIONS ................................................................................................................................................................................. 40 PLANS FOR INFORMATION SYSTEMS CHANGE .......................................................................................................................................... 41 ELECTRONIC HEALTH RECORD STATUS ..................................................................................................................................................... 41 MAJOR CHANGES SINCE LAST YEAR ........................................................................................................................................................... 42 PRIORITIES FOR THE COMING YEAR ........................................................................................................................................................... 42 OTHER SIGNIFICANT ISSUES ........................................................................................................................................................................ 43 MEDI-CAL CLAIMS PROCESSING ................................................................................................................................................................... 43 INFORMATION SYSTEMS REVIEW FINDINGS—IMPLICATIONS ............................................................................................................... 43

SITE REVIEW PROCESS BARRIERS ................................................................................................................................ 45

CONCLUSIONS ...................................................................................................................................................................... 46

STRENGTHS AND OPPORTUNITIES .............................................................................................................................................................. 46 Access to Care .............................................................................................................................................................................................. 46 Timeliness of Services ............................................................................................................................................................................... 46 Quality of Care ............................................................................................................................................................................................. 47 Consumer Outcomes ................................................................................................................................................................................. 47

RECOMMENDATIONS ..................................................................................................................................................................................... 48

ATTACHMENTS ................................................................................................................................................................... 49

ATTACHMENT A—REVIEW AGENDA .............................................................................................................................................................. ATTACHMENT B—REVIEW PARTICIPANTS ................................................................................................................................................... ATTACHMENT C—APPROVED CLAIMS SOURCE DATA ................................................................................................................................ ATTACHMENT D—PIP VALIDATION TOOL ...................................................................................................................................................

COLUSA MENTAL HEALTH PLAN SUMMARY FINDINGS

Beneficiaries served in CY15—523

MHP Threshold Language(s)—Spanish

MHP Size—Small Rural

MHP Region—Superior

MHP Location— Colusa

MHP County Seat—Colusa, CA

MHP-Reported Significant Changes

The MHP hired a new Quality Improvement (QI) Coordinator who will be responsible

for data analysis and trending.

In the spring of 2016, a county administrative officer was hired, with MHP leadership as

a direct report.

The MHP has experienced difficulties with crisis protocols since April 2016 with the

closure of the only hospital in the county.

The MHP has had great difficulty in working with Anthem for Affordable Care Act (ACA)

referrals. Anthem does not seem to have any providers that have openings, and they are

not very responsive to consumers and staff calls.

Performance Measurement Findings from CY15 Claims Data

The MHP’s percentage of high-cost beneficiaries was similar to statewide and showed

an increase from CY14. Its percentage of HCB claim dollars was higher than statewide

and doubled from CY14 to CY15.

Information Systems Findings

The MHP plans to install and begin implementation of a Telechart Patient Portal

following system upgrade in September 2016.

The MHP continues to pursue Meaningful Use, Stage I qualification through

Medicare/Medicaid’s EHR incentive program with a projected target date of October

2016. The MHP is working towards electronic capture of its clinical data and the use of

health IT for continuous quality improvement at the point of care and the exchange of

information.

MHP has no current plans to expand EHR functionality with the addition of eLabs, Level

of Care/Level of Service assessments.

Strengths and Recommendations Findings

Investigate the feasibility to provide more funding or resources to support crisis staff.

Evaluate contributing factors for the increase in hospitalization rates from FY 14/15 to

FY15/16.

Re-establish and formalize the Cultural Competence Committee with regularly

scheduled meetings.

Track timeliness for psychiatry for both children and adults, to determine the impact of

the .5 FTE elimination of child psychiatry and the implementation of the policy

requiring conjoint services for medication child clients.

Expand EHR functionality through consultation with Kings View and nearby counties.

Quantifiably show improvement and expansion through the addition of applications,

such as eLabs, Level of Care/Level of Service, Alerts, etc.

INTRODUCTION

The United States Department of Health and Human Services (DHHS), Centers for Medicare and

Medicaid Services (CMS) requires an annual, independent external evaluation of State Medicaid

Managed Care programs by an External Quality Review Organization (EQRO). External Quality

Review (EQR) is the analysis and evaluation by an approved EQRO of aggregate information on

quality, timeliness, and access to health care services furnished by Prepaid Inpatient Health Plans

(PIHPs) and their contractors to recipients of Managed Care services. The CMS (42 CFR §438;

Medicaid Program, External Quality Review of Medicaid Managed Care Organizations) rules specify

the requirements for evaluation of Medicaid Managed Care programs. These rules require an on-

site review or a desk review of each Medi-Cal Mental Health Plan (MHP).

The State of California Department of Health Care Services (DHCS) contracts with fifty-six (56)

county Medi-Cal MHPs to provide Medi-Cal covered specialty mental health services to Medi-Cal

beneficiaries under the provisions of Title XIX of the federal Social Security Act.

This report presents the fiscal year 2016-2017 (FY 16-17) findings of an EQR of the Colusa

MHP by the California External Quality Review Organization (CalEQRO), Behavioral Health

Concepts, Inc. (BHC).

The EQR technical report analyzes and aggregates data from the EQR activities as described below:

(1) VALIDATING PERFORMANCE MEASURES1

This report contains the results of the EQRO’s validation of eight (8) Mandatory Performance

Measures (PM) as defined by DHCS. The eight performance measures include:

Total Beneficiaries Served by each county MHP

Total Costs per Beneficiary Served by each county MHP

Penetration Rates in each county MHP

Count of TBS Beneficiaries Served Compared to the four percent (4%) Emily Q.

Benchmark (not included in MHP reports; this information is included in the Annual

Statewide Report submitted to DHCS).

Total Psychiatric Inpatient Hospital Episodes, Costs, and Average Length of Stay

Psychiatric Inpatient Hospital 7-Day and 30-Day Rehospitalization Rates

Post-Psychiatric Inpatient Hospital 7-Day and 30-Day Specialty Mental Health Services

(SMHS) Follow-Up Service Rates

1 Department of Health and Human Services. Centers for Medicare and Medicaid Services (2012). Validation

of Performance Measures Reported by the MCO: A Mandatory Protocol for External Quality Review (EQR),

Protocol 2, Version 2.0, September, 2012. Washington, DC: Author.

High Cost Beneficiaries ($30,000 or higher)

(2) VALIDATING PERFORMANCE IMPROVEMENT PROJECTS2

Each MHP is required to conduct two Performance Improvement Projects (PIPs) during the 12

months preceding the review; Colusa MHP submitted two PIPs for validation through the EQRO

review. The PIP(s) are discussed in detail later in this report.

(3) MHP HEALTH INFORMATION SYSTEM CAPABILITIES3

Utilizing the Information Systems Capabilities Assessment (ISCA) protocol, the EQRO reviewed and

analyzed the extent to which the MHP meets federal data integrity requirement for Health

Information Systems (HIS), as identified in 42 CFR §438.242. This evaluation included review of

the MHP’s reporting systems and methodologies for calculating Performance Measures (PM).

(4) VALIDATION OF STATE AND COUNTY CONSUMER SATISFACTION SURVEYS

The EQRO examined available consumer satisfaction surveys conducted by DHCS, the MHP or its

subcontractors.

CalEQRO also conducted one 90-minute focus group with beneficiaries and family members to

obtain direct qualitative evidence from beneficiaries.

(5) KEY COMPONENTS, SIGNIFICANT CHANGES, ASSESSMENT OF STRENGTHS,

OPPORTUNITIES FOR IMPROVEMENT, RECOMMENDATIONS

The CalEQRO review draws upon prior year’s findings, including sustained strengths, opportunities

for improvement, and actions in response to recommendations. Other findings in this report

include:

Changes, progress, or milestones in the MHP’s approach to performance management—

emphasizing utilization of data, specific reports, and activities designed to manage and

improve quality.

Ratings for Key Components associated with the following three domains: access,

timeliness, and quality. Submitted documentation as well as interviews with a variety of

key staff, contracted providers, advisory groups, beneficiaries, and other stakeholders

serve to inform the evaluation of MHP’s performance within these domains. Detailed

definitions for each of the review criteria can be found on the CalEQRO Website

www.caleqro.com.

2 Department of Health and Human Services. Centers for Medicare and Medicaid Services (2012). Validating

Performance Improvement Projects: Mandatory Protocol for External Quality Review (EQR), Protocol 3,

Version 2.0, September 2012. Washington, DC: Author. 3 Department of Health and Human Services. Centers for Medicare and Medicaid Services (2012). EQR

Protocol 1: Assessment of Compliance with Medicaid Managed Care Regulations: A Mandatory Protocol for

External Quality Review (EQR), Protocol 1, Version 2.0, September 1, 2012. Washington, DC: Author.


PRIOR YEAR REVIEW FINDINGS, FY15-16

In this section we discuss the status of last year’s (FY15-16) recommendations, as well as changes

within the MHP’s environment since its last review.

STATUS OF FY15-16 REVIEW RECOMMENDATIONS

In the FY15-16 site review report, the CalEQRO made a number of recommendations for

improvements in the MHP’s programmatic and/or operational areas. During the FY16-17 site visit,

CalEQRO and MHP staff discussed the status of those FY15-16 recommendations, which are

summarized below.

Assignment of Ratings

Fully addressed—

o resolved the identified issue

Partially addressed—Though not fully addressed, this rating reflects that the MHP has

either:

o made clear plans and is in the early stages of initiating activities to address the

recommendation

o addressed some but not all aspects of the recommendation or related issues

Not addressed—The MHP performed no meaningful activities to address the

recommendation or associated issues.

Key Recommendations from FY15-16

Recommendation #1: As recommended last year, collaborate with County

Administration to recruit the unfilled Data Coordinator position in order to continue to

move forward in the development of a data driven system of care.

☐ Fully addressed ☒ Partially addressed ☐ Not addressed

o This MHP has not filled its data analyst and/or an equivalent position within

their county system. The MHP has attempted to obtain this position through the

budget process without success.

o Alternatively, the MHP has recruited a Quality Improvement Coordinator who is

a licensed clinician to begin in August 2016. The QI coordinator may perform

analysis of data such as penetration rates, billing, access, demographics, etc.

Recommendation #2: Make staffing and space priorities for the southern portion of the

county. Track all timeliness measures reflected in the Timeliness Self-Assessment form,

for all regions for adult and children’s services. Allocate staffing and fiscal resources

that are commensurate with the capacity demands, prioritizing the amelioration of wait

lists and timeliness issues in South County.

☐ Fully addressed ☐ Partially addressed ☒ Not addressed

o This Item was not addressed as this recommendation was made in error and not

corrected before the report was finalized last year.

Recommendation #3: Reinvigorate the Cultural Competence Committee and

reestablish routine cultural competence activities and robust outreach efforts that

involve consumers/family members on both a larger scale (community feedback at

meetings, surveys) and also on a smaller scale (i.e., committee level planning and

implementation of outreach). Evaluate the effectiveness of the activities. Establish a

diverse leadership of cultural competency and outreach levels to ensure continuous

efforts.


o Although the MHP does not formally have a Cultural Competence Committee, the

MHP reformed its approach to carrying out the cultural competency goals. This

occurred due to the departure of the ethnic services manager the prior year.

o The MHP implemented a significant outreach effort to the Hispanic community

titled Cultura Es Vida in October 2015 with the Hispanic Heritage Program. The

MHP collaborated with the English Learner Advisory Committees to provide

twelve presentations to the preschools and schools addressing the

understanding of mental health issues. One hundred consumer surveys were

completed at the last presentation.

o The MHP has negotiated a Memorandum of Understanding (MOU) to increase

direct services to the Native American Community.

o The MHP has added bilingual Spanish speaking staff: Two added interns, a

clinician, with extra help (transportation) in January 2016. Additionally, the QI

Coordinator hired in August 2016 speaks Punjabi.

Recommendation #4: Evaluate the effectiveness of new crisis protocols through

quarterly data collection and analysis including comparison to baseline data, over time,

with demographics information as well as provider information to determine best

practices. Consider addressing this through a PIP, with consultation and technical

assistance provided by the EQRO.


o The MHP has been experiencing difficulties with its crisis protocols since April

2016 with the bankruptcy and closure of Colusa Regional Medical Center, the

only hospital in the county.

o The hospital closure has complicated the lives of consumers who in the past

could receive care locally after hours at the hospital 24/7. Since the closure

there is no available emergency room, the MHP crisis staff is impacted.

o The MHP is able to access all statistical data through April 2016. The MHP now

relies on information from Glenn and Sutter-Yuba counties. Glenn County

provides medical clearance for those assessed, while Sutter-Yuba accepts Colusa

MHP 5150s with reimbursement by Colusa County.

Recommendation #5: Create opportunities for line staff to encourage their involvement

and offer cross-training to expand skill sets which can support the system. Establish an

ongoing bi-directional feedback system between leadership, management and staff to

evaluate effectiveness and consumer/staff satisfaction.


o In the fall of 2015, the MHP formed a staff advisory committee (SAC) comprised

of leadership and other system wide staff. The SAC meets quarterly to identify

challenges in providing effective services and to evaluate consumer/staff

satisfaction.

o The staff has advisory committee meetings monthly.

o A staff survey from 2015 indicated that communication could be improved

between leadership and staff particularly around receiving feedback and

communicating changes. The MHP states that they have made some changes

based on the suggestions from staff, however, feedback from staff indicate that

there are still issues remaining.

Recommendation #6: Identify and incorporate a system wide consumer outcome tool

for the children’s system of care, such as the child behavior checklist, into the EHR and

establish monthly reporting of consumer functioning similar to the MHP’s

implementation of the MORs for adults. Use data to develop an evidence based quality

driven system.

☒ Fully addressed ☐ Partially addressed ☐ Not addressed

o The MHP installed the Child Behavior Check List (CBCL) in December 2015,

followed by staff training. System implementation occurred in January 2016.

o CBCL scoring is incorporated into the discharge summary in the EHR and

Medical Necessity form.

o The MHP should be able to perform data analyses with the newly hired QI

Coordinator who commences employment in August 2016.

Recommendation #7: Continue to create and implement ways to increase the peer

employee presence in programs. Increase consumer and family member involvement in

service related committees, including policy making committees. Ensure that there are

methods or venues for Spanish speaking stakeholders to participate.

☒ Fully addressed ☐ Partially addressed ☐ Not addressed

o In January 2016, two peer positions were added and were classified as extra help.

These are paid part time positions with no benefits.

o Three peers have attended certification training and the Department has

participated in regional training for peer certification. Certification involved a

13-month training on Peer Core Competency through the Northern CA WET

Alliance.

o There is peer participation in the QI Committee, the BH Advisory Board, and in

Consumer Leadership for Safe Haven Wellness and Recovery Center.

o The MHP identified bicultural staff within its department for the Cultura de la Vida

outreach program and from there, collaborated with the district learner advisory

committee (DLAC), English learner advisory committee (ELAC), and migrant

education programs. Through these efforts, the MHP is involving peers and

identifying natural leaders to develop a Promotores program.

CHANGES IN THE MHP ENVIRONMENT AND WITHIN THE MHP—IMPACT AND IMPLICATIONS

Changes since the last CalEQRO review, identified as having a significant effect on service provision

or management of those services are discussed below. This section emphasizes systemic changes

that affect access, timeliness, and quality, including those changes that provide context to areas

discussed later in this report.

Access to Care

o The MHP established a Cultura De La Vida program which outreaches to the

Hispanic community. Beginning in October 2015, the program has provided 12

presentations in the community on services, and various mental health topics.

o The local hospital went bankrupt and closed abruptly. There are no emergency

room services. Glenn County provides medical clearance for those assessed.

Sutter Yuba will accept 5150s with reimbursement by Colusa County.

o The wellness center was expanded to be the Safe Haven Wellness and Recovery

Center. The new larger site has more services/programing for consumers

including staff leads groups, as well as peer lead groups.

Timeliness of Services

o The closure of the hospital impacts the timeliness of consumers obtaining

medical clearance for psychiatric hospitalization.

Quality

o The MHP has an in-house psychiatrist full time. The MHP revamped its policy

and procedure regarding the medication of children. They no longer have

medication only child clients. If receiving medication, the child is required to be

in services.

o The MHP plans to install and begin implementation of a Telechart Patient Portal


o The MHP now has a county administrative officer who started in the spring of

2016. Historically the MHP had only a board of supervisors.

o Over the last year, the MHP reorganized its prevention programs to

accommodate staff changes. The previous prevention coordinator left, so the

MHP reorganized the duties and structure of the AOD prevention program. The

MHP’s Spanish speaking clinical intern became a permanent county position. In

addition, the MHP added case management to AOD and added another intern

who also speaks Spanish.

o The MHP hired/promoted several staff members including bringing on QI

Coordinator, who will start August 22, 2016, adding a clinician for adult services

(currently awaiting clearance), and case manager for children’s services.

o The MHP expanded its relationship with the Office of Education by assigning one

of the clinicians to provide services to children on probation that are attending

the Community School. The MHP has a presence in all schools in Colusa County.

Consumer Outcomes

o The MHP hired a new QI Coordinator who will be responsible for data analysis

and trending, including reports based on data from the Client Service

Information (CSI), California Outcomes Measurements System (CalOMS),

penetration rates, billing, walk in appointments, demographics, crisis reports

regarding contacts, outcomes, and hospitalization.

PERFORMANCE MEASUREMENT

CalEQRO is required to validate the following PMs as defined by DHCS:

Total Beneficiaries Served by each county MHP

Total Costs per Beneficiary Served by each county MHP

Penetration Rates in each county MHP

Count of TBS Beneficiaries Served Compared to the four percent (4%) Emily Q.

Benchmark (not included in MHP reports; this information is included in the Annual

Statewide Report submitted to DHCS)

Total Psychiatric Inpatient Hospital Episodes, Costs, and Average Length of Stay

Psychiatric Inpatient Hospital 7-Day and 30-Day Rehospitalization Rates

Post-Psychiatric Inpatient Hospital 7-Day and 30-Day SMHS Follow-Up Service Rates

High Cost Beneficiaries ($30,000 or higher)

TOTAL BENEFICIARIES SERVED

Table 1 provides detail on beneficiaries served by race/ethnicity.

Table 1—Colusa County MHP Medi-Cal Enrollees and Beneficiaries Served in CY15 by Race/Ethnicity

Race/Ethnicity Average Monthly Unduplicated

Medi-Cal Enrollees* Unduplicated Annual Count of

Beneficiaries Served

White 1,258 202

Hispanic 5,377 248

African-American 36 4

Asian/Pacific Islander 103 10

Native American 78 12

Other 671 47

Total 7,521 523

*The total is not a direct sum of the averages above it. The averages are calculated separately.

PENETRATION RATES AND APPROVED CLAIM DOLLARS PER BENEFICIARY

The penetration rate is calculated by dividing the number of unduplicated beneficiaries served by

the monthly average enrollee count. The average approved claims per beneficiary served per year

is calculated by dividing the total annual dollar amount of Medi-Cal approved claims by the

unduplicated number of Medi-Cal beneficiaries served per year.

Regarding calculation of penetration rates, the Colusa County MHP:

☒ Uses the same method as used by the EQRO

☐ Uses a different method

☐ Does not calculate its penetration rate

Figures 1A and 1B show 3-year trends of the MHP’s overall approved claims per beneficiary and

penetration rates, compared to both the statewide average and the average for Small Rural MHPs.

$0

$1,000

$2,000

$3,000

$4,000

$5,000

$6,000

CY13 CY14 CY15

Figure 1A. Overall Average Approved Claims per Beneficiary

Colusa Small-Rural State

0.00%

1.00%

2.00%

3.00%

4.00%

5.00%

6.00%

7.00%

8.00%

9.00%

10.00%

CY13 CY14 CY15

Figure 1B. Overall Penetration Rates


Figures 2A and 2B show 3-year trends of the MHP’s foster care (FC) approved claims per

beneficiary and penetration rates, compared to both the statewide average and the average for

Small Rural MHPs.

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

$16,000

CY13 CY14 CY15

Figure 2A. FC Average Approved Claims per Beneficiary


0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

CY13 CY14 CY15

Figure 2B. FC Penetration Rates


Figures 3A and 3B show 3-year trends of the MHP’s Hispanic approved claims per beneficiary and

penetration rates, compared to both the statewide average and the average for Small Rural MHPs.

$0

$1,000

$2,000

$3,000

$4,000

$5,000

$6,000

CY13 CY14 CY15


Figure 3A. Hispanic Average Approved Claims per Beneficiary

0.00%

1.00%

2.00%

3.00%

4.00%

5.00%

6.00%

7.00%

CY13 CY14 CY15

Figure 3B. Hispanic Penetration Rates


HIGH-COST BENEFICIARIES

Table 2 compares the statewide data for high-cost beneficiaries (HCB) for CY15 with the MHP’s data

for CY15, as well as the prior two years. HCB in this table are identified as those with approved

claims of more than $30,000 in a year.

Table C1 (Attachment C) shows the penetration rate and approved claims per beneficiary for the

CY15 Medi-Cal Expansion (Affordable Care Act [ACS]) Penetration Rate and Approved Claims per

Beneficiary.

Table C2 (Attachment C) show the distribution of the MHP CY15 Distribution of Beneficiaries by

Approved Claims per Beneficiary (ACB) Range for the various categories; under $20,000; $20,000

to $30,000, and those above $30,000.

MHP Year

HCB

Count

Total

Beneficiary

Count

HCB %

by

Count

Average

Approved

Claims

per HCB

HCB Total

Claims

HCB % by

Approved

Claims

Statewide CY15 13,851 483,793 2.86% $51,635 $715,196,184 26.96%

CY15 14 523 2.68% $57,859 $810,021 30.48%

CY14 8 534 1.50% $41,594 $332,754 14.97%

CY13 8 500 1.60% $44,346 $354,765 19.90%

Table 2—High-Cost Beneficiaries

Colusa

TIMELY FOLLOW-UP AFTER PSYCHIATRIC INPATIENT DISCHARGE

Figures 4A and 4B show the statewide and MHP 7-day and 30-day outpatient follow-up and

rehospitalization rates for CY14 and CY15.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Oupatient MHP Outpatient State RehospitalizationMHP

RehospitalizationState

Figure 4A. 7-Day Outpatient Follow-up and Rehospitalization Rates, Colusa MHP and State

CY14 CY15

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Oupatient MHP Outpatient State RehospitalizationMHP

RehospitalizationState

Figure 4B. 30-Day Outpatient Follow-up and Rehospitalization Rates, Colusa MHP and State

CY14 CY15

DIAGNOSTIC CATEGORIES

Figures 5A and 5B compare the breakdown by diagnostic category of the statewide and MHP

number of beneficiaries served and total approved claims amount, respectively, for CY15.

MHP self-reported percent of consumers served with co-

occurring (substance abuse and mental health) diagnoses:

0%

5%

10%

15%

20%

25%

30%

35%

Depression Psychosis Disruptive Bipolar Anxiety Adjustment Other Deferred

Figure 5A. Diagnostic Categories, Beneficiaries Served

Colusa CY15 State CY15

0%

5%

10%

15%

20%

25%

30%

35%

Depression Psychosis Disruptive Bipolar Anxiety Adjustment Other Deferred

Figure 5B. Diagnostic Categories, Total Approved

Colusa CY15 State CY15

20%

PERFORMANCE MEASURES FINDINGS—IMPACT AND IMPLICATIONS

Access to Care

o The MHP’s overall penetration rate has declined each year between CY13 and

CY15 similar to the trend experienced overall and across small rural MHPs. Its

penetration rate has been higher than statewide and small rural MHPs during

the same period.

o The MHP’s foster care penetration rate is similar to small rural MHPs and lower

than the statewide.

o The MHP’s Hispanic penetration rate is the same as small rural MHPs and higher

than statewide. The MHP’s Hispanic penetration rates have declined between

CY13 and CY15 similar to the downward trend experienced statewide.


o The MHP’s 7-day and 30-day outpatient follow-up rates after discharge from

psychiatric inpatient episodes were higher than statewide.

Quality of Care

o The MHP’s percentage of high-cost beneficiaries was similar to statewide and

showed an increase from CY14. Its percentage of HCB claim dollars was higher

than statewide and doubled from CY14 to CY15.

o The MHP’s average approved claims per beneficiary served was lower than

statewide and higher than small rural MHP’s. The MHP’s average approved

claims per beneficiary served has increased consistently for the three years

between CY13 and CY15.

o The MHP’s average approved claims per beneficiary for foster care is higher

than both statewide and small rural MHPs.

o The MHP’s average approved claims per beneficiary for Hispanics is higher than

small rural MHPs but lower than statewide.

o Like statewide, a primary diagnosis of Depressive Disorders accounted for the

largest number of beneficiaries served by the MHP. However, the percentage

was higher than statewide. The MHP had a higher percentage of beneficiaries

with primary diagnosis of anxiety than statewide.

o The MHP appears to use Deferred Diagnosis slightly lower than statewide.

Consumer Outcomes

o None noted.

PERFORMANCE IMPROVEMENT PROJECT VALIDATION

A Performance Improvement Project (PIP) is defined by CMS as “a project designed to assess and

improve processes, and outcomes of care that is designed, conducted and reported in a

methodologically sound manner.” The Validating Performance Improvement Projects Protocol

specifies that the EQRO validate two PIPs at each MHP that have been initiated, are underway, were

completed during the reporting year, or some combination of these three stages. DHCS elected to

examine projects that were underway during the preceding calendar year 2015.

COLUSA MHP PIPS IDENTIFIED FOR VALIDATION

Each MHP is required to conduct two PIPs during the 12 months preceding the review. CalEQRO

reviewed and validated two MHP submitted PIPs as shown below.

Table 3A—PIPs Submitted

PIPs for Validation # of PIPS PIP Titles

Clinical PIP 1 Recovery

Non-Clinical PIP 1 Parent Intervention

Table 3B lists the findings for each section of the evaluation of the PIPs, as required by the PIP

Protocols: Validation of Performance Improvement Projects.4

Table 3B—PIP Validation Review

Step PIP Section Validation Item

Item Rating*

Clinical PIP

Non-Clinical

PIP

1 Selected Study Topics

1.1 Stakeholder input/multi-functional team M M

1.2 Analysis of comprehensive aspects of enrollee needs, care, and services

M M

1.3 Broad spectrum of key aspects of enrollee care and services

M M

1.4 All enrolled populations M M

2 Study Question 2.1 Clearly stated NM M

4 2012 Department of Health and Human Services, Centers for Medicare and Medicaid Service Protocol 3

Version 2.0, September 2012. EQR Protocol 3: Validating Performance Improvement Projects.



Item Rating*

Clinical PIP

Non-Clinical

PIP

3 Study Population 3.1 Clear definition of study population M M

3.2 Inclusion of the entire study population M M

4 Study Indicators

4.1 Objective, clearly defined, measurable indicators

M M

4.2 Changes in health status, functional status, enrollee satisfaction, or processes of care

M M

5 Sampling Methods

5.1 Sampling technique specified true frequency, confidence interval and margin of error

NM NA

5.2 Valid sampling techniques that protected against bias were employed

M NA

5.3 Sample contained sufficient number of enrollees

PM NA

6 Data Collection Procedures

6.1 Clear specification of data M M

6.2 Clear specification of sources of data PM M

6.3 Systematic collection of reliable and valid data for the study population

PM PM

6.4 Plan for consistent and accurate data collection

M PM

6.5 Prospective data analysis plan including contingencies

NM NM

6.6 Qualified data collection personnel M M

7 Assess Improvement Strategies

7.1 Reasonable interventions were undertaken to address causes/barriers

UTD M

8

Review Data Analysis and Interpretation of Study Results

8.1 Analysis of findings performed according to data analysis plan

NM NM

8.2 PIP results and findings presented clearly and accurately

M NA

8.3 Threats to comparability, internal and

external validity NM NA

8.4 Interpretation of results indicating the success

of the PIP and follow-up NM NA

9 Validity of Improvement

9.1 Consistent methodology throughout the study UTD NA

9.2 Documented, quantitative improvement in

processes or outcomes of care NA NA



Item Rating*

Clinical PIP

Non-Clinical

PIP

9.3 Improvement in performance linked to the

PIP NA NA

9.4 Statistical evidence of true improvement NA NA

9.5 Sustained improvement demonstrated

through repeated measures. NA NA

*M = Met; PM = Partially Met; NM = Not Met; NA = Not Applicable; UTD = Unable to Determine

Table 3C gives the overall rating for each PIP, based on the ratings given to the validation items.

Table 3C—PIP Validation Review Summary

Summary Totals for PIP Validation Clinical

PIP

Non-Clinical

PIP

Number Met 13 13

Number Partially Met 3 2

Number Not Met 6 2

Number Applicable (AP)

(Maximum = 28 with Sampling; 25 without Sampling)

24 17

Overall PIP Rating ((#Met*2)+(#Partially Met))/(AP*2) 60.42% 82.35%

CLINICAL PIP—RECOVERY

The MHP presented its study question for the clinical PIP as follows:

“Can the Milestones of Recovery (MORS) rating scale be used to effectively guide

treatment interventions?”

Date PIP began: January 2015

Status of PIP:

☒ Active and ongoing

☐ Completed

☐ Inactive, developed in a prior year

☐ Concept only, not yet active

☐ Submission determined not to be a PIP

☐ No PIP submitted

The MHP, as a result of its participation in the Advancing Recovery Collaborative, collected MORS

scores for all adult consumers beginning in January 2015. Higher than expected MORS scores for

13% of the study population led to the MHP focusing on transitioning consumers to lower levels of

care. The MHP aimed to improve outcomes for adults, including “graduating” from services of the

MHP. The PIP examined which interventions assisted consumers in recovery as measured using the

Milestones of Recovery outcome tool (MORs), leading to shortened and consistent time to

transition to lower level levels of care.

The MHP collected MORS scores monthly and found that individuals at MORS 7 were able to

transition out of services, as well as those at MORS 6. The MHP then focused on individuals with a

MORS score of 5. In contrast to the MORS 6 and 7 individuals, these individuals were not able to

move quickly up the MORS scale. These individuals stayed at MORS 5 throughout the measurement

cycle. Previously successful Interventions were doubled (Strength Assessment and Group

Supervision). Even with these interventions, little movement was seen for the MORS 5 individuals.

The MHP is now testing the outcome tools that may be more sensitive to smaller changes (PHQ 9

and the GAD 7) for individuals with depression or anxiety diagnoses as they can be administered

more frequently.

The study question design did not lend itself to be answerable through the PIP process.

Relevant details of these issues and recommendations are included within the comments found in

the PIP validation tool.

Technical assistance was provided to the MHP by CalEQRO. Recommendations include a discussion

on how the study question design did not lend itself to be answerable through the PIP process.

Recommend that for next PIP, set up question so that it is quantifiable and contains indicators, i.e.

“Will ____ intervention improve ________ (symptoms/problem) as measured by ________outcome tool?”

As it stands, the PIP’s design does not allow the MHP to track the new outcome tools and does not

identify which interventions improve which symptoms. Clarification on whether MORs 6s and 7s

entered at the 5 level when they came in for services initially. This would allow for a meaningful

comparison regarding how long it took to get to the next level. For next steps, the MHP plans to

amend its question, evaluate the impact of strengths assessment on “leveling up” using the MORs, in

conjunction with the anxiety and depression inventory tools. Recommended that this PIP be

considered completed rather than going into year 3 of this PIP. Instead, develop a new PIP

evaluating clinical outcomes post intervention using the anxiety and depression assessments. As it

stands, this process didn’t make an improvement for clients as they were still discharged for the

same reason which was not a result of better implementation of the MORS. Rather, the PIP provided

more information regarding level of care and helped inform the administrative process of

discharge/level of care.

NON-CLINICAL PIP—PARENT INTERVENTION

The MHP presented its study question for the non-clinical PIP as follows:

“Will the provision parental intervention result in shorter duration of care, higher levels

of parent satisfaction with care, and improved functioning as measured by the Child

Behavior Checklist?”

Date PIP began: August, 2015

Status of PIP:

☒ Active and ongoing

☐ Completed


☐ Concept only, not yet active


☐ No PIP submitted

This PIP is at the beginning of its second year. Year 1 focused on shortening duration of services

with the addition of conjoint family services as a standard protocol. The second year of the PIP is

aimed at evaluating the impact of conjoint (automatically assigned) family interventions,

specifically Parent-Child Interaction Therapy (PCIT) on client functioning. The overarching goal of

this PIP will be to improve child functioning in many areas relevant to the child’s progress toward

resilience. The MHP analyzed data that compared the length of treatment for children receiving

family therapy of any kind or duration beyond intake as part of their treatment with children who

had not. The MHP looked at this data for fiscal year 2014/2015 and found that 145 children had

received family therapy, where 62 children had not. When comparing time in treatment for the two

groups, the MHP found that children receiving any form of family involvement (145 children)

averaged 3.763889 months in care and those that received no form of family involvement (63

children) averaged 3.967742 months. Remeasurement has proven to be a challenge. The MHP had

difficulty rolling out the CBCL due to scoring challenges and difficulty in getting the score entered

into the EHR. Additionally, the MHP discovered that they were not getting discharge CBCL scores

because families were choosing to withdraw from care when the clinician began discussing the end

of treatment. Further, no one has been in care for a full year since the implementation of the CBCL.

The MHP experienced similar issues with the parent satisfaction survey. A way to ameliorate this

type of issue would be to collect data on a quarterly basis.

Relevant details of these issues and recommendations are included within the comments found in

the PIP validation tool.

Technical assistance was provided to the MHP by CalEQRO. Feedback included acknowledgment

that the PIP focus seemed to naturally shift to evaluating the PCIT intervention. The study question

needed clarification so that it was measurable an objective. The interventions listed are not truly

the intervention, but rather assigning all mild to moderate clients to PCIT and all moderate to

severe clients to individual as a matter of standard procedure. It was recommended that the MHP

then compare the CBCL scores of from a baseline at start of treatment after a set period of time (i.e.

3 months). This would allow the county to see the true effectiveness of the new administrative

procedure. Data should be collected and analyzed quarterly.

PERFORMANCE IMPROVEMENT PROJECT FINDINGS—IMPACT AND IMPLICATIONS

Access to Care

o If the MHP is successful identifying and implementing interventions that move

clients along the continuum of care, and decrease the level of service provided,

more space for new clients would be available.


o If the MHP is successful identifying and implementing interventions that move

clients along the continuum of care, and decrease the level of service provided,

timeliness for appointments for new clients would improve with the new “slots”

available.

o If client functioning improves and they are able to transition out of service, more

timely appointments are available for new clients coming in.

Quality of Care

o Interventions which are effective in helping clients to improve functioning

contribute to the quality of services provided by the MHP and convey the

message of hope.

o Engaging parents in children’s treatment improves quality of care for children.

Consumer Outcomes

o Engaging parents in children’s treatment results in better outcomes for child

clients.

o Using anxiety and depression symptom inventories would allow for more

nuanced assessment of client outcomes and would prove useful for evaluating

client functioning, involvement and perceptions of treatment.

PERFORMANCE & QUALITY MANAGEMENT KEY COMPONENTS

CalEQRO emphasizes the MHP’s use of data to promote quality and improve performance.

Components widely recognized as critical to successful performance management include an

organizational culture with focused leadership and strong stakeholder involvement, effective use of

data to drive quality management, a comprehensive service delivery system, and workforce

development strategies that support system needs. These are discussed below.

Access to Care

As shown in Table 4, CalEQRO identifies the following components as representative of a broad

service delivery system that provides access to consumers and family members. An examination of

capacity, penetration rates, cultural competency, integration and collaboration of services with

other providers forms the foundation of access to and delivery of quality services.

Table 4—Access to Care

Component Compliant

(FC/PC/NC)* Comments

1A Service accessibility and availability are reflective of cultural competence principles and practices

FC Cultural, ethnic, racial, and linguistic needs are assessed at intake. Over the last year, the MHP prioritized hiring bilingual and bicultural staff as well as identifying current staff for resources within.

Treatment outcomes are evaluated, however, the MHP would benefit from additional staffing for analysis. The MHP developed the Cultural Vida program and provided 12 presentations within the community beginning October 2015. From there, the MHP collaborated with the district level English learner advisory committee (DLAC), and English learner advisory committee (ELAC), both migrant education programs. The Cultural Vida Program is working to identify natural leaders to develop a Promotores Program. The Cultural Vida program are working with the community and the office of education with their presentation “how mental health pertains to you.” The need for this program was identified through a survey where people said they were aware of mental health but didn’t know how it pertained to them. With this presentation, there will be a pre/post survey

The MHP is using MHSA money to supplement staff salaries so that all staff can are involved in outreach and engaging the community.

1B Manages and adapts its capacity to meet beneficiary service needs

FC The MHP tracks clients coming in for appointments and also assesses the need for bilingual services.

The MHP utilizes an office assistant who speaks Spanish and assists psychiatry. The previous prevention coordinator left. As a result, the MHP reorganized duties

Table 4—Access to Care

Component Compliant

(FC/PC/NC)* Comments and structure of the program. A Spanish speaking intern became a permanent county position. The MHP also drew from the children’s department and added another intern for this program who also speaks Spanish.

1C Integration and/or collaboration with community based services to improve access

FC The psychiatry department works with primary care providers in the area. There is also a collaboration for suicide prevention training for medical staff. They do not have contract providers in the area. The MHP has integrated AOD services with mental health. They participate in joint trainings. Trainings were offered to public health but they did not attend.

One Stop manages the transitional housing for CalWorks. However, there is no short term or long term housing in the county.

In April 2016, the local hospital went bankrupt and closed abruptly. There is no access to emergency room services. Glenn and Sutter-Yuba counties are assisting with 5150s. Glenn County provides medical clearance for those assessed. And Sutter Yuba will accept 5150s with reimbursement by Colusa County.

The MHP continues to collaborate with law enforcement agencies for crisis services. However, the MHP’s request to California Highway Patrol (CHP) to limit 5150s to Colusa residents was unsuccessful. CHP still brings out of town folks to 5150. This impacts crisis staff which is an informal mobile crisis unit as there is no official funding stream or designated vehicle.

*FC =Fully Compliant; PC = Partially Compliant; NC = Non-Compliant


As shown in Table 5, CalEQRO identifies the following components as necessary to support a full

service delivery system that provides timely access to mental health services. The ability to provide

timely services ensures successful engagement with consumers and family members and can

improve overall outcomes while moving beneficiaries throughout the system of care to full

recovery.

Table 5—Timeliness of Services

Component Compliant


2A Tracks and trends access data from initial contact to first appointment

FC The MHP sets a standard of 10 days, with 100% of its appointments meeting this standard. The range is 0-9

Table 5—Timeliness of Services

Component Compliant


days for both adults and children with an average of 3 days.

2B Tracks and trends access data from initial contact to first psychiatric appointment

PC The MHP monitors time from assignment to psychiatry (MD) to first psychiatry appointment. The MHP sets a standard of 45 days, with 100% of its appointments meeting this standard. The range is 2-33 days for with an average of 16.6 days. Adult and children services are not tracked separately. In light of the change in policy (fall 2015) regarding medication management by children’s psychiatry and the elimination of meds only practice (all children receiving medications are required to be in services), it is recommended that this measure be tracked and trended separately.

2C Tracks and trends access data for timely appointments for urgent conditions

FC The MHP reported a goal of .125 days (3 hours) wait times for urgent appointments and that they met this goal 80% of the time, with an average of .125 days.

2D Tracks and trends timely access to follow up appointments after hospitalization

FC The MHP reported 24 hospitalizations (22 adult, 2 youth) and that all clients received a follow up appointment within 7 days of discharge. The MHP sets a goal of 7 days. The average length of time for a follow up appointment with clinical staff was 1.7 days, with 100% of the appointments meeting the standard.

Last year, the MHP reported 8 hospitalizations (7 adults, 1 child) from January through June 2015 (6 months). Hospitalization rates increased this last year, even taking into account that the data for 15/16 spanned 12 months rather than last year when the MHP reported on 6 months of data.

2E Tracks and trends data on re-hospitalizations

FC The MHP reports that of the 24 hospitalizations (22 adult, 2 youth), no clients were re-hospitalized. The MHP sets a goal of 0 rehospitalizations within 30 days of discharge.

2F Tracks and trends No Shows

NC The MHP reports that they are just beginning to extract data on this measure and that they have not yet developed a format for tracking/trending.

*FC = Fully Compliant; PC = Partially Compliant; NC = Non-Compliant

Quality of Care

As shown in Table 6, CalEQRO identifies the following components of an organization that is

dedicated to the overall quality of care. Effective quality improvement activities and data-driven

decision making require strong collaboration among staff (including consumer/family member

staff), working in information systems, data analysis, clinical care, executive management, and

program leadership. Technology infrastructure, effective business processes, and staff skills in

extracting and utilizing data for analysis must be present in order to demonstrate that analytic

findings are used to ensure overall quality of the service delivery system and organizational

operations.

Table 6—Quality of Care

Component Compliant


3A Quality management and performance improvement are organizational priorities

FC The MHP hired a new Quality Improvement

Coordinator who may perform analysis of data such as

penetration rates, billing, access, demographics, etc.

3B Data are used to inform management and guide decisions

FC Based on intake tracking data on demographics, client

numbers, language and type of service need, the MHP

changed its intake schedule from 2 clinicians to 3,

including a Spanish speaker.

The MHP assesses client outcomes using the

Milestones of Recovery scale (MORs) for all adults once

per month via reporting from the EHR. The MHP will be

developing its trending ability with its new QI and EHR

coordinator. They are also working to get a depression

inventory tool (PHQ9) and anxiety inventory tool

(GAD7) into to the EHR.

The MHP is using the Child Behavior Checklist (CBCL) for children. All children receive the CBCL with results in the EHR. The MHP is beginning to look at group data.

3C Evidence of effective communication from MHP administration

PC The MHP holds a variety of meetings for

communication both within the MHP and in the

community. These include quarterly all-staff, weekly

clinical teams, Katie A., multidisciplinary teams each

month, QIC meets every other month.

Leadership/management meets weekly.

The MHP has a monthly newsletter in Spanish and

English They also have Facebook pages for

adults/children’s services. Client leadership monthly

meets to oversee and run the drop-in center. Peers


Component Compliant

(FC/PC/NC)* Comments Helping Peers meets twice per month to provide

information on the wellness center.

Clinical staff participation during the review was

limited and not all staff who wished to participate were

permitted.

The MHP formed a representative staff advisory

committee that meets regularly to identify challenges

in providing effective services and to evaluate

consumer/staff satisfaction. However, clinical staff

reported that while the opportunity to provide ideas

and feedback exists, they felt that they had little input

in program and system planning. The consensus

among line staff was that morale is very low, citing a

punitive system if productivity is not met (either by a

little or a lot). The feeling was that there is a big

disconnect with what upper management perceives is

happening vs. what is really going on.

The MHP did make changes to its all staff meetings and

how the meeting is run. The agenda is posted before

the meeting so that those who wish to provide

feedback can. There are now standing items at each

meeting. Most recently, there was a staff survey

regarding how acknowledgements are made.

3D Evidence of stakeholder input and involvement in system planning and implementation

PC CFM participants are actively involved in running the

wellness center.

Because of the low attendance the prior year at the

MHSA stakeholder planning meeting, this year, the

MHP alternatively held meetings within each

community – Maxwell, Arbuckle, Williams and Colusa.

Spring 2016. They were able to update the plan. 10-15

people participated.

During Mental Health Awareness Month, May 2016,

the MHP held 4 events targeting specific groups most

of which had low turnout. However, the most

successful event was a Safe Haven Peer Walk In – with

Veterans centered on a car show with 33 cars. The

MHP outreached and raised $900. There were 100-150

participants.


Component Compliant

(FC/PC/NC)* Comments A staff survey from 2015 provided feedback that

communication was poor regarding feedback and

subsequent changes. The MHP states that they have

made some changes due to suggestions from staff,

however, feedback from staff indicate that there are

still issues regarding how feedback is received and

whether it is acted upon effectively by leadership.

3E Evidence of strong collaborative partnerships with other agencies and community based services

FC As stated above, the MHP has strong collaborative

relationships with community agencies such as law

enforcement (Colusa County Sheriff, Colusa Police

Department., Williams Police Department, and

California Highway Patrol) as well as probation, and

health and human services. The MHP also has a robust

youth program through the school district. The county

has partnered with Glenn and Sutter-Yuba counties to

cover the gap in mental health services previously

provided by the local hospital, which closed abruptly in

April, 2016.

3F Evidence of a systematic clinical Continuum of Care

FC The MHP uses the MORS as an outcome tool in adult

services. This instrument is scored monthly on “MORS

Monday” which is the last Monday of the month. As

noted in the PIP write up, this tool is used to measure

consumer recovery status, and to assist with treatment

planning. The PHQ 9 was tested in the adult division as

a quick measurement tool for depression; and the GAD

7 as a quick measurement tool for generalized anxiety.

The MHP recently added the use of the Child Behavior

Check List (CBCL) in the Child division. This tool is

administered at intake, annually and at discharge. To

date no discharge CBCL’s have been recorded.

Regarding, medication monitoring, the QI plan

referenced that the MHP had not been compliant in

having prescribing practices reviewed since the

departure of psychiatrist Dr. Samson. The MHP added

this as an action item to the annual plan.

3G Evidence of individualized, client-driven treatment and recovery

PC The MHP’s drop-in center moved in June 2016 to a

larger space. The MHP wanted to maintain the peer

drop in center but within the context of wellness and

recovery. The center was renamed Safe Haven


Component Compliant

(FC/PC/NC)* Comments Wellness & Recovery Center. In addition to drop-in

services, the center now has groups – nutrition,

mindfulness, and walking. Staff leads groups, as well as

peers. Wellness Recovery Action Planning (WRAP) will

be starting as some staff are trained WRAP leaders.

While the MHP utilizes depression and anxiety

inventories with clients in session, information on how

consumers are engaged in their own treatment

planning and how a consumer uses level of function

scales/tools, etc. themselves was not provided.

3H Evidence of consumer and family member employment in key roles throughout the system

FC Peer Specialist (a full time benefited position) manages

greeter positions at Safe Haven. Greeters receive

stipends.

The peer specialist recently completed a 13 month

training on Peer Core Competency through the

Northern California Workforce Education and Training

(WET) alliance and will be implementing those

competencies at the wellness center.

There are 5 part time drivers and safety support staff.

They are on call, work no more than 20 hours per

week.

3I Consumer run and/or consumer driven programs exist to enhance wellness and recovery

FC The MHP’s wellness center is the Safe Haven Wellness & Recovery Center. During the site visit, there were at least 3 consumers there; a full-time CFM, and 2 part-time CFM’s. The center is open to whomever comes through the doors. Service and program information were available at the desk. The operating hours are Monday through Saturday 8:30-4:30 pm. Clients receive a calendar of activities in their intake packet at assessment.

3J Measures clinical and/or functional outcomes of consumers served

FC The MHP assesses client outcomes using Milestones of

Recovery Scale (MORs) for all adults once per month

via reporting from the EHR. The MHP will developing

its trending ability with its new QI and EHR

coordinator. They are also working to get a depression

inventory tool (PHQ9) and anxiety inventory tool

(GAD7) into the EHR. The MHP is using the Child

Behavior Checklist (CBCL) for children. All children

receive the CBCL with results in the EHR. This was


Component Compliant

(FC/PC/NC)* Comments implemented August 2015 as part of the medical

necessity form. The MHP is beginning to look at

grouped data.

3K Utilizes information from Consumer Satisfaction Surveys

PC The MHP does provide the perception and outcomes

survey but does not analyze the data prior to State

submission. The State sends raw data back for review.

The MHP does retain the comments. They added

parenting classes as a result of POQI feedback.

*FC = Fully Compliant; PC = Partially Compliant; NC = Non-Compliant

KEY COMPONENTS FINDINGS—IMPACT AND IMPLICATIONS

Access to Care

o The MHP has added bilingual Spanish speaking staff: Two added interns, a

clinician and a QI Coordinator in August 2016, with extra help transportation

positions added in January 2016.

o In April 2016, the local hospital went bankrupt and closed abruptly. There is no

access to emergency room services. The MHP collaborates with Glenn County to

provide medical clearance for those needing assessment while Sutter Yuba MHP

accepts Colusa County 5150s with reimbursement by Colusa County.

o The MHP is also collaborating with law enforcement to assist consumers in

crisis.

o The MHP has an informal mobile crisis unit as there is no official funding stream

or designated vehicle.


o The MHP has reported no change for length of time from first request for service

to first clinical assessment from FY15-16 to FY16-17 – average length of time is

3 days surpassing their goal of 10 days.

o The MHP has reduced their average length of time from first request of service

to first psychiatric appointment from 24.69 days for FY15-16 to 16.6 days for

FY16-17 for all consumers.

o The MHP does not track no shows stating they only recently have had the

capability to track electronically.

Quality of Care

o Based on intake tracking data on demographics, language, and type of service

needed, the MHP changed its intake schedule team from 2 clinicians to 3

including a Spanish speaker.

o They are also working to get a depression inventory tool (PHQ9) and anxiety

inventory tool (GAD7) into the EHR.

o The MHP recently hired a new QI Coordinator who will be responsible for data

analysis and trending, including reports based on data from the Client Service

Information (CSI), California Outcomes Measurements System (CalOMS),

penetration rates, billing, walk in appointments, demographics, crisis reports

regarding contacts, outcomes, and hospitalization.

Consumer Outcomes

o The MHP is using the Child Behavior Checklist (CBCL) for children. All children

receive the CBCL with results in the EHR. The MHP is beginning to look at group

data.

o The MHP assesses client outcomes using Milestone of Recovery (MORs) for all

adults once per month via reporting from the EHR. The MHP plans to develop its

trending ability with its new QI and EHR coordinator.

o The MHP did a Clinical PIP utilizing the MORs to assess how interventions assist

consumers in recovery.

CONSUMER AND FAMILY MEMBER FOCUS GROUP(S)

CalEQRO conducted one (1) 90-minute focus group with consumers and family members during the

site review of the MHP. As part of the pre-site planning process, CalEQRO requested one (1) focus

group with 8 to 10 participants each, the details of which can be found in each section below.

The Consumer/Family Member Focus Group is an important component of the CalEQRO Site

Review process. Obtaining feedback from those who are receiving services provides significant

information regarding quality, access, timeliness, and outcomes. The focus group questions specific

to the MHP reviewed and emphasized the availability of timely access to care, recovery, peer

support, cultural competence, improved outcomes, and consumer and family member involvement.

CalEQRO provided gift certificates to thank the consumers and family members for their

participation.

CONSUMER/FAMILY MEMBER FOCUS GROUP 1

CalEQRO requested a culturally diverse group of adult beneficiaries and parents/caregivers of

child/youth beneficiaries including a mix of existing and new clients who have initiated/utilized

services within the past 12 months. The focus group was held at the MHP offices in Willows, located

at 162 E Carson Street, Colusa, CA 95932.

The group was comprised of 7 women and 3 men, 6 of which received services in the last year.

Number of participants: 10

For the six (6) participants who entered services within the past year, they described their

experience as the following:

Regarding timeliness of first contact to assessment, the responses varied from the next

day, one week, or within two weeks.

Regarding timeliness of assignment to and meeting with a therapist, responses also

varied, from one week, a couple of weeks, to a few weeks.

General comments regarding service delivery that were mentioned included the following:

The consensus of the group was then even if in crisis, it takes two to three weeks to get

help and by that time, the crisis is over.

Impacted by staff turnover or leave, reassignment of therapists takes a long time.

Public transportation is unreliable.

Recommendations for improving care included the following:

Timely response in rescheduling missed appointments or assignment to therapist after

in-take.

Focus groups for veterans.

Provide intakes more than once-a-week.

Provide transportation to outer areas.

Interpreter used for focus group 1: ☐ No ☒ Yes Language(s): Spanish

CONSUMER/FAMILY MEMBER FOCUS GROUP FINDINGS—IMPLICATIONS

Access to Care

o Clients’ experiences with accessing initial care were positive. However, there is

inconsistencies in the process for being assigned a therapist and for ongoing

services.

o Transportation from outer areas contributes to service access issues.


o Services impacted by staff shortages make it challenging for clients to

reschedule appointments in a timely manner.

o Clients reported that access to crisis services was not timely.

Quality of Care

o When receiving services, overall, focus group members reported satisfaction

with the services, and felt they were getting better.

Consumer Outcomes

o No information provided.

INFORMATION SYSTEMS REVIEW

Knowledge of the capabilities of an MHP’s information system is essential to evaluate the MHP’s

capacity to manage the health care of its beneficiaries. CalEQRO used the written response to

standard questions posed in the California-specific ISCA, additional documents submitted by the

MHP, and information gathered in interviews to complete the information systems evaluation.

KEY ISCA INFORMATION PROVIDED BY THE MHP

The following information is self-reported by the MHP in the ISCA and/or the site review.

Table 8 shows the percentage of services provided by type of service provider:

Table 8—Distribution of Services by Type of Provider

Type of Provider Distribution

County-operated/staffed clinics 99.5%

Contract providers 0.5%

Network providers 0%

Total 100%

Percentage of total annual MHP budget is dedicated to support information technology

operations: (includes hardware, network, software license, IT staff)

7.2%

Consumers have on-line access to their health records either through a Personal Health

Record (PHR) feature provided within EHR or a consumer portal or a third-party PHR:

☐ Yes ☐ In Test/Pilot Phase ☒ No

MHP currently provide services to consumers using an tele-psychiatry application:

☐ Yes ☐ In Test/Pilot Phase ☒ No

o If yes, the number of remote sites currently operational:

0

MHP self-reported technology staff changes since the previous CalEQRO review (FTE):

Table 9 – Summary of Technology Staff Changes

Number IS

Staff

Number of New

Hires

Number of Staff Retired,

Transferred, Terminated

Current Number of

Unfilled Positions

3 0 0 0

MHP self-reported data analytical staff changes since the previous CalEQRO review

(FTE):

Table 10 – Summary of Data Analytical Staff Changes

Number

Data Analytical

Staff

Number of New

Hires

Number of Staff Retired,

Transferred, Terminated

Current Number of

Unfilled Positions

0 0 0 0

The following should be noted with regard to the above information:

The MHP has recruited a QI Coordinator who is a licensed clinician to begin in August

2016. Their primary job responsibilities may encompass more intensive analytical

tasks.

CURRENT OPERATIONS

The MHP continues to implement the Cerner Behavioral Health System (CCBH) via an

Application Service Provider (ASP) contract with Kings View Behavioral Health.

Table 11 lists the primary systems and applications the MHP uses to conduct business and manage

operations. These systems support data collection and storage, provide electronic health record

(EHR) functionality, produce Short-Doyle/Medi-Cal (SD/MC) and other third party claims, track

revenue, perform managed care activities, and provide information for analyses and reporting.

Table 11— Primary EHR Systems/Applications

System/Application Function Vendor/Supplier Years Used Operated By

Cerner Community Behavioral Health (CCBH) Client Data

Practice Management Cerner 5 Kings View

CCBH - ATP Assessment and Treatment Plan

Cerner 5 Kings View

CCBH - Scheduling Appointment Scheduler Cerner 5 Kings View

CCBH Doctor's Homepage Clinical and ePrescribing Cerner 5 Kings View

CCBH-Clinician’s Homepage Clinical information and functionality

Cerner 5 Kings View

PLANS FOR INFORMATION SYSTEMS CHANGE

The MHP has no plans for information systems change.

ELECTRONIC HEALTH RECORD STATUS

Table 12 summarizes the ratings given to the MHP for Electronic Health Record (EHR) functionality.

Table 12—Current EHR Functionality

Function System/Application

Rating

Present Partially Present

Not Present

Not Rated

Alerts Cerner x

Assessments Cerner x

Document imaging/storage Cerner EHR x

Electronic signature—consumer Cerner EHR x

Laboratory results (eLab) x

Level of Care/Level of Service Cerner EHR x

Outcomes MORS, CBCL x

Prescriptions (eRx) Cerner EHR x

Progress notes Cerner EHR x

Treatment plans Cerner EHR x

Summary Totals for EHR Functionality 7 3

Progress and issues associated with implementing an electronic health record over the past year

are discussed below:

The MHP has no current plans to implement eLab functionality.

The MHP has no current plans to implement electronic Level of Care/Level of Service

Assessments.

Consumer’s Chart of Record for county-operated programs (self-reported by MHP):

☐ Paper ☐ Electronic ☒ Combination

MAJOR CHANGES SINCE LAST YEAR

The MHP have met Meaningful Use standards for Phase 1 and will be able to use

Citrix/Cerner following system upgrade.

The MHP amended both the Medical Necessity form and Discharge Assessment form to

allow entry and scoring of the Children’s Behavioral Checklist (CBCL), which measures

outcomes.

PRIORITIES FOR THE COMING YEAR

The MHP plans to install a SQL upgrade, followed by installation of CCBH - Promotion

222. The installation of Promotion 222 will enable Ultra-Sensitive Exchange (USX) with

client portal which will allow secure client messaging.

The MHP has created two new assessments to accurately measure outcomes for anxiety

GAD-7 (General Anxiety Disorder – 7 Item) and major depressive disorders PHQ-9

(Patient Health Questionnaire – 9 Item). These assessments were sent to Kings View for

uploading in July 2016, but at the time of the review were not uploaded. The MHP is

projecting these will be implemented in mid-August 2016.

The MHP plans to install and begin implementation of a Telechart Patient Portal


The MHP plan to develop and implement Client Portal documents, client training and

program security.

The MHP continues to pursue Meaningful Use, Stage I attestation for the EHR incentive

program with a projected target date of October 2016.

The MHP plans to begin test, training and implementation of the non-axial diagnosis

review form.

OTHER SIGNIFICANT ISSUES

The MHP states that some of the clinicians are unable to perform the electronic provider

signature function when using their laptops out in the field.

MEDI-CAL CLAIMS PROCESSING

Normal cycle for submitting current fiscal year Medi-Cal claim files:

☒ Monthly ☐ More than 1x month ☐ Weekly ☐ More than 1x weekly

MHP performs end-to-end (837/835) claim transaction reconciliations:

☒ Yes ☐ No

If yes, product or application:

Kings View Reports

Method used to submit Medicare Part B claims:

☐ Clearinghouse ☒ Electronic ☐ Paper

INFORMATION SYSTEMS REVIEW FINDINGS—IMPLICATIONS

Access to Care

o The MHP mandated use of CCBH Scheduler to produce weekly reports to

determine what percentage of consumers are walk-ins without appointments.

Number

Submitted

Gross Dollars

Billed

Dollars

Denied

Percent

Denied

Number

Denied

Gross Dollars

Adjudicated

Claim

Adjustments

Gross Dollars

Approved

14,529 $2,744,177 $31,417 1.14% 255 $2,712,760 $66,501 $2,646,259

Table 13 - Colusa MHP Summary of CY15 Processed SDMC Claims

Note: Includes services provided during CY15 with the most recent DHCS processing date of May 19,2016

o The MHP did not delineate between adults and children when tracking length of

time from initial contact to first psychiatric appointment for FY16-17. The MHP

did delineate between adults and children for FY15-16. The MHP reports this

was due to losing their child psychiatrist in June 2015.


o The MHP does not track and trend no shows. However, the MHP noted that they

only recently were able to extract data from their EHR. The MHP would like

direction in tracking and trending from other counties as well as dialogue on

what other counties’ interventions have been successful to reduce their no show

rate.

Quality of Care

o The MHP plans to install SQL upgrade, followed by CCBH Promotion 222, and

then Ultra-Sensitive Exchange with Client Portal.

o The MHP has plans to develop and implement Client Portal documents, client

training and program security following.

o The MHP has recruited a Quality Improvement Coordinator who is a licensed

clinician to begin in August 2016. Her primary job responsibilities may

encompass more intensive data analytical tasks.

Consumer Outcomes

o The MHP amended both the Medical Necessity form and Discharge Assessment

form to allow entry and scoring of the Children’s Behavioral Checklist (CBCL),

which measures outcomes.

o The MHP did a Clinical PIP on utilizing the MORs to assess how interventions

assist consumers in recovery.

SITE REVIEW PROCESS BARRIERS

The following conditions significantly affected CalEQRO’s ability to prepare for and/or conduct a

comprehensive review:

Review planning was challenging as the MHP’s Compliance Officer/Consultant works

one day per week, with communication being limited to that day. Further, on the date of

the review and due to unforeseen circumstances, the Compliance Officer/Consultant

was not able to attend. Alternatively, a conference call was held the following week for

further discussion.

CONCLUSIONS

During the FY16-17 annual review, CalEQRO found strengths in the MHP’s programs, practices, or

information systems that have a significant impact on the overall delivery system and its

supporting structure. In those same areas, CalEQRO also noted opportunities for quality

improvement. The findings presented below relate to the operation of an effective managed care

organization, reflecting the MHP’s processes for ensuring access to and timeliness of services and

improving the quality of care.

STRENGTHS AND OPPORTUNITIES

Access to Care

Strengths:

o With its Cultura Vida program, the MHP has a robust outreach program in the

Hispanic community, with over 12 presentations on services and mental health

issues.

o The MHP has forged strong collaborative relationships with both Glenn and

Sutter-Yuba MHPs to serve Colusa County consumers post hospital closure.

Opportunities:

o The MHP has been experiencing difficulties with its crisis protocol since April

2016 with the closure of the only hospital in the county.

o The MHP does not have a formalized cultural competence committee, nor has it

updated its Cultural Competency Plan since FY11/12.


Strengths:

o The MHP is responsive in providing intake assessments and responding to

needs for urgent appointments, consistently meeting their timeliness goals.

Opportunities:

o The MHP tracked their adult and children separately on the Timely Self-

Assessment for FY15-16 for the length of time from assignment after

assessment to first psychiatric appointment but did not track separately for

FY16-17. Further, they set a goal of 45 days from assignment after assessment to

first psychiatry appointment.

o The MHP changed its protocol for medication-only clients and eliminated

medication only clients for children’s services, requiring that children be in

other services to be eligible for medication services. The MHP has the

opportunity to track timeliness for psychiatry for both children and adults,

separately as before, to determine the impact of the policy on both timeliness of

service, access and quality of care.

o The MHP has a need for full tracking and trending of crisis data for effective and

efficient staff utilization in response to hospital closure issues and meeting crisis

caseloads. The MHP has a need to evaluate the increase in hospitalization rates.

o The MHP has not yet developed a format for tracking and trending no shows.

Consulting with nearby counties on their criteria and methodology for tracking

and trending no shows as well as creating and enforcing a policy and procedure

could be beneficial.

Quality of Care

Strengths:

o The MHP hired a new QI Coordinator who will be responsible for data analysis

and trending.

o In spring of 2016, a county administrative officer was hired, with MHP

leadership as a direct report.

Opportunities:

o The MHP has recruited a QI Coordinator who is a licensed clinician to begin in

August 2016. Her primary job responsibilities may encompass more intensive

analytical tasks.

o The MHP has the opportunity to improve bi-directional communication and

implementation of staff feedback for effective resolution and involvement of

staff at higher levels of program planning and service delivery.

Consumer Outcomes

Strengths:

o The MHP utilizes both Milestones of Recovery Scores (MORS) for adults and has

utilized aggregate data for their Clinical PIP.

Opportunities:

o CBCL for children was installed December 2015 and training was done in

January 2016. No aggregate reporting and analysis has been done as yet with

CBCL.

RECOMMENDATIONS

Investigate the feasibility of additional funding or resources to support crisis staff,

which currently is an informal mobile crisis unit with no vehicle. This is relevant and

timely with the recent closure of the local hospital and their ER services.

Evaluate contributing factors for the increase in hospitalization rates from FY 14-15 to

FY15-16. Use data to monitor and evaluate consumers’ access to crisis services as it relates

to the hospital closure. Use data on the contributing factors to identify gaps in service and

potential solutions.

Re-establish and formalize the Cultural Competence Committee with regularly

scheduled meetings to address other aspects of cultural responsiveness not limited

to Hispanic outreach, considering underserved populations, and gaps in staff

training. Actively recruit representatives from the community for stakeholder

participation and their input.

Evaluate current timeliness tracking procedures, and expand reporting (separately) on both

adult and children’s measures for all measures contained in the timeliness self-assessment,

including no shows. Track timeliness for psychiatry for both children and adults, to

determine the impact of the .5 elimination of child psychiatry and the implementation of the

policy requiring conjoint services for medication child clients.

Expand EHR functionality through consultation with Kings View and nearby

counties. Quantifiably show improvement and expansion through the addition of

applications, such as eLabs, Level of Care/Level of Service, Alerts, etc.

ATTACHMENTS

Attachment A: Review Agenda

Attachment B: Review Participants

Attachment C: Approved Claims Source Data

Attachment D: CalEQRO PIP Validation Tools

ATTACHMENT A—REVIEW AGENDA

Colusa EQRO Review FY16-17 Final Agenda 1

Behavioral Health Concepts, Inc. - California EQRO

5901 Christie Ave., Suite 502, Emeryville, CA94608 (855) 385-3776 www.caleqro.com

Colusa County MHP CalEQRO Agenda

Day 1 Thursday, August 4, 2016 All sessions located at Colusa County Behavioral Health 162 E. Carson St. Colusa, CA 95932 unless otherwise noted.

Time Activity

8:30 a.m. – 10 :00 am

Opening Session

Introduction to BHC

MHP Team Introductions Review of Past Year

Significant Changes and Key Initiatives

Responses to Last Year’s Recommendations

Use of Data in the Past Year

State Survey (#s, use of) Participants: MHP Leadership, Quality Management Staff, Key Stakeholders

Location: ROOM 102

BHC Staff: All

1 0 : 0 0 a m – 1 0 : 1 5 a m Break

1 0 : 1 5 a m – 1 2 : 0 0 p m MHP Clinical Staff Group Interview 6-8 Clinical line staff, representing various geographical regions of the county, including crisis staff, with no supervisory level staff included.

Location: ROOM 102 BHC Staff: CE, JL

Billing/It Key Staff Group Interview • Recommendation FY15-16 • ISCA Review • Continuous Training Initiative • CHAT Program Implementation • Analysis & Use of Data • Kings View/Dashboards • Process to Review Denied Claims and Claims Reconciliation • EHR Deployment (new functions in past year, pending enhancements) • Consumer Personal Health Record

Location: “SUNSHINE” ROOM

BHC Staff: JT

1 2 : 0 0 p m – 1 : 0 0 p m BHC Working Lunch Meeting

1 : 0 0 p m – 2 : 3 0 p m

Consumer/Family Member Focus Group 8-10 culturally diverse adult beneficiaries representing both high and low utilizers of service.

Access, Timeliness, Outcomes, and Quality • Timeliness Self-Assessment Document • MHP Timeliness Metrics and Procedures • Access • Medi-Cal Penetration Rates


Colusa EQRO Review FY16-17 Final Agenda 2

Time Activity

Location: ROOM 102

BHC Staff: JL

• MHP Cultural Competence Metrics and Procedures Location: “SUNSHINE” ROOM BHC Staff: CE, JT

2 : 3 0 p m – 3 : 3 0 p m Katie A. Implementation •Overview of current Katie A services, coordination, and future strategies •Technical Assistance Location: “Sunshine” Room

BHC Staff: JT

Performance Improvement Projects •Review Clinical PIP •Review Non-Clinical PIP •Technical Assistance

Location: Room 102 BHC Staff: CE, JL

3 : 3 0 p m – 3 : 4 0 p m BHC Staff Travel to Safe Haven Site

3 : 4 0 p m – 4 : 0 5 p m Safe Haven Wellness Center Visit Location: Safe Haven 411 MAIN STREET, COLUSA BHC Staff: All

4: 15 pm - 4 : 45 pm Exit Interview if needed for questions and next steps. • Summary of Findings • Collection of Requested Documentation • Next Steps

Location: ROOM 102

BHC Staff: All

CalEQRO Review Team: Cyndi Eppler – Lead Quality Reviewer [email protected]

Judith Toomasson – Information Systems Reviewer [email protected]

Janyce Leathers – Consumer/Family Member Consultant www.CalEQRO.com

mailto:[email protected]


ATTACHMENT B—REVIEW PARTICIPANTS

CALEQRO REVIEWERS CyndiEppler–LeadQualityReviewerJudithToomasson–InformationSystemsReviewerJanyceLeathers–Consumer/FamilyMemberConsultant

AdditionalCalEQROstaffmemberswereinvolvedinthereviewprocess,assessments,andrecommendations.Theyprovidedsignificantcontributionstotheoverallreviewbyparticipatinginboththepre‐siteandthepost‐sitemeetingsand,ultimately,intherecommendationswithinthisreport.

SITES OF MHP REVIEW

MHP SITES

ColusaCountyBehavioralHealth162E.CarsonSt.Colusa,CA95932SafeHavenWellnessCenter,411MainStreet,Colusa,CA95932

CONTRACT PROVIDER SITES

Nocontractprovidersiteswerevisited.

PARTICIPANTS REPRESENTING THE MHP

Name Position Agency

Angela Shields Family Specialist CCBH

Chantelle Estess Office Supervisor CCBH

Deana Fleming Deputy Director Adult Services CCBH

Elaine S. McCord EHR Coordinator CCBH

Emily Clark Therapist CCBH

Jack Joiner Compliance Officer, Consultant CCBH

James Balderama Case Manager CCBH

Jan Morgan Deputy Director Children’s Services CCBH

Jeannie Scroggins Therapist CCBH

Kim Perry Therapist CCBH

Mark McGregor Clinical Program Manager Children’s Services CCBH

Michael Laffin Deputy Director of Administration & Finance CCBH

Senaida Rangel MHSA Coordinator CCBH

Shannon Piper Clinical Program manager Adult Services CCBH

Terrence Rooney Director of Behavioral Health CCBH

Tomika Arce Billing Department CCBH

Tracy Woo Quality Improvement Coordinator, Consultant CCBH

William McCloud EHR Manager CCBH

ATTACHMENT C—APPROVED CLAIMS SOURCE DATA

ThesedataareprovidedtotheMHPinaHIPAA‐compliantmanner.

TwoadditionaltablesareprovidedbelowonMedi‐CalACAExpansionbeneficiariesandMedi‐Calbeneficiariesservedbycostbands.

TableC1(AttachmentC)showsthepenetrationrateandapprovedclaimsperbeneficiaryfortheCY15Medi‐CalACAExpansionPenetrationRateandApprovedClaimsperBeneficiary.

TableC2(AttachmentC)showsthedistributionoftheMHPCY15DistributionofBeneficiariesbyApprovedClaimsperBeneficiary(ACB)Rangeforthevariouscategories;under$20,000;$20,000to$30,000,andthoseabove$30,000.

Entity

Average Monthly

ACA Enrollees

Number of

Beneficiaries

Served

Penetration

Rate

Total Approved

Claims

Approved Claims

per Beneficiary

Statwide 2,001,900 131,350 6.56% $533,318,886 $4,060

Small‐Rural 17,753 1,992 11.22% $5,569,311 $2,796

Colusa 803 122 15.19% $387,591 $3,177

Table C1 ‐ CY15 Medi‐Cal Expansion (ACA) Penetration Rate and Approved Claims per Beneficiary

Range of ACB

MHP Count of

Beneficiaries

Served

MHP

Percentage

of

Beneficiaries

Statewide

Percentage

of

Beneficiaries

MHP Total

Approved

Claims

MHP

Approved

Claims per

Beneficiary

Statewide

Approved

Claims per

Beneficiary

MHP

Percentage

of Total

Approved

Claims

Statewide

Percentage

of Total

Approved

Claims

$0K ‐ $20K 500 95.60% 94.46% $1,634,554 $3,269 $3,553 61.51% 61.20%

>$20K ‐ $30K 9 1.72% 2.67% $212,700 $23,633 $24,306 8.00% 11.85%

>$30K 14 2.68% 2.86% $810,021 $57,859 $51,635 30.48% 26.96%

Table C2 ‐ Colusa MHP CY15 Distribution of Beneficiaries by ACB Range

ATTACHMENT D—PIP VALIDATION TOOL

Colusa Clinical PIP Validation Tool FY16-16 CE v2.1 Page 1 of 12

PERFORMANCE IMPROVEMENT PROJECT (PIP) VALIDATION WORKSHEET FY16-17

GENERAL INFORMATION

MHP: Colusa ☒ Clinical PIP ☐ Non-Clinical PIP

PIP Title: Recovery

Start Date (MM/DD/YY): January 2015

Completion Date (MM/DD/YY): ongoing

Projected Study Period (#of Months): 3 year

Completed: Yes ☐ No ☒

Date(s) of On-Site Review (MM/DD/YY): 08/04/2016

Name of Reviewer: Cyndi Eppler

Status of PIP (Only Active and ongoing, and completed PIPs are rated):

Rated

☒ Active and ongoing (baseline established and interventions started)

☐ Completed since the prior External Quality Review (EQR)

Not rated. Comments provided in the PIP Validation Tool for technical assistance purposes only.

☐ Concept only, not yet active (interventions not started)



Brief Description of PIP (including goal and what PIP is attempting to accomplish):

The MHP, as a result of its participation in the Advancing Recovery Collaborative, collected MORS scores for all adult consumers beginning in January 2015. Higher than expected MORS scores for 13% of the study population led to the MHP focusing on transitioning consumers to lower levels of care. The MHP aimed to improve outcomes for adults, including “graduating” from services of the MHP. The PIP examined which interventions assisted consumers in recovery as measured using the Milestones of Recovery outcome tool (MORs), leading to shortened and consistent time to transition to lower level levels of care.

The MHP collected MORS scores monthly and found that individuals at MORS 7 were able to transition out of services, as well as those at MORS 6. The MHP then focused on individuals with a MORS score of 5. In contrast to the MORS 6 and 7 individuals, these individuals were not able to move quickly up the MORS scale. These individuals stayed at MORS 5 throughout the measurement cycle. Previously successful Interventions were doubled (Strength Assessment and Group Supervision). Even with these interventions, little movement was seen for the MORS 5 individuals. The MHP is now testing the outcome tools that


may be more sensitive to smaller changes (PHQ 9 and the GAD 7) for individuals with depression or anxiety diagnoses as they can be administered more frequently.

The study question design did not lend itself to be answerable through the PIP process. Recommend that for next PIP, set up question so that it is quantifiable and contains indicators, i.e. “Will ____ intervention improve ________(symptoms/problem) as measured by ________outcome tool?” As it stands, the PIP’s design does not allow the MHP to track the new outcome tools and does not identify which interventions improve which symptoms. Clarification on whether MORs 6s and 7s entered at the 5 level when they came in for services initially. This would allow for a meaningful comparison regarding how long it took to get to the next level. For next steps, the MHP plans to amend its question, evaluate the impact of strengths assessment on “leveling up” using the MORs, in conjunction with the anxiety and depression inventory tools. Recommended that this PIP be considered completed rather than going into year 3 of this PIP. Instead, develop a new PIP evaluating clinical outcomes post intervention using the anxiety and depression assessments. As it stands, this process didn’t make an improvement for clients as they were still discharged for the same reason which was not a result of better implementation of the MORS. Rather, the PIP provided more information regarding level of care and helped inform the administrative process of discharge/level of care.


ACTIVITY 1: ASSESS THE STUDY METHODOLOGY

STEP 1: Review the Selected Study Topic(s)

Component/Standard Score Comments

1.1 Was the PIP topic selected using stakeholder input? Did the MHP develop a multi-functional team compiled of stakeholders invested in this issue?

☒ Met

☐ Partially Met

☐ Not Met

☐ Unable to Determine

The persons involved in this PIP are noted by role above. Specifically they are: Deana Fleming, LCSW, Deputy Director, Clinical, Adult; Janet Morgan, LCSW, Deputy Director, Clinical, Children; Veronica Vasquez, Fiscal; Jack Joiner, LMFT, Consultant.

1.2 Was the topic selected through data collection and analysis of comprehensive aspects of enrollee needs, care, and services?

☒ Met

☐ Partially Met

☐ Not Met


Select the category for each PIP:

Clinical:

☐ Prevention of an acute or chronic condition ☐ High volume services

☒ Care for an acute or chronic condition ☐ High risk conditions

Non-Clinical:

☐ Process of accessing or delivering care

1.3 Did the Plan’s PIP, over time, address a broad spectrum of key aspects of enrollee care and services?

Project must be clearly focused on identifying and correcting deficiencies in care or services, rather than on utilization or cost alone.

☒ Met

☐ Partially Met

☐ Not Met


The study topic of recovery is clearly the most important element in consumers’ “getting better”. We are committed to doing our best as a Department in helping consumers experience improvement in their ability to be an active participant in society

1.4 Did the Plan’s PIPs, over time, include all enrolled populations (i.e., did not exclude certain enrollees such as those with special health care needs)?

Demographics:

☐ Age Range ☐ Race/Ethnicity ☐ Gender ☐ Language ☐ Other

☒ Met

☐ Partially Met

☐ Not Met


Participants in this study come from throughout Colusa County, and are receiving treatment in the only clinic in the County, in Colusa. All participants are adults. All participants are engaged in outpatient treatment and are not “meds only” consumers.

Totals 4 Met Partially Met Not Met UTD


STEP 2: Review the Study Question(s)

2.1 Was the study question(s) stated clearly in writing?

Does the question have a measurable impact for the defined study population?

Include study question as stated in narrative:

Can the Milestones of Recovery (MORS) rating scale be used to effectively guide treatment interventions?

☐ Met

☐ Partially Met

☒ Not Met


Question is not measurable, does not contain indicators. Examples of measurable question with indicator as follows:

Will the use of ________intervention(s) move consumers to the next level of care within _______ time period/within certain number of sessions?

or

What impact will the use of ________intervention(s) have on consumer outcomes as measured by MORs (or other outcome measurement tool)?

Totals Met Partially Met 1 Not Met UTD

STEP 3: Review the Identified Study Population

3.1 Did the Plan clearly define all Medi-Cal enrollees to whom the study question and indicators are relevant?

Demographics:


☒ Met

☐ Partially Met

☐ Not Met


Participants in this study come from throughout Colusa County, and are receiving treatment in the only clinic in the County, in Colusa. All participants are adults. All participants are engaged in outpatient treatment and are not “meds only” consumers.

3.2 If the study included the entire population, did its data collection approach capture all enrollees to whom the study question applied?

Methods of identifying participants:

☒ Utilization data ☐ Referral ☐ Self-identification

☐ Other: Text if checked

☒ Met

☐ Partially Met

☐ Not Met



STEP 4: Review Selected Study Indicators

4.1 Did the study use objective, clearly defined, measurable indicators?

List indicators:

MORS score at least 7 for three months

MORS score of 5 for three months

☒ Met

☐ Partially Met

☐ Not Met


The indicators were objective and measureable.


4.2 Did the indicators measure changes in: health status, functional status, or enrollee satisfaction, or processes of care with strong associations with improved outcomes? All outcomes should be consumer focused.

☐ Health Status ☒ Functional Status

☐ Member Satisfaction ☐ Provider Satisfaction

Are long-term outcomes clearly stated? ☐ Yes ☒ No

Are long-term outcomes implied? ☒ Yes ☐ No

☒ Met

☐ Partially Met

☐ Not Met


MORs scores for the identified individuals were tracked.


STEP 5: Review Sampling Methods

5.1 Did the sampling technique consider and specify the:

a) True (or estimated) frequency of occurrence of the event?

b) Confidence interval to be used?

c) Margin of error that will be acceptable?

☐ Met

☐ Partially Met

☒ Not Met

☐ Not Applicable


As this PIP has progressed the MHP have found that individuals at MORS score 7 were indeed ready for discharge planning and in fact 10 of the 14 identified individuals did leave treatment. The MHP also found that for individuals with a greater level of impairment (as indicated by a MORS score of 5) required a more focused approach. The MHP was unable to focus this greater level of attention on all MORS 5 individuals so instead the MHP required each clinician to identify one client at this level to be the focus the interventions in this PIP. Each identified consumer participated in a Strengths Assessment and a Personal Recovery Plan and each clinician then presented this information in a Group Supervision model. As the PIP focus on MORS level 5 individuals continued it became clear that the MORS was good at identifying recovery level in a broad way but did not reflect the partial changes needed within this level to move forward in recovery. Unlike individuals at MORS 7 who were quickly able to move to discharge, individuals at MORS 5 have much more work to do before they are able to move up to MORS 6.

This sampling technique does not appear to be grounded in a way that prevents bias, rather the individuals were selected by level.


5.2 Were valid sampling techniques that protected against bias employed?

Specify the type of sampling or census used:

Text

☒ Met

☐ Partially Met

☐ Not Met

☐ Not Applicable


All MORs 7, 6, 5 Clients were included.

5.3 Did the sample contain a sufficient number of enrollees?

______N of enrollees in sampling frame

______N of sample

______N of participants (i.e. – return rate)

☐ Met

☒ Partially Met

☐ Not Met

☐ Not Applicable


Sample size was based on what each clinician could accomplish. The result was a sample size of 8. The sample size was not grounded in any statistical analysis. The MHP did not justify this number and show that those benefitting from the PIP are not disproportionately one “type of client.”

Totals 1 Met 1 Partially Met 1 Not Met UTD


STEP 6: Review Data Collection Procedures

6.1 Did the study design clearly specify the data to be collected?

☒ Met

☐ Partially Met

☐ Not Met


MORS scores for the identified individuals.

6.2 Did the study design clearly specify the sources of data?

Sources of data:

☐ Member ☐ Claims ☐ Provider


☐ Met

☒ Partially Met

☐ Not Met


MORS scores monthly on “MORS Monday”

6.3 Did the study design specify a systematic method of collecting valid and reliable data that represents the entire population to which the study’s indicators apply?

☐ Met

☒ Partially Met

☐ Not Met


Methodology description did not provide any specifics or detail.

6.4 Did the instruments used for data collection provide for consistent, accurate data collection over the time periods studied?

Instruments used:

☐ Survey ☐ Medical record abstraction tool

☒ Outcomes tool ☐ Level of Care tools


☒ Met

☐ Partially Met

☐ Not Met


6.5 Did the study design prospectively specify a data analysis plan?

Did the plan include contingencies for untoward results?

☐ Met

☐ Partially Met

☒ Not Met


The data is extracted by a fiscal person who is familiar with the EHR. Analysis plan not specified.


6.6 Were qualified staff and personnel used to collect the data?

Project leader:

Name: Deana Flemming, LCSW

Title: Deputy Director Clinical, Adult

Role: Project leader

Other team members:

Names: PIP committee

☒ Met

☐ Partially Met

☐ Not Met


Clinicians obtained the MORS score with their respective clients.


STEP 7: Assess Improvement Strategies

7.1 Were reasonable interventions undertaken to address causes/barriers identified through data analysis and QI processes undertaken?

Describe Interventions:

MORS Scoring

Strength Assessment

Personal Recovery Planning

Group Supervision

☐ Met

☐ Partially Met

☐ Not Met

☒ Unable to Determine

The interventions did not inform the study question in a way that measurable and quantifiable.

Totals Met Partially Met Not Met NA 1 UTD

STEP 8: Review Data Analysis and Interpretation of Study Results

8.1 Was an analysis of the findings performed according to the data analysis plan?

This element is “Not Met” if there is no indication of a data analysis plan (see Step 6.5)

☐ Met

☐ Partially Met

☒ Not Met

☐ Not Applicable


Data analysis plan was not provided.

8.2 Were the PIP results and findings presented accurately and clearly?

Are tables and figures labeled? ☐ Yes ☐ No

Are they labeled clearly and accurately? ☐ Yes ☐ No

☒ Met

☐ Partially Met

☐ Not Met

☐ Not Applicable



8.3 Did the analysis identify: initial and repeat measurements, statistical significance, factors that influence comparability of initial and repeat measurements, and factors that threaten internal and external validity?

Indicate the time periods of measurements: monthly___________

Indicate the statistical analysis used:__n/a_________________

Indicate the statistical significance level or confidence level if available/known:_______% ______Unable to determine

☐ Met

☐ Partially Met

☒ Not Met

☐ Not Applicable


8.4 Did the analysis of the study data include an interpretation of the extent to which this PIP was successful and recommend any follow-up activities?

Limitations described:

Text

Conclusions regarding the success of the interpretation:

Based on the learning that has and is occurring from this PIP we consider it successful. We are learning the short comings of the MORS and are testing other measurements to see if more minor but important changes can assist in guiding treatment.

Recommendations for follow-up:

Text

☐ Met

☐ Partially Met

☒ Not Met

☐ Not Applicable


This discovery of little apparent progress for the MORS 5 individuals also lead to a question about discharges in general. The MHP queried the EHR to learn what the most frequent reason for discharge was. They were surprised to find that almost all discharges were recorded as “administrative”. We knew this was not clinically correct so we launched a project to compare clinical reasons for discharge versus what the final recorded reason was. The MHP found that there was indeed a difference between what the clinician had recorded as the reason for discharge in one part of the EHR versus what was recorded as the final discharge reason. The two main reasons for the difference were a lack of communication between clinical and clerical staff regarding reason for discharge, and a lack of discharge codes to reflect a consumer who was seen in crisis only and did not require additional services (this could only be recorded as administrative). Additional codes were subsequently added to the system for these brief, self-contained episodes of care. Communications of discharge reasons between clerical and clinical staff were also emphasized.

This process didn’t make an improvement for clients as they were still discharged for the same reason which was not a result of better implementation of the MORS. Rather, the PIP provided more information regarding level of care and helped inform the administrative process of discharge/level of care.

Totals 1 Met Partially Met 3 Not Met NA UTD


STEP 9: Assess Whether Improvement is “Real” Improvement

9.1 Was the same methodology as the baseline measurement used when measurement was repeated?

Ask: At what interval(s) was the data measurement repeated?

Were the same sources of data used?

Did they use the same method of data collection?

Were the same participants examined?

Did they utilize the same measurement tools?

☐ Met

☐ Partially Met

☐ Not Met

☐ Not Applicable

☒ Unable to Determine

For meaningful data comparison, the timeframe for a client with MORs level 6 to become a level 7 cannot be compared to timeframes that a level 5 takes to become a 6 unless, the 6s and 7s came in at a level 5.


9.2 Was there any documented, quantitative improvement in processes or outcomes of care?

Was there: ☒ Improvement ☐ Deterioration

Statistical significance: ☐ Yes ☒ No

Clinical significance: ☒ Yes ☐ No

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


The MHP was not able to do the analysis of 9.3 and 9.4, therefore there isn’t documented improvement which is known to be the result of the PIP.

9.3 Does the reported improvement in performance have internal validity; i.e., does the improvement in performance appear to be the result of the planned quality improvement intervention?

Degree to which the intervention was the reason for change:

☐ No relevance ☐ Small ☐ Fair ☐ High

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


9.4 Is there any statistical evidence that any observed performance improvement is true improvement?

☐ Weak ☐ Moderate ☐ Strong

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


9.5 Was sustained improvement demonstrated through repeated measurements over comparable time periods?

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


Totals 2 Met Partially Met Not Met 3 NA UTD


ACTIVITY 2: VERIFYING STUDY FINDINGS (OPTIONAL)


Were the initial study findings verified (recalculated by CalEQRO) upon repeat measurement?

☐ Yes

☒ No

n/a

ACTIVITY 3: OVERALL VALIDITY AND RELIABILITY OF STUDY RESULTS: SUMMARY OF AGGREGATE VALIDATION FINDINGS

Conclusions:

Met 13

Partially Met 3

Not Met 6

UTD 2

# Not applicable 4

# Applicable 24

Score 60.42%

Recommendations:

The study question design did not lend itself to be answerable through the PIP process. Recommend that for next PIP, set up question so that it is quantifiable and contains indicators, i.e. “Will ____ intervention improve ________(symptoms/problem) as measured by ________outcome tool?” As it stands, the PIP’s design does not allow the MHP to track the new outcome tools and does not identify which interventions improve which symptoms. Clarification on whether MORs 6s and 7s entered at the 5 level when they came in for services initially. This would allow for a meaningful comparison regarding how long it took to get to the next level. For next steps, the MHP plans to amend its question, evaluate the impact of strengths assessment on “leveling up” using the MORs, in conjunction with the anxiety and depression inventory tools.

This process didn’t make an improvement for clients as they were still discharged for the same reason which was not a result of better implementation of the MORS. Rather, the PIP provided more information regarding level of care and helped inform the administrative process of discharge/level of care.

Check one: ☐ High confidence in reported Plan PIP results ☐ Low confidence in reported Plan PIP results

☐ Confidence in reported Plan PIP results ☐ Reported Plan PIP results not credible

☒ Confidence in PIP results cannot be determined at this time

Colusa Non-Clinical PIP Validation Tool FY16-17 CE v2.1 Page 1 of 14

PERFORMANCE IMPROVEMENT PROJECT (PIP) VALIDATION WORKSHEET FY16-17

GENERAL INFORMATION

MHP: Colusa ☐ Clinical PIP ☒ Non-Clinical PIP

PIP Title: Parent Intervention

Start Date (MM/DD/YY): August 2015

Completion Date (MM/DD/YY): ongoing

Projected Study Period (#of Months): 2 years

Completed: Yes ☐ No ☒

Date(s) of On-Site Review (MM/DD/YY): 08/04/16

Name of Reviewer: Cyndi Eppler

Status of PIP (Only Active and ongoing, and completed PIPs are rated):

Rated

☒ Active and ongoing (baseline established and interventions started)

☐ Completed since the prior External Quality Review (EQR)

Not rated. Comments provided in the PIP Validation Tool for technical assistance purposes only.

☐ Concept only, not yet active (interventions not started)



Brief Description of PIP (including goal and what PIP is attempting to accomplish):

This PIP is at the beginning of its second year. Year 1 focused on shortening duration of services with the addition of conjoint family services as a standard

protocol. The second year of the PIP is aimed at evaluating the impact of conjoint (automatically assigned) family interventions, specifically Parent-Child

Interaction Therapy (PCIT) on client functioning. The overarching goal of this PIP will be to improve child functioning in many areas relevant to the child’s progress toward resilience. The MHP analyzed data that compared the length of treatment for children receiving family therapy of any kind or duration beyond

intake as part of their treatment with children who had not. The MHP looked at this data for fiscal year 2014/2015 and found that 145 children had received family therapy, where 62 children had not. The When comparing time in treatment for the two groups, the MHP found that children receiving any form of

family involvement (145 children) averaged 3.763889 months in care and those that received no form of family involvement (63 children) averaged 3.967742

months. Remeasurement has proven to be a challenge. The MHP had difficulty rolling out the CBCL due to scoring challenges and difficulty in getting the score entered into the EHR. Additionally, the MHP discovered that they were not getting discharge CBCL scores because families were choosing to withdraw from

care when the clinician began discussing the end of treatment. Further, no one has been in care for a full year since the implementation of the CBCL. The MHP experienced similar issues with the parent satisfaction survey. A way to ameliorate this type of issue would be to collect data on a quarterly basis.


Feedback included acknowledgment that the PIP focus seemed to naturally shift to evaluating the PCIT intervention. The study question needed clarification

so that it was measurable an objective. The interventions listed are not truly the intervention, but rather assigning all mild to moderate clients to PCIT and all moderate to severe clients to individual as a matter of standard procedure. It was recommended that the MHP then compare the CBCL scores of from a

baseline at start of treatment after a set period of time (i.e. 3 months). This would allow the county to see the true effectiveness of the new administrative procedure.


ACTIVITY 1: ASSESS THE STUDY METHODOLOGY

STEP 1: Review the Selected Study Topic(s)


1.1 Was the PIP topic selected using stakeholder input? Did the MHP develop a multi-functional team compiled of stakeholders invested in this issue?

☒ Met

☐ Partially Met

☐ Not Met


Director (Terry Rooney), two Deputy Directors (Jan Morgan and Deana Fleming), a data person (Veronica Vasquez) and a consultant (Jack Joiner). Also planned is the inclusion of parents who have received this service, initially via the locally developed satisfaction survey.

1.2 Was the topic selected through data collection and analysis of comprehensive aspects of enrollee needs, care, and services?

☒ Met

☐ Partially Met

☐ Not Met


Select the category for each PIP:

Clinical:

☐ Prevention of an acute or chronic condition ☐ High volume services

☐ Care for an acute or chronic condition ☐ High risk conditions

Non-Clinical:

☒ Process of accessing or delivering care All children entering system of care will receive conjoint family services.

1.3 Did the Plan’s PIP, over time, address a broad spectrum of key aspects of enrollee care and services?

Project must be clearly focused on identifying and correcting deficiencies in care or services, rather than on utilization or cost alone.

☒ Met

☐ Partially Met

☐ Not Met


1.4 Did the Plan’s PIPs, over time, include all enrolled populations (i.e., did not exclude certain enrollees such as those with special health care needs)?

Demographics:


☒ Met

☐ Partially Met

☐ Not Met


Children entering care after start of PIP, which to date, was 208 children. Approximately 63 children were moderate to severe, while 145 children were mild to moderate.



STEP 2: Review the Study Question(s)

2.1 Was the study question(s) stated clearly in writing?

Does the question have a measurable impact for the defined study population?

Include study question as stated in narrative:

“Will the provision parental intervention result in shorter duration of care, higher levels of parent satisfaction with care, and improved functioning as measured by the Child Behavior Checklist?”

☒ Met

☐ Partially Met

☐ Not Met



STEP 3: Review the Identified Study Population

3.1 Did the Plan clearly define all Medi-Cal enrollees to whom the study question and indicators are relevant?

Demographics:


☒ Met

☐ Partially Met

☐ Not Met


Children entering care after start of PIP. Approximately 63 children were moderate to severe, while 145 children were mild to moderate.

3.2 If the study included the entire population, did its data collection approach capture all enrollees to whom the study question applied?

Methods of identifying participants:

☒ Utilization data ☐ Referral ☐ Self-identification

☐ Other:

☒ Met

☐ Partially Met

☐ Not Met


Children entering care after start of PIP. Approximately 63 children were moderate to severe, while 145 children were mild to moderate.


STEP 4: Review Selected Study Indicators

4.1 Did the study use objective, clearly defined, measurable indicators?

List indicators:

Child Behavior Checklist Score

Length of time in care

Scores on parental satisfaction survey after parental intervention

☒ Met

☐ Partially Met

☐ Not Met



4.2 Did the indicators measure changes in: health status, functional status, or enrollee satisfaction, or processes of care with strong associations with improved outcomes? All outcomes should be consumer focused.

☐ Health Status ☒ Functional Status

☐ Member Satisfaction ☒ Provider Satisfaction

Are long-term outcomes clearly stated? ☐ Yes ☒ No

Are long-term outcomes implied? ☒ Yes ☐ No

☒ Met

☐ Partially Met

☐ Not Met


Totals Met Partially Met Not Met UTD

STEP 5: Review Sampling Methods

5.1 Did the sampling technique consider and specify the:

a) True (or estimated) frequency of occurrence of the event?

b) Confidence interval to be used?

c) Margin of error that will be acceptable?

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


5.2 Were valid sampling techniques that protected against bias employed?

Specify the type of sampling or census used:

From UR data.

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


All children receiving services were/will be included once all numbers of sessions are completed.

5.3 Did the sample contain a sufficient number of enrollees?

______N of enrollees in sampling frame

______N of sample

______N of participants (i.e. – return rate)

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


Totals Met Partially Met Not Met UTD


STEP 6: Review Data Collection Procedures

6.1 Did the study design clearly specify the data to be collected?

☒ Met

☐ Partially Met

☐ Not Met


CBCL score, Parent Satisfaction score, Length of treatment.

6.2 Did the study design clearly specify the sources of data?

Sources of data:

☒ Member ☒ Claims ☒ Provider

☒ Other: EHR, Survey, Outcome tool

☒ Met

☐ Partially Met

☐ Not Met


6.3 Did the study design specify a systematic method of collecting valid and reliable data that represents the entire population to which the study’s indicators apply?

☐ Met

☒ Partially Met

☐ Not Met


The data will be collected by clinical staff for CBCL and Parent Satisfaction. Length of treatment will be calculated by fiscal support staff. Parents not staying to complete the CBCL is interfering with data collection.

6.4 Did the instruments used for data collection provide for consistent, accurate data collection over the time periods studied?

Instruments used:

☒ Survey ☐ Medical record abstraction tool

☒ Outcomes tool ☐ Level of Care tools


☐ Met

☒ Partially Met

☐ Not Met


The CBCL score will be entered into the EHR. Parent Satisfaction scores will be tabulated by a PIP Committee member. Length of treatment will be calculated as noted above. Parents not staying to complete the CBCL is interfering with data collection.

6.5 Did the study design prospectively specify a data analysis plan?

Did the plan include contingencies for untoward results?

☐ Met

☐ Partially Met

☒ Not Met


The data will be analyzed by the Committee. No detail or methodology was described.


6.6 Were qualified staff and personnel used to collect the data?

Project leader:

Name: Janet Morgan

Title: Deputy Director, Clinical, Children

Role: Project Leader

Other team members:

Names: PIP Committee

☒ Met

☐ Partially Met

☐ Not Met


The data will be collected by clinical staff for CBCL score and entered into the EHR. Parent Satisfaction scores will be tabulated by a PIP Committee member. Length of treatment will be calculated as noted above.



STEP 7: Assess Improvement Strategies


7.1 Were reasonable interventions undertaken to address causes/barriers identified through data analysis and QI processes undertaken?

Describe Interventions:

Week 1 intervention: The focus will be on engagement, education and goal setting. The clinician will review the diagnosis and history and offer acknowledgement of parent’s strengths and areas of concern.

Week 2: After signing the treatment plan, the focus will be on educating parent regarding parenting styles (authoritative, permissive, supportive versus disciplinary) using the Berkeley study.

Week 3: The focus will be on addressing parent’s ability to provide support/warmth to their child through at least one of the following evidence based models:

(a) Teaching PRIDE skills (modified for children above age 8);

(b) Introducing skills through the Love and Logic model;

(c) Using the Nurturing Parent model.

Week 4: The focus will be on discipline/structure and help clarify between troubled and normal behaviors using Ericson’s Developmental Stages model and/or Transactional Analysis model. These models introduce parents to nurturing versus critical parenting and help parents review their ability to nurture and structure their inner child. Parents will also receive support for structured and supportive parenting.

Week 5: The focus will be on discipline/structure and help clarify between troubled and normal behaviors using Ericson’s Developmental Stages model and/or Transactional Analysis model. These models introduce parents to nurturing versus critical parenting and help parents review their ability to nurture and structure their inner child. Parents will also receive support for structured and supportive parenting.

Week 5: The focus will be on parent’s barriers to implementing skills that have been learned in the previous weeks. Possible evidence based models may include the communication

☒ Met

☐ Partially Met

☐ Not Met


Note: The intervention is the changed process of automatically assigning child clients to conjoint individual and family services. The weekly interventions reflect implementation of the process. The process of automatically assigning mild to moderate clients to parent intervention has begun.


Totals 1 Met Partially Met Not Met NA UTD

STEP 8: Review Data Analysis and Interpretation of Study Results

8.1 Was an analysis of the findings performed according to the data analysis plan?

This element is “Not Met” if there is no indication of a data analysis plan (see Step 6.5)

☐ Met

☐ Partially Met

☒ Not Met

☐ Not Applicable


Because the children on average have been in therapy for less than 4 months, they should have some data analysis comparing to the baseline data collected in Sept. 2015. Data collection and analysis should occur at least quarterly. Quarterly data collection would allow the MHP to identify issues sooner and allow changes to be implemented in the process to benefit others.

8.2 Were the PIP results and findings presented accurately and clearly?

Are tables and figures labeled? ☐ Yes ☐ No

Are they labeled clearly and accurately? ☐ Yes ☐ No

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable



8.3 Did the analysis identify: initial and repeat measurements, statistical significance, factors that influence comparability of initial and repeat measurements, and factors that threaten internal and external validity?

Indicate the time periods of measurements:___________________

Indicate the statistical analysis used:_________________________

Indicate the statistical significance level or confidence level if available/known:_______% ______Unable to determine

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


8.4 Did the analysis of the study data include an interpretation of the extent to which this PIP was successful and recommend any follow-up activities?

Limitations described:

Text

Conclusions regarding the success of the interpretation:

Text

Recommendations for follow-up:

Text

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


Totals Met Partially Met 1 Not Met 3 NA UTD

STEP 9: Assess Whether Improvement is “Real” Improvement

9.1 Was the same methodology as the baseline measurement used when measurement was repeated?

Ask: At what interval(s) was the data measurement repeated?

Were the same sources of data used?

Did they use the same method of data collection?

Were the same participants examined?

Did they utilize the same measurement tools?

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable



9.2 Was there any documented, quantitative improvement in processes or outcomes of care?

Was there: ☐ Improvement ☐ Deterioration

Statistical significance: ☐ Yes ☐ No

Clinical significance: ☐ Yes ☐ No

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


9.3 Does the reported improvement in performance have internal validity; i.e., does the improvement in performance appear to be the result of the planned quality improvement intervention?

Degree to which the intervention was the reason for change:

☐ No relevance ☐ Small ☐ Fair ☐ High

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


9.4 Is there any statistical evidence that any observed performance improvement is true improvement?

☐ Weak ☐ Moderate ☐ Strong

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


9.5 Was sustained improvement demonstrated through repeated measurements over comparable time periods?

☐ Met

☐ Partially Met

☐ Not Met

☒ Not Applicable


Totals Met Partially Met Not Met 5 NA UTD


ACTIVITY 2: VERIFYING STUDY FINDINGS (OPTIONAL)


Were the initial study findings verified (recalculated by CalEQRO) upon repeat measurement?

☐ Yes

☒ No

NA

ACTIVITY 3: OVERALL VALIDITY AND RELIABILITY OF STUDY RESULTS: SUMMARY OF AGGREGATE VALIDATION FINDINGS

Conclusions:

Met 13

Partially Met 2

Not Met 2

UTD 0

# Not applicable 11

# Applicable 17

Score 82.35%

Recommendations:

MHP is experiencing clients leaving PCIT groups early and not completing all weeks of treatment. If parents are not finishing and given that the PCIT has its own timeline, the MHP can’t meaningful compare “length of time in treatment” (esp. if required to complete all assigned sessions). Feedback included acknowledgment that the PIP focus seemed to naturally shift to evaluating the PCIT intervention. The study question needed clarification so that it was measurable an objective. The interventions listed are not truly the intervention, but rather assigning all mild to moderate clients to PCIT and all moderate to severe clients to individual as a matter of standard administrative procedure. It was recommended that the MHP then compare the CBCL scores from a baseline at start of treatment for groups of children (mild/moderate, moderate/severe) after a set period of time of either individual or PCIT intervention (i.e. 3 months, or 6 months). This would allow the MHP to see the true effectiveness of the new administrative procedure. Because the children on average have been in therapy for less than 4 months, they should have some data analysis comparing to the baseline data collected in Sept. 2015. Data collection and analysis should occur at least quarterly. Quarterly data collection would allow the MHP to identify issues sooner and allow changes to be implemented in the process to benefit others.


Check one: ☐ High confidence in reported Plan PIP results ☐ Low confidence in reported Plan PIP results

☐ Confidence in reported Plan PIP results ☐ Reported Plan PIP results not credible

☒ Confidence in PIP results cannot be determined at this time

mhp final report - caleqrocaleqro.com/data/reports_and_presentations/fiscal year 2016-2017... ·...

Documents