Primary Care Behavioral Health Clinical and Process Outcomes: Program Evaluation in the

Department of Defense

Anne C. Dobmeyer, PhD, ABPP, Chief Psychologist, PCBH Directorate, Deployment Health Clinical Center

Christopher L. Hunter, PhD, ABPP, DoD Program Manager for Behavioral Health in Primary Care

Jennifer L. Bell, MD, Associate Director, PCBH Directorate, Deployment Health Clinical Center

Collaborative Family Healthcare Association 17th Annual ConferenceOctober 15-17, 2015 Portland, Oregon U.S.A.

Session # B5October 17, 2015

Faculty Disclosure

The presenters of this session have NOT had any relevant financial relationships during the past 12 months.

Learning Objectives

At the conclusion of this session, the participant will be able to:

• Identify two process metrics useful to assess as part of a primary care behavioral health (PCBH) program evaluation

• Identify two outcome metrics useful to assess as part of a PCBH health program evaluation

• Describe one barrier to PCBH program evaluation in large medical systems and one strategy for addressing the barrier

Bibliography / References

1. Bryan, C. J., Corso, M. L., Corso, K. A., Morrow, C. E., Kanzler, K. E., & Ray-Sannerud, B. (2012). Severity of mental health impairment and trajectories of improvement in an integrated primary care clinic. Journal of Consulting and Clinical Psychology, 80, 396-403.

2. Hunter, C. L., & Goodie, J. L. (2012). Behavioral health in the Department of Defense Patient-Centered Medical Home: history, finance, policy, work force development, and evaluation. Translational Behavioral Medicine, 2, 355-363.

3. Hunter, C. L., Goodie, J. L., Dobmeyer, A. C., & Dorrance, K. A. (2014). Tipping points in the Department of Defense’s experience with psychologists in primary care. American Psychologist, 69 (4), 388-398.

4. Peek, C. J., Cohen, D. J., & deGruy, F. V. (2014). Research and evaluation in the transformation of primary care. American Psychologist, 69, 430-442.

5. Ray-Sannerud, B. N., Dolan, D. C., Morrow, C. E., Corso, K. A., Kanzler, K. E., Corso, M. L., & Bryan, C. J. (2012). Longitudinal outcomes after brief behavioral health intervention in an integrated primary care clinic. Families, Systems, and Health, 30, 60-71.

Learning Assessment

• A learning assessment is required for CE credit.

• A question and answer period will be conducted at the end of this presentation.

Disclaimer

The views expressed are those of the authors and do not reflect the official policy of the Department of Defense (DoD), the United States Public Health Service (USPHS) or the U.S. Government.

Acknowledgements

The authors would like to acknowledge Justin Curry, PhD,who also contributed to this presentation.

Overview

•Primary Care Behavioral Health (PCBH) program evaluation in the Department of Defense (DoD)

‒ Context‒ Evaluation plan‒ Early outcomes‒ Barriers and solutions

•Application to other settings: Monitoring and evaluation (M&E) ‒ Overview of M&E concepts‒ Monitoring and evaluating local innovations/programs‒ Potential areas to monitor/evaluate

•Worksheet and Discussion

Context: PCBH in the DoD

•DoD Military Health System (MHS) serves 3.3 million beneficiaries in military facility primary care (PC) clinics¹•DoD policy requires all PC clinics with > 3,000 enrollees have a full-time behavioral health consultant (BHC) using PCBH model¹

‒ 313 BHCs; 313 clinics•Program evaluation is facilitated by use of the DoD Electronic Health Record (EHR)

‒ Some data (demographics, visit type and length, procedure coding, diagnosis) can be extracted from EHR

‒ Additional PCBH module in EHR allows extraction of additional data

¹Hunter, C. L., Goodie, J. L., Dobmeyer, A. C., & Dorrance, K. A. (2014). Tipping points in the Department of Defense’s experience with psychologists in primary care. American Psychologist, 69 (4), 388-398.

Evaluation Plan

Excerpted from: Hunter, C. L., & Goodie, J. L. (2012). Behavioral health in the Department of Defense Patient-Centered Medical Home: history, finance, policy, work force development, and evaluation . Translational Behavioral Medicine, 2, 355-363.

DoD Strategic Imperative Potential Measures

Readiness: Enhance psychological health and resiliency

Anxiety and Depression•% screened for anxiety and depression•% screening positive managed in PCMH•% attending initial BHC appointment•Pre/post improvement in symptoms and functioning•% referred to specialty MH care

Population Health: Engage patients in healthy behaviors

Obesity•% enrollees with BMI > 30•% screened for obesity•% working with BHC for weight change•BMI change from 12 months post-treatment initiation•Average BMI change for all enrollees with BMI > 30

Evaluation Plan (cont’d)

Excerpted from: Hunter, C. L., & Goodie, J. L. (2012). Behavioral health in the Department of Defense Patient-Centered Medical Home: history, finance, policy, work force development, and evaluation . Translational Behavioral Medicine, 2, 355-363.

DoD Strategic Imperative Potential Measures

Experience of Care: Optimize access to care

• Same day access to BHC appointment• Time to same day available new and 3rd

return open BHC appointment• % who desire same day BHC

appointment who receive it• Satisfaction with getting timely care• % of family members seen by BHC

(potentially recaptured network care)

Per Capita Cost:Manage healthcare costs

• Annual percent increase in per capita costs

• Emergency room visits per 100 enrollees per year for BH presentation

Per Capita Cost:Enable better decisions

• % use of standard electronic medical record note templates for PCBH

Early Outcomes

•Monitoring period: Fiscal Year 2015 Quarter 1‒ Oct 1, 2014 – Dec 31, 2014‒ Patient scores also gathered from prior 3 months (Jul – Sep 2014)

•Quantity and penetration of BHC services‒ BHCs saw 23,068 unique patients during 3 month monitoring period

‒ This represents 1.66% of all patients seen in primary care during the monitoring period

‒ BHCs conducted 38,855 unique encounters‒ This represents 1.20% of all primary care encounters

Early Outcomes: Fidelity to Model

•Approximately half of BHC patients were seen only once (during 6 month window)

‒ 54.7% ‒ 12,612 patients

•Nearly all BHC patients were seen 4 or fewer times‒ 92.8%‒ 21,403 patients

•Very few BHC patients were seen 9 or more times‒ 1.2%‒ 277 patients

Early Outcomes: Fidelity to Model (cont’d)

•Feedback given to PCP at every visit (Goal = 100%)‒ 54.5%‒ 12,573

•Questions raised‒ Is feedback occurring, but not documented in the appropriate place?‒ Are there differences based on Service?

‒ Yes…. Range from 42% to 75%‒ Suggests variability in training and leadership

Early Outcomes: Fidelity to Model (cont’d)

•Required outcome measure administered (Goal = 100%)‒ 46.9%‒ 10,820

•Questions raised‒ Did policy change affect outcome? (The required outcome measure

changed from DUKE to BHM-20 at the start of the monitoring period)‒ Is measure being administered but not documented in appropriate

location?‒ Are there differences based on Service?

‒ Yes…. Range from 30% to 65%‒ Suggests variability in training and leadership

Early Outcomes: Clinical

•BHM-20 Global Mental Health Scale‒ Composite score including life satisfaction, psychological symptoms,

social relations, and life functioning‒ Compared initial and last BHM-20 administrations (paired samples t-test)‒ Only 2,667 patients (11.6%) had both 2 or more appointments and 2 or

more administrations of BHM-20‒ Not a true random sample; difficult to generalize

‒ Statistically significant improvements in BHM-20 Global Mental Health scale were seen for patients with at least two appointments t(2,666)=17.04, p<.000001

‒ Time 1 M=2.65, SD=0.72‒ Time 2 M=2.87, SD= 0.75

Early Outcomes: Clinical

•Depression scores measured by PHQ-9‒ Categorized patients into clinical outcome trajectories based on reliable

change index (RCI)‒ Restricted sample to those who had a depression-related diagnosis, 2 or

more appointments, and at least 2 PHQ-9 scores (n=554)‒ 44.8% of patients (n= 248) with depression-related diagnosis

demonstrated reliable improvement on PHQ-9

•Anxiety scores measured by GAD-7‒ Categorized patients into clinical outcome trajectories based on RCI‒ Restricted sample to those who had an anxiety-related diagnosis, 2 or

more appointments, and at least 2 GAD-7 scores (n=566)‒ 40.3% of patients (n= 228) with anxiety-related diagnosis demonstrated

reliable improvement on GAD-7

Early Outcomes: Clinical

•Post-traumatic stress scores measured by PCL‒ Categorized patients into clinical outcome trajectories based on RCI‒ Restricted sample to those who had a PTSD diagnosis, 2 or more

appointments, and at least 2 PCL scores (n=21)‒ Note very small sample size! Not a true random sample; difficult to

generalize‒ 61.9% of patients (n= 13) with PTSD diagnosis demonstrated reliable

improvement on PHQ-9

Barriers and Solutions

• What goes into the EHR does not always come out‒ Data entered into a non-mineable field cannot be easily retrieved! ‒ (And of course, data that is not entered can never be retrieved)‒ Work with IT to improve EHR options for data mining‒ Train BHCs and PCPs on where and what to document

• Periodic (e.g., quarterly) monitoring may not capture change over time when visits are spaced

‒ Sequential monthly (versus weekly) appointments may not be captured in a data pull over one quarter

‒ Consider expanding time range (e.g., 6 months)• When mean number of visits is low (between 1 to 2), assessing

clinical change becomes a challenge!‒ Look for alternate ways of assessing impact‒ Consider ways to obtain follow-up data later in time (e.g., assessment

measures administered at future PCP appointments)

Barriers and Solutions (cont’d)

•Large data sets will be inaccurate (over and over again)‒ Plan time and approach for substantial data scrubbing‒ Incorporate regular updates of new and departed personnel

•Large data sets will be unwieldy to use‒ Identify and purchase statistical software with capability to manage large

sets of data•Large data sets will yield statistical significance (even when change is not clinically meaningful or reliable)

‒ Determine reliable change threshholds for each measure, when possible‒ Evaluate for clinically meaningful change

Barriers and Solutions (cont’d)

•Not all providers diagnose and code the same way‒ E.g., Consider the number of different diagnoses that may be used for

presentation of depressive symptoms ‒ Clearly define full range of acceptable codes for data pulls

•Large data sets include disparate problem areas, diverse patient populations, and varying degrees of severity

‒ This is a strength but also a challenge‒ Meaningful clinical change for sub-populations can get lost in larger data

‒ Would we expect PHQ-9 scores to change for everyone?‒ Would we expect BHM-20 psychological health scores to change for

those seen for health behavior change?‒ Run analyses on distinct sub-groups of interest, based on demographics,

problem area/diagnosis, severity, etc.

Application to Other Settings:Monitoring and Evaluation

Monitoring & Evaluation Concepts

• Both monitoring and evaluation (M&E) involve the collection of data

• M&E is not research (although many research methods and practices are used)– Does not seek to make generalizable statements– Is not intended to contribute to a scientific literature– Accepts a lower standard of evidence than research

• M&E is a management tool that collects and uses information to inform decision-making around…– Innovation implementation– Innovation performance– Innovation impact

Monitoring & Evaluation Definitions

• Monitoring– A continuous effort targeting processes and intermediate outcomes– Most useful for measuring implementation factors that influence quality or practice

fidelity– What are the critical components of your innovation that, if not done correctly or

on time, will threaten the desired impact? These are the things you need to monitor!

• Evaluation– A more periodic effort (usually once or twice over the life of a project)– Assesses impact of an innovation and seeks to derive lessons-learned– How will you know that your innovation has been successful and how will you

understand what contributed to or limited that success? These are questions that evaluations seek to answer!

Measurement Burden

• Information from M&E efforts inform decision-making

• But information comes at a cost (time and resources)– Data collection– Data management– Data analysis– Measurement burden

• What to measure and when?– Think about what kinds of decisions you need to make and how often you

need to make them– Think about what information you need to inform your decisions– Measure only what you need to – don’t try to measure everything

These factors represent both an opportunity cost (i.e., time spent NOT doing patient care) and a threat to data validity

Monitoring & Evaluation

The Science of monitoring and evaluation is measurement, the Art of M&E is efficiency.

Monitoring & Evaluating Local Innovations

1) Define the problem– Identify what needs to be changed– Be concrete and specific– Analyze the current state

• Who is involved?• What are the current practices?• What structures, policies, practices are currently limiting performance?

2) Define the desired end-state– Envision what the situation should look like after the problem has been

resolved– Again, be concrete and specific– Write a SMART Goal

Monitoring & Evaluating Local Innovations

3) Define the innovation– Map out the resources required and the steps that need to be taken to

logically move you from current state to your desired end-state

ResourceResource

ResourceResource

Step 1aStep 1a

Step 2Step 2

Intermediate Outcome

Intermediate Outcome

Intermediate Outcome

Intermediate Outcome

End-StateEnd-State

ResourceResource

ResourceResource

Step 1bStep 1b

Monitoring & Evaluating Local Innovations

4) Identify critical steps and outcomes in your innovation model– These are the things that need to be monitored

ResourceResource

ResourceResource

Step 1aStep 1a

Step 2Step 2

Intermediate Outcome

Intermediate Outcome

Intermediate Outcome

Intermediate Outcome

End-StateEnd-State

ResourceResource

ResourceResource

Step 1bStep 1b

Monitoring & Evaluating Local Innovations

5) Define the measurement strategy– For each critical step and intermediate outcome...

• Determine if information collection mechanisms already exist• If not, determine how information will be collected

‒ Questionnaires/Surveys‒ Counts‒ Interviews‒ Focus Groups

• Determine who will collect the information• Determine how often the information will be collected and analyzed

– For the desired end-state…• Revisit your SMART Goal and determine how you will assess whether or not

you’ve met your goal• Define success in terms of degree of change in the measure you will use• Measure the current state BEFORE you initiate the innovation

Monitoring & Evaluation: Some Things to Keep in Mind

• Complex innovations will have more complex M&E systems

• Favor sufficiency over comprehensiveness in choosing measurement strategies– A 500-item survey may give you the best information– Asking your staff a couple of questions during a meeting or conducting an AAR

may give you sufficient information

• Ensure sufficiency and relevance of your measurement strategies– Simple methods are better, BUT…– this is only true if the simple method is sufficient to meet your information needs

• Be intentional in your data collection especially when using qualitative approaches (e.g., After Action Reports or informal staff interviews)– Follow your information collection plan (follow the schedule, ask all the questions

that you have mapped out)

Monitoring & Evaluation: Additional Things to Keep in Mind

• For the most part, information collection in the service of quality assurance and program evaluation is exempt from research and survey regulations.

• However, large scale or sensitive information collection strategies could trigger requirements for research or survey determinations.

• Consult with your local IRB whenever in doubt

Broad Categories to Monitor/Evaluate

• Patient outcomes• Clinic-level or system outcomes• Team outcomes• Cost outcomes

• Clinical variables• Process variables

Potential Areas to Monitor/Evaluate

Patient Outcomes: Clinical

• Behavioral health symptom improvement – Scores on symptom-specific behavioral health measures

• Patient Health Questionnaire-9 (PHQ-9 for depression)• Generalized Anxiety Disorder-7 (GAD-7)• PTSD Checklist (PCL)• Insomnia Severity Index (ISI)

– Scores on broad-based behavioral health measures• DUKE Health Profile• Behavioral Health Measure-20 (BHM-20)

• Functioning or quality of life improvement– DUKE Health Profile– BHM-20

Patient Outcomes: Clinical

• Health behavior improvement– Successful tobacco cessation– Decrease in at-risk alcohol use– Increase in adherence to medication as prescribed

• Medical outcome improvement– BMI decrease– A1C decrease– Improved lipids– Decreased chronic pain intensity and interference (PEG-3 )

Patient Outcomes: Process

• Patient satisfaction– % satisfied with quality of care– % satisfied with convenience of care– % who believe that BHC helped them with their concern– % of patients who would recommend seeing a BHC to others

• Patient engagement– % of patients who felt involved in treatment decision-making– % of patients who came back for a second BHC appointment (when

recommended)– % of patients who complete expected number of appointments as part of a

clinical pathway (e.g., 4 appointments for tobacco cessation)

Practice-Level Outcomes: Process

• Access to care– Patient wait times– Same day access

• Proportion of eligible patients referred• Proportion of referred patients who kept BHC appointment• Utilization rates of BHC

– # unique patients seen– # patient encounters– # same day patient encounters– Average # visits per patient– Utilization rates of BHC for particular conditions or issues– Average appointments per workday

BHC Practice Fidelity

• Appointment length – % of BHC appointments that were < 30 minutes)

• Feedback to PCP– % of appointments with documentation that feedback was provided to PCP

• Use of outcome measures– % of appointments in which required outcome measure documented

• Population-based care– % of patients seen for more than 4 appointments (per episode of care)

Cost-Related Outcomes

• Reduction in referrals to specialty care• Reduction in use of emergency care services• Reduction in inpatient hospitalization rates

Team Factors: Barriers to Referral

• PCP perceptions of barriers to referral – Not sure how to refer– Don’t want to interrupt BHC– No time to talk to patient about BHC referral– Patient unlikely to benefit– Forget to refer– Patient refuses

Team Factors: Communication

• Team communication– (Existence of) daily team huddles– (Existence of) regular provider meetings including BHCs and PCPs– Ease of communicating between team members– Staff satisfaction with communication– Frequency (and helpfulness) of “curbside consults”– Frequency of face-to-face PCP feedback after BHC appointment– Use of shared care plans in EHR

Team Factors: BHC Satisfaction

• BHC satisfaction with core aspects of their PCBH work– Seeing many patients– Rapid pace; same-day availability– Providing services for a wide range of referral problems– Giving PCPs feedback about patients– Providing group services– Working with PCPs to develop new programs– Belief that BHC services are helpful to their patients

Team Factors: PCP Satisfaction/Attitudes

• PCP satisfaction with BHC services– Satisfaction with PCP’s and patients’ access to BHC– Perception of how helpful BHC is to PCP and patients– Rating of quality of BHC services– Assessment of extent BHC meets both PCP’s and patient’s needs– Overall satisfaction with BHC services

• PCP attitudes towards BHC– Belief that BHC is value added– Belief that BHC is competent– Belief that a BHC knows how to deal with various conditions– Belief that BHC is capable of learning about and treating a patient with a

condition unfamiliar to the BHC– Belief that the PCBH model can and will work

Discussion and Worksheet

Q & A/Summary

•What are two process metrics useful to assess as part of a primary care behavioral health (PCBH) program evaluation?•What are two outcome metrics useful to assess as part of a PCBH health program evaluation?•What is one barrier to PCBH program evaluation in large medical systems? What is a potential strategy for addressing the barrier?

Session Evaluation

Please complete and return theevaluation form to the classroom monitor before

leaving this session.

Thank you!