copyright © bhl, 2009 two decades of top research it’s about improving treatment not just...

Copyright © BHL, 2009

Two decades of TOP research

It’s about improving treatment not just measuring it

David R. Kraus PhDFounder/CEO

October 7, 2009

1

Copyright © BHL, 2009Copyright © BHL, 2009

Treatment Outcome Package

2


TOP EvolutionVersion # Items # Subjects Factors

1 -- 1993 250 432 15

2 -- 1995 112 2,217 15

3 -- 1996 93 5,288 13

4 -- 2003 58 19,801 12

Horowitz, L. M., Lambert, M. J., & Strupp, H. H. (Eds.) (1997). Measuring patient change in mood, anxiety, and personality disorders: Toward a core battery. Washington, D. C.: American Psychological Association Press.

GOAL: Meeting the requirements of the Core Battery Conference

WORKSOCIALSEXUAL

FUNCTIONAL

QUALITYOF LIFE

GENERAL DISTRESS

DEPRESSIONSUBSTANCE ABUSEPSYCHOSISINSOMNIA / SLEEP

SUICIDEVIOLENCEANXIETYBIPOLAR

SYMPTOMS

3


Treatment Outcome Package

Built from tens of thousands of clinical cases

Excellent construct validity

Excellent discriminant and convergent validity

e.g. TOP Depression Scale vs. BDI (r = .92)

Excellent sensitivity to change

Excellent predictive validity

4


The tool is free

Data collection is free

Real-time client reports are free

FREE SERVICES

5


EMPIRICALLY SUPPORTED INTERVENTIONS

Outcome-based Referrals sm

Client Feedback ReportsBenchmarking and CQI

Hospital Prevention

BHL provides multiple pathways to improving service quality with

6


• 150 Empirically Supported Treatments• Accounting for 1 – 2% of outcome variance (Wampold, 2001)

• The therapeutic alliance• Accounts for 5% of the variance• But most of this variance is between therapists

• It is at the therapist level where results happen• 8.6% of variance across 10 RCTs (Crits-Christoph & Mintz, 1991)• Estimates as high as 50% in other studies (Blatt, Sanislow, Zuroff, & Pilkonis, 1996; Elkin,

Falconnier, Martinovich, & Mahoney, 2006; Crits-Christoph, et al., 1991; Huppert, Bufka, Barlow, Gorman, Shear, & Woods, 2001)

• Assigning cases based on skill sets • Can increase outcome effect sizes by as much as 400%• (Kraus & Castonguay, in preparation)

Outcome-Based Referrals SM

patent pending

7


Definitions

“Effective therapist” – Average patient is reliably improved

“Poor therapist” – Average patient is reliably worse


patent pending

Reliable Change (Jacobson & Truax, 1991) RCI = (X2 − X1)⁄SEDIFF SEDIFF = (2(SE)2)1⁄2

8



patent pending Areas of Effectiveness Therapists

0 4%

1 6%

2 9%

3 11%

4 12%

5 14%

6 12%

7 11%

8 9%

9 7%

10 3%

EXPERTISE AREASDepressionSubstance abuseManiaPanicPsychosisViolenceSuicideSleepSocial conflictSexWorkQuality of life

(Kraus & Castonguay, in preparation) 9



patent pending

DEPRS LIFEQ MANIA PANIC PSYCS SA SCONF SEXFN SLEEP SUICD VIOLN

LIFEQ .469

MANIA .326 .044

PANIC .491 .196 .240

PSYCS .428 .171 .335 .369

SA .264 .159 .211 .151 .239

SCONF .350 .193 .150 .252 .367 .134

SEXFN .310 .164 .118 .226 .218 .094 .310

SLEEP .419 .208 .184 .384 .333 .149 .238 .160

SUICD .531 .264 .229 .322 .454 .345 .309 .205 .290

VIOLN .281 .102 .207 .232 .321 .228 .283 .222 .181 .421

WORKF .331 .127 .258 .265 .253 .144 .251 .228 .213 .264 .241

Correlations between therapist skills by domain

(Kraus & Castonguay, in preparation)

DEPRS: Depression; LIFEQ: Quality of Life; PSYCS: Psychosis; SA: Substance Abuse; SCONF: Social Conflict; SEXFN: Sexual Functioning; SUICD: Suicide; VIOLN: Violence; WORKF Work Functioning

10



patent pending

ALL THERAPISTSCohen’s d effect size0.91 – A large effect accordingTo Lipsey’s criteria (1990)

pre post0

0.5

1

1.5

2

2.5

Depression

11



patent pending

Effect size-1.05

Effect size0.04

Effect size1.40



Reliably worseNo better or worseReliably better


patent pending

67%

30%

3%

Not all therapists are the same

Some make their patients:

Reliable Change (Jacobson & Truax, 1991) RCI = (X2 − X1)⁄SEDIFF SEDIFF = (2(SE)2)1⁄2

694 Clinicians treating at least 10 depressed patients



• Based on the philosophy of the “good-enough mother” (Winnicott, 1953)

• At intake, TOP feedback includes a list of therapists with proven, good-enough therapeutic skills to handle this patient’s issues

• Clinics should assign cases based on these recommendations


patent pending

14



patent pending

Avoid these and achieve9% improvement

Improving service quality overnight

Rely on these and achieve 54% increase


15



patent pending


Good Therapists

Poor Therapists

Effect size (All therapists)

Effect size (Good therapists)

Effect size increase

Depression 67% 3% 0.91 1.41 54%

Quality of life 47% 5% 0.68 1.51 122%

Mania 0.7% 0.3% Not enough data Not enough data N/A

Panic/Anxiety 43% 10% 0.42 1.17 179%

Psychosis 46% 9% 0.43 1.00 133%

Substance Abuse 50% 16% 0.47 1.14 143%

Social Conflict 45% 14% 0.48 1.46 204%

Sexual Functioning 29% 12% 0.27 1.48 448%

Sleep 54% 9% 0.57 1.20 111%

Suicide 58% 7% 0.64 1.30 103%

Violence 38% 16% 0.31 1.02 229%

Work Functioning 35% 7% 0.44 1.52 245%

16



patent pending

Rankings vary depending on domainTake the highest ranking Depression therapist

Domain Ranking

Depression 1

Quality of Life 3

Social Conflict 9

… …

Violence 550

Mania 692






Hospital Prevention


18


Exceptional sensitivity to change

LSQ2 TOP–DEPRS3

TOP–FULL3

Improvement 20% 54% 91%

No change 72% 32% 5%

Deterioration 8% 14% 4%

2. Developed by UBH, the LSQ is a 30–item short form of the OQ–45. Doucette, A. (2006) From Ordinal Observation to Linear Measurement, paper presented at the North American meeting of the Society for Psychotherapy Research. N>140,0003. Kraus, D. R., Seligman, D, Jordan, J. R., (2005). Validation Of A Behavioral Health Treatment Outcome And Assessment Tool Designed for Naturalistic Settings: The Treatment Outcome Package. Journal of Clinical Psychology, Vol 61(3), 285-314. N>20,0004. Reliable Change (Jacobson & Truax, 1991) RCI = (X2 − X1)⁄SEDIFF SEDIFF = (2(SE)2)1⁄2

19


Construct Validity Confirmatory Factor AnalysesGFI CFI TLI RMSEA

Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: Guilford.

Acceptable fit >.90 >.90 >.90 <.08

Good fit >.95 >.95 >.95 <.05

Mueller, R. M., Lambert, M. J. & Burlingame, G. M. (1998) Construct Validity of the Outcome Questionnaire: A Confirmatory Factor Analysis, Journal of Personality Assessment, 70(2), 248-262.

OQ-Total Score .827 .195 Not listed Not listed

OQ-2 factor model .824 .177 Not listed Not listed

OQ-3 factor model .822 .157 Not listed Not listed

Kraus, D. R., Seligman, D., & Jordan, J. R. (2005). Validation of a Behavioral Health Treatment Outcome and Assessment Tool Designed for Naturalistic Settings: The Treatment Outcome Package. Journal of Clinical Psychology , 61(3), 285-314.Kraus, D., Boswell, J., Wright, A., Castonguay, L., & Pincus, A. (submitted). Factor Structure of the Treatment Outcome Package for Children. Journal of Clinical Psychology.

TOP Adult (12 factor) .952 .947 .940 .035

TOP Child (13 factor) .966 .973 .969 .035

20


A Psychometric Evaluation of the Treatment Outcome Package -

TOPMark A. Blais, PsyD, Samuel J. Sinclair, PhD &

Hal Shorey, PhDMGH-Psychological Evaluation

and Research Laboratory

INDEPENDENT VALIDATION FROMA PRIMARY COMPETITOR

21


Client Report

23


Client Report

24


Empirically Supported Feedback

Can therapists handle complex feedback?

Each TOP administration comes with detailed assessment results.

There are no directives on how to, or whether to, read the report.

So what happens if therapists are allowed to do as they please?

25


Case Vignettes• Most therapist attitudes towards outcomes begins with

skepticism, even hostility.

• As a clinician I typically try to see the world through the eyes of my patients.

• It is typically in one of these eye-opening experiences that therapist attitudes towards outcomes change.– Sainz (2009) When metrics matter: The Case of Frank and the Treatment Outcome Package

• Often it is easier to be more honest about how bad things are on a questionnaire (Carr & Ghosh, 1983; Erdman, Klein, & Greist, 1985; Hile & Adkins,

1997; Lucas, 1977; Searles, Perrine, Mundt, & Helzer, 1995; Turner et al., 1998)

26


Empirically Supported Feedback

Clie

nt

Impr

ovem

ent

Number of clients with feedback

1 2 3 4 5 6 7 8 9

Those that don’t look

Those that look

27


Empirically Supported FeedbackΔ µ1 Δ µ2 σ1 σ2 T df α Initial µ1 Initial µ2

TOP total score** 2.6 4.1 11.2 10.1 2.4461 946 0.0146 24.2 25

Depression* 0.56 0.65 0.93 0.9 1.6639 1149 0.0964 1.88 2.06

Life quality*** 0.12 0.28 0.87 0.84 3.1266 1121 0.0018 1.81 1.98

Mania*** 0.13 0.23 0.51 0.6 3.0473 1143 0.0024 -0.16 -0.12Panic** 0.29 0.45 1 1.1 2.5713 1137 0.0103 1.65 1.74

Psychosis 0.55 0.65 1.3 1.3 1.2968 1139 0.195 1.47 1.53

Substance abuse 0.48 0.54 2.3 1.9 0.4489 1003 0.6536 1.15 1.12

Social conflict 0.33 0.31 1 1 0.3349 1123 0.7377 1.13 1.09

Sexual functioning 0.2 0.18 0.88 0.87 0.3749 1077 0.7078 0.48 0.49Sleep 0.37 0.43 0.79 0.8 1.277 1146 0.2019 1.13 1.24Suicide 0.55 0.54 1.7 1.4 0.1081 1147 0.8452 1.41 1.39

Violence*** 0.54 0.29 1.7 1.5 2.6263 1144 0.0087 1.04 0.93

Work functioning* 0.2 0.29 0.79 0.98 1.6694 1078 0.0953 -0.11 -0.04

Group1 = First 5 clients with feedback; Group2 = Second 5 clients with feedback

Δ µ = Average pre-post change; Initial µ = Average pre-treatment score* significant α <0.1; ** significant α <0.05; *** significant α <0.01 28




Client Feedback Reports

Benchmarking and CQIHospital Prevention


29


• An outcome management system with fully integrated quality improvement report cards.

• Integrated evidenced-based treatment libraries that have been clinically proven to improve outcomes.

EBT and CQI Support

30


Kraus, D. Castonguay, L. (2006) Integrating Evidence Based Practices with Outcomes Management. Paper presented at NASPR Annual Conference.

Adelman, R. (2008) Methods of Reconstruction with Adolescent Substance Abusers: Combining REBT and Constructivism. In Raskin, J.D. & Bridges, S. (Ed.), Studies in Meaning 3: Constructivist Psychotherapy in the Real World, New York: Pace University Press.

Adelman. R. (2007) Reducing Anger in Adolescents: An REBT Approach, Center City, Minnesota: Hazelden Press.

Adelman, R. (2006) The Angry Adolescent & Constructivist REBT. In Cummins, P. (Ed.), Working with Anger: a Practical Constructivist Approach, London: John Wiley & Sons.

Adelman, R. et. al, (2005) Reducing adolescent clients’ anger in a residential substance abuse treatment facility. Journal on Quality and Patient Safety, 31, (6), 325-327.

EBT and CQI Support

31


not sig

-1

-0.5

0

0.5

1

1.5

Dep

ress

ion

Inte

rper

son

al

Qu

ality

of L

ife

Pan

ic

Psy

cho

sis

Sex

Su

icid

ality

Sle

ep

Wo

rk

Vio

len

ce

Man

ia

Agency X: raw, pre/post outcomesall change scores are significant unless specified

Not sig

32


Provider Benchmarks

33


Quartile Analysis

0

1

2

-2

-1

Suicide

BENCHMARK

RANGE OF RAW

DATA

TOP quartile

Bottom quartile

mean

34


“The National Leader in Outcomes Management”Quartile analysis of same data

Clients with excellent results are left two bars in each domain, followed by clients with very poor results on right. Note, the second group of suicide clients are getting worse during the course of treatment. 35


Demographics Medical Stressful Events Best

Outcomes Worst

Outcomes Best

Outcomes Worst

Outcomes Best

Outcomes Worst

Outcomes Category # % # % Category # % # % Category # % # % Homeless 2 7% 4 8% HealthExclnt 2 7% 6 14% Avg.Severity 29.7 21.6 Male 9 35% 29 67% HealthVryGd 1 4% 7 16% StdDev 23.7 19.1 Female 17 65% 14 33% HealthGood 12 44% 20 47% Death 7 35% 14 36% Single 13 46% 28 57% HealthFair 7 26% 7 16% SeriousIll 10 48% 13 32% Married 3 11% 3 6% HealthPoor 5 19% 3 7% IlnessofSelf 7 35% 12 31% Divorced 10 36% 17 35% Physician/year 4 2 RelationEnd 11 48% 11 30% Separated 2 7% 1 2% Physician/2Mo 1 1 RelationProb 14 64% 13 35% Widowed 0 0% 0 0% Prescriptions 3 2 SupportProb 12 52% 14 38% Christian 19 70% 29 62% NoMedHosp 23 82% 35 70% Ed.Prob 6 27% 8 21% Hindu 0 0% 0 0% MedHospAvg 1.1 6.2 FinancialProb 14 64% 24 59% Jewish 1 4% 0 0% CaffeineAvg 3 4 HousingProb 11 52% 13 34% Muslim 0 0% 0 0% NonSmokers 10 36% 13 26% WorkProb 10 45% 17 46% No Religion 3 11% 9 19% Avg.PsychHos 3.8 1.8 HealthProb 8 36% 13 33% OtherReligion 4 15% 9 19% Avg.Therapists 3 2 LegalProb 9 41% 18 45% White 23 79% 45 85% ExerciseFreq 2 2 TraumaProb 5 24% 7 19% Black 0 0% 0 0% BackProblems 10 36% 13 26% DailyHassles 13 62% 12 32% Asian 0 0% 0 0% HeartDisease 3 11% 3 6% Client Defined Goals East Indian 1 3% 1 2% Diabetes 4 14% 5 10% Avg.Sever#1 7.5 7.6 Am.Indian 3 10% 4 8% Cancer 5 18% 4 8% Avg.Year#1 7.4 7.8 Hispanic 2 7% 3 6% LungDisease 4 14% 4 8% Avg.Sever#2 7.9 6.7 Mixed Race 2 7% 3 6% KidneyDisease 2 7% 1 2% Avg.Year#2 7.7 9.8 Income $0+ 15 60% 13 34% LiverDisease 2 7% 1 2% Avg.Sever#3 7.4 6.8 Income $10+ 5 20% 12 32% Anemia 3 11% 5 10% Avg.Year#3 10.1 9.3 Income $20+ 3 12% 6 16% ImmuneDis. 1 4% 4 8% Miscellaneous Income $30+ 0 0% 1 3% Ulcer 4 14% 5 10% AssistYes 10 38% 5 11% Income $40+ 0 0% 0 0% Asthma 3 11% 5 10% AssistNo 16 62% 40 89% Income $50+ 2 8% 6 16% BloodPressure 5 18% 4 8% TOPExclnt 6 24% 5 12% Income $75+ 0 0% 0 0% Education TOPVeryGd 4 16% 4 10%

Income$100+ 0 0% 0 0% Avg.Yrs.Sch. 12 12 TOPGood 10 40% 23 55% Income$200+ 0 0% 0 0% ElementaryEd 3 11% 4 10% TOPFAir 3 12% 7 17% Empl.FlTm 1 4% 23 46% HighSchoolEd 19 68% 29 72% TOPPoor 2 8% 3 7% Unemployed 16 57% 19 38% College2yr 5 18% 5 13% VoluntaryNo 4 14% 22 44% Average Age 40 34 College4yr 0 0% 1 3% LifeGood% 59 62 Avg.# Chldrn 1.4 1.4 GraduateSchool 1 4% 1 3%

Quartile analysis of same data

36


Best

Outcomes Worst

Outcomes Best

Outcomes Worst

Outcomes Best

Outcomes Worst

Outcomes Category # % # % Category # % # % Category # % # % DeprsMean .9 .3 QOLMean 1.4 -0.3 ViolnMean 1.3 0.8 DeprsNormal 36 64% 40 71% QOLNormal 16 50% 35 71% ViolnNormal 38 69% 41 77% DeprsMild 6 11% 3 5% QOLMild 2 6% 0 0% ViolnMild 2 4% 1 2% DeprsMod. 5 9% 4 7% QOLMod. 11 34% 6 12% ViolnMod. 3 5% 3 6% DeprsSevere 9 16% 9 16% QOLSevere 3 9% 8 16% ViolnSevere 12 22% 8 15% IntprMean 0.6 -0.1 SexMean -0.3 -0.5 WorkMean 0.4 -0.2 IntprNormal 27 84% 45 92% SexNormal 30 97% 46 96% WorkNormal 24 80% 38 83% IntprMild 1 3% 2 4% SexMild 0 0% 0 0% WorkMild 2 7% 3 7% IntprMod. 3 9% 2 4% SexMod. 0 0% 1 2% WorkMod. 1 3% 3 7% IntprSevere 1 3% 0 0% SexSevere 1 3% 1 2% WorkSevere 3 10% 2 4% Substance Abuse ManicMean 0.0 -.3 SleepMean 1.1 0.3 DrugWorry 5 15% 14 27% ManicNormal 49 88% 51 94% SleepNormal 30 56% 36 68% DUI 7 21% 21 40% ManicMild 3 5% 2 4% SleepMild 6 11% 2 4% ExcessDrink 0.5 0.5 ManicMod. 1 2% 1 2% SleepMod. 8 15% 6 11% NoDrugsUsed 19 58% 36 69% ManicSevere 3 5% 0 0% SleepSevere 10 19% 9 17% Alcohol 11 33% 9 17% Maijuana 4 12% 7 13% PanicMean 1.5 .5 SuicdMean 2.0 1.5 Cocaine 1 3% 1 2% PanicNormal 29 52% 41 73% SuicdNormal 38 68% 40 74% Other Drugs 2 6% 3 6% PanicMild 7 13% 1 2% SuicdMild 2 4% 1 2% Hallucinogens 0 0% 1 2% PanicMod. 7 13% 3 5% SuicdMod. 1 2% 3 6% PanicSevere 13 23% 11 20% SuicdSevere 15 27% 10 19% PsycsMean 1.9 .9 PsycsNormal 26 48% 40 71% PsycsMild 6 11% 2 4% PsycsMod. 6 11% 3 5%

PsycsSevere 16 30% 11 20%

Quartile analysis of same data

37


Summary of quartile analysis findings Agency X: suicide

• Knowing what patients to watch out for:– Involuntary treatment– Males more than females– Who endorse few problems and especially

state their quality of life is great– … BUT, who ironically endorse clinically

significant suicidal ideation

38


Empirically Supported Treatment Manual

39





Hospital Prevention


42


Predictive Alerts on First Assessment

• Hospitalization alerts– Who will be hospitalized in next six months– Could save US more than a half billion each

year by increasing appropriate out-patient tx• On what recovery path is this patient?

– Rapid responder– Slow responder– Negative responder

43


Stelk, W., & Berger, M. (2009). Predictive Modeling: Using TOP Clinical Domain Items to Identify Adult Medicaid Recipients at Risk for High Utilization of Behavioral Health Services in a Managed Care Provider Network. 40th SPR International Annual Meeting. Santiago de Chile.

BCBSMA Warehouse studies: Potential savings of $6.6M per year. Key is identifying patients heading over a cliff, but before they start through the revolving door.

Reasonable sensitivity (40%), especially when considering the prediction of future behavior, and excellent specificity (99%).

Predicting Hospitalizations

44


WellnessCheck®

Infrastructure

with

Hands-Free Outcomes sm

45


Hands-Free Outcomes sm

Three simple steps to measurement:

1. Sign informed consent to use patient’s email2. Register patient on WellnessCheck.net® 3. BHL handles the rest with sophisticated patent-

pending No-PHI sm security protections

46


Patented “One Click Assessments”You have been asked by your doctor to complete an important

WellnessCheck.net assessment of your health. This process is highly confidential and your doctor will receive a critical laboratory health assessment report that he/she can go over with you at your next appointment. Please do not reply to this automated email message from WellnessCheck but communicate directly with your doctor if you have any questions.

Click on the following link and answer the security question (answer given to you by your doctor in a separate handout or verbal communication):

https://demo.wellnesscheck.net/oltop/t/welcome.jsf?accessId=NTcyMDI2OjE1ODoxOTYwOjA6MTo2NzM3

47




WELLNESSCHECK® assess patient health & treatment outcomes online

A secure site for doctors and their patients

Log in

WellnessCheck.net

Free outcome toolsFree data collection enginesFree client feedback reports

48

http://demo.wellnesscheck.net/oltop/home.jsf


Three administration options:• In office over web• In office on paper (print and fax forms to BHL)• At home administration

50


Pharmaceutical Outcomes

51

copyright © bhl, 2009 two decades of top research it’s about improving treatment not just...

Documents