total clinical outcomes management: theory and method richard a. epstein, ph.d., m.p.h. michael j....

Total Clinical Outcomes Management:Theory and Method

Richard A. Epstein, Ph.D., M.P.H.Michael J. Cull, Ph.D., M.S.N.

Department of Psychiatry

2

Road Map

• The importance of shared vision• Managing information asymmetry• The role of standardized assessment• Implementation examples

3

“To organize the world’s information and make it universally accessible and useful.”

4

“Our investment in data-mining is part of our drive to deliver what our customers want: the item, at the right store, at the right time, at the right price.”

“To organize the world’s information and make it universally accessible and useful.”

5

“Our investment in data-mining is part of our drive to deliver what our customers want: the item, at the right store, at the right time, at the right price.”

“The behavioral health system, at all levels, should always make decisions based on the needs and well-being of the people served.”

“To organize the world’s information and make it universally accessible and useful.”

6

Step 1: Sit downStep 2: Hold on for 8 seconds

7

Landing a passenger jet on the Hudson

8

A Successful Landing• Capt. Sullenberger credited teamwork,

preparation and strict adherence to protocols for the successful landing.

9

Success in Health Care

10

International Comparison, 2004

2.8 2.8 3.1 3.2 3.3

4.4

5.3

6.8

Source: Commonwealth Fund National Scorecard on U.S. Health System Performance, 2008

Infant Mortality Rate

Deaths per 1,000 births

11

Ambulatory Care Visits for Adverse Drug Effects

20

22

18

21

17

9

28

1716

22

0

10

20

30

Total

Mal

e

Femal

e

White

Black

Other

Northea

st

Mid

west

SouthW

est

Visits per 1,000 population per year

Source: Commonwealth Fund National Scorecard on U.S. Health System Performance, 2008

11.312.9

15.3

11.3

17.4

19.8

1999 2000 2001* 2002 2003 2004

Total

Physician Office

Hospital Emergency Department Visits

Hospital Outpatient Department Visits

By Gender, Race, and Region, 2004 Annual Averages, by Care Setting

12

13

Health Care Failures

1. Ignorance – science has given us incomplete information

2. Ineptitude – science has provided a solution, we fail to apply it correctly

Atul Gwande, 2009

14

Ignorance

• Evidence-based practice

• Practice-based evidence

15

Ineptitude

• Organizing information

• Standardize practice

• Support rational decision making

16

The cost of ineptitude• Poor Quality• Inefficiency• Poor resource allocation• Excess costs

17

Combating ineptitude

• Bias• Complexity• Goal incongruence

18

Bias• Stereotyping• Moral hazard• Treatment Framing• Probability• Confirmation bias

19

Complexity• Co-morbidity• Treatment options• Governance and financing

20

Goal Incongruence• Multiple perspectives• Information asymmetry• Motive/”agenda”

21

The Case for Decision Support• Eliminates bias• Manages complexity• Supports creation of a shared vision

22

Shared Vision: Child/Family• Child and family centric• Strengths based• Culturally and contextually sensitive

23

“12 Angry Men” Video Clip

24

“12 Angry Men”• Multiple perspectives?• Group think? Strong-arming?• Disagreement?• Bias?• How will they discuss their disagreement?• Can they reach a shared vision?

25

Managing Complexity: Aviation• Checklists developed in 1935 to manage complexities

of the Boeing model 299 long-range bomber

• In 1968 NTSB recommended a re-evaluation of checklists after the crash of an American World Airways B-707

• Pre-flight checklists are now “standard of care”

26

Reducing Bias & Complexity: Medicine• Critical care

– Pronovost et al. (2003) – Use of “daily goals” checklists associated with improved communication and reduced length of ICU stay.

• Catheter related infections– Pronovost et al. (2006) – Use of checklists and team

empowerment associated with 66% reduction in catheter-related infections.

• Surgery– Makary et al. (2007) – Use of operating room briefing tool

improved team communication and reduced risk of “wrong-side” surgeries

– Haynes et al. (2009) – Use of checklists associated with reduced risk of death and other surgery outcomes

27

Example: Surgical Safety• Surgery is integral to global health

– 234 million operations each year• Many complications are preventable• Evaluated the effectiveness of improving

team communication on outcomes

28

Hospitals• Toronto, Canada• New Delhi, India• Amman, Jordan• Auckland, New Zealand• Manila, Philippines• Ifakara, Tanzania• Long, England• Seattle, Washington

29

Intervention• Measured outcomes before and after

intervention• Intervention was a 19-point checklist• Checklist had three parts:

– Sign in– Time out– Sign out

30

Sign In• Before anesthesia, team verbally confirms:

– Verification of patient identity, surgical site, procedure, and consent

– Surgical site is appropriately marked– Pulse oximeter is on the patient and functioning– All team members are aware of patient allergies– Patient risk of airway obstruction and aspiration

has been evaluated and appropriate equipment is available

– Blood and fluids are available

31

Time out• Before incision, team verbally confirms:

– All team members are introduced by name and role

– Patient identity, surgical site, and procedure– Review of anticipated critical events– Administration of prophylactic antibiotics

before incision or that they are not indicated

– Essential imaging results are displayed

32

Sign Out• Before patient leaves the OR:

– Nurse reviews aloud with the team:• Name of the procedure as recorded• That needle, sponge, and instrument counts are

complete• That specimen is correctly labeled• Whether there are any equipment issues

– Team reviews aloud key concerns for recovery and patient care

33

Complications Before After p value

Surgical site infection 6.2 3.4 < .001

Death 1.5 0.8 < .01

Process Measures

Prophylactic antibiotics given 56.1 82.6 < .001

Orally confirm patient’s identity 54.4 92.3 < .001

Results

34

Summary• Introducing surgical checklist:

– Reduced surgical site infections and death by ½

– Increased compliance with safety process measures

• What did the checklist accomplish?– Ensured similar information for the team – Improved team communication– Supported shared vision

35

Break

36

Road Map• The importance of shared vision• Managing information asymmetry• The role of standardized assessment• Implementation examples

37

Data-driven decision making

38

Service planning exercise• Case vignette• System-level service plan• Importance of measurement and

communication

39

“Bender” Video Clip 1

40

Identify 3 service plan needs

41

“Bender” Video Clip 2

42

Okay, now what?

43

Measurement as Communication• Purpose of all measurement is to communicate

– Time (e.g., days), Temperature (e.g., Fo, C

o), Value

(e.g., dollars)

• Physical sciences– Concern phenomena accessible via instrumentation– Instrumentation improves; measurement stays the

same

• Human and social sciences– Sometimes concern phenomena similarly measured– Many times involves phenomena not readily

accessible

44

Social Science Measurement• Logical empiricist perspective

– Measurement is application of procedure to show verifiable truth from observation

– Implication is that some real phenomenon exists and that this “truth” can be revealed

• Classical test and item response theories are based on a similar philosophies

45

Communication Theory• Traditional perspective also empiricist

– Knowledge transfers from one mind to another

• Main critique of the “transfer” model– Overly linear– A B with no room for impact of B on A and

information coming from A• Constitutive / constructivist model

– Communication as shared meaning making

46

Shared Meaning Making

47

Communication and Measurement• From a communication / constructivist

perspective, measurement is:– Less about “revealing truth”– More about “creating shared meaning”

• Classical test theory – Multiple items; internal consistency

reliability

48

Internal Consistency Reliability• Cronbach’s alpha is the statistic used to define

internal consistency reliability• The equation is as follows:

49

Classical Test and Communication Theories

• Complex systems require clear and concise communication about multiple constructs

• Classical test theory: reliability = redundancy• Inter-rater reliability is therefore both the

– Relevant measure of reliability– Concept most related to shared meaning– Directly facilitates decision support

50

• Reduce bias• Integrate multiple perspectives• Attend to all relevant information• Communicate and use information at all

system levels

Measurement Helps

51

TCOM Grid of ActivitiesFamily & Youth

Program System

Decision Support

Service Planning

Eligibility Resource Management

Quality Improvement

Case Management & Supervision

AccreditationCQI

TransformationSystem Evolution

Outcome Monitoring

Service Transitions & Celebrations

Evaluation Performance Contracting

52

Computer

User

CANS Assessment

Interactive Training

Environment

Public house

Agency

Group

Rater Groups

System

AnalyticsServer

Trained Rater

Child and Family

Using assessment data to support decision-making

rater data creates training opportunities

53

Tennessee DCS Implementation• Case managers are annually trained to reliability• Children are assessed at several points during a custody

episode– Entry and discharge– Placement transitions and regular intervals

• Support is provided by network of Centers of Excellence (COEs)– Training, consultation, independent 3rd party review– Data management and analytics

• Data used to:– Monitor child outcomes; Support placement

decisions– Inform treatment plans; Plan array of services

54

Training: January 2008 – April 2009Training % Completed

Initial 100

Recertification 95

PBC Agencies 85

• Initial and recertification trainings are ongoing• Increasingly use CANS for decision support• Initiated provider trainings and “Guide Program”

55

Assessment Volume: July - April 2009Month No. of completed assessments

July 1185

August 1310

September 1219

October 1349

November 1127

December 1230

January 1642

February 1560

March 1607

Total 12,229

56

Algorithm Services Intensity Recommendation: As of April 2009 (Initial assessments = 5119)

Services intensity recommendation

Frequency Percent

1 2391 46.7

2 1632 31.9

3 840 16.4

4 256 5.0

57

Secondary data utilization• Review of high-risk children• Placement quality team review• Regional utilization review support• Well being team meetings• Unified Assessment process• Assessment of Services Quality (ASQ)

58

Special projects• Restructuring the continuum model project (DCS)• Transition age youth (GOCCC)• Ad-hoc region level reports (DCS)• Surveillance of foster care youth with scores of ‘2’ or

‘3’ on sexual aggression, sexually reactive, and danger to others items (TAC)

• TFACTS placement matching project (DCS)• Substance abuse (T-ACT, GOCCC)• Sexually problematic behaviors (SPB Task Force,

GOCCC)• OJJDP juvenile court screening pilot

59

Characteristics of DCS children, age 5 years or more,

assessed between 2/18/08 – 2/31/09 Demographics # %Age in years

5 – 12 2663 29.8 13 – 16 3790 42.5 17 – 19 2473 27.7Race / ethnicity

African-American 2737 30.7 Non-Hispanic White 5699 63.8 Other 241 2.7Gender

Female 3723 41.7 Male 5203 58.3Adjudication Status

Dependent / Neglect / Unruly 5168 57.9 Delinquent 2074 23.23 regions with the most assessments

East Tennessee 1433 16.1 Shelby 1010 11.3 Mid Cumberland 994 11.1*Totals may not add up to 100% due to missing data

60

Three most prevalent Risk Behaviors by ageActionable Risk Behaviors 5 – 12 13 – 16 17 – 19

Suicide 1.3 4.4 2.5

Self Mutilation 1.5 3.6 1.7

Other Self Harm 2.0 7.9 5.9

Danger to Others 5.1 14.0 8.5

Runaway 0.8 13.4 15.7

Firesetting 0.6 0.9 0.3

Social Behavior 14.8 37.8 27.8

Sexually Reactive Behavior 7.0 5.7 3.5

Sexual Aggression 1.6 4.1 2.3

Delinquency 2.2 28.4 24.2

Substance Use 0.2 14.4 18.1

61

Age at which each risk behavior is most prevalent

Actionable Risk Behaviors 5 – 12 13 – 16 17 – 19

Suicide 1.3 4.4 2.5

Self Mutilation 1.5 3.6 1.7

Other Self Harm 2.0 7.9 5.9

Danger to Others 5.1 14.0 8.5

Runaway 0.8 13.4 15.7

Firesetting 0.6 0.9 0.3

Social Behavior 14.8 37.8 27.8

Sexually Reactive Behavior 7.0 5.7 3.5

Sexual Aggression 1.6 4.1 2.3

Delinquency 2.2 28.4 24.2

Substance Use 0.2 14.4 18.1

62

Three most prevalent Behav. / Emo. Needs by age

Actionable Behavioral and Emotional Needs

5 – 12 13 – 16 17 – 19

Psychosis 1.1 2.1 1.7

Impulsivity 21.3 26.5 18.6

Depression 4.8 17.5 13.8

Anxiety 6.7 11.1 7.6

Oppositional 6.2 21.3 15.7

Adjustment to Trauma 12.5 14.3 7.9

Attachment 4.4 4.8 2.8

Anger Control 12.4 32.6 22.2

Emotional Control 10.5 23.2 16.0

63

Age group in which each Behav. / Emo. Need is most prevalent

Actionable Behavioral and Emotional Needs

5 – 12 13 – 16 17 – 19

Psychosis 1.1 2.1 1.7

Impulsivity 21.3 26.5 18.6

Depression 4.8 17.5 13.8

Anxiety 6.7 11.1 7.6

Oppositional 6.2 21.3 15.7

Adjustment to Trauma 12.5 14.3 7.9

Attachment 4.4 4.8 2.8

Anger Control 12.4 32.6 22.2

Emotional Control 10.5 23.2 16.0

64

Difficulties with adjustment to trauma by SPB category

0

20

40

60

80

100

No SPB SR SA SR +SA

No difficulties with adjustment to trauma

Difficulties with adjustment to trauma

Perc

enta

ge

65

7

7.5

8

8.5

9

9.5

10

10.5

-1 -0.5 0 0.5 1 1.5 2

Years (vs Start Date)

Item

Ave

rage

(x

10)

TOT (ALL) YCM CMO TRH GRH PCR RES

Start

Hinge analysis of outcome trajectories prior to and after program initiation

66

Figure 5.2 Survival analysis of time to placement disruption for children/youth whose placement matches CANS recommendations (Match=0), those whose placed is at a lower intensity than recommended (match=1) and those

whose placement is more intensive than recommended (match=-1).

67

Modeling crisis decisions (Leon et al., 1999)

• A 5 item model predicted 78% of crisis decisions• Items - Suicide Risk, Danger to Others, Impulsivity, Emotional

Disturbance, Behavioral Disturbance• Replicated by He et al. (2004)

Predicted, # (%)

Observed, # (%) Hospitalize Deflect Total

Hospitalize 667(70)

285(30)

952(43)

Deflect 203(16)

1050(84)

1253(57)

Total 870(39)

1335(61)

2205(100)

68

Illinois Department of Children and Family Services

DCFS Wards

69

Mental Health Services

Illinois Department of Children and Family Services

70

Mental Health ServicesDCFS Wards

Illinois Department of Children and Family Services

71

Figure 5.1 Proportion of low-risk youth hospitalized by race over the duration of an decision support initiative

00.050.10.150.20.25

0.30.350.40.450.5

1998 1999 2000 2001 2002

72

Change in Total CSPI Score by Intervention and Hospitalization Risk Level (FY06)

51.2

34.134.231.0

24.4

17.5

47.4

35.2

26.4

22.1 24.218.0

0

10

20

30

40

50

60

SASS Assessment End of SASSEpisode

Mea

n C

SP

I S

core

HOSP (high riskgroup)

ICT (high risk group)

HOSP (medium riskgroup)

ICT (medium riskgroup)

HOSP (low riskgroup)

ICT (low risk group)

73

Summary

• TCOM can help systems:– Reduce bias– Integrate multiple perspectives/shared vision– Attend to all relevant needs and strengths– Communicate and use information at all system levels– Allocate resources and measure/improve outcomes

74

Selected references1. Gawande A. (2010). The Checklist Manifesto: How to Get Things Right. New York: Metropolitan Books.2. The Commonwealth Fund National Scorecard on U.S. Health System Performance. (2008).3. U.S. Government Accountability Office (http://www.gao.gov).4. Lyons JS. (2004). Redressing the Emperor: Improving Our Children’s Public Mental Health System. Westport, CT: Praeger.5. Lyons JS & Weiner DA. (2009). Behavioral Healthcare: Assessment, Service Planning, and Total Clinical Outcomes Management. Kingston, NJ: Civic Research

Institute.6. Lyons JS. (2010). Communimetrics: A Communication Theory of Measurement in Human Service Settings. New York: Springer.7. Lyons JS. (2008). Total Clinical Outcomes Management. Presentation at the Annual Child and Adolescent Needs and Strengths (CANS) Conference. Nashville, TN.8. Lyons JS. (2009). Total Clinical Outcomes Management in the service of children with behavioral and emotional needs: an update. Presentation to the Tennessee

Council on Children’s Mental Health. Nashville, TN.9. Shiv B & Fedorikhin A. (1999). Heart and mind in conflict: the interplay of affect and cognition in consumer decision making. Journal of Consumer Research, 26(3),

278-292.10. Pronovost P, Berenholtz S, Dorman T, et al. (2003). Improving communication in the ICU using daily goals. Journal of Critical Care, 18, 71-75.11. Rawal PH, Anderson TR, Romansky JR & Lyons JS. (2008). Using decision support to address racial disparities in mental health service utilization. Residential

Treatment for Children and Youth, 25(1), 73-84.12. Pronovost P, Needham D, Berenholtz S., et al. (2006). An intervention to decrease catheter-related bloodstream infections in the ICU. New England Journal of

Medicine, 355, 2725-2732.13. Makary MA, Mukherjee A, Sexton B, et al. (2007). Operating room briefings and wrong-site surgery. Journal of the American College of Surgeons, 204(2), 236-243.14. Haynes AB, Weiser TG, Berry WR et al. (2009). A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of

Medicine, 360(5), 491-499.15. Keysar B, Barr DJ, Balin JA & Brauner JS. (2000). Taking perspective in conversation: the role of mutual knowledge in comprehension. Psychological Science, 11(1),

32-38.16. Video clips are from: “12 Angry Men” and “The Breakfast Club”17. Epstein RA. (2008). Needs and strengths of children entering state custody. Presentation at the Annual Child and Adolescent Needs and Strengths (CANS)

Conference. Nashville, TN.18. Epstein RA & Cull M. (2009). Tennessee Department of Children’s Services CANS Implementation. Presentation to the Tennessee Council on Children’s Mental

Health. Nashville, TN.19. Leon SC, Lyons JS, Uziel-Miller ND, Tracy P. (1999). Psychiatric hospital utilization of children and adolescents in state custody. Journal of the American Academy of

Child and Adolescent Psychiatry, 38, 305-310.

75

Questions

Richard A. Epstein, Ph.D., M.P.H.richard.a.epstein@vanderbilt.edu

Michael J. Cull, Ph.D., M.S.N.michael.cull@vanderbilt.edu