Understanding Cutoffs, Norms, Trajectories, and everything else that

freaks you out: A Primer for Dummies

G.S. (Jeb) Brown, Ph.D.

Center for Clinical Informatics

But I’m a clinician……

• Statistics aren’t necessary to be a good therapist.

• I was never good at math.

• I’ll let the researchers worry about statistics, I’m interested in helping clients.

• I know when my clients are getting better; I don’t need an outcome measure to tell me this.

• I dissociate when confronted with numbers.

Stats phobia desensitization• Close you eyes and breath slowly

• Visualize a lovely hill… green grass, gentle breeze, warm sun

• See how the grand rises slowly at first, then steeper; notice the sensuous lines and beautifully rounded summit.

• As you exhale slowly, say to yourself …

“I love the Bell Curve”.

• Repeat as needed.

Mantra: I love the Bell Curve

Mantra: I love the Bell Curve

0%

2%

4%

6%

8%

10%

20 26 32 38 44 50 56 62 68 74 80

What’s normal?

• All outcome questionnaires appear to measure a common “factor”… global distress/happiness

• Like almost every other human trait, misery is normally distributed.

0%

20%

40%

60%

80%

100%

20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80

Seriously unhappy

Us normal folk

Pathologically happy

Calculating clinical cutoff

• Clinical cutoff scores used to estimate the boundary between “normal” and “clinical”.

• Take another deep breath… here comes a formula……

• Thanks to Jacobson & Truax, 1991

(SD1)(mean2)+(SD2)(mean1) C = SD1+SD2

Idealized clinical cutoff

0%

2%

4%

6%

8%

10%

10 16 22 28 34 40 46 52 58 64 70 76 82 88

Normal Sample

Clinical Sample

Clinical cutoff

Nothing is perfect….

• The Bell Curve is beautiful in the abstract, but reality is messier.

• Many tests may floor and ceiling effects.

• Distributions may depart from normality….

Floor and ceiling effects

• Example – the mean of a in a non clinical sample completing the ORS was 30 and the standard deviation 6.2.

• But wait! The ORS has a maximum of score of 40, only 1.6 standard deviations above the mean.

Example samples (sample samples?)

• Accountable Behavioral Healthcare Alliance– Five county community mental health system in Oregon serving

adults and children. ORS administered administered at every session. Data collected at an estimated 80% of all sessions.

• RFL-Resources for Living– EAP company providing telephonic counseling services to adults.

ORS administered telephonically at every session. Approximately 80% of clients have a single session (one call).

• SAIC– Clinics in multiple countries serving children of active duty US

military personnel and military contractors.

• Community sample- Small sample of non clients.

ORS score distributions 4 clinical & 1 community samples

-5%

0%

5%

10%

15%

20%

25%

30%

35%

0 6 12 18 24 30 36

ABHA -Adults

ABHA_Child &Adolescent

RFL

SAIC

CommunitySample

Non normal curves……

Clinical cutoff scores

ABHA AdultsABHA Child &

AdolescentRFL

AdultsSAIC

Adolescents

standard deviation normal 6.8 6.8 6.8 6.8

standard deviation clinical 9.3 7.7 9 8.2

mean normals 29.7 29.7 29.7 29.7

mean clinical 18.3 21.4 19.5 25.8

cuttoff = 24.9 25.8 25.3 27.9

When is change “real”

• All measurement has “error”

• How do we know if change on a test is simply the result of random error?

• The Reliable Change Index is a common metric to determine if the difference between two scores is greater than expected from random error.

• Here we go again….Jacobson & Truax, 1991 RCI = 1.96 * Sdiff

Sdiff = 2SE2

SE = SD1 - rxx

When is change “real”

• Standard error of measurement (SE) is defined as the standard deviation multiplied by he square root of 1 minus the reliability of test as calculated by the coefficient alpha, a measure of internal consistency.

• Estimates of coefficient alpha….

ABHA = .89

RFL=.82

SAIC=.97

RCI estimates

Did I mention that all measurement is approximation?

ABHA AdultsABHA Child &

AdolescentRFL

AdultsSAIC

Adolescents

Standard deviation 9.3 7.7 9 8.2

reliability 0.89 0.87 0.81 0.97

standard error of measure 3.1 2.8 3.9 1.4

Sdiff 4.4 3.9 5.5 2.0

RCI 5.6 5.0 7.1 2.6

Criteria for recovery

• Jacobson & Truax (1991) proposed a two fold criteria for recovery

1. Change score exceeds the RCI

2. Scores moves from clinical range to normal range.

• Two problems with this criteria…

1. A substantial percentage of patients start treatment in the normal range

2. The probability of change exceeding the RCI is a function of severity

% of cases with “real” change(Change exceeds 5 points)

All casesClinical range cases (ORS intake =< 25)

0%

20%

40%

60%

80%

100%

ABHA RFL SAIC0%

20%

40%

60%

80%

100%

ABHA RFL SAIC

Worse

No change

Improved

Case mix adjustment

• We can’t evaluate outcomes without answering the question “Compared to what?”

• Differences in severity (intake score) and other client characteristics make it difficult to compare outcomes

• Case mix adjustment uses statistical methods to adjust for differences in clients when comparing outcomes from one site to another

• Case mix adjustment is always imperfect.

Regression to the rescue

Regression artifacts are always present in repeated measures

• If a measure has test-retest reliability (correlations between two points in time) then it will exhibit regression to the mean.

• Correlations between measures will tend to decrease over time.

• The intake score is almost always the strongest predictor of change.

• Recommended reading – A Primer on Regression Artifacts (Campbell & Kenny, 1999)

Multivariable GLM

• The General Linear Model can incorporate multiple variables to predict a continuous dependent variable, such as a change score.

• Both continuous and categorical variables can be incorporated into the model.

• Multivariate analysis of variance using GLM suggest that most of the variance in change scores is explained by the intake score. Variables such as diagnosis, age and sex account for a relatively small percentage of the variance.

Multivariable Regression

Excel and regression

• The slope and intercept values for a simple linear regression can be calculated in Excel.

• Amazingly, the slope function returns the coefficient for the slope and the intercept function returns the value for the intercept.

• Different slopes and intercepts can be calculated for subgroups (age group, diagnosis, etc).

Benchmarking outcomes

• GLM permits “benchmarking” of outcomes, by comparing an actual score (or change score) with the predicted score derived some the comparison (or normative) sample.

• Difference between actual and predicted score is know as the residual score (change score)

• “Benchmark Score” or Change Index Score” maybe have more intuitive meaning than “Residual Score”. Or not.

Trajectory of Change Graph

• The Trajectory of Change graphs displays a client’s actual scores to the predicted scores.

• GLM used to predict scores are subsequent measurement intervals (sessions, weeks) using intake score and any other variables available.

• Distribution of residuals used to plot percentiles at different intervals.

Graphing scores over time

Trajectory of change

0

5

10

15

20

25

30

35

40

Inta

ke

Sessio

n 2

Sessio

n 3

Sessio

n 4

Sessio

n 5

Sessio

n 6

Sessio

n 7

Sessio

n 8

Sessio

n 9

Session

Ou

tco

me

Sc

ore

Actual score

Average Score

25th percentile

75% percentile

Clinical cuttoff

Graphing 3 way interactions• Trajectory of Change graphs can be used to

display 3 way interactions involving severity, time and a third variable of interest (treatment, age group, diagnosis, etc.)

• Slope and intercept at each measurement point are calculated separately for grouping of interest.

• Following are two examples from the ABHA data.

Trajectories of changeAdults and Children/adolescents (AHBA)

05

10152025303540

Inta

ke

Sessio

n 2

Sessio

n 3

Sessio

n 4

Sessio

n 5

Sessio

n 6

Sessio

n 7

Sessio

n 8

Sessio

n 9

Session

Ou

tco

me

Sco

re

75% percentile

Clinical cuttoff

Children/adolescents

Adults

25th percentile

Trajectories of changeBefore and after feedback (AHBA)

0

5

10

15

20

25

30

35

40

Inta

ke

Sessio

n 2

Sessio

n 3

Sessio

n 4

Sessio

n 5

Sessio

n 6

Sessio

n 7

Sessio

n 8

Sessio

n 9

Session

Ou

tco

me

Sco

re 75% percentile

Clinical cuttoff

Before feedback

After feedback

25th percentile

Norms and Benchmarks

• Regression formulas can be used to create norms for change.

• Outcomes can be “Benchmarked” by determining the difference between the expected change (using the regression formula) and the actual change

• By contributing data to common data repository, ORS users can assure that norms are continuously refined and updated.

References

1. Science Cartoons by Georg Meixner http://www.vias.org/science_cartoons/

2. Jacobson NS & Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. 1991;59:12-19.

3. Campbell, DT & Kenny, DA. A Primer on Regression Artifacts. The Guildford Press, 1999.

About the presenter

G.S. (Jeb) Brown is a licensed psychologist with a Ph.D. from Duke University. He served as the Executive Director of the Center for Family Development from 1982 to 19987. He then joined United Behavioral Systems (an United Health Care subsidiary) as the Executive Director for of Utah, a position he held for almost six years. In 1993 he accepted a position as the Corporate Clinical Director for Human Affairs International (HAI), at that time one of the largest managed behavioral healthcare companies in the country.

In 1998 he left HAI to found the Center for Clinical Informatics, a consulting firm specializing in helping large organizations implement outcomes management systems. Client organizations include PacifiCare Behavioral Health/ United Behavioral Health, Department of Mental Health for the District of Columbia, Accountable Behavioral Health Care Alliance, Resources for Living and assorted treatment programs and centers throughout the world.

Dr. Brown continues to work as a part time psychotherapist at behavioral health clinic in Salt Lake City, Utah. He does measure his outcomes.