cricket mitchell, phd senior associate, cimh tf-cbt program performance and outcome evaluation

Cricket Mitchell, PhDCricket Mitchell, PhD

Senior Associate, CIMHSenior Associate, CIMH

TF-CBT Program Performance TF-CBT Program Performance and Outcome Evaluationand Outcome Evaluation

TF-CBT Evaluation: TF-CBT Evaluation: What You Should Have What You Should Have • Guide to Evaluation and All Measures (PDF File)

– TFCBT Outcome Evaluation Guide and Forms-3– “Quick-Start” instructions for evaluation– Measures of symptom-specific and global youth

mental health functioning (PTSD-RI and YOQ)

• Database (Excel File)– TFCBT Data Entry Shell2– Holds all data for all children served– Submit to CIMH twice a year; keep entering new data

into same database (ongoing record)

2

Overview of Training Overview of Training • Standardized Measures Used in the TF-CBT

Evaluation– UCLA Post Traumatic Stress Reaction Index©

(PTSD-RI)• Description, Administration, Scoring, Clinical Utility

– Youth Outcome Questionnaires© (YOQ & YOQ-SR)

• Description, Administration, Scoring, Clinical Utility

• Instructions for TF-CBT Data Entry & Data Submissions

3

Overview of Training Overview of Training • Standardized Measures Used in the

TF-CBT Evaluation– UCLA Post Traumatic Stress Reaction Index©

(PTSD-RI)• Description, Administration, Scoring, Clinical Utility

– Youth Outcome Questionnaires© (YOQ & YOQ-SR)

• Description, Administration, Scoring, Clinical Utility

• Instructions for TF-CBT Data Submissions

4

Importance of Program Performance Importance of Program Performance and Outcome Evaluationand Outcome Evaluation

• Assessment is the beginning of developing a relationship with the child and parents– Demonstrates a desire to know what the child

is experiencing– By incorporating standardized measures of

functioning, the efficiency and thoroughness of assessment is enhanced

5

Importance of Program Performance Importance of Program Performance and Outcome Evaluationand Outcome Evaluation

• Using standardized measures of functioning…– Assists in initial clinical impressions– Provides valuable information to guide

treatment/interventions– Assesses sufficiency of treatment delivered– Demonstrates treatment-related improvements

in child functioning

6

7

UCLA Post-Traumatic Stress UCLA Post-Traumatic Stress Reaction IndexReaction Index©© (PTSD-RI) (PTSD-RI)

CIMH PTSD-RI TrainingCIMH PTSD-RI Training

• Content for today’s training courtesy of:– National Center for Child Traumatic Stress at

UCLA• 2004 article in Current Psychiatry Reports by Alan

Steinberg, Melissa Brymer, Kelly Decker and Robert Pynoos (included in LA PEI MAP Evaluation Guide and Forms PDF file)

– National Child Traumatic Stress Network• Video-taped training on the administration and scoring

of the PTSD-RI (Alan Steinberg, William Saltzman and Melissa Brymer)

– Personal communications with Laura Murray

8

PTSD-RI DescriptionPTSD-RI Description• Symptom-specific measure of functioning• Assesses the frequency of occurrence of

children’s post-traumatic stress reactions– Parent/caregiver report for children 3 and older– Self-report for children 7 and older

• Readability is age 12• Preferred method of administration is to read aloud to

informant

• Sensitive to clinical change over time• Valid and reliable

9

PTSD-RI DescriptionPTSD-RI Description

• Part I: 14 items– Assesses lifetime history

of exposure to trauma– Yes or No– If more than one trauma,

focus on event most currently bothersome

• Part II: 13 items– Assesses objective and

subjective features of the trauma exposure

– Maps on to DSM-IV Criteria A1 & A2

• Part III: 20 items– Assesses the frequency of

PTS symptoms during the past month

– Maps on to DSM-IV Criteria B, C & D

– 5-point Likert scale response options

• 0 = None (of the time)• 1 = Little (of the time)• 2 = Some (of the time)• 3 = Much (of the time)• 4 = Most (of the time)

10

Let’s take a look at the Let’s take a look at the PTSD-RI...PTSD-RI...

(pgs 6-10 in the TF-CBT (pgs 6-10 in the TF-CBT Outcome Evaluation Guide and Outcome Evaluation Guide and Forms pdf)Forms pdf)

11

PTSD-RI AdministrationPTSD-RI Administration• Readability is age 12

– Preferred method of administration is to read measure aloud to informant

• Ask parents and youth to respond to the questionnaires as honestly as possible– Informants can easily be influenced by the attitude of

the person administering the scale– Let them know that this questionnaire will help you, as

a clinician, better understand how the child is doing overall

• Ask parents and children to complete all items– “Don’t Know” responses are not scored

12

PTSD-RI AdministrationPTSD-RI Administration

• Part I & II: Lifetime history of exposure• Part III: “How much of the time during the past

month?”– For Part III, guide the respondent through the

Frequency Rating Scale prior to administration• Tear off Page 5 so they can see it while they respond• Explain each response option• Ensure understanding of each response option

– “Suppose I ask you how often in the past month you…» … had a headache?”» … did your homework?”» … had green hair?”

13

Let’s take a look at the Let’s take a look at the PTSD-RI Scoring Worksheet...PTSD-RI Scoring Worksheet...

(pg 27 in the TF-CBT Outcome (pg 27 in the TF-CBT Outcome Evaluation Guide and Forms pdf)Evaluation Guide and Forms pdf)

14

PTSD-RI ScoringPTSD-RI Scoring

• Each DSM-IV Criterion is established separately• DSM-IV Criterion A

– Exposure to trauma (Part 1)• At least 1 “Yes” on Q#s 1-13

– Criterion A1 (Part II)• To be met, >1 “Yes” on Q#s 15-21

– Criterion A2 (Part II)• To be met, >1 “Yes” on Q#s 22-26

– Criterion A• To be met, exposure to trauma and A1 and A2 met

15

PTSD-RI ScoringPTSD-RI Scoring• DSM-IV Criteria B, C, and D

– Based on Part III: Questions 1-20– Transfer each item’s response score onto the scoring

sheet next to the appropriate Question #– PTSD-RI items map directly on to DSM-IV criteria

• Except Q14 & Q20 which assess associated features

• Severity score for each Criterion is the sum of the items that map on to that Criterion

• Each Criterion is met (to assist in your diagnostic impressions) if a minimum number of symptoms are “present” (see slides 17-19)– Symptom Cutoff Score >3– A score of 3 or 4 (much or most of the time) indicates

that a symptom is “present”

16

PTSD-RI ScoringPTSD-RI ScoringDSM-IV Criterion B (re-experiencing)

DSM-IV Criteria• B1) recurrent and intrusive thoughts• B2) recurrent distressing dreams• B3) acting or feeling as if event recurring• B4) intense psychological distress at cues• B5) physiological reactivity to cues

PTSD-RI Items• Q#3*• Q#5• Q#6• Q#2• Q#18

17

Met if >1 of the 5 symptoms present* (*score 3 or 4)

PTSD-RI ScoringPTSD-RI ScoringDSM-IV Criterion C (avoidance)

DSM-IV Criteria• C1) avoids thoughts, feelings or talks• C2) avoids activities, places or people• C3) inability to recall important aspect• C4) decreased interest in activities• C5) feelings of detachment• C6) restricted range of affect• C7) sense of foreshortened future

PTSD-RI Items• Q#9• Q#17• Q#15• Q#7• Q#8• Q#10 or Q#11• Q#19

18

Met if >3 of the 7 symptoms present* (*score 3 or 4)

PTSD-RI ScoringPTSD-RI ScoringDSM-IV Criterion D (increased arousal)

DSM-IV Criteria• D1) difficulty falling or staying asleep• D2) irritability or outbursts of anger• D3) difficulty concentrating• D4) hypervigilance• D5) exaggerated startle response

PTSD-RI Items• Q#13• Q#4• Q#16• Q#1• Q#12

19

Met if >2 of the 5 symptoms present* (*score 3 or 4)

PTSD-RI ScoringPTSD-RI ScoringPTSD Severity Overall/Total Score• Based on Part III: Questions 1-20• Transfer each item’s response score onto the scoring

sheet next to the appropriate Question #– For one item score, transfer only the higher number of two

Question responses• Q#s 10 or 11• Note that the Parent Scoring Worksheet also states to select higher

score between Q#s 3 and 21.– Items 14 and 20 are omitted (Associated Features)

• A total of 17 items are summed to determine the PTSD Severity Overall/Total Score• Post-TF-CBT administration only includes Part III –

the frequency of symptom items

20

Clinical Utility of the Clinical Utility of the PTSD-RIPTSD-RI• PTSD Severity Overall/Total Score

– Possible scores range from 0 to 68– Clinical cutpoint is 38 or higher– Scores in the high 20’s and 30’s indicate sub-

clinical, yet significant levels of PTS reactions that are appropriate for intervention

21

Clinical Utility of the Clinical Utility of the PTSD-RIPTSD-RI• Asks about a broad range of traumatic

events– Primary reason for referral may not be only

history of trauma exposure– Structured questions can help elicit additional

information about exposure to traumatic events

– Often, children have not been asked directly about traumatic events they’ve experienced

22

Clinical Utility of the Clinical Utility of the PTSD-RIPTSD-RI• Although not a diagnostic tool, the PTSD-RI

can inform clinical impressions– Items map directly onto DSM-IV Criteria for

PTSD, 309.81

– Each item/question in Part III is labeled with subscript indicating the specific DSM-IV 309.81 criterion (letter and number) that it assesses

• e.g., 1D4 … 3B1 … 7C4

• AF = Associated Feature (i.e., guilt, avoidance)

23

Clinical Utility of the Clinical Utility of the PTSD-RIPTSD-RI• Informs clinician about specific post-

traumatic stress reactions that are most bothersome to this particular child– Helps prioritize symptoms for intervention– Guides specific techniques that will be used– Guides psycho-education

• Not all symptoms need to be normalized for each child presenting with PTSD or PTS reactions

24

Clinical Utility of the Clinical Utility of the PTSD-RIPTSD-RI• Research has shown that certain types of

treatment approaches are better for certain aspects of PTS symptoms– e.g.,

• avoidance responds best to in vivo types of exposure

• sleep disturbances would suggest the use of behavioral regimens and/or relaxation techniques

• significant rumination and self-blame would indicate the need for cognitive interventions

25

Clinical Utility of the Clinical Utility of the PTSD-RIPTSD-RI• Comparisons of pre/post scores reveal

areas of clinical improvement– e.g.,

• Does the child’s Overall/Total PTSD Severity Score decrease substantially?

• Does the child’s symptomotology improve in all domains of post-traumatic stress reactions?

26

27

Youth Outcome QuestionnairesYouth Outcome Questionnaires©© (YOQ & YOQ-SR)(YOQ & YOQ-SR)

CIMH YOQ & YOQ-SR TrainingCIMH YOQ & YOQ-SR Training• Information on the administration, scoring,

and clinical utility of the YOQ & YOQ-SR was obtained from each measure’s respective Administration and Scoring Manual published by OQ Measures, LLC

28

YOQ & YOQ-SR DescriptionYOQ & YOQ-SR Description• General measure of functioning

• Assesses the global mental health functioning of children– Parent/caregiver report for children ages 4-18– Self-report for adolescents 12-18

• Sensitive to clinical change in short periods of time

• Valid and reliable

29

YOQ & YOQ-SR DescriptionYOQ & YOQ-SR Description• Parallel versions of

the same measure• 64 items• 5-point Likert scale

response options*– Never or Almost Never– Rarely– Sometimes– Frequently– Almost Always or

Always*response values vary by item

• Six Scale Scores– Intrapersonal Distress (ID)– Somatic (S)– Interpersonal Relations (IR)– Social Problems (SP)– Behavioral Dysfunction (BD)– Critical Items (CI)

• Total Score

30

Let’s take a look at the Let’s take a look at the YOQ-SR...YOQ-SR...

(pgs 37-38 in the TF-CBT (pgs 37-38 in the TF-CBT Outcome Evaluation Guide and Outcome Evaluation Guide and Forms pdf)Forms pdf)

31

YOQ & YOQ-SR AdministrationYOQ & YOQ-SR Administration

• “… during the past 7 days.”• Ask parents and youth to fill out the

questionnaires as honestly as possible– Informants can easily be influenced by the

attitude of the person administering the scale– Let them know that this questionnaire will help

you, as a clinician, better understand how the child is doing overall

• Ask parents and youth to complete all items

32

YOQ & YOQ-SR ScoringYOQ & YOQ-SR Scoring• For each item, transfer the value corresponding to the

selected response into the box at the right-hand side of the page titled, ‘For Office Use Only’– Each item loads onto one of the six scales (e.g., ID, SP, IR)– Note that some items have negative response option values

• Sum the items in each scale on Side1– Add the numbers in all boxes under the heading ID and enter

that subtotal into the ID box at the bottom of the page– Repeat for each scale

• Sum the items in each scale on Side 2• Transfer the subtotals from Side 1 to Side 2• Sum the subtotals to determine Scale Scores• Sum the Scale Scores to determine Total Score

– Note that it is possible to have negative values for scores

33

YOQ & YOQ-SR ScoringYOQ & YOQ-SR Scoring

• Missing Data (items that are left blank)– If 5 or more items are missing, consider the

questionnaire invalid– Substitute a mean item response for the

missing item• Determine in which scale the missing item belongs• Add up the other items in that scale, and determine

their average• Substitute the average score for the missing

response

34

Clinical Utility of the Clinical Utility of the YOQ & YOQ-SRYOQ & YOQ-SR• Total Score

– Possible scores range from -16 to 240 – Clinical cutpoints

• 46 or higher on the YOQ• 47 or higher on the YOQ-SR

– Lower scores indicate more normative, non-clinical, aspects of general mental health functioning

• Elevations on certain scales indicate areas of specific distress for the child

35

Clinical Utility of the Clinical Utility of the YOQ & YOQ-SRYOQ & YOQ-SR• Intrapersonal Distress (ID) Scale

– Possible scores range from -4 to 68 – Clinical cutpoints

• 16 or higher on the YOQ• 17 or higher on the YOQ-SR

– Assesses the amount of emotional distress in the child, including anxiety, depression, fearfulness, hopelessness, and self-harm

– High scores indicate a considerable degree of intrapersonal distress in the child

36

Clinical Utility of the Clinical Utility of the YOQ & YOQ-SRYOQ & YOQ-SR• Somatic (S) Scale

– Possible scores range from 0 to 32– Clinical cutpoints

• 5 or higher on the YOQ• 6 or higher on the YOQ-SR

– Indicates change in somatic distress or physical complaints

– High scores indicate the parent/caregiver is aware of, or the youth is experiencing, a high number of somatic symptoms; while low scores indicate either absence or unawareness of them

37

Clinical Utility of the Clinical Utility of the YOQ & YOQ-SRYOQ & YOQ-SR• Interpersonal Relations (IR) Scale

– Possible scores range from -6 to 34 – Clinical cutpoints

• 4 or higher on the YOQ• 3 or higher on the YOQ-SR

– Assesses issues relevant to the child’s relationship with parents, other adults, and peers

– High scores indicate significant interpersonal difficulty; while low scores reflect a cooperative, pleasant interpersonal demeanor

38

Clinical Utility of the Clinical Utility of the YOQ & YOQ-SRYOQ & YOQ-SR• Social Problems (SP) Scale

– Possible scores range from -2 to 68 – Clinical cutpoints

• 3 or higher on the YOQ & YOQ-SR

– Assesses problems that are socially related including aggression and delinquency

– A feature of these items is that they are slow to change; whereas, content tapped by many of the other scales often changes over a period of time as a result of treatment intervention

39

Clinical Utility of the Clinical Utility of the YOQ & YOQ-SRYOQ & YOQ-SR• Behavioral Dysfunction (BD) Scale

– Possible scores range from -4 to 40 – Clinical cutpoints

• 12 or higher on the YOQ• 11 or higher on the YOQ-SR

– Assesses inattention, hyperactivity, impulsivity, concentration, ability to organize tasks, and ability to handle frustration

40

Clinical Utility of the Clinical Utility of the YOQ & YOQ-SRYOQ & YOQ-SR• Critical Items (CI) Scale

– Possible scores range from 0 to 36 – Clinical cutpoints

• 5 or higher on the YOQ• 6 or higher on the YOQ-SR

– Assesses areas such as paranoia, obsessive-compulsive behaviors, hallucination, delusions, suicide, mania, and eating disorders

41

Clinical Utility of the Clinical Utility of the YOQ & YOQ-SRYOQ & YOQ-SR• Assesses a variety of specific areas of

difficulty in youth mental health functioning• Assists in initial clinical impressions• Provides valuable information to guide

treatment/interventions• Comparisons of pre/post scores reveal

areas of clinical improvement as well as areas of potential unmet need

42

Summary of YOQ/YOQ-SR Score Summary of YOQ/YOQ-SR Score Ranges and Clinical CutpointsRanges and Clinical Cutpoints

43

YOQ/YOQ-SR Scale

Range of possible scores

Clinical Cutpoint for YOQ

Clinical Cutpoint for YOQ-SR

Intrapersonal Distress (ID) -4 to 68 16 17

Somatic (S) 0 to 32 5 6

Interpersonal Relations (IR) -6 to 34 4 3

Social Problems (SP) -2 to 30 3 3

Behavioral Dysfunction (BD) -4 to 40 12 11

Critical Items (CI) 0 to 36 5 6

Total Score -16 to 240 46 47

44

TF-CBT Data Entry & TF-CBT Data Entry & Data SubmissionsData Submissions

TF-CBT Data Entry Shell2 – TF-CBT Data Entry Shell2 – ‘Demos, etc’ Spreadsheet‘Demos, etc’ Spreadsheet

45

Client Information: Early Therapy Information: End of Therapy Information:

(DSM-IV

code) Date of Date of Date of Total # CompletedClient

ID DOB Gender EthnicityPrimary

Axis ITherapist

ID ReferralFirst

SessionLast

Sessionof

Sessions TF-CBT?

The last three fields are for end-of-therapy

information. Data should only be

entered when a child is no longer in TF-CBT.

TF-CBT Data Entry Tips – TF-CBT Data Entry Tips – ‘Demos, etc’ Spreadsheet ‘Demos, etc’ Spreadsheet (slide 1 of 2)(slide 1 of 2)

• Client ID automatically populates into the other spreadsheets (pre and post outcome data on the PTSD-RI and YOQ/YOQ-SR measures)– Use a unique identifier for each child (e.g., chart #)

• Pull-down menus are provided for all categorical data (gender, ethnicity, and Completed TF-CBT?)

• DSM-IV Axis I diagnosis should be entered as the numerical code– Enter only one DSM-IV numerical code– If more than one Axis I diagnosis applies, enter the

Primary Axis I

46

TF-CBT Data Entry Tips – TF-CBT Data Entry Tips – ‘Demos, etc’ Spreadsheet ‘Demos, etc’ Spreadsheet (slide 2 of 2)(slide 2 of 2)

• The last three fields (Date of Last Session, Total # of Sessions, and Completed TF-CBT?) are end-of-therapy fields only– They should be blank/empty for children currently

being seen– They should be filled in for children no longer in TF-

CBT• Completed TF-CBT?

– Select ‘yes’ if the child completed the intervention– Select ‘no’ if the child dropped out of TF-CBT prior to

completing the intervention (e.g., moved away, stopped coming)

47

TF-CBT Data Entry Excel File – TF-CBT Data Entry Excel File – Pre & Post PTSD-RI SpreadsheetsPre & Post PTSD-RI Spreadsheets

48

Pre-TF-CBT PARENT REPORT: CHILD REPORT:

PTSD-RI Total Total

Client ID Severity Score Severity Score

0    

0    

Post-TF-CBT PARENT REPORT: CHILD REPORT:

PTSD-RI Total Total

Client ID Severity Score Severity Score

0

0

TF-CBT Data Entry Tips – Pre TF-CBT Data Entry Tips – Pre and Post PTSD-RI Spreadsheetsand Post PTSD-RI Spreadsheets• There is a separate spreadsheet for the

Pre- PTSD-RI data and the Post- PTSD-RI data– Enter the PTSD Overall/Severity Score for

the Parent and Child informants into their respective fields

• Do not enter text into these fields

• If data are missing, leave the fields blank

49

TF-CBT Data Entry Excel File – TF-CBT Data Entry Excel File – Pre & Post YOQ SpreadsheetsPre & Post YOQ Spreadsheets

50

Pre- TF-CBT

PARENT/CAREGIVER REPORT: 

YOUTH SELF-REPORT: 

YOQ (Scale Scores and Total Score) YOQ-SR (Scale Scores and Total Score)

Client

ID ID S IR SP BD CI Total ID S IR SP BD CI Total

0                           

0                           

Post- TF-CBT

PARENT/CAREGIVER REPORT: 

YOUTH SELF-REPORT: 

YOQ (Scale Scores and Total Score)

YOQ-SR (Scale Scores and Total Score)

Client ID ID S IR SP BD CI Total ID S IR SP BD CI Total0                           0                           

TF-CBT Data Entry Tips – Pre TF-CBT Data Entry Tips – Pre and Post YOQ Spreadsheetsand Post YOQ Spreadsheets• There is a separate spreadsheet for the

Pre- YOQ & YOQ-SR data and the Post- YOQ & YOQ-SR data– Enter the Scale Scores and the Total Score

for the Parent and Adolescent informants into their respective fields

• Do not enter text into these fields

• If data are missing, leave the fields blank

51

TF-CBT Data SubmissionsTF-CBT Data Submissions• Data submissions occur twice a year

– The end of February (reflecting children served thru January)– The end of August (reflecting children served through July)– Note that this is the anticipated schedule; actual data

submission dates may vary slightly• An email notice is sent approximately one month in

advance of each data submission deadline• After data are submitted, sites continue to enter new data

into the same database (always reflects an ongoing, historical record of children served)

• Every effort is made to distribute reports within two months of data submission– Aggregate and site-specific reports

52

53

QuestionsQuestions

54

The EndThe End

Contact InformationContact Information•Cricket Mitchell, PhD

•Email: cmitchell@cimh.org•Cell phone: 858-220-6355