Evaluating the Effectiveness of Day Treatment Programs from Multiple Perspectives

Prepared by:

Sandra Cunning, Ph.D., Former Director of Research

The George Hull Centre for Children and Families

Diane Bartlett, M.A., Research Manager

The George Hull Centre for Children and Families

Table of Contents

Area Page Executive Summary 3 Background 4 Project Summary 4 Method 5 Results 8 Conclusions, Recommendations & Next Steps 15 Knowledge Exchange Plan 17 Appendix A: Budget 20

2

Executive Summary The purpose of the current study was to complete an evaluation of two day treatment programs delivered by the George Hull Centre for Children and Families: The School Program and the Clear Directions Program. The School Program serves youth who exhibit acute or chronic behavioral, emotional, learning, or developmental difficulties. The purpose of the program is academic achievement and the development of cognitive, social and linguistic skills that are necessary for successful adjustment at home, school and in the community. Clear Directions is a collaborative program provided by The George Hull Centre for Children and Families, Breakaway and The Toronto District School Board. The program is designed to assist youth, 18 years and under, who are struggling with serious substance abuse, family difficulties, and mental health issues and who live in the Greater Toronto Area. The evaluation of these two programs was revised and implemented in the 2001-02 academic year and continued into the 2005-06 academic year. Students and guardians consenting to participate in the program evaluation process completed a series of questionnaires including the Child Behavior Checklist, Youth Self-Report, Self-Esteem Index, Substance Abuse Subtle Screening Inventory, and Beck Depression Inventory-II at entry and exit to the program. BCFPI and CAFAS data also were collected according to Ministry guidelines. Results of the study indicate that BCFPI data speak to the heightened and specific needs of the youth and families accessing the School and Clear Directions programs including the fact that over 80% of School and Clear Directions referrals were in the clinical range for Child Functioning and that over 80% of Clear Directions referrals also were in the clinical range for Family Activities and Family Comfort. CAFAS results demonstrated that School Program and Clear Directions cases showed significant improvements between entry and exit to services in overall functioning and in key areas such as School, Community, Managing Mood and Substance Use. Over 75% of all School and Clear Directions students showed some degree of improvement in functioning with over 61% of School and 52% of Clear Directions students improving 30 points or more between entry and exit. In addition, students in the School program received approximately 3 credits on average while Clear Directions students received 3.5 credits. Less clear were outcomes associated with additional measures. While CAFAS results clearly indicated improvement in key areas, other indicators (e.g., BECK-II or SASSI) did not detect such shifts possibly reflecting factors such as lack of sensitivity of the measures, the small size of the samples, or the characteristics of the youth themselves. Youth in these programs present with problems in attention, school, and mood. As well, these students may have significant learning disabilities making lengthy paper-pencil tasks daunting. Further to this, challenges with the flow of information between programs and the evaluation process resulted in a loss of clients eligible for the evaluation process. The results of the study indicate that youth are making significant improvements between entry and exit of the School and Clear Directions programs. However, the results also speak to the need to review the evaluation process. The requirement to complete additional, and in many cases lengthy forms, requires significant resources on the part of youth, their families, staff, and researchers with questionable results. The programs need to review whether or not the collection of additional data above the CAFAS, BCFPI and school performance indicators is worth the additional resource demands. Recommended next steps include dissemination of results and ongoing data analyses.

3

Background The purpose of the current study was to complete an evaluation of two day treatment programs delivered by the George Hull Centre for Children and Families: The School Program and the Clear Directions Program. School Program The School Program offers academic programming to 24 youth through three Section 20 classrooms. The youth who attend the school exhibit acute or chronic Behavioral, emotional, learning or developmental difficulties. The purpose of the program is academic achievement and the development of cognitive, social and linguistic skills that are necessary for successful adjustment at home, school and in the community. All youth who attend the School Program have access to psychiatric assessment. The consultant psychiatrist provides individual assessments, cognitive behavior therapy groups, program and team consultation and staff development seminars.

Clear Directions Clear Directions is a collaborative program provided by The George Hull Centre for Children and Families, Breakaway and The Toronto District School Board. The program is designed to assist youth, 18 years and under, who are struggling with serious substance abuse, family difficulties, and mental health issues and who live in the Greater Toronto Area. A continuum of care encompassing outreach, individual, family and group therapy, day treatment, psychiatric consultation, residential care, case management and follow up is offered. Due to the nature of ‘addiction’ problems and the high rate of ‘relapse’, the follow up and ongoing community care component of the program is significant. It is offered to all youth and families for as long as they find it useful. They are assured that they may return to the program at any time and there will be no wait for service.

Project Summary The evaluation of the School and Clear Directions Programs began in the 2001-02 academic year. To evaluate how the programs were meeting their goals, a literature review and meetings with staff were conducted to select appropriate measures. These measures include: The Child Behavior Checklist, Teacher Report Form, Youth Self Report, and the Self-Esteem Index. Additionally, to assess issues around depression and substance use, the Beck Depression Inventory II and the Substance Abuse Subtle Screening Inventory also have been collected in the Clear Directions Program. Finally, BCFPI and CAFAS data was collected for clients in these programs as well as academic credits achieved. The design for the study is pre-post with data collected from multiple perspectives: youth, parent, teacher, and clinician. Although an evaluation process had been put in place, there was limited capacity to score and analyze data. Preliminary data analyses were conducted in the first year of data collection with some support from graduate students in subsequent years. The purpose of the proposal was to hire a data analyst to: complete data scoring, verify old data transferred to new software, complete data entry, clean data, merge BCFPI and CAFAS data with other measures, develop SPSS to produce automatic and specialized reports, and to analyze and report on data to various stakeholders including families and staff. Outcomes of the process were predicted to add to the quality assurance and program evaluation aspects of the Centre.

4

Method Participants A total of 96 individual students were enrolled in the School Program between 2001-02 and 2005-2006 while 36 students were enrolled in the Clear Directions Program. Sixty-four programs were delivered for Clear Directions students while 139 programs were delivered to School students in those same years. Cases with BCFPI and complete pre-post data are included in Table 1. Fifty-two School students had matching BCFPI and CAFAS while the same was true for 20 Clear Directions Students. The average age of School and Clear Directions students at admission was 14.5 years and 15.9 years respectively.

Table 1 Cases with complete data for School and Clear Directions students.

Measures n School 96 BCFPI 76 CAFAS 56 SEI 29 CBCL 3 TRF 18 YSR 27 Clear Directions 36 BCFPI 29 CAFAS 27 SEI 25 CBCL 5 TRF 30 YSR 25 BDI 24 SASSI 25

Materials A review of relevant literature pertaining to the evaluation of day treatment programs suggested the need to capture multiple perspectives in the evaluation process (Baenen, Parris Stephens, & Glenwick, 1986; Barkley, Shelton, Crosswait, Moorehouse, Barrett, Jenkins, & Metvia, 2000; Blackman, Pitcher, Rauch, 1986; Grizenko & Papineau, 1992; Kotsopoulos, Walker, Beggs, & Jones, 1996; Milin, Coupland, Walker, & Fisher-Bloom, 2000; Rey, Enshire, Wever, & Apollonov, 1998; Sayegh & Grizenko, 1991). This, in addition to existing processes, led to selection of the Child Behavior Checklist (CBCL), Teacher Report Form (TRF), Youth Self-Report (YSR), and Self-Esteem Index (SEI) as measures for both the School and Clear Directions Programs. Additionally, the Substance Abuse Subtle Screen Inventory and Beck Depression Inventory-II (BDI-II) were chosen for the Clear Directions Program. Finally, the BCFPI and CAFAS were collected for all clients (where possible) across the Centre according to Ministry guidelines. CAFAS & BCFPI. The BCFPI is a standard screening questionnaire completed at intake based on caregiver/guardian information. Teacher and adolescent forms also exist. The BCFPI takes approximately 30 minutes to complete and asks the informant to provide demographic information, information about child and family functioning, barriers to service, and information about abuse. Areas addressed in child functioning include: Regulation of Attention and Impulsivity (RAIA), Conduct (CD), Cooperativeness (CO), Separation (SP), Managing Anxiety (MA), Managing Mood

5

(MM), Self-Harm (SH), Social Participation (SOC), Quality of Relationships (QREL), and School Functioning (SCHOOL). RAIA, CD and CO combine to provide an index of Externalizing Behaviors (EX) while SP, MA, MM, SH are combined for Internalizing Behaviors (IN). All scales comprise a measure of Total Mental Health (TMH). Global Child Functioning (CHF) is based on SOC, QREL, and SCHOOL Subscales. Family Comfort (FCF) and Family Activities (FACT) scales combine for Global Family Functioning (GFS). All scale scores are reported as T-scores (Mean = 50, SD = 10). CAFAS is administered as close as possible to the beginning of outpatient services and again at exit or annually. The CAFAS is a clinician-rated tool that assesses child functioning in 8 domains including: School/Work, Home, Community, Behavior Towards Others, Managing Mood, Self-Harm, Substance Use, and Thinking. Family Material Support and Social Support are assessed in two additional caregiver subscales. Clinicians select the behavioral description in each domain that reflects the most severe behavior in the month prior to the assessment. Items are grouped in levels of functionality that range from 30 (severe impairment) to 0 (minimal/no impairment). All child/youth functioning subscales are combined to create a total score. CBCL, TRF, YSR. A review of the literature pertaining to day treatment programs reveals the use of Achenbach measures to assess a youth behavioral and emotional problems from the perspective of the teacher (TRF), youth (YSR), and parent (CBCL) (Barkley et al., 2000; Kotsopoulos et al., 1996; Milin et al., 2000; Rey et al., 1998). The CBCL, TRF, YSR follow a similar format. Guardian, youth, and/or teacher are asked to rate the youth in particular areas of functioning (e.g., activities). They are also asked to rate the degree to which 112 problem items are true of the youth (e.g., Not True, Somewhat or Sometime True, or Very True or Often True). Eight scale scores are generated that include: Withdrawn/Depressed, Somatic Complaints, Anxious/Depressed, Social Problems, Thought Problems, Attention Problems, Delinquent Behavior, and Aggressive Behavior. Internalizing, Externalizing and Total Problem Scores are also generated. All scale scores are reported at T-scores (Mean = 50, SD = 10. The Achenbach series is well-normed with extensive evaluation of reliability and validity. SEI. Following a review of various self-esteem and self-concept measures (Chan, 1997; Harter, 1982; Peterson & Austin, n.d.; Piers & Harris, 1969; Miller, 2000), the SEI was chosen as being the most appropriate for the youth enrolled in the School and Clear Directions programs and the instrument that best reflected the goals of the programs. The SEI (Brown, & Alexander, 1991) consists of 80 items that assess self-esteem in the areas of Familial Acceptance (FA), Academic Competence (AC), Peer Popularity (PP) and Personal Security (PS). The FA scale assesses the degree to which a youth “perceive and value themselves as members of their families” (Brooke, 1996, p. 28) while the PA scale taps into the individual’s perception of themselves within a school setting or academic endeavors. How well the youth sees themselves as interacting with peers is assessed the PP scale while the PS scale measures general well-being. The SEI is a paper-pencil test that takes approximately 30 minutes to complete and may be administered either individually or in groups. Items are scored on a 4-point, Likert-type scale based on level of agreement with statements (i.e., “Always true, Usually true, Usually false, Always false). Scale scores are converted to standard scores (Mean = 10, SD = 3) and an overall Self-Esteem Quotient (Mean = 100, SD = 15) with norms based on a sample of 2,455 participants ranging in age from 8-0 to 18-11 across the United States. In terms of psychometric properties, developers detail the process involved in content validation of items and report Cronbach Alphas for scales ranging

6

from .75 to .94 (Brown & Alexander, 1991). In terms of construct validity the scale scores and overall SEQ on the SEI show low (.10) to strong correlations with other measures of self-esteem including the Teach Evaluation of Student Self-Esteem, Piers-Harris Children’s Self-Concept Scale-Revised, Self-Esteem Inventories-School Form, and Index of Personality Characteristics. Factor analyses (varimax rotation) indicate a general factor accounting for 87% of the variance with four factors corresponding to FA (50%), AC (16%), PP (11%), and PS (10%). Brooke (1996) and King and Daniel (1996) however do raise concerns about the lack of information regarding test-rest reliability. SASSI. The SASSI is a screening measure designed to identify adolescents (12 through 18 years) likely to have a substance abuse disorder. The SASSI is comprised of two sections. In the first section youth are asked to read 72 statements and identify if the statements are true or false. These statements comprise 10 scales that assess risk for substance use in the family (FRISK), attitudes about substances and alcohol (ATT), substance misuse symptoms (SYM), five subtle scales unrelated to substance use (OAT, SAT, DEF, SAM, COR), a validity scale (VAL) and a subscale differentiating substance abuse and substance dependence (SCS). The second section of the SASSI includes 12 items identifying alcohol use (FVA) and 16 items identifying use of other substances (FVOD). In this section youth are required to indicate how frequently (i.e., Never, Once or Twice, Several Times, Repeatedly) they engage in various activities related to substance use. The measure was selected for use by the program prior to the onset of the current evaluation based on training by a research associate. BDI-II. The BDI-II is an instrument used to assess level of depressive symptoms in adults and adolescents that correspond to depressive disorders listed on the DSM-IV (Beck, Steer, & Brown, 1996). The BDI-II contains 21 items scored on a 4-point scale. Each item consists of a series four statements. Participants are asked to select from the statements contained in each item the one that best describes the way the have been feeling for the past two weeks. In terms of reliability and validity, Cronbach alphas were .92 and .93 for outpatient and student samples respectively. One-week test re-test reliability was .93 for a sample of 26 outpatients. Items were selected to represent DSM-IV criteria. BDI-II is significantly correlated with BDI-IA (.84) as well as with the Beck Hopelessness Scale (.68), the Scale for Suicidal Ideation (.37), the Beck Anxiety Inventory (.60), and the Hamilton Psychiatric Rating Scale (.71). Procedures At entry to the School and Clear Directions Programs, students and guardians were provided with consent forms to participate in the program evaluation process. Consent forms were reviewed and signed by interested guardians and youth. For those consenting to participate, guardians were asked to complete the CBCL at admission. Approximately 2 weeks after beginning the School or Clear Directions Programs, youth met with the Centre research staff to complete pre-test forms (i.e., School – SEI, YSR; Clear Directions – SEI, YSR, BDI, SASSI). At approximately 3 weeks into the program, Child and Youth Workers assigned to students completed the TRF. All forms were completed again at exit to the program. CBLC forms were sent home to parents with self-stamped return envelopes.

7

Results BCFPI A total of 1549 valid BCFPI’s were conducted between April 2002 and March 2007 across the Centre. Numbers of cases with complete data for all BCFPI subscales are found in Table 2. The percentage of all School Program, and Clear Directions referrals scoring within the clinical range on BCFPI subscales are shown in Figure 1. Table 2. BCFPI subscale scores for Total (N = 1459), School (n = 77), and Clear Directions (n = 30) referrals between April 2002 and March 2007.

BCFPI Subscale

All

School

CD

N X SD n X SD n X SD Regulation of Attention/Impulsivity 1516 63.91 11.96 76 66.68 12.79 29 69.95 9.87 Cooperativeness 1513 65.70 12.78 76 68.98 11.43 29 65.67 10.61 Conduct 1507 60.82 18.61 76 67.51 22.52 29 69.76 17.43 Externalizing 1506 66.81 12.37 76 70.90 12.97 29 71.30 10.05 Separation 1494 56.89 13.49 75 58.22 14.36 28 57.67 12.49 Managing Anxiety 1494 58.26 13.36 74 59.26 13.55 28 63.22 11.66 Managing Mood 1500 64.01 16.71 72 70.18 16.77 29 76.04 15.71 Self-Harm* 831 74.60 16.43 52 80.07 16.34 23 81.78 15.22 Internalizing 1481 62.10 13.56 72 65.38 13.96 28 69.40 12.77 Total Mental Health 1486 66.40 11.42 72 70.30 12.00 28 72.83 10.75 Social Participation 1445 69.89 17.51 73 77.87 17.51 26 78.49 17.31 Quality of Relationships 1437 63.24 12.76 72 69.05 14.35 25 67.82 11.17 School Performance 1439 65.52 15.38 73 76.00 13.40 25 73.23 12.54 Child Functioning 1485 69.81 14.00 76 79.09 12.90 18 79.09 11.24 Family Activity 1211 69.06 26.67 61 81.35 32.90 19 82.85 24.75 Family Comfort 1363 70.79 13.67 71 74.31 14.50 23 75.98 11.45 Global Family 1381 73.73 18.99 70 81.26 21.18 22 82.34 16.19 Age 1549 12.21 3.16 77 14.63 1.52 30 15.73 1.14

* Self-harm questions are asked only when Managing Mood subscale is within the clinical range (t = 70).

8

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

RAI A C O CD EX S P M A M M S H I N TM H S OC QREL S C H CHF FA CT FCF GFS

BCFPI Subscales

% o

f Cas

es All

School

CD

Figure 1. Percentage of all (N = 1549), School (n = 77), and Clear Directions (n = 30) referrals with in the clinical range by BCFPI subscale.

CAFAS The total sample included 315 cases with complete pre-post CAFAS data. Overall, 73% cases showed some level of improvement (i.e., decrease of a minimum of 10 pts. in Total score) with 47% of cases showing clinically significant improvement of 30 points or more. Average amount of change from pre- to post-test was 28.16 (SD = 43.04) [t (1, 314) = 13.72, p < .01]. Pre-post scores are shown in Table 3. A 8 (Subscale) x 2 (Time) MANOVA revealed significant main effects for Subscales [F (7, 308) = 178.32, p<.01] and Time [F (1, 314) = 256.25, p<.01], and a Subscales x Time [F (7, 308) = 37.36, p<.01] interaction. Paired t-tests revealed significant decreases between pre-test and post-test for School, Home, Community, Behavior, Mood, Substance, and Thinking subscales. Non-parametric Wilcoxon tests confirmed parametric results. In comparison to the total sample, 56 School cases had complete CAFAS pre-post data. The average amount of improvement in Total scores was 34.82 (SD = 38.99) [t (1, 55) = 6.68, p < .01]. Overall, 76.8 % of cases demonstrated some level of improvement between entry and exit CAFAS ratings. A total of 61% of School students improved 30 points or more from T1 (Entry) to T14 (Exit) as seen in Table 3. A 8 (Subscale) x 2 (Time) MANOVA revealed significant main effects for Subscales [F (7, 49) = 51.90, p<.01] and Time [F (1, 55) = 44.67, p<.01], and a significant Subscales x Time [F (7, 49) = 8.81, p<.01] interaction. Paired t-tests confirmed significant pre-post differences on all subscales, except self-harm, in the direction of reduced scores at post-test. Non-parametric Wilcoxon tests confirmed parametric results.

9

10

Finally, 77.8% of Clear Directions (n = 27) cases showed some level of improvement between Entry and Exit CAFAS with an average decrease of 42.22 (SD = 60.40) in Total scores [t (1, 26) = 3.62, p < .01]. Fifty-two percent of cases improved over 30 points. See Table 3. A 8 (Subscale) x 2 (Time) MANOVA also was conducted for pre-post CAFAS data for the Clear Directions Program Overall, MANOVA results revealed significant main effects for Subscale [F (7, 20) = 57.33, p<.01] and Time [F (1, 26) = 13.19, p<.01], while a Subscales x Time interaction approached significance [F (7, 20) = 2.34, p<.06]. Again, paired t-tests indicated significant differences pre-post with non-parametric tests confirming results. Table 3. CAFAS subscale and total scores for all, School Program, and Clear Directions cases with complete pre-post data.

All Cases (n = 315) X SD X SD School 16.25 11.84 9.65** 11.07 Home 15.81 10.51 9.65** 9.69 Community 4.73 9.18 2.13** 6.20 Behavior 12.03 8.61 7.87** 8.04 Mood 15.59 8.25 10.38** 8.05 Self-Harm 3.52 7.86 1.11 4.55 Substance 4.83 9.42 4.67** 8.67 Thinking 2.00 5.12 1.14** 4.14 Total 74.76 41.40 46.60** 37.79

School Program (n = 56) School 23.93 8.67 15.54** 12.05 Home 20.18 11.19 10.89** 10.14 Community 7.86 11.71 3.21** 8.34 Behavior 15.00 7.14 10.18** 8.20 Mood 18.75 8.75 13.57** 7.73 Self-Harm 5.54 9.13 2.32** 6.87 Substance 5.18 9.14 7.14 9.67 Thinking 2.50 5.80 1.25* 3.84 Total 98.93 36.76 64.11** 40.22

Clear Directions (n = 27) School 25.19 8.93 12.59** 14.03 Home 20.00 12.09 15.19 13.41 Community 13.33 11.77 6.67* 10.00 Behavior 12.59 10.95 10.00 9.61 Mood 17.78 8.47 12.22* 8.47 Self-Harm 5.19 10.14 2.22 5.06 Substance 25.93 5.72 21.11* 7.51 Thinking 4.07 7.47 1.85 4.83 Total 124.07 43.88 81.85** 50.54

* p < .05, ** p < .01

BCFPI & CAFAS Correlations between BCFPI and entry CAFAS subscales for all cases indicated significant positive correlations. The CAFAS School subscale was correlated with several BCFPI (Table 4) subscales with the strongest correlations occurring with Quality of Relationships and School subscales. The Home subscale was most strongly correlated with Cooperativeness and Conduct while Community was also correlated with Conduct. Behavior Toward Others was most strongly correlated with Quality of Relationships and Family Activities. Mood and Self-Harm subscales on the CAFAS were most strongly associated with the Managing Moods and Self-Harm subscales on the BCFPI. Substances Use also was correlated with Managing Mood. Thinking was not correlated with any BCFPI subscales.

Table 4. Pearson correlations between BCFPI and T1 (Entry) CAFAS subscales. RAIA CO CD Ex SP MA MM SH IN TMH SOC QREL SCH CHF FACT FCF GFS School .16* .11 .18** .19** -.10 -.06 .21** .12 .03 .13 .11 .33** .40** .34** .19* .15* .17* Home .27** .34** .37** .40** -.11 -.08 .21** .21* .06 .24** .09 .31** .20** .25** .30** .19** .27** Community .11 .07 .26** .16* -.08 -.02 .05 -.03 -.12 .08 -.02 .17* .24** .15* .15 .09 .17 Behavior .19** .20** .21** .25** -.02 .05 .21** .14 .10 .22** .10 .32** .26** .28** .30** .12 .26** Mood -.01 .01 -.06 -.03 .06 .02 .29** .29** .16* .10 .08 .00 .16 .10 -.11 -.12 -.10 Self-Harm -.04 -.03 .05 -.02 -.03 -.04 .28** .26** .10 .06 .13 .10 .03 .12 .05 .01 .04 Substance .06 .02 .16* .08 -.07 -.01 .23** .15 .07 .10 .11 .06 .20** .20** .10 .17* .12 Thinking .06 -.05 .04 .02 .04 .05 .10 .04 .08 .07 .10 -.08 .01 .04 -.10 .03 .09 Total .18** .16* .28** .25** -.08 -.03 .33** .24** .10 .22** .14 .29** .34** .34** .22** .15* .20**

* p < .05, ** p < .01 (n = 552 to 186)

SEI A 4 (Subscale) x 2 (Time) MANOVA of SEI subscales for the Clear Directions Program revealed significant main effects for Time [F (1, 24) = 10.77, p < .01] and Subscale [F (3, 22) = 14.14, p < .01]. No significant interaction was found. The same analysis for the School Program resulted in a significant main effect for Subscale [F (3, 26) = 2.99, p < .05]. No other effects were found. Follow-up univariate tests revealed a significant increase in Academic Competence between pre- and post-test [t (1, 24) = -3.09, p < .01] for the Clear Directions Program. An increase in Familial Acceptance approached significance [t (1, 24) = -1.84, p < .08]. Overall, a significant increase was found for SEQ for the Clear Directions Program [t (1, 24) = -4.00, p < .01] and approached significance for the School Program [t (1, 28) = -1.86, p < .08]. Means and standard deviations are presented in Table 5. Table 5. SEI subscale and total scores for School and Clear Directions programs with complete pre-post test data.

School (n = 29)

Clear Directions (n = 25)

SEI Subscales Pre Post Pre Post X SD X SD X SD X SD Familial Acceptance 7.90 3.30 8.62 2.82 7.24 2.83 8.00* 2.60 Academic Competence 8.52 3.94 9.59 4.05 6.60 3.20 7.84** 2.98 Peer Popularity 9.45 3.62 9.97 2.81 10.32 2.59 10.88 2.30 Personal Security 9.24 3.72 9.83 3.65 10.64 2.86 10.92 2.50 SEQ 90.38 20.03 95.72 14.61 90.60 13.74 96.00** 12.33

YSR, TRF & CBCL An 8 (Subscale) x 2 (Time) MANOVA of TRF subscales for the School Program revealed significant main effects for Time [F (1, 17) = 7.60, p < .05] and Subscale [F (7, 11) = 4.85, p < .01]. The Subscale by Time interaction was not significant [F (7, 11) = 0.69, n.s.]. Follow-up univariate tests revealed significant increases in T-scores between pre-post for Social Problems [t (1, 18) = -2.27, p < .05], Delinquent Behavior [t (1, 18) = -2.33, p < .05], and Aggressive Behavior [t (1, 18) = -2.96, p < .01]. Increases in T-Scores between pre-test and post test for Attention Problems approached significance [t (1, 24) = -2.03, p < .06]. Externalizing [t (1, 15) = -3.23, p < .01] and Total Problems [t (1, 15) = -2.14, p < .05] T-scores on the TRF also increased significantly between pre- and post-test. Means and standard deviations are presented in Table 6. An 8 (Subscale) x 2 (Time) MANOVA of TRF subscales was also conducted for Clear Directions and revealed significant main effects for Time [F (1, 32) = 28.86, p < .01] and Subscale [F (7, 26) = 44.69, p < .01] and a significant Subscale x Time interaction was found [F (7, 26) = 2.47, p < .05]. Follow-up univariate tests revealed significant increases in T-scores between pre-post for Withdrawn/Depressed [t (1, 32) = -2.20, p < .05], Somatic Complaints [t (1, 32) = -2.45, p < .05], Anxious/Depressed [t (1, 32) = -2.27, p < .05], Social Problems [t (1, 32) = -5.01, p < .01], Attention Problems [t (1, 32) = -2.55, p < .05], Delinquent Behavior [t (1, 32) = -3.86, p < .01], and

Aggressive Behavior [t (1, 32) = -3.85, p < .01]. Internalizing [t (1, 29) = -3.02, p < .01], Externalizing [t (1, 29) = -4.52, p < .01, and Total Problems [t (1, 29) = -4.40, p < .01] T-scores on the TRF also increased significantly between pre- and post-test. Means and standard deviations are presented in Table 6.

Table 6. TRF and YSR subscales and total scores for School and Clear Directions Programs.

TRF

School Program (n = 18) Clear Directions (n = 33) Pre Post Pre Post X

SD X SD X SD X SD

Withdrawal 64.56 8.70 64.50 9.15 60.97 5.87 63.00* 6.05 Somatic Complaints 59.06 6.92 62.17 9.72 58.21 6.21 60.79* 7.15 Anxiety/Depressed 61.67 7.59 65.00 10.81 61.48 8.76 64.12 9.68 Social Problems 60.39 7.79 63.89* 9.13 56.15 7.03 60.88** 6.55 Thought Problems 56.11 7.95 58.50 9.13 55.70 9.08 57.67 8.29 Attention Problems 56.56 4.50 58.56 8.18 56.12 6.74 58.45** 7.95 Delinquent Behavior 61.22 7.46 65.94* 10.18 69.82 7.90 75.39** 8.07 Aggressive Behavior 59.22 4.72 63.11** 7.28 55.73 5.74 60.03* 8.36 Internalizing 62.38 6.11 65.38 8.72 61.73 7.26 64.50** 6.65 Externalizing 59.75 5.52 63.81** 7.55 60.63 5.97 65.00** 6.70 Total 60.75 5.17 63.81* 7.83 60.07 6.75 64.20** 7.30

YSR School Program (n = 27) Clear Directions (n = 25) Pre Post Pre Post X

SD X SD X SD X SD

Withdrawal 61.11 9.81 58.89 9.98 59.12 7.79 55.48** 7.72 Somatic Complaints 57.70 9.51 57.22 10.61 62.20 11.26 60.88 11.56 Anxiety/Depressed 58.74 12.13 56.81 10.66 57.80 8.89 57.00 7.31 Social Problems 55.70 8.46 56.81 8.81 57.44 9.38 55.44 8.55 Thought Problems 57.48 10.48 55.26 7.04 58.16 10.49 55.96* 8.38 Attention Problems 57.26 11.05 56.78 8.57 63.12 13.44 60.80 11.14 Delinquent Behavior 62.56 9.56 59.22 8.46 74.44 8.94 72.44 5.92 Aggressive Behavior 56.52 8.19 56.37 7.94 61.28 9.07 59.48 7.62 Internalizing 56.78 14.27 53.15 15.74 59.32 9.77 56.12* 10.22 Externalizing 57.37 11.14 54.00 13.04 67.40 9.72 65.16* 8.35 Total 57.48 13.11 53.81 14.97 63.20 10.82 60.72* 9.75

* p < .05, ** p < .01

13

An 8 (Subscale) x 2 (Time) MANOVA of YSR subscales for the School Program revealed no significant main effects for Time [F (1, 26) = 0.96, n.s.] or Subscale [F (7, 20) = 1.65, n.s.]. A significant Subscale x Time interaction was found [F (7, 20) = 2.54, p < .05]. Follow-up univariate tests revealed a significant decrease in T-scores between pre-post Delinquent Behavior [t (1, 18) = -2.51, p < .05]. No significant differences were found for Internalizing [t (1, 26) = 1.27, n.s.], Externalizing [t (1, 26) = 1.47, n.s.], or Total Problems [t (1, 26) = 1.28, n.s.] between pre- and post-test. Means and standard deviations are presented in Table 6. An 8 (Subscale) x 2 (Time) MANOVA of YSR subscales also was conducted for Clear Directions and revealed significant main effects for Time [F (1, 24) = 7.00, p < .05] and Subscale [F (7, 18) = 22.80 p < .01]. The Subscale x Time interaction was not significant [F (7, 18) = 0.51, n.s.]. Follow-up univariate tests revealed significant decreases in T-scores between pre-post for Withdrawn/Depressed [t (1, 24) = 2.91, p < .01] and the difference between Thought Problems approached significance [t (1, 24) = 1.85, p < .07]. Internalizing [t (1, 24) = 2.29, p < .05] and Total Problems [t (1, 24) = 2.22, p < .05] T-scores on the YSR decreased significantly between pre- and post-test. Decreases in Externalizing scores approached significant [t (1, 24) = 1.83, p < .08]. Means and standard deviations are presented in Table 6. BECK & SASSI The BDI-II and SASSI were administered to only Clear Directions students. A total of 25 students had complete SASSI pre-post data while the same was true for 24 students for the BDI-II. Paired t-tests for BDI-II [t (1, 23) = 1.61, n.s.] and SASSI Alcohol [t (1, 24) = -0.21, n.s.] and Other Drug Use [t (1, 24) = 0.11, n.s.] showed no significant difference between pre-post. Means and Standard Deviations are presented in Table 7.

Table 7. BDI-II (n = 24) and SASSI (n = 25) total scores for Clear Directions Program with complete pre-post test data.

Pre Post X SD X SD SASSI - Alcohol 10.52 6.68 10.36 7.449 SASSI – Other Drugs 25.12 7.11 25.16 9.79 BDI-II 19.63 13.32 17.00 12.14

School Performance Based on available data, the average number of credits achieved was 2.7 and 3.6 for School and Clear Directions clients respectively. Fifty percent of school students achieved 2 credits or more while 50% of Clear Directions students earned 3 credits or more by the end of their programs.

14

Conclusions, Recommendations & Next Steps Outputs The most significant outcome of the current study was the capacity to score forms, collate data, and develop databases and syntax that can be used for automated reporting and more extensive data analyses. Over 250 protocols were scored and entered into an SPSS database for School and Clear Directions students that contained 257 variables. The study also resulted in 1549 and 555 lines of data verified for BCFPI and CAFAS data, respectively, with combined BCFPI and CAFAS data for 389 cases. Program service data also was combined with BCFPI and CAFAS data. Results BCFPI data for all referrals to the Centre compared to School Program and Clear Directions students reveal interesting patterns. Compared to all referrals, average scores for School Program and Clear Directions referrals were in the clinical range for Externalizing Behaviors, Managing Mood, Total Mental Health, School Performance, Child Functioning, and Family Activities. Regulation of Attention and Conduct scores for Clear Directions students also were in the clinical range. The percentage of cases within the clinical range also speaks to this pattern. Perhaps most striking is that over 80% of School and Clear Directions referrals were in the clinical range for Child Functioning. Over 80% of Clear Directions referrals also were in the clinical range for Family Activities and Family Comfort. The scores speak to the heightened and specific needs of the youth referred to these programs. CAFAS results indicated that all outpatient cases as well as School Program and Clear Directions cases showed significant improvements between entry and exit to services. On average, School and Clear Directions clients showed clinically significant improvements with differences of 35 and 42 points, respectively, between T1 and T14 ratings. Over 75% of all School and Clear Directions students showed some degree of improvement with over 61% of School and 52% of Clear Directions students improving 30 points or more. While clients in the School Program showed improvement on all subscales of the CAFAS except Substance Use, students in the Clear Directions program showed the greatest level of overall improvement with areas of significant improvement including School, Community, Managing Mood and Substance subscales. In addition, students in the School program received approximately 3 credits on average while Clear Directions students received 3.5 credits. Overall, CAFAS scores and credits achieved provide important data that may be used to assess outcomes for these programs. Outcomes attached to the additional measures collected at entry and exit to these services must be viewed with greater caution due to the small sample size and the degree to which the sample actually reflects the School and Clear Directions populations. Seventy-nine percent of School and 81% of Clear Directions cases had BCFPI data while the same was true for 58% and 75% of cases, respectively, regarding CAFAS data. As seen in Table 1, the SEI was the measure with the highest percentage of complete data, 30%, for School students while only 3% had complete CBCL data. Over 65% of Clear Directions cases had complete data but only 5 cases had complete CBCL data. Results of the SEI suggest that self-esteem increases from entry to exit of the Clear Directions program with specific areas of improvement being Academic Competence and Familial Acceptance. BECK-II and SASSI Alcohol and Other Drug Use scales, however, show no improvement beginning to end. Yet, CAFAS results do in fact show improvements in Mood and Substance Use. The most interesting results come from the TRF and YSR. Youth see themselves improving over the course of the Clear Directions programs. Youth reported decreases Internalizing problems,

15

specifically Withdrawn/Depression and Thought Problems, and in Total problems. Child and Youth Workers, however, consistently reported increases in various problems areas for both the School and Clear Directions problem between program start and end, contrary to the experiences of clinicians assigned to youth and the youth themselves. This contradictory finding must be explored further and may reflect shifts in program staff that occurred at various points in the evaluation process. An important part of this study was the ability to combine BCFPI and CAFAS data. The correlation patterns between the BCFPI and CAFAS subscales contribute important information regarding the validity of the two measures. Although the BCFPI and CAFAS are done at different points in time (i.e., referral vs. onset of service) and with information from different sources (i.e., guardian/caregiver vs. caregiver, youth, various professionals) the correlation between subscales suggests that the measures are tapping common constructs. For example, the Home subscale on the CAFAS was correlated with Conduct and Cooperativeness while Behavior Toward Others was correlated with Quality of Relationships and Family Activities. One the hand, Thinking, which taps organic or psychiatric issues (e.g., Schizophrenia) is not correlated with the mental health subscales of the BCFPI which target socioemotional problems. Recommendations and Next Steps Evidence from the BCFPI suggests that youth are being appropriately selected and streamed into the School and Clear Directions programs. CAFAS results and credits achieved clearly indicate that youth in the School and Clear Directions are improving in key areas such as school, mood, substance use, and overall functioning while other indicators (e.g., BECK-II or SASSI) are not detecting these shifts. This may reflect several factors such as lack of sensitivity of the measures to detect change within these populations, the small size of the samples, or the characteristics of the youth themselves. Youth in these programs present with problems in attention, school, and mood. As well, these students often have significant learning disabilities making completing lengthy paper-pencil tests a difficult and daunting task. Another challenge of the evaluation was the completion of consent forms and CBCL forms. Many students in School Program are also being served in the residential programs at the Centre. Few guardians provided complete pre-post CBCL data. As with youth, this may reflect various factors. Requiring families to complete additional forms as part of an already involved intake process may be overwhelming for families. Guardians themselves may have learning difficulties or have difficulties completing forms due to factors such as ESL or their own mental health concerns. A major stumbling block in the evaluation was the flow of information between programs and the evaluation. In many cases, youth and guardians coming into residential programs may have consented to participating in the evaluation process yet consents were not forwarded from the residential programs to the School or Clear Directions programs or the research department until the file was closed resulting in a reduced sample size. Although the results of the study indicate that youth are making significant improvements between entry and exit of the School and Clear Directions programs the results also speak to the need to review the evaluation process in. The requirement to complete additional, and in many cases lengthy forms, requires significant resources on the part of youth, their families, staff and researchers. The programs need to review whether or not the collection of additional data above the CAFAS, BCFPI and school performance indicators (e.g., credits achieved) is in fact worth the

16

additional resource demands. If the programs feel that this is useful or necessary, an improved flow of information is required to ensure that as many youth as possible are included in the evaluations. Obvious next steps are further analyses of the data. Further analyses should include examining program length in relation to outcomes, gender differences, and factors that contribute to positive outcomes. The syntax and program files also allow for additional information regarding BCFPI and CAFAS utility.

Knowledge Exchange Plan Several steps should be included in the Knowledge Exchange Plan that must be decided by the new Manager of Research and Evaluation and management of the Centre:

1. Results of the study should be translated into a Power Point presentation and shared with School and Clear Directions staff. This will provide staff with feedback regarding outcomes of the program as well as their significant contribution to the evaluation process. The timing of the report is significant in that staff have the opportunity to review the process and determine if this is an efficient use of resources or if the evaluation should be retooled for the upcoming school year.

2. Results of the current study can also be used in the annual quality assurance report to share with staff and Board Members.

3. A summary fact sheet should be developed to provide to youth and their families who have participated in the evaluation as well as those who may be entering the program. The summary fact sheet should also be distributed to other staff in programs across the Centre.

4. Finally, the results of the study should be disseminated to the larger mental health community by presenting at local CAFAS Community of Practice Meetings and the annual Children’s Mental Health Ontario Conference in the Fall of 2007.

17

References Baenen, R., Parris Stephens, M., & Glenwick, D. (1986). Outcomes in psychoeducational day school programs: A review. American Journal of Orthopsychiatry, 56, 263 – 270. Barkley, R., Shelton, T., Crosswait, C., Moorehouse, M., Fletcher, K., Barrett, S., Jenkins, L., & Metvia, L. (2000). Multi-method psycho-educational intervention for preschool children with disruptive behavior: Preliminary results at post-treatment. Journal of Child Psychology and Psychiatry, 41, 319 – 322. Beck, A., Steer, R., & Brown, G. (1996). The BDI-II Manual. Hartcourt Brace & Company: Toronto: ON. Blackman, M., Pitcher, S., & Rauch, F. (1986). A preliminary outcome study of a community group treatment programme for emotionally disturbed adolescents. Canadian Journal of Psychiatry, 31, 112 – 118. Brooke, S. (1996). Critical analysis of the Self-Esteem Index. Measurement and Evaluation in Counseling and Development, 28, 233 – 238. Brown, L., & Alexander, J. (1990). The Self-Esteem Index Examiner’s Manual. Pro-Ed: Austin, TX. Chan, D. (1997). Self-concept domains and global self-worth among Chinese adolescents in Hong Kong. Personality ad Individual Differences, 22, 511 – 520. Grizenko, N., & Papineau, D. (1992). A comparison of the cost-effectiveness of day treatment and residential treatment for children with severe behavior problems. Canadian Journal of Psychiatry, 37, 393 – 400. Harter, S. (1982). The perceived competence scale for children. Child Development, 53, 87 – 97. King, D., & Daniel, L., (1996). Psychometric integrity of the Self-Esteem Index: A comparison of normative and field study results. Educational and Psychological Measurement, 56, 537 – 550. Kotsopoulos, S., Wlaker, S., Beggs, K., & Jones, B. (1996). A clinical and academic outcome study of the children attending a day treatment program. Canadian Journal of Psychiatry, 41, 371 – 378. Milin, R., Coupland, K., Walker, S., & Fisher-Bloom, E. (2000). Outcome and follow-up study of an adolescent psychiatric day treatment school program. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 320 – 328. Miller, F., Renn, W., and Lazowksi, L. (2001). The Adolescent SASSI-A2. Baugh Enterprises: Bloomington: IN. Miller, H. (2000). Corss-cultural validity of a model of self-worth: Application to Finnish children. Social Behavior and Personality, 28, 105 – 118. Peterson, C., & Austin, J. (n.d.). Review of Coopersmith Self-Esteem Inventories. Mental Measurements Yearbook.

18

Piers, E. & Harris, D. (1969). The Piers-Harris Children’s Self-Concept Scale. Western Psychological Services: Los Angeles: CA. Rey, J., Enshire, E., Wever, C., and Apollonov, I. (1998). Three-year outcome of disruptive adolscents treated in a day program. European Child and Adolescent Psychiatry, 7, 42 – 48. Sayegh, L., & Grizenko, N. (1991). Studies of the effectiveness of day treatment programs for children. Canadian Journal of Psychiatry, 36, 246 – 253.

19

Appendix A

Budget

20

21

Description Units Rate Beneftits Amount

Data analysis coordinator 540 $23.50 1.14 $14,466.60• Scoring, data entry, cleaning and verification � • Database development/merge � • Syntax development � • Data analysis and reporting

Computer 1 $1,500.00Office supplies $400.00SPSS software 1 $85.00Dissemination $500.00Total $16,951.60

Note: Reduced cost of the SPSS software license was due to licensing through University of Toronto with funds being redirected to support work of the data analysis coordinator, which involved extensive scoring, entry and analysis of data.