November 18th, 2010

San Diego, California

CERA Conference

The California Statewide ASES Evaluation

2 / 27

Purpose of Study

1. Prop 49 becomes effective in Sept 2006

2. Afterschool funding increases to 550 million annually

3. One of the stipulation is that the CDE shall contract for an Independent Statewide evaluation on the effectiveness of programs receiving funds.

4. CRESST will conduct 2 separate evaluations- ASES and ASSET

5. Evaluation will cover analyzing STAR and other CDE data from 2006-2007 through 2009-2010

3 / 27

Deliverables

• Annual reports for ASES & ASSETs including achievement and behavioral outcomes

• Annual reports on afterschool profiles and changes

• In year 3 and 4 of the study, youth development outcomes will also be included in the annual reports.

• Final report to the Governor and the Legislature on December 1st 2011.

Study Theory

Best practices evidenced in the literature on out-of-school time suggest that several critical components such as:

• goal-oriented programs

• program structure

• and program process

These components contribute to the effectiveness and success of programs.

5 / 27

Instructional Features

ProgramEnvironment

Program Orientation

Staff Efficacy

Setting Features

Expectation

Aspirations

Positive Youth Development

Academic

Relationships

Safety

Goals

Management

External Connections

Evaluation system

Alignment

Program Climate

Satisfaction

STAR

CASHEE

School Attendance

MonitoringMonitoring

Resources

Professional Development

Theoretical model

Student Engagement

Student Engagement

6 / 27

Data Sources

CDE will provide:

• STAR

• CAHSEE

• CELDT

• SSID maintenance system

• Graduation rate for High School students

• Annual accountability reports with student level data, including attendance in school and afterschool

7 / 27

Data Collected by CRESST

• Profiling survey distributed to all grantees

program goals, structures, and process

• In depth site visits

program satisfaction, program process, community partnerships, program impacts

8 / 27

Study Sample Definition

• Study Sample I

All programs participating in the ASES and ASSET programs

• Study Sample II

100 ASES and 30 ASSET districts

• Study Sample III

All programs returning the profiling questions

• Study sample IV

Small sample for in depth analyses

9 / 27

The 6 evaluation questions for Study Sample IV

1. What is the impact of after school programs on the academic performance of participating students? Does participation in after school programs appear to contribute to improved academic achievement?

2. Does participation in after school programs affect other behaviors such as: school day attendance, homework completion, positive behavior, skill development, and healthy youth development?

3. Examine the similarities and differences in program structure and implementation. Describe how and why implementation has varied across programs and schools, and what impact these variations have had on program participation, student achievement, and behavior change.

4. What is the level of student, parent, staff, and administration satisfaction concerning the implementation and impact of after school programs?

5. What is the nature and impact of organizations involved in local partnerships?

6. What unintended consequences have resulted from the implementation of the after school programs?

10 / 27

Preliminary Afterschool Attendance Findings

• Elementary students tend to attend more regularly

• About 50% attended over 100 days

• Middle school students less regularly

• 26% less than 10 days, about 20% attended 100 or more days

11 / 27

Preliminary School Attendance Outcomes

• Statistically significant effect on School attendance

• Effect higher for frequent attendees

• No difference on school suspension-Sample II

12 / 27

Program Characteristics

• Most Programs are free for all

• First come first serve

• Majority of the programs stated that they have an academic goal

• Top three programming activities offered are:

homework assistance, academic enrichment and tutoring

• Top three programming activities offered are: physical fitness/sports, arts/music, and recreation

• Top developmental activities offered are: ASES school safety, youth development

13 / 27

Example: Goal Setting & Progress

Goal Set goal Met goal

Progressed towards

goal

Set goal, but not

evaluated

Failed to

progressAcademic improvement (n = 298)

93.6% 15.4% 49.1% 35.5% 0.0%

Improved day school attendance (n = 275)

68.0% 27.8% 36.9% 34.8% 0.5%

Improved homework completion (n = 279)

80.6% 29.3% 40.9% 29.3% 0.4%

Positive behavior change (n = 270)

69.3% 19.8% 50.8% 28.3% 1.1%

Improved program attendance (n = 274)

86.9% 43.7% 35.3% 19.3% 1.7%

Increased skill development (n = 260)

62.3% 23.5% 45.1% 30.2% 1.2%

14 / 27

Professional Development

• Site coordinators are the most likely to receive professional development

• The three top format are: training/workshop, program level meetings, and site level meetings

• The top three topics are:

classroom management, behavioral management, and federal mandated trainings

15 / 27

Program Orientation

• Most programs use a collaborative management approach

• Having prior work experienced is important

• Most site claimed to have at least some credentialed staff on site at any day—especially in the urban areas

• Support of educational goals is the most likely recruitment strategy for ASES programs

• Staff recognition and is the most mentioned strategies for retaining staff, followed by promotion opportunities

16 / 27

Community Partnership

• Majority of the programs maintained community partnership

• ASES programs partnered with parents, school and district staff and high school students

• The entity most school sites partnered with is the local LEAs of school districts or colleges

• LEA helps in providing operational assistance (data collection), site management, and professional developments

• Parents and other community members generally were involved in setting and revising goals, providing goods and helping out with the implementation and fundraising of the programs.

• Other community members also played a role in fundraising for the programs.

17 / 27

Additional Insights from Preliminary Findings of Sample IV

• Students perceived the program climate to be safe physically and emotionally. They felt cared for and supported by the staff, while the staff reported to encourage students’ efficacy and aspiration.

• In general, staff reported to have sufficient writing materials, space to work, and enough time to work with the students. Both site staff and site coordinators were likely to report that they did not have time to prepare for activities.

• staff efficacy was enhanced by professional development. Both site coordinators and site staff agreed that the professional development offered was aligned with program goals and that the training was provided annually.

• Parent surveys reviewed a low degree of involvement, though they also appeared to be satisfied with the way programs connected with them.

18 / 27

Evaluation Systems

• Most site coordinators reported that they tracked the different stakeholders’ levels of satisfaction

• They also responded that they had an internal method of evaluating the program activities for program improvement

19 / 27

Perceived Outcomes

• Overall, students, parents, and after school staff all have positive perceptions on the effects of program participation. Both students and their parents felt that attending a program led to students feeling more academically efficacious. They also believed participation increased the students’ cognitive competence, & socio-emotional competence.

• Students reported that they felt that they had the skills, knowledge, and desire to become successful adults as a result of attending an after school program.

• High school students further reported that attending a program helped them resist the lure of alcohol, drugs, and gangs.

20 / 27

Obstacles for Parent Involvement

ASES

• 60% visited the program

• One in three attended any event

• One in five volunteered or given feedback

21 / 27

Obstacles to Student Recruitment and Retention

ASES-NONE