Overview of Outcome Measures Online ToolboxDeborah Lowe Vandell, PhD. Principal Investigator

Professor and Founding DeanSchool of EducationUniversity of California, Irvine

afterschooloutcomes.org

Project Researchers:Valerie Hall, Ph.D.

Pilar O’Cadiz, Ph.D.Andrea Karsh, M.A.

afterschool@uci.edu

Background and Rationale for Online Toolbox Research indicates high quality afterschool programs can have

beneficial effects on students both socially and academically The Afterschool Outcome Measures Project at UC Irvine

developed an Online Toolbox in response to the State’s legislative mandate for alternative accountability measures to standardized test scores for publically funded afterschool programs in California*

Online Toolbox includes reliable and valid measures of Positive Behavior Change and Skill Development for use by diverse afterschool programs

Field Test of online toolbox at over 200 afterschool and summer learning program sites

Online Toolbox Statewide Implementation in Fall 2012/ Spring 2013

Development and Testing of Outcome MeasuresTHREE PHASES

PHASE 1—INSTRUMENT DEVELOPMENT (2008-2009) Literature review Survey of the field (expected outcomes and current practices) Selection of measures with input from the CDE and

Accountability and Outcomes Subcommittee (representation from diverse programs)

PHASE 2—PILOT STUDY (2009-2010) Pre-piloting at 9 sites to test procedures and instructions & to

determine feasibility of online administration of surveys Piloting at 28 sites to test protocols under varying conditions:

• Range of program size, geographic location, and level of operation• Sites with/without internet access

PHASE 3—FIELD STUDY (2010-2011) Test of measures & online administration across the State of

California• 196 field test sites representing diverse program sizes and

populations served by 29 programs in 9 regions statewide

NEXT PHASE: Statewide Implementation of Online

Toolbox for California ASES & 21stCCLC

Programs Opportunity to participate open to all publically funded afterschool programs in California:

Afterschool Education and Safety Program (ASES) 21st Century Community Learning Center (21stCCLC)

Up to 1000 sites statewide in 2012-2013

Up to 2000 sites to participate by 2014

Representation of diverse programs across 11 regions in the state

WHAT MEASURES ARE IN TOOLBOX ?

WHO TAKES THE SURVEYS?

HOW DO PROGRAMS ADMINISTER SURVEYS?

WHEN DO ASSESSMENTS TAKE PLACE?

WHY DO IT?

Outcome Measures Online Toolbox

Overview of Outcome Measures Web-based surveys of student performance collected from three

respondents: 1. Students (grades 3-12)2. Program staff3. Classroom teachers

PRE and POST Surveys collected at TWO time points, for example: Time 1 (early Fall/Winter) & Time 2 (Spring) Beginning of Summer & End of Summer

Technical Assistance provided through: Online Instructions and Resources Email and Telephone Communication

Individualized reports are provided to all participating programs. These reports enable programs to compare their students’

skills and behaviors over time, and to compare their program and site-level outcome scores to aggregate scores across the State.

Description of Measures

STUDENT SELF-REPORTS Skill Development •Work Habits•Reading Efficacy •Math Efficacy•Science Efficacy

Positive Behavior Change• Social Competencies• Reductions in Misconduct

Program Quality (post only)• Affiliation with Peers• Relations with Program Staff• Interest in Program Activities

PROGRAM STAFF & CLASSROOM TEACHER REPORTS—of Student Performance

Skill Development• Work Habits• Task Persistence• Academic Performance (classroom

teachers only)

Positive Behavior Change• Reductions in Aggressive Behaviors • Prosocial Behavior with Peers• Social Skills

Programs have option to administer one, two, or three surveys:

(1) Student (grades 3-12); (2) Program Staff; (3) Classroom Teachers

Field Test Key Findings

The Online Toolbox is an easy-to-use, inexpensive method of assessing skill development and positive behavior.

Elementary and middle school students reported having high quality experiences at their ASES programs.

Quality of experiences at ASES programs is linked to students’ skill development and positive behaviors.

Significant improvements in student performance were found between the fall and spring assessments.

SIGNIFICANT FINDINGS 2010-2011 Field Test

PRE-POST OUTCOMES in MATCHED SAMPLEStaff Reports of Elementary Students Increase in Prosocial Behavior **

Staff Reports of Middle School Students Increase in Prosocial Behavior **

Teacher Reports of Elementary Students Increase in Academic Performance ** Decrease in Aggressive Behavior**

Student Reports of Quality Program Experience Positive student reports of program experience linked

to positive results in ALL student outcomes** p<.01

Staff Reports on Student Prosocial Behavior

Increase in pre-post scores—matched sample

Fall Spring1.34

1.35

1.36

1.37

1.38

1.39

1.4

1.41

1.42

1.43

1.37

1.42

Fall Spring1.22

1.24

1.26

1.28

1.3

1.32

1.34

1.36

1.38

1.4

1.28

1.39

Elementary School Students Middle School Students

SIGNIFICANT FINDINGS

Teacher Reports on Elementary Students

Fall Spring2.6

2.7

2.8

2.9

3

3.1

3.2

2.79

3.13

Fall Spring0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.35

0.28

Improvements in Academic Performance

Reductions in Aggressive Behavior

SIGNIFICANT FINDINGS

Program Quality Distribution

Staff & Activities Peer Affiliation

Low Fair Good Excellent0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

5.5%

25.9%

34.3% 34.3%

9.9%

39.3%

31.9%

18.9%

Elementary Middle

Low Fair Good Excellent0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

5.4%

24.8%

30.1%

39.7%

7.1%

32.6% 32.8%

27.4%

Elementary Middle

Student reports of higher quality experiences with staff & activities are linked to greater Math Efficacy

Student reports of higher quality experiences with peers are linked to less Aggressive Behavior

Quality to Outcomes

Low Quality Mid Quality High Quality0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.60

0.45

0.34

0.64

0.53

0.43

Elementary Middle

Low Quality Mid Quality High Quality2

2.2

2.4

2.6

2.8

3

3.2

3.4

3.6

3.8

3.13

3.49

3.63

2.73

3.20

3.47

Elementary Middle

How does it work?http://afterschooloutcomes.org/

EASY TO ADMINISTERONLINE SURVEYS

Step-by-Step Instructions available to download from website

To APPLY to participate in

Statewide ImplementationCLICK HERE

Sign-up & Instructions on Website

To download Instructions for Administering

SurveysCLICK HERE

Program Staff & Teacher SurveysReports on Students in Afterschool Program

Program Staff and Classroom teachers are assigned individual access codes

Use both individual access code and the ID code for each student for whom they complete a report

Each student report takes 3-5 minutes to complete. Student reports may be completed in multiple

sessions. For example: A staff completing reports for 20 students can do four reports a day (a 20-minute session) on five different days.

Why the Online Toolbox? Provides a battery of valid, reliable, and easy to

administer youth outcome measures

Expands accountability tools beyond standardized test scores and other academic measures

Value in tracking individual student performance over time with matched pre-post surveys, providing more powerful & interpretable data• MATCHED = Respondent data are matched for both pre and post

survey administrations (i.e. same students, staff and/or teachers in fall and spring)

Information can be useful to guide program improvement based on student reports of program experience AND on student performance

For more information on measures:Forum for Youth

Investment publication (September 2011)

• Review of measures in Toolbox

with a technical appendix• Down Load at:

forumfyi.orgOr, link from our website:

http://afterschooloutcomes.org/

Contact Info:EMAIL: afterschool@uci.edu