evaluation practices from top to bottom
DESCRIPTION
Data sharing and data-driven decision making are a critical component for successful collaborations that drive toward student achievement. At this session, we will discuss best practices for developing a data driven, results-based organization, learning from Higher Achievement’s experiences successfully submitting to a third party evaluation, customizing a management information system for in-house use, and regularly using internal and external data to make strategic and programmatic decisions.TRANSCRIPT
Evaluation from Top to Bottom
Rachel GwaltneyChief of Programs, Higher Achievement
www.higherachievement.org
AGENDA• Program overview• Third-party research study findings• Internal data analysis
– Participant outcomes– Program quality– Staff
• Interactive discussion– Benefits and best practices for data and systems– Challenges of data and systems
• Resources
Higher Achievement’s Theory of Change• After school and summer program
offering middle school students 650 hour extended learning beyond 900 hours of school
• Preparing scholars for college and career readiness
• Combined culture and content model
• Founded 35 years ago in DC• Started national expansion 2008
• Increased • Academic • Interest
• Increased Academic
Opportunities
• Increased Academic • Effort
• Increased Academic Achievem
ent
Who Are Our Scholars?
• 5th – 8th grade• Starting GPA: 2.5• 99% minority• 81% FARM-eligible• 79% will be
first-generation college graduates
• Most are recommended by teachers
Scholars commit to 650 hours per year, beyond the 1000 hours in school
Culture- high
expectations - praise for effort- student voice
and choice- learning is fun!
http://www.youtube.com/wat
ch?v=EOEZZvI2aKU&feature=re
lated
Results and Impact
Annual outcomes:• Significant improvements in
grades, test scores, and school attendance
8th grade graduates (2010):• Improved their average GPA
from 2.2 to 3.2• 95% were placed in a top high
school program• 85% improved or maintained an
A or B in math and reading
Third-party research:• The intensive year-round
program had a significant impact on youth's standardized reading and math test scores.
• 64%of parents of children attending the program confirming at their first-year follow-up that they spoke to Higher Achievement staff about their child's progress at least once a month.
Research Partners• Principal Investigators
– Carla Herrera (Public/Private Ventures)– Jean Baldwin Grossman (P/PV, Princeton)– Leigh L. Linden (The University of Texas at Austin)
• Funders of published work to date– The Atlantic Philanthropies– The William T. Grant Foundation– The Wallace Foundation
• Data Collection– Survey Research Management
Research design• Overview
– Evaluation of the Higher Achievement program• After-school and summer program
– One-year, two-year and summer findings– Four-year evaluation in progress
• Recruitment and Randomization– 951 students applied to Higher Achievement– More students applied than Higher Achievement could serve– Randomly chose students to offer admission to Higher Achievement– Remainder became a control group
• Advantages of design– Gold standard evaluation strategy– Sample comprises “types” of children served by Higher
Achievement
Outcomes Measured• Key outcomes and variables of interest:
– Standardized test scores• Abbreviated SAT 10 Problem Solving• Abbreviated SAT 10 Reading Comprehension
– Behavior– Academic attitudes– Perceptions of peer and adult support– Participation in Higher Achievement and other OST
programs– Activities related to high school application
• Analyzed separately:– Parent and child assessments of OST programs– Mentor and teacher surveys within Higher Achievement– Qualitative data on Higher Achievement
Timing of Data Collection2006 2007 2008 2009 2010 2010 2011 2012Spring Spring Spring Spring Spring Fall Spring Spring
Survey Round Baseline FU1 FU2 FU4Grade Entering 5th/6th 6th/7th 7th/8th 9th/10th
Survey Round Baseline FU1 FU2 FUSp FUFa FU4Grade Entering 5th/6th 6th/7th 7th/8th 8th 8th 9th/10th
Survey Round Baseline FU1 FU2/FUSp FUFa FU4Grade Entering 5th/6th 6th/7th 7th/8th 7th/8th 9th/10th
Cohort 1 (N=276)
Cohort 2 (N=276)
Cohort 3 (N=399)
Note: FU1 = One-Year Follow-Up FU2 = Two-Year Follow-Up FU4 = Four-Year Follow-Up FUSp = Spring FU for the Summer Study FUFa = Fall FU for the Summer Study
Standardized Test Scores
• Significant effects after two years– Problem Solving: 0.12 Standard Deviations– Reading Comp: 0.09 Standard Deviations
• Effect sizes are larger than those reported for other OST programs evaluated by large-scale RCT studies.
• No effects after one year• No difference during summer 2010
Behavior• Asked youth about their engagement in several
negative behaviors– In-school: e.g., principal’s office, tardies, skipping– Out-of-school: e.g., taking or breaking something, hitting
• At both the one- and two-year follow-ups treatment students were more likely to report engaging in some of these behaviors.
Academic Attitudes• Six measures
– Industry and Persistence– Creativity– Self-Perceptions of Academic Abilities– Enjoyment of Learning– Curiosity– Ability to Change the Future through Effort
• Overall, treatment students have more negative attitudes than control students after the first year.
• No overall differences at the second year.• Effects vary by the grade at which youth enter HA.• Gains in Enjoyment of Learning during Summer 2010
Program Participation
• Higher Achievement provides opportunities that scholars would not otherwise have.– Without access to HA, 35 percent attend an academic OST; – Access to HA increases this by 52 percentage points.– Treatment students average more time in academic OST
programs• 10.3 hours more a week during the academic year• 19.8 hours more a week during the summer
Activity Participation• Treatment youth were more likely to report
engaging in a wide range of activities. For example:– Visiting a college campus– Speaking to a group about youth’s ideas or work– Speaking to an adult about high school, college
and jobs– Going to events outside youth’s neighborhood– Writing poems, stories, etc. not for school– Going to events outside of school
High School Application Activities
• Only tested in Summer 2010 and four-year follow-up
• Students were more likely to report engaging in various preparatory activities. For example:– 14 percentage point difference in visiting high schools– 15 percentage point difference in getting application
information on a school
• Significant increase in students wanting to attend competitive high schools.– A relative increase of 16 percentage points
How are these outcomes achieved?• This study cannot rigorously answer this question.• But Higher Achievement has several
characteristics that make it stand out as a strong program:– Long-term and intensive– Broad range of academic and enrichment activities– Guided by grade-level curricular standards– Staff are well trained and supported– Strives to involve parents– Focus on small-group instruction– Opportunities for leadership
Conclusions• Participation in well-structured, long-term, academically
focused out-of-school-time programs can boost student achievement.
• Gains take time, emerging only after two years of access to the program.
• Gains coincided with increased reports of negative behavior and without an improvement in academic attitudes.– Requires further investigation
• Engagement in activities related to high school application process is promising.– Fourth-year data collection to be completed this summer
and published the following year.• Lack of test score differences in summer does not mean
that the summer program is not an important component of the program.
Implications:• Organizations• Sector• Policy
Lessons Learned
• Addressing moral question of denying access to program for research purposes
• Ensuring staff capacity to recruit and support• Feedback from researchers was invaluable for
program improvement• Retention is critical• Plan regarding communicating findings• Work with research team to secure investment
Internal Data Practices – Scholar DataFREQUENCY DATA TRACKED
Daily Individual feedback on session participation and progress
Quarterly Report card data (grades, attendance)Scholar Action Plan
Biannually (twice/year) Attitudes and behavior (360 survey)
Annually Scholar outcomes: Standardized test scores, GPA, school attendance, high school placement
Internal Data Analysis - ProgramFREQUENCY DATA TRACKED
Weekly Feedback from volunteer mentors
Biweekly (every two weeks) Dashboard data
Periodically Program quality observations • Internal tool correlated to core
program elements• YPQA external tool
Triannually (three times/year)
Quality assurance reports compiled from observations
Annually Scholar outcomes
Internal Data Analysis - Staff
FREQUENCY DATA TRACKED
First 90 days Completion of orientation goals
Quarterly Progress toward workplan goals
Annually • Evaluation against workplan goals and organizational culture• Scholar outcomes
Lessons Learned
• Invest in the right systems• Set up and enforce strong systems
for data collection• Train staff to report on, understand,
and act on analysis of data• Make data-driven improvement part
of organizational culture
Discussion
Resources• Harvard Family Research Project: www.hfrp.org
– Afterschool Evaluation 101• ChildTrends: www.childtrends.org
– Data-Driven Decision-Making in Out-of-School Time Programs• Forum for Youth Investment: www.forumfyi.org
– From Soft Skills to Hard Data• Wallace Foundation: www.wallacefoundation.org
– Hours of Opportunity– The Cost of Quality OST Programs
• Public/Private Ventures: www.ppv.org
Thank You!