moira denike consulting · structured interview tool for school principals and beacon directors to...

0

Moira DeNike Consulting

SAN FRANCISCO BEACON INITIATIVE EVALUATION 2011-12

1

Table of Contents

Background ....................................................................................................................................................................................... 1

Beacon Management Structure .......................................................................................................................................... 1

Evaluation Framework ........................................................................................................................................................... 1

Program Model ................................................................................................................................................................................ 2

Program Goals ................................................................................................................................................................................. 3

Research Methods.......................................................................................................................................................................... 4

Youth Survey ............................................................................................................................................................................... 4

Parent and Community Participant Survey .................................................................................................................. 4

Structured Interview Tool for CBO-School Partnership ........................................................................................ 4

District Attendance and Achievement Data ................................................................................................................. 5

Findings .............................................................................................................................................................................................. 5

Youth Skills, Academic Engagement and Satisfaction ............................................................................................. 5

Family Skills, Engagement and Satisfaction .............................................................................................................. 16

School Partnerships .............................................................................................................................................................. 18

Community Impact ................................................................................................................................................................ 23

Conclusion ...................................................................................................................................................................................... 25

References ...................................................................................................................................................................................... 28

Appendix – Survey Tools ......................................................................................................................................................... 30

.............................................................................................................................................................................................................. 30

1

San Francisco Beacon Initiative Evaluation

Background The San Francisco Beacon Initiative (SFBI) was launched in 1994 with the mission of transforming

public schools in low-income areas into youth and family centers that become a beacon of activity

for the surrounding neighborhood. SFBI serves approximately 10,000 youth and adults each year,

making it one of the largest youth development efforts in the Bay Area. Beacon activities aim to

promote the healthy development of children, youth, families, and communities, before and after-

school, in the evenings, on weekends and during the summer. Each Beacon Center offers

neighborhood-based programs for children, youth and families in education, career development,

arts & recreation, leadership, and health. Beacon Center programming is designed to support a

healthy community, and to give youth an increased sense of well-being, increased positive

connection to school, and improved academic performance.

Beacon Management Structure

To support its goals, the Beacon Initiative has developed a broad base of partnerships including city

departments, San Francisco Unified School District, community organizations, and private

foundations. The Beacon Initiative is led by the San Francisco Department of Children, Youth and

Their Families (DCYF), the San Francisco Unified School District (SFUSD), and a group of several

private funders including the Evelyn & Walter Haas, Jr. Fund, the W. Clement & Jessie V. Stone

Foundation, and The San Francisco Foundation.

Oversight for the Beacon Initiative is provided by the Beacon Steering Committee, which includes

representatives of each stakeholder group. Each Beacon Center is managed by a non-profit,

community-based organization that serves as the lead agency, and has a Director and staff. The lead

agencies are responsible for staffing programs and creating an effective partnership with the school

that hosts the “hub” site as well as any satellite sites where programming occurs. The initiative is

managed by a small SFBI staff that convenes Steering Committee meetings, forges and nurtures

partnerships with funders and stakeholders, maintains quality programming by offering Beacon

Center staff professional development and peer learning forums, and ensures that evaluation takes

place.

Evaluation Framework For the 2011-12 evaluation SFBI contracted evaluator Moira DeNike to lay out an evaluation

framework that would capture the multi-layered work of the initiative, including work happening

at the community, school and family levels which previous evaluation efforts had not emphasized.

The proposed framework was approved by DCYF and the Beacon Steering Committee and crafted in

order to align with SFBI’s concurrent effort in 2011 to develop a new communication strategy. This

new evaluation framework forms the structure and methods of this evaluation study.

2

Program Model The Beacon model rests on the assumption that by transforming public schools into safe spaces

where children, youth, parents and community members can engage in positive, educational,

enriching and healthy activities during extended hours, youth in the community will benefit. This

assumption is reinforced by the well-established social ecological model of child development,

which posits that there are multiple spheres of influence, often depicted as concentric circles. These

spheres emanate out from micro-level influences like family, to larger, more structural or macro-

influences like school, neighborhood, and cultural or economic context (Bronfenbrenner, 1977;

Bronfenbrenner, 1979; Santrock, 2007; Kail & Cavanaugh, 2010). The job of the Beacons, is

therefore, seen as not only providing youth development and academic learning supports to youth,

but also supporting parents in skill development, helping families navigate schools, coordinating

school-day and out-of-school-time efforts, deepening participating youths’ connection to their

communities, and ensuring the neighborhood is a healthy place in which to grow up.

Community Level: Connecting community to school and

children & youth to community

School Level: Coordination between schools and out of

school time providers

Family Level: Parents and families participating in school-

located activities

Youth Taste &Spark: Afterschool enrichment

for children & youth

Youth Deep Engagement:

Youth consistently participating in skill-

building programming

3

Program Goals The new framework recognizes that Beacon activities take place at the five distinct levels laid out in

the model, targeting five different audiences: 1) youth who engage in a sustained manner in Beacon

programs, 2) youth who drop in at Beacon Centers to have a safe and productive place to be after

school, 3) family members of students attending host schools or the Beacon, 4) program staff and

personnel at the host schools, and 5) community members who partake in Beacon programs.

Program goals are also framed according to these five spheres. The table below lists five basic

program goals as well as the activities, outputs and measurable outcomes associated with each.

Activities Outputs Outcomes

Goal 1: Through consistent participation in Beacon programs youth will gain skills, develop supportive relationships, and show growth on youth development and academic measures

Activities centered on leadership, skill development, and community outreach/involvement

Number of unduplicated youth participants participating at an intensive level; Number of youth participating in skill-building programs

Highly engaged youth show improved skill-building, supportive relationships, leadership & community involvement; improved school attendance and achievement compared with youth not deeply engaged

Goal 2: All youth, including those who only drop in to Beacon Centers, will experience a safe space that offers quality programs

Drop-in after school enrichment (including sports, recreation, arts, etc.) and academic support

Number of unduplicated student participants; Average Daily Attendance (ADA)

Youth and child participant satisfaction with program

Goal 3: Beacon programs will help family members navigate schools more effectively, gain skills and connect with the community

Classes/activities for parents during evenings and weekends, including parenting classes, youth performances, and other events

Percent of adult community program participants who are parents; Number of adults participating in family-oriented programs

Increase in family participants’ 1) connection to schools, 2) access to support and skill-developing, and 3) connection with the community.

Goal 4: Host hub schools and Beacon lead agencies will be more coordinated and aligned

Beacon acts as a conduit for communication and coordination between lead agencies &school sites, and helps to align program and school-day goals

Amount of space and number of hours that Beacon Centers occupy school sites; Meetings between program and school-day personnel

Mutually satisfying and supportive relationship between program and school-day; Increased alignment of goals

Goal 5: Community programs will improve the health of communities and increase community investment in youth, families and the school

Classes for adults during evenings and weekends, including dance, recreation, computer skills, and other educational programs

Number of unduplicated adult participants; ADA

Intensified investment in community, neighborhood families, children, youth and school

4

Research Methods Methods of inquiry were designed to capture an impression of what is happening at each of these

levels of Beacon work. These included a youth survey to gauge youth satisfaction and program

quality in terms of youth development principles, an adult survey to collect information from

parents, family members and community members who participate in adult programming, a

structured interview tool for school principals and Beacon Directors to enable a better

understanding of the strengths and weaknesses of current partnerships, and the use of SFUSD data

for academic outcome measurement.

Youth Survey

The Youth Survey was designed by Public Profit, the firm responsible for evaluating SFUSD’s ExCEL

out of school time programs. Because all Beacon Centers blend ExCEL program funds, SFBI worked

with DCYF, SFUSD ExCEL and Public Profit to arrive at an agreement that the ExCEL survey would

be used to gauge student impressions of program quality. The survey questionnaire asks students

questions that assess the program on multiple domains, including: Program Environment and

Safety; Promoting Diversity, Access, Equity and Inclusion; Youth Development; and Intentional Skill

Building. Items on the Youth Survey instrument were drawn from the California Healthy Kids

Survey, DCYF Customer Satisfaction Survey, as well as questions that Public Profit has developed

over its years of experience evaluating youth-serving programs. Two versions were distributed

among Beacon youth – one for 3rd -5th graders and one for youth in 6th grade and above. Both

questionnaires cover roughly the same subject matter, but language on the instruments was

adjusted according to reading level. Surveys were distributed to Beacon youth in ExCEL-funded

program components as well as youth in program components that are not ExCEL-funded. Sites

were asked to administer surveys online or via paper, depending on their technology capacity.

Parent and Community Participant Survey A single adult survey questionnaire was designed to measure how well the program is

accomplishing the dual aims of community and family programming. The analysis of the adult

survey data takes into consideration whether respondents were family members, and the

questionnaire includes questions pertaining to family needs such as the ability to navigate school

systems and the development of skills that help participants support students. The questionnaire

also includes questions around community strengthening ,social networking and connection to the

school, as many of the adult respondents were not family members to Beacon youth or students at

the host schools, and a key aim of the Beacon (and broader community schools model) is to engage

the surrounding community. Surveys were collected at Beacon hub sites using drop-boxes so that

survey respondents did not have to worry that staff would see their responses.

Structured Interview Tool for CBO-School Partnership

The organizational structure of the Beacon Initiative facilitates close coordination between the

schools that host Beacon hubs and the providers of out-of-school-time (OST) programming. New

models of collaboration between schools and community-based organizations (CBOs), such as Full

Service Community Schools, Promise Neighborhoods, and New Day for Learning, all hinge upon

strong substantive partnerships between school-day and OST providers. Heretofore there have

been no systematic methods in place to measure the effectiveness that the SFBI-facilitated

5

coordination produces at the Beacon Centers or at the schools themselves, beyond interviews and

focus groups, but those have taken place inconsistently over the years. The new evaluation

framework includes structured interviews that were informed by the Coalition for Community

Schools’ Indicators of Capacity tool (Coalition for Community Schools, 2012). School principals at all

eight Beacon hub sites were interviewed by phone using the same set of questions. These questions

prompted school principals to provide numerical ratings as well as commentary on the strengths

and weaknesses of the partnership with the Beacon lead agency. Telephone interviews were also

conducted with all eight Beacon Directors who were asked a similar set of questions. Interviews

took place just before and after the summer of 2012. In the event that a school principal was

transitioning out of the school (which was the case at a few of the schools), the outgoing principal

was interviewed.

District Attendance and Achievement Data Finally, the San Francisco Beacon Initiative and the evaluator worked with SFUSD to access school

attendance and achievement data for Beacon youth participants. In order to analyze whether youth

who engage deeply in Beacon programming experience measurable improvements in academic

performance (during one academic year), as compared with youth who do not, and whether these

changes differ based on variables such as gender and ethnicity, SFBI’s participation and

demographic data had to be merged with SFUSD data. SFUSD provided a) attendance, b) GPA, and c)

standardized test data (wherever available), for Spring 2011 and Spring 2012. A data merge

between SFBI data and SFUSD data was conducted, and student identifiers were removed from the

dataset before analysis to ensure that student confidentiality was maintained, in compliance with

the Family Education Rights and Privacy Act (FERPA). The data merge enabled a pre-post analysis

of attendance and academic indicators that took program participation levels (dosage) into

consideration. It also enabled an analysis that disaggregated according to demographics and

provided useful information for each Beacon Center. All reports include only aggregate data and no

analysis considers such a small disaggregated student group (e.g., African American fourth-graders

at one school) to allow for student identification to reasonably take place.

Data obtained through the above methods inform the content of this report.

Findings

Youth Skills, Academic Engagement and Satisfaction The value of Out of School Time (OST) programs has been well documented as a way to increase

youth safety during the riskiest time of the day (Eccles & Gootman, 2002; Simpkins, 2003) and to

improve school day attendance (Arbreton et al., 2009; Kauh, 2011). Youth development programs

have been shown to increase community investment, goal-setting, and leadership skills (Benson et

al., 2006; Philliber Research, 2005). The evaluation is designed to measure 1) how well Beacon OST

programs are engaging youth during those high risk hours, 2) whether participation correlates to

improved school engagement and achievement, and 3) from the youth participant perspective, how

well the program promotes emotional safety and quality relationships, intentional skill-building,

youth development principles, cultural competence, and academic support.

6

Program Engagement and Participation

For the 2011-12 year Beacon Centers, at both hub and satellite sites, served 11,989 unduplicated

individuals in total. Among these 9,470 were children and youth and 2,519 were adults. In 2010-11

the initiative served 10,146 children, youth, and adults.

Beacon youth participation is fairly proportionate to the ethnic composition of the district, with

slightly lower white and Latino representation, in part attributable to the lower representation of

these groups at hub and satellite schools, and higher Polynesian and African American

representation. Comparing the ethnic distribution among all Beacon youth with that of youth who

participated in programming for 35 or more days (high level participants), there are few

differences with the exception that the proportion of Asian students rises and that of Polynesian

students drops. In general, however, there is no perceptible attrition problem with most ethnic

groups, including African American and Latino students.

Ethnic Category District Beacon Schools

Beacon Youth

High Level Participants

American Indian/Alaska Native 0.5% 0% 0% 0% Asian 40% 51% 40% 47% Native Hawaiian/Pacific Islander 1% 1% 5% 2% Filipino 5% 8% 6% 6% Hispanic/Latino 24% 18% 19% 19% Black or African American 11% 9% 13% 13% White 11% 7% 4% 4% Two or More Races 3% 1% 5% 5% Other/None Reported 4% 4% 8% 4% Total 100% 100% 100% 100%

Among the 9,470 youth served, SFUSD could provide data pertaining to socioeconomic status on

5,143. These data showed that approximately 70% of all youth served initiative-wide, and 69.3% of

high-level participants are designated socioeconomically disadvantaged.1

Initiative-wide Average Daily Attendance (ADA) was 2,049, though there was a good deal of

variation among individual Beacon Centers’ ADA.

Beacon Center Unduplicated Adult Unduplicated Youth ADA

Bayview Beacon 102 708 112

Mission Beacon Center 106 611 127

North Beach-Chinatown Beacon Center 223 899 107

OMI/Excelsior Beacon Center 566 1270 210

Richmond Village Beacon 979 1875 428

Sunset Neighborhood Beacon Center 329 2067 698

Visitacion Valley Beacon 105 1022 167

Western Addition Beacon 109 1018 200

Grand Total Initiative Wide 2519 9470 2049

1 Defined by CDE as students eligible for FRMP or whose parents have less than a high school education.

7

Nearly half of the services received by Beacon participants fell into the category of academic

support. Arts programming, physical recreation, youth leadership development, and civic

engagement comprised 19%, 18%, 4% and 2% of total activity time for the year, respectively. Other

activities and supports, including family support, vocational assessment and career awareness and

exploration, science, technology, engineering, and mathematics-STEM, health/violence prevention

education, culture, identity, and diversity, life skills training, vocational/employment training,

comprised the remainder of service hours.

It should be noted that Beacons blend ExCEL funding with core Beacon funding from DCYF and

other funders. The SFUSD-sponsored evaluation of the ExCEL program looked at participation and

ADA at non-Beacon and Beacon ExCEL sites (this excludes Beacon activities and program

components that are not funded by ExCEL), and found that Beacon sites outperform non-Beacon

ExCEL OST programs in terms of participation. Beacon sites generally had a higher ADA than other

ExCEL programs serving the same grade level. Furthermore, while Beacons represent only 18% of

all ExCEL sites, in 2011-12 Beacon youth represented nearly 27% of San Francisco’s ExCEL

participants.2

2 This information was provided by Public Profit, which conducted a comprehensive evaluation of the ExCEL program for SFUSD.

49%

19%

18%

4%

2% 8%

Beacon Participation by Activity Category

Academic Support andAcademically-Linked Activities(ACA)

Arts, Music and CulturalActivities (ART)

Sports, Physical Activity &Recreation (REC)

Youth Leadership Development(YOU)

Civic Engagement & CommunityOrganizing (CEO)

Other

8

Academic Engagement and Achievement

Because “dosage” has been shown to be an important consideration when measuring an OST

program’s impact on school attendance and academic outcomes (Harvard Family Research Project,

2004), the evaluation has been designed to measure program impact according to levels of program

participation. Consistent with what is recommended in the OST dosage literature, rather than set

the critical dosage threshold according to a pre-formulated notion of how many days would be

necessary to effect a positive impact, our analysis grouped participation levels a number of ways,

including based on a 100-day threshold and an 80-day threshold. In the end, dividing Beacon

participants into equal thirds based on days of participation, with “Low” participation level defined

as fewer than 10 days of program participation, “Medium” as 10-34 days, and “High” as 35 days or

more, emerged as the most meaningful division for running statistical inquiries and recognizing

program impact.3 These participation levels were used, alongside SFUSD data, to gauge whether

Beacon program participation had an impact on school-day attendance and academic achievement.

In both cases program participation was found to correlate with greater school engagement and

performance.

Impact on School Absences

SFUSD data showed that high-level Beacon program participants as a group had 43% fewer school

absences than low-level participants. This relationship between program participation level and

school absences was found to be statistically significant (p<.001) using Pearson’s Correlation and

Kendall’s tau measures of correlation for both elementary school students and middle/high school

students. The finding, however, may be more meaningful for the middle and high school students

who exercise a greater degree of choice in terms of both school and program attendance than do

their younger counterparts.

It should also be noted that there were no discernable differences in school absences among the

three groups of middle and high school students during the year preceding the program year,

increasing the confidence that observed differences can be reasonably attributed to program

participation (by contrast, elementary school students in the high participating group were more

likely to have lower absences in the preceding year). Furthermore, since 73% of Beacon youth

survey respondents indicated that the Beacon program has helped them, “plan to go to school

everyday,” there a statistically significant relationship between program participation and school

absences, and, among middle and high school students there is compelling evidence that this

relationship is causal.

3 Low level youth participants participated on average 11 hours total between Summer 2011 and Spring 2012; Medium level youth participants participated on average 61 hours; High level participants participated 248 hours.

9

The relationship was stronger with some ethnic groups than others. Among Asian, Black, and Latino

youth the relationship between program participation and fewer school absences is clear (this was

also true among Middle Eastern students, although their numbers were too low to draw a

generalizable finding). The relationship, however, did not appear in the data with White students

(the relationship was also nonexistend with Polynesian, Native American, and Multiethnic, although

their numbers were too low to draw a generalizable finding from this result).

The program effect on school absences was found to be slightly stronger for Socioeconomically

Disadvantaged youth (Low SES) – high-level participants experienced 46% fewer absences than

low-level participants – and significantly stronger for Special Ed youth – high-level participants

experiencing 65% fewer absences. No gender differences were found.

6.8957

5.1003

3.8318

7.0877

5.6052

4.0427

.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

Low Medium High

School Absences for 2011-12 by Beacon Program Participation Level

Elementary School

Middle & High School

.002.004.006.008.00

10.0012.0014.0016.0018.0020.00

Black(626)

Latino(941)

PacificIslander

(121)

White(189)

Asian(2774)

Low SES(3537)

Special Ed(591)

Average School Absences by Program Participation Level for Key Subgroups

Low Participation

Medium Participation

High Participation

10

These findings indicate that the Beacon programs encourage better school engagement, and that

this is particularly true among key subgroups, including African American and Special Ed students,

who, without program participation, show on average more than 12 days of absences in the school

year.

Impact on Academic Performance

Beacon program participation levels also correspond to academic performance. Differences in

achievement between final grades in the 2011-12 and 2010-11 Spring semesters were calculated

for Beacon participants in the three participation level categories. Because elementary school

grading is organized around a different principle than the A-F grading used in the middle and high

school levels, the findings on this measure are presented separately. Middle and high school

students’ academic performance is represented by GPA, while elementary school students’

academic performance is represented by a numeric mean of the multiple academic standards listed

on students’ report cards (note: measures of classroom behavior and physical coordination are

excluded from the academic mean calculation).

.1247 .1512

.2979

.00

.05

.10

.15

.20

.25

.30

.35

Low Medium High

Report Card Differences from 2010-11 to 2011-12 by Program Participation Level for Elementary School

.0076

.0239

.0424

.0000

.0050

.0100

.0150

.0200

.0250

.0300

.0350

.0400

.0450

Low Medium High

GPA Improvement from 2010-11 to 2011-12 by Program Participation Level for Middle & High School

11

Differences between pre (2010-11) and post (2011-12) measures for both GPA and elementary

school report card means show that higher levels of participation are associated with greater gains.

Among elementary school students these findings were statistically significant (p<.001). Differences

for middle and high school students also show overall improvements, but these do not reach the

level of statistical significance, largely because the difference measured was small (representing

approximately a 25% improvement on a single course grade, assuming 7 courses per report card).

Among all Beacon participants for whom there were available data (n=4,060) it was found that

academic gains were experienced by a total of 49%. This also varied significantly (p<.005)

according to participation level, with only 45% of low-level participants experiencing gains, 48% of

medium-level participants experiencing gains, and 52% of high-level participants experiencing

gains. This relationship was found consistent whether dosage was measured as overall

participation or participation in academic support program elements only (measured as a threshold

at 83 hours of academic support).

When the data were disaggregated according to Special Education and socioeconomic disadvantage

status, this pattern held, although the relationship between participation and improvement in

report card gains is stronger for non-Special Education-designated students and non-

socioeconomically disadvantaged students. When the data were disaggregated by school level, the

relationship appeared far stronger among elementary students than middle and high schoolers.

When disaggregated by ethnic group the data showed that greater participation is associated with

academic gains among Multiethnic, Asian and White youth, but not among Black, Polynesian or

Latino youth. There were no gender differences measured.

Percentage of Students in Sub-Categories Experiencing Academic Gains between Spring 2010-11 and Spring 2011-12 by Participation Level Student Category (n) Low

Participation Medium

Participation High

Participation Overall

Not Low SES (1231) 43.4% 48.9% 54.3% 49.7% Low SES (2829) 46.4% 47.6% 51.0% 48.7%

Not Special Ed (3598) 45.2% 47.8% 51.9% 48.8%

Special Ed (462) 47.9% 50.0% 52.8% 50.9%

Asian (2263) 45.4% 47.2% 54.7% 49.8%

Black (481) 46.1% 49.2% 47.9% 47.8%

Latino (732) 47.7% 47.9% 47.2% 47.5%

Multiethnic (174) 36.5% 47.5% 53.7% 47.1%

Polynesian (96) 43.2% 33.3% 43.8% 40.6%

White (135) 41.7% 59.6% 57.7% 54.1%

Data on CST in Math and English Language Arts (ELA) were available for approximately 4,800

Beacon youth. Overall, a minority of Beacon youth experienced improvement in their raw scores on

both subjects (Math: 35%; ELA: 40%), although high-level participants were more likely to show

improvement (Math: 37%; ELA 41%). While this difference was very modest, it was found to be

statistically significant (p<.05), and was consistent whether high participation was measured as 35

or more days attending program or 83 or more hours of academic support services.

12

Safety Index: I feel safe in this program This program makes me feel safer after school

Diversity Index: In this after school program I learn to get along

with people who are different The people in my after school program respect my

language and culture School Engagement Index: This program has helped me to feel like a part of

my school This program has helped me plan to go to school

everyday This program has helped me to care about my

school Skill Development Index: Because of this program I am better at something I

used to think was hard I learn new things [in this program] In this program I learned a new skill that I’m

excited about In this program I learned new information about a

topic that interests me Positive Relationship Index: Since coming to this program I am better at making

friends This program has helped me feel like I am

important to my school The staff in this program expects me to try hard to

do my best No one is left out when we are doing physical

activities The staff members here listen to what I have to say The staff here tells me when I do a good job I feel like I belong at this program There is an adult at this program who really cares

about me I feel like people are happy to see me here Since coming to this program I have met an adult

who I can trust Youth Development Index: Since coming to this program I am better at telling

others about my ideas In this program I get to help other people In this program I get to decide things like activities

and group agreements Since coming to this program I am better at setting

goals for myself Since coming to this program I am better at

listening to other people Academic Support Index: This program has helped me to learn good study

skills (like reading directions, taking tests) I am more interested in my education because of

this program Things I learned in this program make me want to

learn more This program makes learning fun

Due to the minimal effect that was found, no

disaggregated analyses were conducted on CST

improvements.

Youth Development

Students at all eight Beacon Centers were

surveyed using the instrument that Public Profit

designed for participants in SFUSD’s ExCEL-

funded after school programs. Public Profit

provided youth survey data for Beacon students

in ExCEL-funded program components (with

student identifiers removed). Beacon youth who

were in non-ExCEL-funded program components

were surveyed separately using the same tool.

Between these two data collection efforts, a total

of 1,532 surveys were collected from non-

duplicated Beacon youth. This is nearly two-times

the number of surveys that has been collected for

past Beacon evaluations, a success which is due

largely to the generous collaboration of Public

Profit.

The tool is designed to capture an “insider

perspective on program quality” from

participating youth, and asks questions that

correspond to SFUSD’s program expectations,

including perceived safety, skill development,

academic support and relationship-building. For

the purposes of the Beacon evaluation, index

scores were created by combining responses on

key questions (see panel to the right for a listing

of the questions that comprise each composite

index).

The index indicators were constructed to align

with the Beacon Initiative’s guiding principles, but

they are also consistent with the approach that

underlies the validated and nationally-recognized

Youth Program Quality Assessment (YPQA). This

approach posits that before authentic youth

engagement can take place (where youth are

really reached substantively and youth

development principles such as youth leadership

and voice can be enacted), quality interaction and

support must be in place. A precursor to quality

13

interaction and support is physical and emotional safety. In other words, quality in the higher

domains depends on quality in the lower domains, as demonstrated in the diagram below, which

shows the YPQA pyramid and how it aligns with SFBI’s pyramid for Learning without Limits.

It is anticipated that it would be easier for a program to achieve high ratings in the lower two

domains of safety and supportive environment than the higher domains of interaction and

engagement, where the more complex work of effecting positive youth outcomes would be

expected to take place.

Overall survey findings align with this assumption, with large majorities of Beacon youth

respondents indicating that they feel safe, that diversity is promoted, and that positive peer and

adult relationships are cultivated. These data help to establish that the program is approaching goal

#2 that, “all youth, including those who only drop in to Beacon Centers, will experience a safe space

that offers quality programs.”

Engagement

Interaction

Supportive Environment

Program Safety

83%

84%

86.47%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Positive Relationship Index

Diversity Index

Safety Index

Student Survey Findings: how many students agree with the statements in the...

Belonging, Safety and Acceptance

Curiosity to Knowledge

Learning without Limits

Growth

SFBI Creating Learning without Limits

14

As predicted by the YPQA model, smaller majorities of youth survey respondents indicated that

program participation promoted school engagement, skill development, youth development

principles like empowerment and self-expression, and academic achievement. These differences

along the hierarchy of domains are most pronounced among elementary school students who were

highly inclined to indicate positive impressions of program safety and diversity, but the least likely

to agree with statements in the youth development index (Skill Development is omitted because

questions pertaining to this domain were not included in the elementary school version of the tool).

Differences along the index measures are least pronounced among high school respondents who

were significantly (p<.005) more likely than younger students to indicate that programs nurtured

the development of skills, youth development principles, and academic support. This may reflect

the success of SFBI efforts in recent years to target older youth through the national Beacon Youth

Adolescents (BYA) initiative, which has included specific strategies to engage and retain older

youth through things like sequential skill-building programming and internships. There were no

significant differences found between hub and satellite sites on any of the survey indexes.

While Beacon programs show some room for growth in the higher-level areas of promoting

relevant skills and academic behaviors, it is important to recognize that the Beacons are doing

relatively well on all of these measures. Among all school levels, Beacon middle school responses

averaged lower than elementary or high school responses on all of the index measures. However,

Beacon middle school participant responses compare positively against those of non-Beacon ExCEL

middle school respondents. This is a worthwhile subpopulation to consider because middle school

students comprise over 50% of survey respondents and comprise a large portion of SFBI clients in

general. The Beacon-non-Beacon middle school analysis found that Beacon youth were:

Over 40% more likely to report new skills and experiences

0% 20% 40% 60% 80% 100%

Academic Support Index*

Youth Development Index*

Skill Development Index*

School Engagement Index

Relationship Index*

Diversity Index

Safety Index*

Student Survey Findings: mean percentage of students who agree with statements contained in the...

Elementary

Middle

High

*p<.005

15

Over 40% more likely to report improved academic behaviors Over 50% more likely to report increased school engagement About 30% more likely to report opportunities for engagement within program activities About 50% more likely to indicate positive impressions About 60% more likely to report that their programs promote diversity

These differences all reached the level statistical significance, implying that what is being observed

is more likely to be the result of a true dissimilarity than the result of sampling error or simple

chance. There were no significant differences between Beacon and non-Beacon middle school youth

in terms of reported safety, adult support, or support for skill-building.4

There were no gender differences found among Beacon respondents along perceived safety or any

other of the index measures, though the small number (4) of transgender-identified youth

respondents did indicate markedly more positive impressions of the programs in terms of youth

development, school engagement, relationship-building and safety.

Notably, when disaggregated by ethnicity (ethnicity data were available for 1,722 of the 1,992

youth surveyed), African American, Latino and other non-white students were more inclined to

indicate program strengths than were Asian or white student respondents. These differences are

statistically significant (p<.001) for both the Youth Development and Academic Support indexes,

and approach statistical significance on the Skill Development index (p=.09).

Looking more closely at student perceptions of safety inside and outside of the program, a pattern

emerges wherein Beacon students’ perceived safety during their commute home and in their

neighborhoods declines with age, with a precipitous low among 9th graders. Feelings of safety rise

again as youth progress through high school.5 The line graph below shows average (mean)

4 This information also provided by Public Profit, which had access to non-Beacon ExCEL student responses. 5 Among the 1,517 responses on which this chart is based, there were between 49 and 391 responses in each grade category, with lower numbers of respondents in the 3rd and 4th grade, higher numbers in 6th and 7th

.0000

.1000

.2000

.3000

.4000

.5000

.6000

.7000

.8000

.9000

Skill Development Index Youth DevelopmentIndex*

Academic Support Index*

Ethnic Breakdown on Higher-Level Survey Indexes

African American

Other non-white

Latino

Asian

White

16

perceptions of safety (with 1=Very Unsafe and 4=Very Safe) among Beacon youth in their

neighborhoods, going home, and in the program. As the graph demonstrates, student perceptions of

safety within the program are relatively consistent across all grades. While there is a slight decline

during the middle years, it is far more gradual, and 9th graders appear to be shielded from the

increased feelings of vulnerability during the time that they are in the program.

Family Skills, Engagement and Satisfaction Research has long supported the idea that family engagement is a critical predictor of students’

academic success (Dearing et al., 2004; Houtenville & Conway, 2008). Engaging families, however,

particularly in lower-income areas, can be a challenge, especially if parents feel intimidated by

school bureaucracy or cannot find an entry point into the school community (Bouffard, 2008; Crew,

2007). The Harvard Family Research Project has invested time and effort into this issue and has

identified a number of best practices for engaging family members.6 Among these practices is the

idea of offering adult classes at the school that are based on the needs and interests of parents and

the community (Westmoreland et al., 2009).

In line with this recommendation, Beacon parent and community programs are designed according

to the preferences and needs of the adults in the community, as identified by the neighborhood lead

agency, usually in collaboration with the hub school. Adult programming ranges from adult

education (e.g., ESL, computer literacy, etc.), to workshops to help parents support students more

effectively (e.g., parenting skills, School Loop tutorials, etc.) to healthy recreation (e.g., Tai Chi,

dance, yoga, ceramics, etc.). By design these programs are open to all community members, but a

grade. A total of 124 in the 9th grade responded to the listed safety questions, a high enough number to merit a moderate level of conclusiveness to the finding. 6 See Harvard Family Research Project’s Family Involvement Makes a Difference series at http://www.hfrp.org/publications-resources/publications-series/family-involvement-makes-a-difference

2.50

2.70

2.90

3.10

3.30

3.50

3.70

3.90

3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th

Perceptions of Safety Inside and Outside of Program by Grade Level

I feel safe in thisafter schoolprogram

I feel safe in myneighborhood

I feel safe goinghome

http://www.hfrp.org/publications-resources/publications-series/family-involvement-makes-a-difference

17

significant portion of adult participants are adult family members of students who attend Beacon

site schools or the Beacon program itself.

At total of 240 adult surveys were collected, 27% of which were from the family members of

Beacon or host school students. Because there is no requirement for adult program participants to

indicate whether or not they are family to a student at a Beacon or at a host school, there are no

accurate records for the number of family members that are served by the program. It is possible,

however, to extrapolate from survey findings and overall adults served by the program (8,783) and

arrive at an estimate that approximately 2,371 unduplicated family members were served by the

Beacons.

The survey indicated that large majorities of adult family member survey respondents feel more

positive about their children’s experience the school (83%) and about the after school program

(92%) because of their own participation in Beacon programs. This underscores the role that the

Beacons play in improving parents’ confidence in the school system and might be used by the

Beacon Initiative as it moves forward in further strengthening school partnerships.

Survey responses also show that a large majority of family members feel more comfortable

talking with school administrators, faculty and staff because of their participation in Beacon

programs (85%). Since finding an entry point to the school is a key barrier to family engagement

in their children’s education, this is an important finding.

A large proportion of family members (85%) report that at the Beacon they have learned to be a

better parent or family member. This is an interesting finding because most Beacon adult

programming is not specifically geared toward parenting skills. In fact, when asked about what they

expect to get out of Beacon programs, a large majority (73%) of family members indicate that they

are not expecting to learn about ways to help kids (their own or others). A cross-tabulation of

family members who learned something that helped them to be a better parent/family member

with those who expected to learn about helping kids, showed less of a difference than one might

85%

85%

83%

92%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

learn to be a better parent/family member

feel more comfortable talking with schooladministrators, faculty and staff

feel positive about the experience their childrenare having at the school

feel positive about the after school program at theschool

Family Connection to the School: because of the Beacon parents and family members...

18

expect. Family members with the expectation were more likely to “strongly agree” that they learned

something to help them be a better parent, but those with no such expectation were nearly as likely

to agree (81.6% v. 85.7%). So, while family members who expressly go to the Beacon to gain skills

and supports around parenting generally perceive that this expectation is met, family members

who were not specifically seeking these skills were nearly as likely to find that through the Beacon

Center they have gained some knowledge or capacity to be a more supportive family member. It

also implies that programming need not be explicitly geared toward parenting skills to improve

family supports.

I have learned something at the Beacon which has helped me to be a better parent/family member Expecting to Learn about Ways to Help Kids

Strongly Disagree

Disagree Neither Agree Strongly Agree

Total

No (38) 5.3% 5.3% 7.9% 42.1% 39.5% 100.0% Yes (12) 7.1% 7.1% .0% 21.4% 64.3% 100.0%

School Partnerships

Based on a thorough investigation of research and expert opinion, the Harvard Family Research

Project has concluded that in order for partnerships in education to be successful, a number of

factors must be in place, including, a shared vision, good relationships among partners, blended

staffing, regular data sharing, and strong connections with families and other community resources

(Harvard Family Research Center, 2010). The OMG Center for Collaborative Learning echoes these

ideas in its 2011 evaluation of the Hartford Community Schools:

“Evidence suggests that community schools have forged strong relationships and a

sense of trust at the community school director and principal levels, often calling

on one another for support and integrating efforts. Community school directors

indicate that they are continuing to develop stronger relationships with school

leaders. They attribute this development to better data-sharing, more access to

core school meetings and trainings, as well as seamless communication.” (OMG,

2011, emphasis added).

The Coalition for Community Schools concurs that these things are key for successful collaborations

between schools and community-based partners. The Coalition’s Rationale and Results Framework

suggests that assessing school-community partnerships should include looking closely at how

leadership, data use, relationship-building, planning and decision making, and parent engagement

are coordinated between schools and their community partners (Coalition for Community Schools,

2012).

Beacon Directors and School Principals at all eight Beacon hubs were asked to rate the depth and

collaborative nature of their partnerships, using an interview protocol guided by the Coalition for

Community Schools framework. Closed-ended questions prompted respondents to quantify a

number of measures on a 1-5 scale. These questions were combined to create key composites

around communication, shared goal-setting, shared data analysis and problem solving. A scale was

also created for the number of school-based meetings into which Beacon staff were integrated, and

19

both sets of parties were asked to rate how well this partnership compared with other school/CBO

partnerships they had observed, and their overall satisfaction with this partnership.

In general the results show very high levels of collaboration at all of the Beacon hub sites. Beacon

Principals were slightly more likely to perceive a large number of shared meetings and a high

degree of shared goal setting, data analysis and effective problem-solving when conflicts arise

between the Beacon and the school. Beacon Directors were slightly more likely to indicate that the

Beacon hub school partnership compared favorably to other school-CBO partnerships. On average,

both Beacon Directors and hub Principals rated satisfaction with the partnership at 4.75 out of 5.

Although there were no non-Beacon comparison sites that could be used to provide a backdrop

against which the above referenced scales might be understood, the interviews contained open-

ended questions, as well, in which interviewees provided specific insight into the strengths of the

partnerships. Among the themes that clearly emerged from the open-ended discussions were:

The value of developing a shared vision The recognition of roles and each other’s value The development of trust and mutual respect The importance to the CBO partner when the school makes a seat at the table The symbolic and practical significance of colocation

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

School Partnership Ratings: mean composite scores from Beacon Directors and Hub Principals

BeaconDirector

Principal

20

“[What is needed for a good

partnership is] communication,

alignment to school goals, distribution

of resources, sharing scheduling,

providing space, and meeting

frequently, once a week preferred. Also

inviting and attending each other’s

planning meetings, data analysis

activities, and events so you see first-

hand how the other is working. And

respecting one another and the work

we each do, and realizing that this is a

partnership that, when it works, is

extremely beneficial.” – Principal

On developing a shared vision, one Beacon hub Principal put it this way:

“They [the Beacon] have had a hand in crafting in the vision, so they know the vision not just

because I’ve told them the vision. They have a different angle from which they understand the

challenges and they can come to solutions based on that.

There’s a sense of appreciation that goes deeper when you're

working in close partnership and you can see the problems

from outside of just your own perspective – there’s

something to be said about that.”

In a similar vein, a Beacon Director said the following:

“The co-creating of the vision is really important - we're not

afraid to take a risk around power sharing - lose a little, gain

a lot. Build trust and keep the vision going. I can make

mistakes because we have a good relationship.”

It is no accident that shared visioning emerged as a prevalent

theme. The Beacon Initiative has pushed Lead Agencies to

engage in collaborative visioning with their hub schools, something that may not happen in every

school-CBO partnerships and had rarely happened at Beacon Centers before SFBI began to urge

Lead Agencies in this direction. Notably, ExCEL sites are also asked to do collaborative visioning

now.

Another prominent theme was the recognition of each entity’s role. Beacon Directors were

consistently emphatic that establishing role-clarity with their schools, and particularly with the

principal, was essential for the CBO partner to be seen as valuable to the school and for the

maintenance of a productive relationship. Below are some examples of Beacon Director statements

on the subject:

“Alignment which doesn't mean duplication. I would never expect us to provide

hard core academic support. But how do you take what youth need and use your

flexibility that teachers or school based staff don't have, and bring the information

to the Beacon space. If not, you're running a generic program. You don’t' want to just

extend the school day - it has to be something different. What can you provide that

the school can't (because of money, curriculum, willingness or unwillingness to go

down the youth engagement continuum - we have flexibility).”

“One is open communication, but also really clear communication. Understanding of

context, so as much work as we do to try to understand a school context, having

them work to understand our broader context (how funding works, etc.) - optimally

we should both have a good understanding of each other’s context.”

“[The principal] gets it. That is the most important support you can get from an

administration. He gets youth development principles… We are a part of the school -

we're not going anywhere… Schools didn't used to be opening up their doors to

21

community partners – the district has evolved in terms of recognizing that they can’t

do it alone. [The principal here] is very clear that they don't do it alone.”

In particular, Beacon Directors emphasized that the role they provided in reaching out to families,

was distinct from what school personnel could accomplish on their own.

“The biggest thing that principals love is people who have relationships with

parents, because [family members] are so difficult to engage.”

“Offering parent supports and programs draws parents and folks from the

community so the school recognizes the value of what we have to offer - beyond

what the school can offer by themselves.”

When asked about the value of the partnership, Principals frequently cited parent engagement and

outreach, as well, confirming the Beacon Directors’ impression that engaging parents was a way for

the Beacons to demonstrate their contribution to the school’s objectives.

Principals also talked about establishing trust and mutual respect, and emphasized that open

communication is key to establishing a good, trusting relationship:

“I really feel very comfortable and trust [the Lead Agency director and Beacon

Director] and we have a very open line of communication and I think that makes a

big difference. I know [the Beacon Director] goes out of her way to communicate

with me and keep me in the loop. [What is needed for a good partnership is] clear

and effective communication, relational trust, and shared mission and purpose.”

“It's a good partnership. There’s a lot of communication. They respond to needs that

we have and vice versa are good, there is mutual respect cooperation and

understanding. It's a pretty good marriage.”

Beacon Directors said similar things, underscoring how important it is to have the principal’s trust:

“The closeness and the trust and the reciprocity in terms of communication and

respect – I’ve never experienced anywhere else.”

“At other sites, where the trust isn't there, where we

are not invited to the table, we are constantly putting

out fires. It depends on the leadership of the school to

make the trust happen.”

The idea of giving the CBO partner a seat at the “table” was

echoed by several of the Principals. This principal explains

his view that without integrating the Beacon into the

structure of the school, and empowering the Beacon Director

as a true partner, the full benefit of the collaboration will not

be felt:

The first thing is you have to treat

them as equals, as true partners.

You have to give them space, and let

them know they're part of the act,

you have to empower them to be

part of the school. In a true

partnership you have to have open

discussions of goals, how you can

support each other, what you can

and can't do, and then it all falls

into place. - Principal

22

“One of the things that work with us is that we embedded the Beacon in the team

structure. It wasn’t that way in the beginning. I did have a hand in hiring the Beacon

Director, and then I created the weekly check in meetings, and I realized it wasn’t a

good approach. It was like I was supervising her. If I wanted to have her as a high

end partner, I had to approach this differently. I would encourage principals to

elevate the Beacon Director to the level that you work with an Assistant Principal.

Stop thinking of them as a CBO, but as a partner in the fabric of your school. I feel

free to call them into the office and ask them to do a favor. I feel that the Beacon staff

are at the same level as my teachers.”

Beacon Directors felt that a crucial way to integrate oneself into the school structure was to attend

meetings:

“Be present all the time - you have to be at every meeting from the ground up.”

“Attend school meetings -as many meetings as possible - get 5 minutes on the

agenda to keep the school community informed of what your program is doing.”

Colocation is important not just because of the practical aspects of having a space on campus, which

is invaluable, but also because of the symbolic stature it gives the CBO partner, and the

seamlessness it provides to students and families. Here three Beacon Directors explain why having

a space at the school is so meaningful:

“I feel like we're really integrated in the school. For example we are fixtures in

regular school meetings. I co-lead the student services department. I feel like we are

really seen as important players in terms of providing services. It helps being school

based. I don't think people make a distinction between school staff and beacon staff.

The space we have is a form of recognition - we have an office and it’s right near the

main office. I’ve seen other CBO programs where they come in for their program

time and don’t have a chance to connect with the school.”

“At other schools we have had space issues, and there are certain teachers who have

issues with it, but not as much here - the administration and teachers are more

willing to work with us. Here we are located within the school community -

colocation is important. Teachers come to us to get support on different kinds of

events, field trips etc.”

“Colocation - we love the fact that we can have dedicated space. Space and things

like that matters, but the other things are so critical, if the relationship and

expectations aren't there, it will crumble. Where if it is there the school will

advocate for you and what you need because they understand what those needs

are.”

In the interviews Principals were asked to rate their confidence that the collaboration with their

Beacon Centers was promoting good outcomes, supporting teachers, improving attendance, and

supporting the social and personal needs of students and families. Beacon Directors were asked

how confident they were that schools perceived that the Beacon brought value in these four areas.

23

Both school principals and Beacon Directors perceive that the Beacon Centers promote positive

youth and family outcomes and support the social and personal needs of students and families.

School principals believe that the Beacon impact on school attendance as high, but are slightly less

confident that Beacon programs support the work of classroom teachers. Beacon directors were not

confident that school personnel perceive their work as promoting either attendance or the work of

classroom teachers.

The one weakness in the Beacon-school partnerships that emerged from these interviews is

captured in the above table. The confidence among both school Principals and Beacon Directors

that classroom teachers in general recognized the work of the Beacon was not high. While many of

the Principals expressed an appreciation that the Beacon does have an impact on school

achievement (one Principal said, “Sometimes the only reason kids come to school is the Beacon”)

there was also a sense that school personnel don’t see the Beacon as promoting the academic

agenda. As one Principal put it, “The staff is not aware of some of the things the Beacon does.”

Another said, “I believe they do support classroom teachers, but because 1/3 [of Beacon

programming] is recreation, there is that tension.” Findings from this evaluation which

demonstrate the close relationship between Beacon participation and school day attendance

(especially for older youth) and the relationship between Beacon participation and academic

achievement (especially for younger students) might be used at the site level to improve teacher

impressions of the Beacon.

Community Impact Community programs at Beacon hubs include evening and weekend classes open to members of the

community. As mentioned earlier, Beacon Centers served a total of 2,519 unduplicated adults,

among whom it is estimated 73% were community members with no direct family link to the

1.00

2.00

3.00

4.00

5.00

Positive outcomesfor youth and

families

The work ofclassroom teachers

School attendance Social and personalneeds of students

and families

School Perceptions of Outcomes Promoted: do Beacon Centers support...

BeaconDirector

Principal

24

students enrolled at the school or the Beacon Center. Reaching these community members is part of

the Beacon (and broader Community Schools) design. This design follows a well-established body

of research that points to a strong correlation between the strength of social networks and a

population’s physical and mental health, academic achievement, and local economic development,

as well as lower rates of homicide, suicide, and alcohol and drug abuse (Buka, 1999; Kawachi &

Berkman, 2000; Wandersman, 1998). Research has shown that children tend to be mentally and

physically healthier in neighborhoods where adults talk to each other (Wilkerson, 1999), and in

Latino communities, high levels of social support are linked to a number of positive health benefits

(Policy Link, 2002). Closer social networks are associated with community members’ willingness to

participate in community-building activities, helping to supervise community children, and

maintaining public order (Putnam 2001; Putnam et al 1997). Participation in social networks also

promotes the enforcement of social sanctions by community members to diminish negative

behavior and reduce the incidence of crime, juvenile delinquency, and access to firearms within

communities (Putnam 1995; Veenstra, 2002).

Beacon Center data show that the ethnicity of adult

participants is somewhat different from that of the

youth served by the Initiative overall. Specifically, the

African American and Latino proportions are smaller,

and the White proportion is higher.

An adult survey was distributed at seven out of eight

Beacon Centers. The survey instrument was designed to

measure the extent to which adult program participants

perceive that their engagement with the Beacon Center

is enhancing their social networks and increasing their

sense of connection to the community, the schools, and

the youth in the neighborhood. The following findings are based on 240 adult surveys collected at

Beacon hub sites.

The survey found that substantial majorities of community program participants (over 90%) report

that the Beacon makes the neighborhood a more positive place and increases their sense of

connection to the community. A large majority (89%) indicated that through Beacon programs they

had met people from the neighborhood they otherwise would not have met, implying that the

Beacons are accomplishing the aim of strengthening social networks and creating a neighborhood

where adults talk with one another.

Smaller but still substantial majorities indicated that coming to the Beacon made them feel more

connected to the schools (84%) and kids in the neighborhood (69%). This is a remarkable finding

given that adult programming at the Beacon Centers seldom involves students or school personnel.

This implies that simply participating in activities housed in a school setting may alter how a

community member views that school and the children who go there.

Ethnicity %

American/Alaskan Native 0%

Asian 43%

Black 5% Hispanic/Latino 10%

Middle Eastern 1%

Multiracial/Multiethnic 1%

Pacific Islander 2%

White 16%

Other/None Reported 20%

Total 100%

25

Another piece of information that contributes to the understanding of what Beacon community

programs are achieving comes from the youth survey. Student survey results show that youth

enrolled in ExCEL programs at Beacon hub sites, where the community programs take place, are

significantly more likely to report feeling “very safe” on their way home from school than those at

sites where no community programming takes place (53% versus 38%). While this finding is not

enough to establish a causal relationship between community programs and neighborhood safety,

it is a potentially important discovery that supports the assumption that community programming

may improve the safety of the neighborhood for youth.

Conclusion The San Francisco Beacon Initiative is premised on a complex model for the support of children and

youth in San Francisco. Some of the supports offered by the Beacon Centers are meant to directly

benefit youth (e.g., homework help, recreation, youth development), while others are designed to

benefit them more indirectly (i.e., building a strong relationship between the Beacon Center

leadership and the school leadership so that students and families experience seamless and

consistent support, supporting family members as they work to support students, and fostering a

healthy social environment in the surrounding neighborhood so that students are surrounded by a

safe and caring community). The model aligns with and is founded upon research, but past

evaluation efforts have not conclusively established that the San Francisco Beacon Initiative bears

out the assumptions upon which it is built. This evaluation was able to show that, at the multiple

levels on which SFBI work takes place, Beacons are having the impact they seek.

69%

84%

89%

92%

94%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Coming to the Beacon helps me feel more connected to thekids in this neighborhood

Coming to the Beacon helps me feel more connected to theschools in this neighborhood

At the Beacon I have met people from the neighborhood Iprobably would not have met otherwise

Coming to the Beacon helps me feel more connected to thecommunity

I believe that having a Beacon Center helps make thisneighborhood a more positive place

Community Connection: percentage of respondents who agree with the following statements

26

Goal 1: Through consistent participation in Beacon programs youth will gain skills, develop supportive

relationships, and show growth on youth development and academic measures

This evaluation found that consistent participation in Beacon programs has an overall positive and

statistically significant impact on school day attendance. This relationship was found to be stronger

for middle and high school youth, those in Special Education, and those designated

Socioeconomically Disadvantaged. The data also showed an overall positive and statistically

significant relationship between program participation and academic gains. These findings were

stronger for elementary school and less disadvantaged students. Student survey findings indicated

that students developed supportive relationships, gained academic and personal expression skills,

and grew on youth development measures, as well. High school youth and African American

students were significantly more likely to affirm that the program provided academic support and

fostered youth development.

Goal 2: All youth, including those who only drop in to Beacon Centers, will experience a safe space

that offers quality programs

Student survey findings showed that very large majorities of students, including those who may not

participate consistently, experience the Beacon Centers as safe places. Respondents of all ages

tended to report that they felt safe at the Beacon Centers – this was significant because 9th graders

indicated on the same survey that they felt substantially less safe than younger and older students

in their neighborhoods and while going home. Vast majorities also indicated that the Beacon

Centers are places where diversity is promoted and positive relationships are nurtured.

Goal 3: Beacon programs will help family members navigate schools more effectively, gain skills and

connect with the community

A survey of family members participating in Beacon programs revealed that Beacon programs help

families feel more at ease talking with school administrators, faculty and staff. It also showed that

family members perceive that participating in the Beacon programs equips them to be better family

members, even when they are participating in programs that are not expressly geared toward

increasing parenting skills. In interviews school principals affirmed that the work of the Beacons

increases the schools’ ability to conduct outreach and engage families.

Goal 4: Host hub schools and Beacon lead agencies will be more coordinated and aligned

Interviews with Beacon Directors and Principals showed a large degree of agreement on criteria for

healthy school-CBO partnerships, including strong two-way communication, shared visioning, and

inclusion of CBO partners in regular school meetings. This evaluation has helped to establish a

model for school-CBO partnership. To be truly persuasive, however, that the partnerships at Beacon

hub schools are objectively robust, future evaluations should include a comparison group of school-

CBO partnerships that are not supported by the structure that SFBI offers.

Goal 5: Community programs will improve the health of communities and increase community

investment in youth, families and the school

A survey of 240 adult participants showed that adults who participate in Beacon programs make

more neighborhood connections, and feel more affinity to the community, the school, and the kids

in the neighborhood. Findings from this evaluation support the long-held Beacon philosophy that

27

school-based community programming is ultimately valuable to the school community and to

students.

While the overall findings from this evaluation are very positive, room for growth did appear in

some areas.

Promoting Academic Achievement with Vulnerable Sub-Groups

Beacon participation was found to have a measurable positive relationship with school-day

attendance. This finding that was supported by student survey results and Principal interviews, and

the relationship was found to be strongest among African American youth and students designated

Special Education or Socioeconomically Disadvantaged. These findings are generally very

encouraging, but the data showed that for these same sub-groups greater Beacon participation did

not translate to academic gains as much as it did for youth not in these categories. Why Beacon

participation would be associated with better attendance but not improved academic achievement

for youth in these categories may have more to do with the multiplicity of challenges some of these

students may be facing than with a specific shortfall of the programs. Nevertheless, the finding

provides a valuable opportunity for Beacon Directors and the Initiative as a whole to consider ways

they might more effectively reach all youth, particularly those in these important subgroups.

Extending School Partnership into the Classrooms

Interviews with Beacon Directors and Principals revealed a very strong set of relationships and

working collaborations at the hub schools. But both Principals and Beacon Directors indicated that

they perceive classroom teachers to be less convinced of the value that the Beacons bring to the

school. Both sets of interviewees acknowledged that there are some teachers at the school that are

Beacon advocates, and that Beacon participation in regular school meetings and teacher

involvement in Beacon staffing promote more trust among classroom teachers. But there was also

an acknowledgement that other teachers either did not know much about Beacon programs or did

not recognize that what happens in the Beacon can support their work in the classroom. Given that

the evaluation found positive correlations between program participation and school attendance

and achievement, this also appears as an area for reflection moving forward.

Looking More Closely at the Meaning of Quality School Partnerships

Finally, this evaluation revealed that while the relationships between Beacon lead agencies and

their host schools are strong, without the ability to compare the strength of those relationships

with other, non-Beacon school-CBO partnerships, the apparent quality is less meaningful. Future

evaluation efforts should include interviews with program directors and school Principals from a

comparable list of non-Beacon schools for a meaningful comparison.

28

References Arbreton, A. et al. (2009). Making Every Day Count: Boys & Girls Clubs’ Role in Promoting Positive Outcomes for Teens. Philadelphia: Public/Private Ventures. Retrieved 10/30/2012 http://www.sp2.upenn.edu/ostrc/docs/document_library/cade/Adolescents%20and%20Youth%20Culture/Making%20Every%20Day%20Count%20-%20Teens.pdf

Benson, P.L., Scales, P.C., et al. (2006). “Positive Youth Development So Far: Core Hypotheses and Their Implications for Policy and Practice.” Insights & Evidence. Search Institute. November 2006, vol 3, no. 1.

Bouffard, S. (Ed.). (2008). “Building the future of family involvement.” The Evaluation Exchange, 14(1 & 2).

Bronfenbrenner, U. (1977). “Toward an experimental ecology of human development.” American Psychologist, 32, 513-531.

Bronfenbrenner, U. (1979). The Ecology of Human Development: Experiments by Nature and Design. Cambridge, MA: Harvard University Press.

Buka S. 1999. “Results from the project on human development in Chicago neighborhoods.” Presented at: 13th Annual California Conference on Childhood Injury Control; October 25-27, 1999; San Diego, CA.

Coalition for Community Schools. “Community Schools: Promoting Student Success, A Rationale and Results Framework.” Retrieved 11/9/12 www.communityschools.org/assets/1/AssetManager/CS_Results_Framework.pdf.

Crew, R., with Dyja, T. (2007). Only connect: The way to save our schools. New York: Farrar, Straus and Giroux.

Dearing, E., McCartney, K., Weiss, H. B., Kreider, H., & Simpkins, S. (2004). “The promotive effects of family educational involvement for low-income children’s literacy.” Journal of School Psychology, 42, 445–460.

Eccles, J. S., & Gootman, J. A. (Eds.). (2002). Community programs to promote youth development. Washington, DC: National Academy Press.

Harvard Family Research Project (2010). Partnerships for Learning: Profiles of Three School-Community Partnership Efforts. Retrieved 11/19/2012 http://www.hfrp.org/out-of-school-time/publications-resources/partnerships-for-learning-profiles-of-three-school-community-partnership-efforts

Harvard Family Research Project. (2004). "Understanding and Measuring Attendance in Out-of-School Time Programs." Issues and Opportunities in Out-of-School Time Evaluation. Number 7, August 2004.

Houtenville, A. J. & Conway, K. S. (2008). “Parental effort, school resources, and student achievement.” Journal of Human Resources, 43(2), 437–453.

Kail, R. V., & Cavanaugh, J. C. (2010). The Study of Human Development. Human Development: A Life-span View (5th ed.). Belmont, CA: Wadsworth Cengage Learning.

Kauh, Tina J. (2011). AfterZone: Outcomes for Youth Participating in Providence’s After-School System. Public/Private Ventures. Retrieved 10/30/2012 http://www.wallacefoundation.org/knowledge-center/after-

http://www.sp2.upenn.edu/ostrc/docs/document_library/cade/Adolescents%20and%20Youth%20Culture/Making%20Every%20Day%20Count%20-%20Teens.pdf

http://www.sp2.upenn.edu/ostrc/docs/document_library/cade/Adolescents%20and%20Youth%20Culture/Making%20Every%20Day%20Count%20-%20Teens.pdf

http://www.communityschools.org/assets/1/AssetManager/CS_Results_Framework.pdf

http://www.hfrp.org/out-of-school-time/publications-resources/partnerships-for-learning-profiles-of-three-school-community-partnership-efforts



http://www.wallacefoundation.org/knowledge-center/after-school/evaluations/Documents/AfterZone-Outcomes-Youth-Participating-Providences-Citywide-After-School-System.pdf

29

school/evaluations/Documents/AfterZone-Outcomes-Youth-Participating-Providences-Citywide-After-School-System.pdf

Kawachi, I. and Berkman, L. “Social Cohesion, Social Capital, and Health.” In: Berkman LF, Kawachi I eds. Social Epidemiology. New York: Oxford University Press; 2000.

OMG Center for Collaborative Learning (2011). Hartford Community Schools Evaluation. Retrieved 12/10/2012 http://www.communityschools.org/assets/1/AssetManager/HCS%20Final%20Report%20(2-6-12).pdf

Philliber Research Associates and the American Camp Association. (2005). Directions: Youth Development Outcomes of the Camp Experience. Retrieved 10/30/2012 http://www.acacamps.org/sites/default/files/images/research/directions.pdf

PolicyLink. Reducing health disparities through a focus on communities. A PolicyLink Report. Oakland, CA: 2002.

Putnam R. 1995. “Bowling alone: America’s declining social capital.” Journal of Democracy. 1995;6:65-78.

Putnam Sampson RJ, Raudenbush SW, Earls F. 1997. “Neighborhoods and violent crime: a multilevel study of collective efficacy.” Science. 1997;277:918-924.

Putnam, R.D. 2001. “Social Capital: Measurement and Consequences.” ISUMA: Canadian Journal of Policy Research. 2(Spring 2001):41-51.

Santrock, John W. (2007). A Topical Approach to Life-Span Development. New York, NY: McGraw-Hill.

Simpkins, S. (2003). “Does youth participation in out-of-school time activities make a difference?” The Evaluation Exchange, 9(1).

Veenstra G. 2002. “Social capital and health (plus wealth, income inequality and regional health governance).”. Social Science & Medicine. 54 (2002) 849–868.

Vile, J.D., Arcaira, E. & Reisner, E.R. (2009). Progress toward high school graduation: Citizen Schools’ youth outcomes in Boston. Washington, D.C.: Policy Studies Associates, Inc.

Wandersman A, Nation M. 1998. “Urban neighborhoods and mental health: psychological contributions to understanding toxicity, resilience, and interventions.” American Psychologist. 1998;43:647-656.

Westmoreland, H., Rosenberg, H.M., Lopez, E., and Weiss, H. 2009. "Seeing is believing: promising practices for how school districts promote family engagement." Harvard Family Research Project Issue Brief, July 2009.

Wilkenson R. 1999. “Income inequality, social cohesion, and health: clarifying the theory – a reply to Muntaner and Lynch.” International Journal of Health Services. 1999;29:525-545.



http://www.communityschools.org/assets/1/AssetManager/HCS%20Final%20Report%20(2-6-12).pdf

http://www.communityschools.org/assets/1/AssetManager/HCS%20Final%20Report%20(2-6-12).pdf

http://www.acacamps.org/sites/default/files/images/research/directions.pdf

30

Appendix – Survey Tools