power of d stem year 1 (2011-12) report of ......stem programming implementers who serve as mentors...

UNIVERSITY OF CALIFORNIA, IRVINE

BERKELEY • DAVIS • IRVINE • LOS ANGELES • MERCED • RIVERSIDE • SAN DIEGO • SAN FRANCISCO SANTA BARBARA • SANTA CRUZ

School of Education 3200 Education Irvine, CA 92697-5500 (949) 824-5118 (949) 824-9103 FAX

POWER OF DISCOVERY: STEM2 YEAR 1 (2011-12)

REPORT OF FINDINGS

OCTOBER 2012

Deborah Lowe Vandell, Principal Investigator Valerie Hall, Ph.D.

Maria del Pilar O’Cadiz, Ph.D. Andrea Karsh, M.A.

Power of Discovery: STEM2 Year 1 Report i

TABLE OF CONTENTS

SECTION I — INTRODUCTION 1 SECTION II — METHODS 3 SECTION III DESCRIPTIVE STATISTICS 8 SECTION IV — PROGRAM STAFF REPORTS 9 A. STAFF CHARACTERISTICS 9 B. PROGRAM PRACTICES 11 C. STAFF BELIEFS AND ATTITUDES 13 SECTION V — QUANTITY AND QUALITY OF PROGRAM STEM ACTIVITIES 14 A. STEM ACTIVITY DOCUMENTATION FORM DATA 14 B. YOUTH REPORTS OF QUALITY OF PROGRAM EXPERIENCES 16 SECTION VI — STUDENT OUTCOMES 18 A. STUDENT SELF-REPORTS OF PRE/POST OUTCOMES 18 B. ASSOCIATIONS BETWEEN STEM-RELATED FACTORS & STUDENT OUTCOMES 21 C. ASSOCIATIONS BETWEEN PROGRAM EXPERIENCES & STUDENT OUTCOMES 22 SECTION VII — CONCLUSIONS AND RECOMMENDATIONS 26 REFERENCES 29 APPENDICES 31 APPENDIX A. DEMOGRAPHIC PROFILE OF PILOT STUDY SITES 31 APPENDIX B. COMPUTER AND INTERNET ACCESS AT PILOT STUDY SITES 36

Power of Discovery: STEM2 Year 1 Report ii

LIST OF TABLES AND FIGURES

Figure 1.1 Logic Model for the Power of Discovery: STEM2 Initiative 2 Table 2.1 Power of Discovery: STEM2 Year 1 Evaluation Sites 4 Table 2.2 Gender and Grade Level, Student Respondents 5 Table 3.1 Descriptive Statistics 8 Figure 4.1 Race and Ethnicity of Program Staff 9 Table 4.2 Experience in Current Position 10 Table 4.3 Instruction Implemented in Current Position 10 Table 4.4 Implementation of STEM Activities in Current Position 11 Figure 4.5 Time Implementing STEM Per Week 11 Figure 4.6 Frequency of Staff Meetings About STEM 12 Figure 4.7 Staff Interactions with Teachers About STEM Topics 12 Figure 4.8 STEM-Related Events with Parents 13 Table 4.9 Program Staff Beliefs and Efficacy Scores 13 Figure 5.1 Number of STEM Activities, by Student Grade Level 14 Figure 5.2. Number of STEM Activities, by STEM Area Focus 15 Figure 5.3. Level of Student Engagement in STEM Activities, by Level of Challenge 16 Table 5.4 Program Experiences, Student Self-Reports 17 Table 6.1 Student Outcome Scores, Skill Development 19 Table 6.2 Student Outcome Scores, Attitudes and Beliefs 20 Table 6.3 Student Outcome Scores, Positive Behavior 21 Table 6.4. Effects of Staff Beliefs about STEM on Student Post-Participation Outcome Scores 22 Table 6.5. Effects of Quality of Program Experiences on Student Post-Participation Outcome Scores 23 Figure 6.6 Associations Between Program Experiences and Outcomes: Math and Science Efficacy 24 Figure 6.7 Associations Between Program Experiences and Outcomes: Work Habits, View of Future 24 Figure 6.8 Associations Between Experiences and Outcomes: Social Competencies, Misconduct 25

Power of Discovery: STEM2 Year 1 Evaluation Report, October2012

1

SECTION I – INTRODUCTION

The Power of Discovery: STEM2 Initiative is an ambitious, comprehensive project that seeks to transform young people’s engagement, interest, and understanding of science, technology, engineering and mathematics (STEM) subject matter by capitalizing on the opportunities that could be afforded by high quality experiences during out-of-school time.

To accomplish this agenda, the Power of Discovery: STEM2 Initiative seeks to increase the

capacity of Out-of-School-Time (OST) programs in California to offer rich, hands-on learning opportunities in the STEM domain. Multiple strategies, including technical assistance from Regional STEM programming implementers who serve as mentors and coaches to OST programs, the development of a Virtual Innovation Center to provide on-line assistance to programs, and systematic training and professional development for OST staff, are being implemented in afterschool programs across the State of California. The ultimate goal of the Power of Discovery: STEM2 Initiative is to specifically improve students’ STEM engagement, interest, and understanding as well as academic and youth development more generally.

Figure 1 graphically represents the logic model that guides the Power of Discovery: STEM2

Initiative. The model is a sequential one in which Initiative training and support activities in Staff Professional Development, Curricular Innovations, and Online Virtual Supports (the blue boxes on the left hand side of the figure) are expected to yield improvements in Beliefs and Competencies of program staff and in the quantity and quality of Program Offerings in STEM subjects. These improvements in Staff Competencies and Program Offerings are expected to be mutually reinforcing. The Staff Competencies and Program Offerings are then expected to yield desired improvements in students’ outcomes as measured by interests, engagement, and career aspirations in the STEM domain.

The determination of the success of Power of Discovery: STEM2 Initiative requires a

comprehensive evaluation strategy that provides a careful documentation of the implementation of the Power of Discovery: STEM2 Initiative AND a determination of the effects of the Initiative on the quantity and quality of program activities, program staff, and young people.

To this end, the Year 1 evaluation was conducted by two organizations. Researchers at the

University of California Davis were responsible for the documentation of the implementation of the Professional Development, Curricula Innovations, and Virtual Innovation Center activities—the “treatment” designed to drive the STEM in OST improvements. Researchers at the University of California Irvine were responsible for the determination of the effects of the “treatment” on program activities, program staff, and students.

In its work during Year One, the UC Irvine research team sought to address two broad issues.

The first was to establish the feasibility of a state-wide study of STEM-related beliefs and practices. Much of the previous research and evaluations had been conducted at a small number of sites. The issue was whether these measures could be administered reliably across the State at a large number of sites serving a large number of diverse students. The second issue was whether the first year data could be used to identify early trends in the effects of the Power of Discovery: STEM2 Initiative on program activities, program staff, and students.


2

Figure 1.1 Logic Model for the Power of Discovery: STEM2 Initiative

In this report of Year One, findings pertaining to both of these broad issues are presented as they

pertain to Program Offerings, Staff Competencies, and Student Outcomes. Each of these areas represents a critical and necessary research activity. All are inter-related, and many of the core analyses combine information across aims.


3

SECTION II – METHODS

Participating Programs

A total of 15 (of the 17) participating JumpStarting STEM programs were recruited to participate in the Year 1 phase of the Power of Discovery: STEM2 Initiative. These programs were distributed across nine Regions (1, 2, 3, 4, 7, 8, 9, 10, 11). Table 2.1 provides a list of all participating programs and sites. As indicated on Table 1, the sample consisted of 47 elementary, 9 middle school and 4 K-8 program sites. Twelve different STEM curricula were utilized at different study sites in addition to other STEM curricula that programs chose to implement.1

The study sites were selected to ensure a diverse demographic profile representative of students in the State of California with regard to student ethnic and socio-economic backgrounds, English Language Learners, and Free and Reduced Lunch (FRL). Sites ranged from 46% to 98% students on FRL and from having no students designated English Language Learners (ELL) to up to 87% ELL students [see Appendix A for demographic profile of participating sites].

In order to participate in the study, sites needed to have access to the Internet to administer the online student and staff surveys. See Appendix B. Computer and Internet access at participating sites. Staff Participants

During the Year 1 Evaluation of the Power of Discovery: STEM2, 135 program staff completed the online surveys (88 pre-participation; 47 post-participation). A total of 37 program staff completed 990 STEM Activity Documentation Forms.

Student Participants

A total of 1029 students in grades 3-12 completed surveys at Time 1 (Fall 2011 or early Winter 2012) and 1,278 completed Time 2 surveys in late Spring 2012. About half of the students were female (49% PRE; 52% POST), and the majority of students were in grades 3-5 (87% PRE; 83% POST). Table 2.2 provides the gender and grade level distributions of the student participants.

Students were able to complete the pre- and post-participation surveys in a reasonable amount of time. 90% of all students report completing each survey in 15 minutes or less; 68% report taking 9 minutes or less. About 9-10% of students report completing each survey in 5 minutes or less. The majority of students also report that the questions on the surveys are easy to read, understand, and answer. For both pre- and post-participation surveys, about 85-86% of all students indicate it is “mostly true” or “really true” that the questions are easy to read, understand, and answer; only 4-6% indicate it is “not at all true.”

1CEES report: 71% of program administrators report implementing other STEM activities, in addition to JumpStarting STEM curricula.


4

Table 2.1 Power of Discovery: STEM2 Year 1 Evaluation Study Sites

Region Participating Programs Total # of Program

Sites

Elementary Study Sites

Middle School Study Sites

Total Study Sites

1 BOYS & GIRLS CLUBS OF CENTRAL SONOMA COUNTY

9

Calmecac (K-8) Healdsburg Cloverdale

Calmecac (K-8)

3

2 BUTTE COUNTY OFFICE OF EDUCATION RURAL COUNTIES

24

Cedarwood Wyandotte Four Winds (K-8)

Ishi Hills Palermo Four Winds (K-8)

5

3 GIVE EVERY CHILD A CHANCE Central Valley Manteca Unified School District (10); San Joaquin County Office of Education (3)

13 Golden West (K-8) August Knodt Banta

Golden West (K-8)*

3

3 TWIN RIVERS SCHOOL DISTRICT, Sacramento

6 0 MLK Jr. Tech Academy

1

4 BAY AREA COMMUNITY RESOURCES Oakland, San Rafael, San Francisco, AUSD, OUSD, WCCUSD, San Rafael Unified, SFUSD (4 districts)

30

ML King Wilson Ruby Bridges Guadalupe San Pedro

Davidson 6

4 BAY AREA AFTERSCHOOL ALL STARS San Jose

5

Grant Lowell

0 2

7 PRO-YOUTH HEART Visalia

12

Anne R. Mitchell Mountain View Washington Houston Strathmore Woodville

0 6

8 BRIGHT TOMORROWS Lucia Mar Unified School District

10 Nipomo Grover Beach Oceano

0 3

9 SAY (Social Advocates for Youth), SDCOE, San Diego USD

12 Jones Encanto

0 2

9 ANAHEIM ACHIEVES YMCA Anaheim City SD, Magnolia SD, Savanna SD

23

Ross Gauer Cerritos Disney

Mann Ball

6

9 THINK TOGETHER Tustin USD (6); Santa Ana USD (15)

6; 15 Heideman Veeh Walker King Greenville Washington

0 6

10 THINK TOGETHER Moreno Valley USD; Ontario USD

19; 23 Butterfield Rainbow Ridge Central (K-8)

Central (K-8) Danks MS

4

10 CAPS—Creative Before and Afterschool Programs For Success, San Bernardino USD

21 Cypress Hunt Lytle Creek

0 3

11 WOODCRAFT RANGERS Los Angeles USD

23 Nevada 99th Street

Nimitz 3

11 POMONA UNIFIED SCHOOL DISTRICT

27 Alcott Allison Arroyo Harrison Westmont Vejar

Madison

7

TOTAL: 15 Participating Programs (47 Elem.; 9 MS; 4 K-8 Study Sites) 60


5

Table 2.2 Gender and Grade Level, Student Respondents PRE POST N % N % Gender Female 501 49% 665 52% Male 591 51% 613 48% Grade Level 3rd grade 241 24% 315 25% 4th grade 339 33% 363 28% 5th grade 309 30% 382 30% 6th grade 108 11% 121 10% 7th grade 10 1% 59 5% 8th grade 10 1% 32 3% 9th-12th grade 3 .3% 6 1%

Measures

Staff Surveys—During the Fall survey administration, staff reported their educational and professional background and experience implementing STEM in afterschool program settings. In Fall and Spring, staff reported on the professional development and support they received around STEM, and the kinds of STEM activities they implemented at their program site. They also reported their beliefs about the value of STEM and their perceived competencies in delivering STEM-related activities to students at their program sites.

Two measures on the pre- and post-participation Staff Surveys were Beliefs about STEM in the afterschool program and Efficacy Implementing STEM activities. Items for the two measures were rated using a 5-point rating scale (1= strongly disagree, 3= neither agree/nor disagree, 5 =strongly agree). Examples of items from the Beliefs about STEM measure include: “I think the students enjoy doing STEM activities, and “I don’t think there is enough time at the program for students to learn much about STEM.” Items from the Efficacy Implementing STEM measure include: “I have a strong background in at least one area of STEM” and “I do not know enough about Science, Technology, Engineering and/or Mathematics to teach any of them well.”

Stem Activity Documentation Form—This assessment was created and pilot tested as a possible means of documenting the nature of the actual activities being implemented at the study sites. STEM implementers were provided with hard copy forms and instructions and prepaid and addressed envelopes for returning completed forms to UC Irvine. The forms allowed staff to record the following data about each STEM activity they implemented: (1) Date and duration of each activity; (2) Number of students participating in a given activity; (3) Name of activity and STEM content area addressed; (4) Four-point ratings of level of student engagement, level of challenge, and overall assessment of success of activity.

Student Reports of Program Experiences —As part of the post-participation survey, students were asked to report the quality of their experiences at their program. The Student Self-Report of Program Experiences measure consists of 16 items that are rated using 4-point rating response: (1) not at all true; (2) a little true; 3) mostly true; (4) really true. The measure included two subscales, Staff &


6

Activities and Peer Affiliation. Examples of items in the Staff & Activities experiences subscale are “I like the activities here” and “I trust the teachers here”. The Peer Affiliation subscale measures students’ experiences with other students in the afterschool program with items such as “I get to know other kids really well here”.

Student Surveys—This student self-report outcome measure included valid and reliable scales

used in previous studies to measure changes in students’ beliefs, attitudes about STEM, their curiosity and engagement in STEM activities, their skill development and STEM career aspirations, and students’ experiences with staff and activities in the afterschool program. The pre- and post-participation student surveys included eight measures of self-reported outcomes, in the following three categories: (1) Skill Development, (2) Attitudes and Beliefs, and (3) Positive Behaviors. Students reported their own skill development in the three areas of work habits, math efficacy, and science efficacy. Students reported their attitudes and beliefs in the three areas of science interest, science career, and view of future. Students reported their positive behaviors in two areas: social competencies and misconduct (with low levels of misconduct viewed as a positive indicator).

These assessments are described below.

1. SKILL DEVELOPMENT

Work Habits - The student self-report of Work Habits is an adaptation of the teacher and

program staff report Mock Report Card: Work Habits (Pierce et al., 1999). The six items are reworded for the student self-report and the responses assessed on a 4-point scale (1 = not at all true, 4 = really true). Sample items on the work habits scale include: “I work well by myself” and “I finish my work on time.” An overall work habits score is computed as the mean of the items. Cronbach’s coefficient alpha in the California Afterschool Outcome Measures Project was .77 for elementary and .80 for middle school student reports.

Math Efficacy - Math Efficacy includes four items such as “I expect to do well in math” and “I

am interested in math.” The response scale is a 4-point scale (1 = not at all true, 4 = really true). An overall score is computed as the mean of the items. The Math Efficacy scale was adapted from a measure developed by Eccles and colleagues (Eccles et al., 1993; Jacobs et al., 2002) to assess efficacy or competence beliefs. The original 7-point scale was changed to a 4-point scale. Cronbach’s coefficient alpha in the California Afterschool Outcome Measures Project was .85 for elementary and .88 for middle school student reports.

Science Efficacy - Science Efficacy includes four items such as “I expect to do well in science”

and “I am interested in science.” The response scale is a 4-point scale (1 = not at all true, 4 = really true). An overall score is computed as the mean of the items. The Science Efficacy scale was adapted from a measure developed by Eccles and colleagues (Eccles et al., 1993; Jacobs et al., 2002) to assess efficacy or competence beliefs. The original 7-point scale was changed to a 4-point scale. The Cronbach’s coefficient alpha is .89-.91.


7

2. ATTITUDES AND BELIEFS

Science Interest (PEAR) - The student self-report of Science Interest was developed at the Harvard University Program in Education, Afterschool & Resiliency (PEAR), and includes 24 items such as “I get excited about learning about new discoveries or inventions,” and “Science is one of my favorite subjects.” It is assessed on a 4-point scale (1 = not at all true, 4 = really true). An overall score is computed as the mean of the items. The Science Career subscale includes twelve items such as “I will get a job in a science-related area.” (Cronbach’s coefficient alpha .78 - .94).

Science Career and View of Future - The student self-report of View of Future includes two

subscales: Science Career and View of Future. Both subscales are assessed on a 4-point scale (1 = not at all true, 4 = really true). An overall score for each is computed as the mean of the items for that subscale. The Science Career subscale includes 12 items such as “I will get a job in a science-related area” (Cronbach’s coefficient alpha .78 - .94) and is derived from Tyler-Wood, Knezk, & Christensen, 2010. The View of Future subscale includes seven items. A sample item is “I will go to college.” The items are adapted from a set used in the National Youth Survey – Prediction of Adult Success (Elliott & Menard, 1996).

3. POSITIVE BEHAVIOR Social Competencies - Social Competencies are assessed by seven items such as “I work well

with other kids” and “I can tell other kids what I think, even if they disagree with me.” These skills are rated on a 4-point scale (1 = not at all true, 4 = really true) and an overall social competencies score is computed as the mean of the items. The scale is derived from the Social Self-Efficacy scale on the Self-Efficacy Questionnaire for Children (SEQ-C) by Muris (2001) to measure youths’ perceived capability for peer relationships and assertiveness. The original response scale was a 5-point scale. Cronbach’s coefficient alpha in the California Afterschool Outcome Measures Project was .69 for elementary and .80 for middle school student reports.

Misconduct - Misconduct Scale items are rated on a 4-point scale (0 = never, 3 = more than once a week). Sample items include “I have gotten into a fight at school” or “I have taken something that belongs to someone else.” An overall misconduct score is computed as the mean of the nine item scores. The Misconduct Scale is an adaptation of the Self-Reported Behavior Index (Brown, Clasen, & Eicher, 1986) created by Posner and Vandell (1994) for the longitudinal Ecological Study of After-School Care. Cronbach’s coefficient alpha in the California After School Outcome Measures project was .82 for elementary student reports and .83 for middle school student reports. Analysis Plan

The Power of Discovery: STEM2 Year One findings are presented below in three sections. First, a summary of program staff background data are presented followed by a summary of program staff pre/post outcome data and the STEM Documentation Forms. In the third section, student pre/post outcome data are presented including pre/post mean scores and pre/post categorical scores for the whole sample.


8

SECTION III – DESCRIPTIVE STATISTICS

Table 3.1 provides the descriptive statistics and psychometrics for all of the measures on the Program Staff Reports and the Student Self-Reports. All of the measures have good distributions of scores and good internal consistency for both pre-participation and post-participation survey administrations. The Cronbach’s alphas for the measures are also strong. Overall, the descriptive statistics and psychometrics show that these measures will work well for the Power of Discovery: STEM2 Year 2 evaluation.

Table 3.1 Descriptive Statistics

Pre-Participation Post-Participation n Mean SD Range Alpha n Mean SD Range Alpha Program Staff Reports

Beliefs about STEM 88 3.71 0.46 2.50-5.00 .68 45 3.72 0.48 2.33-4.78 .67 Efficacy Implementing STEM 88 3.73 0.62 2.14-5.00 .84 45 3.89 0.51 2.83-5.00 .72

Student Self-Reports

Work Habits 1,020 3.12 0.66 1.00-4.00 .81 1,278 3.08 0.70 1.00-4.00 .84 Math Efficacy 983 3.29 0.77 1.00-4.00 .83 1,254 3.20 0.84 1.00-4.00 .86 Science Efficacy 982 3.14 0.84 1.00-4.00 .86 1,253 3.07 0.89 1.00-4.00 .89 Science Interest (PEAR) 974 3.06 0.61 1.00-4.00 .93 1,250 3.01 0.66 1.00-4.00 .94 Science Career 957 2.91 0.80 1.00-4.00 .84 1,223 2.88 0.83 1.00-4.00 .85 View of Future 961 3.63 0.53 1.00-4.00 .92 1,224 3.63 0.58 1.00-4.00 .94 Social Competencies 989 2.94 0.66 1.00-4.00 .73 1,257 2.92 0.69 1.00-4.00 .76 Misconduct 1,006 1.43 0.50 1.00-4.00 .85 1,275 1.48 0.51 1.00-4.00 .84 Staff & Activities 1,212 3.10 0.68 1.00-4.00 .87 Peer Affiliation 1,207 3.15 0.78 1.00-4.00 .88


9

SECTION IV – PROGRAM STAFF REPORTS

A. STAFF CHARACTERISTICS

A total of 88 program staff completed an initial survey in late fall 2011 or early winter 2012. They responded to questions about their background (gender, age, ethnicity, education, professional experience) and their experiences in their current position.

The initial survey indicated that the line staff in the participating programs was heavily female (76%) and ethnically diverse (See Figure 2). Staff educational background varied widely: 8% had a M.A. degree, 28% had a B.A., and 17% an Associates (AA) degree. Ninety-five percent reported to have attended “some college.” Many of the line staff were relatively young: 33% were between18 and 25 years of age, 38% were between 26 and 35 years old.

.

Figure 4.1 Race and Ethnicity of Program Staff Professional Experience Staff professional experience was also variable: 78% have some experience in school settings (other than current position).

! 16% as a School Administrator ! 39% as a Classroom Teacher ! 24% as an Instructional Specialist (arts, ELL, P.E., special education) ! 64% as a Classroom Aide or TA ! 15% as School Administrative Staff

!"#$%&'()*(+%'(,)!-'./'()0'1%2#)

34)

5%-%6%(7)34)

819#$):4)

;'&%<%&)*.-'(+#$)=4)!.%'()!"#$%&'()

>4)

!?$%&'()!"#$%&'()3>4)

@%.6'(%&,)A'1%(7)=B4)

C'D&'.%'()=E4)


10

Experience in Current Position ! 46 of the staff reported they are Site Coordinators ! 36 are Activity Leaders ! 3 are Program Coaches working at multiple sites ! 43% have been in their current position for 2 years or more

Figure 4.2 Experience in Current Position Instruction Implemented in Current Position

! Over 60% report implementing instruction in Math, Language Arts ! Over 50% report implementing instruction in Science, Athletics

Figure 4.3 Instruction Implemented in Current Position

Implementation of STEM Activities in Current Position

! Prior to their current position, 70% of staff had not implemented STEM activities ! 43% had not received any training relating to STEM during the past academic year ! In their current position, 55% implement activities in Applied Mathematics ! 44% report implementing activities in Life Science ! 40% report implementing activities in Technology, Physical Science

!"#$

%%#$

!&#$

!'#$

(&#$

"#$ &#$ !"#$ !&#$ ("#$ (&#$ %"#$ %&#$

&)*$+,-./$

()&$+,-./$

!)($+,-./$

0$12345/$42$!$+,-.$

6,//$45-3$0$12345/$

!'#$

(%#$

%%#$

%0#$

&%#$

&'#$

0&#$

00#$

"#$ !"#$ ("#$ %"#$ 7"#$ &"#$ 0"#$ '"#$

839:3,,.:39$$

;<42.:39=$>21,?2.@$>,AB$

;,C532A29+$$

D,.E2.1:39FG:/<-A$H.4/$

IC:,3C,$

H45A,4:C/$

6-39<-9,$H.4/$

J-45$$


11

Figure 4.4 Implementation of STEM Activities in Current Position

The above findings show that afterschool program staff typically have a large turnover with

relatively few staff members staying at a program site for many years. Many program directors need to re-train new staff annually. Even though most staff have prior teaching experience, they have little experience in STEM curricula, and especially lacking training in the implementation of Technology and Engineering. B. PROGRAM PRACTICES

This section summarizes changes in staff reports about the implementation of STEM, from pre to post. On both pre-participation and post-participation surveys, program staff were asked how much time per week they spent implementing STEM activities with students. From pre to post, the amount of time increased; t(41) = -2.95, p = .005. On the pre-participation survey 33% of staff reported implementing 1 to 3 hours of STEM per week. This increased to 60% on the post-participation survey. Similarly, the percent of staff implementing STEM three or more hours per week increased from 11% to 15%. The percent of staff who reported that they did not implement any STEM activities decreased from 23% to 4% (see Figure 4.5). Figure 4.5 Time Implementing STEM Per Week

(%#$

(0#$

('#$

7"#$

7"#$

77#$

&&#$

"#$ !"#$ ("#$ %"#$ 7"#$ &"#$ 0"#$

839:3,,.:39$KL<:AM:39=$.2L24:C/N$

8-.45$IC:,3C,$

;,C532A29+$K422A/=$1-C5:3,/N$

D5+/:C-A$IC:,3C,$

;,C532A29+$KC21B<4,./=$M:9:4-A$1,M:-N$

6:E,$IC:,3C,$

HBBA:,M$J-45,1-4:C/$

(%#$%%#$

!!#$7#$

0"#$

!&#$

"#$

!&#$

%"#$

7&#$

0"#$

O23,$ !)%$52<./$ %$52<./$2.$12.,$

D.,$K3$P$QQN$ D2/4$K3$P$7&N$


12

Staff also reported about the frequency of meetings about STEM issues. Figure 4.6 shows the frequency of meetings, at pre and post. Staff showed an increase in weekly STEM staff meetings from 26% to 35% and a decrease from 23% to 13% in their reporting of “Never discussing STEM at staff meetings”. Figure 4.6 Frequency of Staff Meetings About STEM

Figure 4.7 shows there was an increase in the frequency of staff interactions with classroom

teachers about STEM topics taught in the classroom; t(41) = -2.29, p = .027. On the pre-participation survey, 43% of staff reported that during the past academic year they discussed STEM with teachers once a month or more. This increased to 60% on the post-participation survey. There was also a decrease from pre to post in the percent of staff who never talked with teachers about STEM concepts taught in the classroom (31% to 13%). Figure 4.7 Staff Interactions with Teachers About STEM Topics

(%#$(0#$

!%#$

%&#$

"#$

!"#$

("#$

%"#$

7"#$

O,R,.$ !)%$4:1,/$-$?,,@$

D.,$K3$P$QQN$ D2/4$K3$P$7&N$

%!#$

7%#$

!%#$

0"#$

"#$

!&#$

%"#$

7&#$

0"#$

O,R,.$ S3C,$-$J2345$2.$J2.,$

D.,$K3$P$QQN$ D2/4$K3$P$7&N$


13

There was an overall increase in the percent of staff who reported holding STEM-related events for parents; t(40) = -2.01, p = .051. Figure 4.8 shows that the percent of staff who reported holding STEM-related events once a month or more increased from 33% to 37% and the percent of staff who reported holding events less than once a month increased from 36% to 47%. There was also a decrease in the percent of staff who reported never holding STEM-related events for parents (31% to 16%). Figure 4.8 STEM-Related Events with Parents

C. STAFF BELIEFS AND ATTITUDES

Pre and post-participation scores for both Staff Beliefs about STEM and Efficacy Implementing STEM measures showed that staff overall had positive beliefs about STEM and positive efficacy about implementing STEM activities. Matched Pair analyses showed no significant change in these scores from pre to post. Table 4.9 Program Staff Beliefs and Efficacy Scores PRE POST N Mean N Mean Beliefs about STEM 88 3.71 45 3.72 Efficacy Implementing STEM 88 3.73 45 3.89

%!#$%0#$ %%#$

!0#$

7'#$

%'#$

"#$

!"#$

("#$

%"#$

7"#$

&"#$

O,R,.$ 6,//$45-3$S3C,$-$J2345$ S3C,$-$J2345$2.$J2.,$

D.,$K3$P$QQN$ D2/4$K3$P$7&N$


14

SECTION V – QUANTITY AND QUALITY OF PROGRAM STEM ACTIVITIES

A. STEM ACTIVITY DOCUMENTATION FORM DATA

A total of 38 staff submitted the STEM Activity Documentation Form and documented the

implementation of 990 STEM activities. The activity forms included the following information about each activity: duration, STEM content area(s), number of participating students, student grade level, level of student engagement, level of challenge to the students, and overall success of the activity.

The total number of activities reported by staff ranged from 5 to 150 and the average number of

activities reported by each staff was 26. Staff reported that the number of students participating in the activity ranged from 1 to 71; the

average number of participating students per activity was 20. The duration of the activities ranged from 15 minutes to 2 hours. Most of the activities (n = 454, 46%) lasted 45 minutes to one hour. 325 (33%) of the activities lasted less than 45 minutes, and 203 (21%) of the activities lasted longer than one hour.

Student grade level was reported for 450 of the 990 activities, with all of these activities

occurring with students in grades 1 to 6. Most activities were reported by staff to include students at multiple grade levels. The majority of activities included students in grades 4 and 5. Figure 5.1 shows the number of STEM activities, by grade level, of participating students. Figure 5.1 Number of STEM Activities, by Student Grade Level

981 of the 990 documented activities identified a specific STEM focus (Science, Technology, Engineering, Mathematics). Of these, the majority of activities (67%) focused on one STEM area only. 157 (16%) of the activities focused on two STEM areas, 92 (9%) included three areas, and 67 (7%) included all four STEM areas in the activity. Of the four STEM areas, science was most often the focus of the activity. Figure 5.2 shows the number of activities that focused on each of the four STEM areas.

(0$

'0$

!'7$!*'$

((&$

**$

"$

&"$

!""$

!&"$

(""$

(&"$

!/4$T.-M,$ (3M$T.-M,$ %.M$T.-M,$ 745$T.-M,$ &45$T.-M,$ 045$T.-M,$

O<1L,.$2E$I;8J$HC4:R:4:,/$


15

Figure 5.2 Number of STEM Activities, by STEM Area Focus

Overall, staff reported that the STEM activities went well. Over 75% of the activities (n = 755) went “very well” (35%) or “mostly well” (42%). Only 7% of the activities were reported by staff to go “not at all well”.

For 962 of the documented STEM activities, staff rated how challenging the activity was for the students. The majority of activities (80%) were reported by staff to be either “very challenging” for the students (n = 210, 22%) or “somewhat challenging” (n = 562, 58%). 160 (17%) of the activities were considered “not at all challenging” and 30 of the activities (3%) were reported by staff as “too challenging”.

Staff also rated students’ level of engagement in the activities. For the majority of activities (n = 738, 76%), students were either “very engaged” (41%) or “mostly engaged” (35%). Students were reported to be “not at all engaged” for only 7% of the activities.

Students’ level of engagement with an activity was positively associated with the level of challenge. For activities that were “somewhat challenging” or “very challenging”, students’ level of engagement increased. For activities that were “not at all challenging”, students’ level of engagement decreased. Figure 5.3 shows the level of student engagement by level of challenge.

'"0$

("7$(*0$ %!'$

"$

(""$

7""$

0""$

Q""$

IC:,3C,$ ;,C532A29+$ 839:3,,.:39$ J-45,1-4:C/$

O<1L,.$2E$I;8J$HC4:R:4:,/$


16

Figure 5.3 Level of Student Engagement in STEM Activities, by Level of Challenge

B. YOUTH REPORTS OF QUALITY OF PROGRAM EXPERIENCES

In addition to collecting program staff reports of STEM activities, the UC Irvine Team also collected surveys from students about the overall quality of their experiences at the afterschool programs. Table 5.4 summarizes the Program Experiences scores for all students in the Power of Discovery: STEM2 Year 1 Study. Scores are presented in two ways: as mean scores and as categorical scores. Mean scores are the aggregate scores of all participating students and the categorical scores are based on ranges in mean scores. Mean scores from 1.0 to 1.9 are categorized as “Low”, scores from 2.0 to 2.9 are “Fair”, scores from 3.0 to 3.59 are “Good”, and scores from 3.6 to 4.0 are “Excellent.” In the Power of Discovery: STEM2 Year 1 Study, 29% of all students reported excellent quality experiences with staff and activities (mean scores of 3.6 to 4), 34% reported good quality experiences in this area (scores of 3.0 to 3.59), 30% reported fair quality experiences (scores of 2.0 to 2.9) and 7% reported low quality experiences (scores less than 2). In terms of positive experiences with peers, 41% reported excellent experiences, 27% reported good experiences, 25% reported fair experiences and 8% reported low quality experiences (see Table 5.4). As a comparison to the scores from the STEM Year 1 Study, Table 5.4 also shows scores from the California Field Test. For all elementary school students in the Field Test, 34% reported excellent quality experiences with staff and activities, 34% reported good quality experiences, 26% reported fair quality experiences, and 6% reported low quality experiences. In terms of positive experiences with peers, 40% reported excellent experiences, 30% reported good experiences, 25% reported fair experiences and 5% reported low quality experiences.

%"#$

!#$ !#$

!'#$(%#$

!*#$

!"#$!'#$

(7#$

%0#$ %*#$

%"#$(%#$

77#$&"#$

%'#$

"#$

!"#$

("#$

%"#$

7"#$

&"#$

0"#$

O24$-4$-AA$C5-AA,39:39$ I21,?5-4$C5-AA,39:39$ G,.+$C5-AA,39:39$ ;22$C5-AA,39:39$

#$O24$-4$-AA$,39-9,M$ #$I21,?5-4$,39-9,M$ #$J2/4A+$,39-9,M$ #$G,.+$,39-9,M$


17

Table 5.4 Program Experiences, Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)

% Excellent (3.6-4.0)

STEM2 Year 1 Study (All Students) Staff & Activities 1,212 3.10 7.0% 30.3% 34.0% 28.7% Peer Affiliation 1,207 3.15 7.7% 24.8% 26.6% 40.9% CA Field Test (Elementary Students) Staff & Activities 5,704 3.19 5.5% 25.9% 34.3% 34.3% Peer Affiliation 5,688 3.18 5.4% 24.8% 30.1% 39.7% N= number of students; Mean = average score


18

SECTION VI – STUDENT OUTCOMES

A. STUDENT SELF-REPORTS OF PRE/POST OUTCOMES

Across all sites in the Power of Discovery: STEM2 Year One Pilot Study, the majority of students (68%) reported good (38% PRE; 34% POST) or excellent work habits (29% PRE; 29% POST). Only about 5-6% of students reported low performance in work habits. About 70-75% of students reported good (29% PRE; 28% POST) or excellent (46% PRE; 42% POST) math efficacy. About 7-9% of students reported low efficacy in math. A substantial proportion reported good (30% PRE; 30% POST) or excellent (38% PRE; 37% POST) science efficacy. About 9-12% of students reported low efficacy in science.

In comparison, for elementary school students in the California Field Test, about 69% were

reported by classroom teachers to have good (40% PRE; 40% POST) or excellent work habits (29% PRE; 29% POST). About 4-5% of the students were reported to have low scores in work habits. In math efficacy, 80% of the elementary grade students reported good (25% PRE; 26% POST) or excellent (54% PRE; 54% POST) math efficacy. Less than 5% of students reported low efficacy in math.

As comparison for the Science Efficacy scores, the scores for middle/high school students at the Tiger Woods Learning Center (TWLC) are reported. About 73-78% of the students reported good (35% PRE and 33% POST) or excellent (43% PRE and 40% POST) efficacy in science, and about 22-27% of students reported fair (19% PRE and 23% POST) or low (4% PRE and 4% POST) science efficacy.

These figures suggest that the students being served in program participating in the STEM Initiative are comparable to other students in publicly funded afterschool programs in California.


19

Table 6.1 Student Outcome Scores, Skill Development

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


STEM2 Year 1 Study (All Students) Work Habits, PRE 1,020 3.12 4.8% 29.0% 37.5% 28.6% Work Habits, POST 1,278 3.08 6.2% 31.1% 34.0% 28.6% Math Efficacy, PRE 983 3.29 6.8% 18.2% 29.1% 45.9% Math Efficacy, POST 1,254 3.20 9.3% 20.7% 27.6% 42.3% Science Efficacy, PRE 982 3.14 9.3% 22.6% 30.3% 37.8% Science Efficacy, POST 1,253 3.07 11.7% 22.5% 28.9% 36.9% CA Field Test (Elementary Students) Work Habits, PRE 3,917 3.16 3.5% 27.7% 40.1% 28.7% Work Habits, POST 5,757 3.14 4.6% 26.9% 40.0% 28.5% Math Efficacy, PRE 3,917 3.41 4.6% 16.2% 25.0% 54.2% Math Efficacy, POST 5,733 3.42 4.4% 15.3% 26.4% 53.9% TWLC Scores, (Middle/High Students) Science Efficacy, PRE 408 3.30 4.2% 18.6% 34.6% 42.6% Science Efficacy, POST 299 3.25 4.0% 23.1% 33.1% 39.8% N= number of students; Mean = average score

With respect to Science Interest, across all sites in the Power of Discovery: STEM2 Year 1 Study, about 52-53% of students reported that their interest and attitude toward science were good (38% PRE; 36% POST) or excellent (15% PRE; 16% POST). About 5-8% of students reported low interest in science. Across all sites, about 92% of students reported that the likelihood of being successful in the future was good (27% PRE; 25% POST) or excellent (65% PRE; 67% POST). Only about 1-2% of students reported low likelihood of future success. A little over half of the students viewed the likelihood of pursuing a career in science as good (32% PRE; 33% POST) or excellent (24% PRE; 22% POST). About 9-11% of students reported low likelihood of pursuing a career in science.

In comparison, about 60% of middle/high school students at the TWLC reported that their

interest and attitude toward science were good (38% PRE and 35% POST) or excellent (22% PRE and 25% POST). Less than 5% of students reported low interest in science. About 99% of middle/high school students reported that the likelihood of being successful in the future was good (22% PRE and 22% POST) or excellent (77% PRE and 77% POST). About 1% of students reported low likelihood of future success. About 65-69% of middle/high school students viewed the likelihood of pursuing a career in science as good (39% PRE and 36% POST) or excellent (26% PRE and 33% POST). Less than 5% of students reported low likelihood of pursuing a career in science. The TWLC program is one that is specially designed to focus on STEM learning and the higher proportion of STEM interests in its participants likely reflects student selection into the program.


20

Table 6.2 Student Outcome Scores, Attitudes and Beliefs

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


STEM2 Year 1 Study (All Students) Science Interest (PEAR), PRE 974 3.06 4.8% 42.5% 37.5% 15.2% Science Interest (PEAR), POST 1,250 3.01 7.6% 40.8% 36.0% 15.6% Science Career, PRE 957 2.91 9.1% 35.4% 31.8% 23.7% Science Career, POST 1,223 2.88 10.5% 33.8% 33.3% 22.4% View of Future, PRE 961 3.63 1.1% 7.7% 26.5% 64.6% View of Future, POST 1,224 3.63 2.3% 5.5% 24.9% 67.3% TWLC Scores, (Middle/High Students) Science Interest (PEAR), PRE 408 3.10 2.7% 36.8% 38.2% 22.3% Science Interest (PEAR), POST 299 3.09 4.7% 35.5% 35.1% 24.7% Science Career, PRE 404 3.07 4.2% 31.2% 39.1% 25.5% Science Career, POST 298 3.13 4.0% 27.5% 35.9% 32.6% View of Future, PRE 405 3.78 1.2% 22.2% 76.5% View of Future, POST 298 3.78 1.3% 21.8% 76.8% N= number of students; Mean = average score

Regarding Positive Behavior (higher Social Competencies and lower Misconduct), the majority of students in the Power of Discovery: STEM2 Year 1Study reported good (42% PRE; 36% POST) or excellent (13% PRE; 16% POST) social competencies. Less than 8% of students reported low social competencies. In the Power of Discovery: STEM2 Year 1 Study, about 85-88% of students reported low levels of misconduct. About 2% of students reported high levels of misconduct.

In comparison, about 60% of the elementary grade students in the California Field Test reported

good (41% PRE; 43% POST) or excellent (18% PRE; 16% POST) social competencies. Less than 7% of students reported low social competencies (7%; PRE; 5% POST). Almost 90% of the elementary school students reported low levels of misconduct. Less than 2% of students reported high levels of misconduct.


21

Table 6.3 Student Outcome Scores, Positive Behavior

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


STEM2 Year 1 Study (All Students) Social Competencies, PRE 989 2.94 7.5% 37.6% 42.1% 12.8% Social Competencies, POST 1,257 2.92 7.7% 40.6% 35.7% 16.0% CA Field Test (Elementary Students) Social Competencies, PRE 3,917 3.01 6.5% 34.9% 40.8% 17.8% Social Competencies, POST 5,724 3.03 4.6% 36.4% 42.5% 16.4%

N

Mean

% Low

(1.0-1.9)

% Medium (2.0-2.9)

% High

(3.0-4.0)

STEM2 Year 1 Study (All Students) Misconduct, PRE 1,006 1.43 87.7% 10.2% 2.1% Misconduct, POST 1,275 1.48 84.9% 12.9% 2.2% CA Field Test (Elementary Students) Misconduct, PRE 3,917 1.41 89.0% 9.5% 1.5% Misconduct, POST 5,742 1.45 86.3% 11.7% 2.0% N= number of students; Mean = average score

B. ASSOCIATIONS BETWEEN STEM-RELATED FACTORS AND STUDENT OUTCOMES

Regression analyses were conducted to examine associations between STEM-related factors on students’ post-participation outcome scores. These three factors were used as predictor variables in the regression analyses: staff-reported beliefs about STEM, staff-reported sense of efficacy about implementing STEM, and staff-reported amount of time spent on STEM activities.

The staff beliefs and staff efficacy scores are continuous variables, ranging from 1.00 to 5.00. Time on STEM is a categorical variable with three categories: less than 1 hour, 1-2 hours, more than 2 hours. All three variables were taken from scores on the staff post-participation survey.

Two control variables were also used in the regression model: student gender and site-level pre-participation student outcome scores. Site-level pre-participation scores were used in place of student-level scores, because only site-level identifiers were used on the survey. In the analyses, the pre-participation outcome score for each student was the aggregate mean score for that student’s site.

Significant results for the regression analyses are summarized in table 6.4 below. Controlling for gender and pre-participation scores, stronger staff beliefs about the importance and benefit of STEM activities for students were significantly related to higher post-participation outcome scores in science efficacy, science interest, and work habits. Greater amount of time spent on STEM activities was significantly related to higher scores in math efficacy and greater student-reported likelihood of a future career in science. A greater sense of staff efficacy in implementing STEM activities was significantly related to a lower sense of student efficacy in science.


22

Table 6.4 Effects of Staff Beliefs about STEM on Student Post-Participation Outcome Scores Predictor Outcome Beta Staff Beliefs about STEM Science Efficacy 0.20 Science Interest (PEAR) 0.13 Work Habits 0.13 Time Spent on STEM Math Efficacy 0.15 Science Career 0.11 Staff Efficacy in STEM Science Efficacy -0.17

C. ASSOCIATIONS BETWEEN PROGRAM EXPERIENCES AND STUDENT OUTCOMES Regression Analyses Regression analyses were conducted to examine associations between the two Program Experiences variables on student post-participation outcome scores. The two Program Experiences variables (Staff & Activities and Peer Affiliation) were used as predictors in the analyses. The Program Experiences scores range from range from 1 to 4, and taken from the post-participation student survey. As in the analyses summarized in the previous section, two control variables were used in the regression model: gender and the site-level pre-participation student outcome scores. Results are summarized in Table 6.5 below. Significant results for the regression analyses are summarized in the table below. Controlling for gender and pre-participation scores, greater quality of student-reported experiences with staff and program activities were significantly related to higher scores in math efficacy, science interest, science efficacy, science career, and work habits (p < .01), a more positive view of the future (p <.05), and lower scores in misconduct (p < .01). Students who reported higher quality of experiences with peers had significantly higher scores in all student outcomes except misconduct (p < .01).


23

Table 6.5 Effects of Quality of Program Experiences on Student Post-Participation Outcome Scores Predictor Outcome Beta Staff & Activities Math Efficacy** .21 Science Interest (PEAR)** .29 Science Efficacy** .26 Science Career** .16 Work Habits** .21 View of Future* .10 Social Competencies NS Misconduct** -.30 Peer Affiliation Math Efficacy** .20 Science Interest (PEAR)** .19 Science Efficacy** .22 Science Career** .19 Work Habits** .26 View of Future** .21 Social Competencies** .41 Misconduct NS

*p<.05, **p<.01 = statistically significant relationship between predictor and outcome score

ANOVAs Analyses (ANOVAs) were conducted to test associations between student reports of the quality of their program experiences and the self-reported student outcome scores that were collected at the end of their programs (post-participation). For these analyses, the quality of program experiences was categorized as low (ratings of 1.0 to 2.9), medium (ratings of 3.0 to 3.59), and high (ratings of 3.6 to 4.0). Figures 6.6-6.8 show student self-reported outcome scores, for each level of Program Experiences scores. Results show that students’ reports of higher quality experiences with program staff are associated with lower scores in misconduct and higher scores in math efficacy, science efficacy, work habits, and social competencies (p < .01). Results were similar for the associations between student reports of Peer Affiliation and student outcomes. Students’ reports of higher quality relations with other students in the program are associated with lower scores in misconduct and higher scores in the other five outcomes (p < .01)


24

Figure 6.6 Associations between Program Experiences and Outcomes: Math Efficacy and Science Efficacy

Math Efficacy Science Efficacy

(UQ($

%U(Q$

%U&'$

(UQ0$%U!*$

%U70$

(U'"$(UQ"$(U*"$%U""$%U!"$%U("$%U%"$%U7"$%U&"$%U0"$

62?FV-:.$W<-A:4+$ T22M$W<-A:4+$ 8XC,AA,34$W<-A:4+$

I4-EE$Y$HC4:R:4:,/$K3$P$!=(!(NZZ$

D,,.$HE[:A:-4:23$K3$P$!=("'NZZ$

(U'%$

%U!%$

%U7&$

(U0*$

%U!"$

%U%0$

(U0"$(U'"$(UQ"$(U*"$%U""$%U!"$%U("$%U%"$%U7"$%U&"$

62?FV-:.$W<-A:4+$ T22M$W<-A:4+$ 8XC,AA,34$W<-A:4+$

I4-EE$Y$HC4:R:4:,/$K3$P$!=(!(NZZ$

D,,.$HE[:A:-4:23$K3$P$!=("'NZZ$

Figure 6.7 Associations between Program Experiences and Outcomes: Work Habits and View of Future

Work Habits View of Future

(U'0$

%U!'$

%U7!$

(U'!$

%U!%$

%U%&$

(U0"$(U'"$(UQ"$(U*"$%U""$%U!"$%U("$%U%"$%U7"$%U&"$

62?FV-:.$W<-A:4+$ T22M$W<-A:4+$ 8XC,AA,34$W<-A:4+$

I4-EE$Y$HC4:R:4:,/$K3$P$!=(!(NZZ$

D,,.$HE[:A:-4:23$K3$P$!=("'NZZ$

%U7&$

%U0Q$

%UQ($

%U7%$

%U0'$

%U'Q$

%U%"$

%U7"$

%U&"$

%U0"$

%U'"$

%UQ"$

%U*"$

62?FV-:.$W<-A:4+$ T22M$W<-A:4+$ 8XC,AA,34$W<-A:4+$

I4-EE$Y$HC4:R:4:,/$K3$P$!=(!(NZZ$

D,,.$HE[:A:-4:23$K3$P$!=("'NZZ$


25

Figure 6.8 Associations between Program Experiences and Outcomes: Social Competencies and Misconduct

Social Competencies Misconduct

(U'&$

(U*%$%U!7$

(U0"$

(U*%$

%U!Q$

(U&"$

(U0"$

(U'"$

(UQ"$

(U*"$

%U""$

%U!"$

%U("$

62?FV-:.$W<-A:4+$ T22M$W<-A:4+$ 8XC,AA,34$W<-A:4+$

I4-EE$Y$HC4:R:4:,/$K3$P$!=(!(NZZ$

D,,.$HE[:A:-4:23$K3$P$!=("'NZZ$

!U'!$

!U7($

!U(0$

!U0'$!U77$

!U%'$

!U("$

!U%"$

!U7"$

!U&"$

!U0"$

!U'"$

!UQ"$

62?FV-:.$W<-A:4+$ T22M$W<-A:4+$ 8XC,AA,34$W<-A:4+$

I4-EE$Y$HC4:R:4:,/$K3$P$!=(!(NZZ$

D,,.$HE[:A:-4:23$K3$P$!=("'NZZ$


26

SECTION VII – CONCLUSIONS AND RECOMMENDATIONS

The year one study of the Power of Discovery: STEM2 initiative tested evaluation measures at multiple sites serving diverse communities. Student Surveys and Program Staff Surveys were administered at 15 Jumpstarting STEM programs from 9 Regions. Data collected included: 135 Staff Surveys (88 pre-participation; 47 post-participation); 2,298 Student Surveys (1020 pre-participation; 1278 post-participation); and 990 STEM Activity Documentation Forms.

KEY FINDINGS

The background of program staff reflected the diversity of the students and communities served. About 70% of staff were Hispanic/Latino or Caucasian, about 16% were African American, and about 14% were Asian American, Pacific Islander, Filipino, or Other. 95% of program staff had attended some college, and 45% had completed a 2-year or 4-year college degree. Over 90% of staff had some experience in other afterschool programs (other than current position), and over 75% had other experience in school settings.

From the pre-participation Staff Survey to the post-participation Staff Survey, the amount of time

per week that staff reported spending on STEM activities increased. Staff reports also showed an increase, from pre to post, in the frequency of staff meetings about STEM issues, in the frequency of interactions with classroom teachers about STEM topics taught in the classroom, and in the frequency of STEM-related events that staff held for parents.

Program staff completed and submitted documentation of 990 STEM activities. The majority of

activities (67%) focused on just one STEM area; about 16% of the activities focused on three or four STEM areas. 80% of the activities were reported by staff to be either “very challenging” for the students (22%) or “somewhat challenging” (58%), and students were either “very engaged” (41%) or “mostly engaged” (35%). Overall, staff reported that the activities went well: 35% went “very well” and 42% went “mostly well.”

Students in grades 3-12 completed 2,298 surveys, reporting on their own skill development, positive behavior, and program experiences. Over 60% of students reported good or excellent quality experiences with staff and activities at their program, and over 50% of student reported good or excellent experiences with peers. Analyses of associations between program experiences and outcome scores were all significant: students who reported higher quality of experiences with staff, activities, and peers, also had lower scores in misconduct and higher scores in math efficacy, science efficacy, work habits, and social competencies (p < .01). Regression analyses, which controlled for gender and pre-participation scores, were also positive and significant. Greater quality of student-reported experiences with staff and program activities was significantly related to higher scores in math efficacy, science interest, science efficacy, science career, and work habits (p < .01), a more positive view of the future (p <.05), and lower scores in misconduct (p < .01). Greater quality of student-reported experiences with peers was significantly related to higher scores in all student outcomes except misconduct (p < .01).


27

Analyses of the effects of staff beliefs and efficacy on student outcomes were also positive and significant. Stronger staff beliefs about the importance and benefit of STEM activities for students was significantly related to higher post-participation outcome scores in science efficacy, science interest, and work habits. Greater amount of time spent on STEM activities was significantly related to higher scores in math efficacy and greater student-reported likelihood of a future career in science. A greater sense of staff efficacy in implementing STEM activities was significantly related to a greater sense of student efficacy in science.

PROPOSED YEAR 2 EVALUATION APPROACH Overall, the year one study of the Power of Discovery: STEM2 Initiative showed that the administration of the three measures used—student self reports, staff reports on students, and completion of STEM Activity Documentation forms by staff—was successful. The measures yielded valid and reliable data and the analyses provided positive preliminary results for the purpose of informing further evaluation of programs participating in the Power of Discovery: STEM2 Initiative.

However, our approach to several underlying challenges should be made explicit. First, there is

always a tradeoff between measures of breadth and depth of the implementation of the Initiative across the State versus more in depth examination at a smaller number of programs and sites —The question of breadth versus depth of coverage. We have opted to combine both depth and breadth. In our proposed Year 2 evaluation, online surveys will be used to obtain information from as many sites, staff, and students as possible. More intensive observations and interviews will be conducted at a selected subsample of sites. Our proposed research design, methods and aims are described below.

Preferred research design—A descriptive or correlational study that takes advantage of natural variation in implementation versus a quasi-experimental study that uses a treatment versus comparison group design versus an experimental study that uses a random assignment at the individual, site/school, and program level to assess program impacts. Each of these approaches has its strengths and advantages. Each has its limitations and disadvantages. We have opted to use all four strategies, guided by particular research aims detailed below.

Preferred research methods—The Evaluation Team is proposing a multi-method

approach that includes online surveys, on-site observations, document reviews, monitoring website usage, focus groups, interviews with key informants, program attendance and standardized test scores.

Research Aims—UC Irvine’s evaluation work will focus on outcomes at the program site, staff,

and student levels which are the result of the efforts of the three RISP to provide technical assistance to support STEM learning. Building on the first year evaluation, the proposed year two evaluation study (academic year 2012-2013) incorporates four major aims:

Aim 1: Use on-line surveys to document demographic characteristics and educational background of program staff, including experience implementing STEM-related activities. These data are critical to future efforts to address the educational and training needs of the afterschool


28

STEM workforce because training needs to align with the skills and competencies of the workforce.

Aim 2: Use on-line surveys, observations, and interviews to document changes in STEM-related program practices, resulting from the efforts of the Regional Innovation Support Providers.

Aim 3: Use on-line surveys, observations, and interviews to examine changes in the interests, skills, beliefs, and attitudes of program staff in the STEM domain, and in other areas such as youth development, particularly for staff with limited STEM background.

Aim 4: Use on-line surveys, observations, and interviews to assess student outcomes in STEM areas of interest and engagement in STEM learning, STEM career aspirations and effects on broader youth outcomes and the quality of their afterschool program experience with staff, activities and peer relationships over the course of the 2012-13 Power of Discovery: STEM2

Initiative.

Each of these aims represents a critical and necessary research activity. Each aim will be studied using multiple methods of assessment, the standard in comprehensive evaluation research.


29

REFERENCES Brown, B. B., Clasen, D. R., & Eicher, S. A. (1986). Perceptions of peer pressure, peer conformity,

dispositions, and self-reported behavior among adolescents. Developmental Psychology, 22, 521-530.

Chaput, S. (August 2004). Understanding and Measuring Attendance in Out-of-School Time Programs. The Evaluation Exchange. Harvard Family Research Project. Vol. X, No. 7.

Eccles, J. and Gootman, J.A. (Eds.) (2002). Community Programs to Promote Youth Development. Washington, DC: National Academies Press.

Eccles, J., Wigfield, A., Harold, R. D., & Blumenfeld, P. (1993). Age and gender differences in children’s self- and task perceptions during elementary school. Child Development, 64, 830- 847.

Elliott, D. S., & Menard, S. (1996). Delinquent friends and delinquent behavior: Temporal and developmental patterns. In J. D. Hawkins (Ed.), Delinquency and crime: Current theories (pp. 28-67). New York: Cambridge University Press.

Hussar, K., Schwartz, S., Boiselle, E., and Noam, G. G. (August 2008). Toward a Systematic Evidence-Base for Science in Out-of-School Time: The Role of Assessment. Program in Education, Afterschool & Resiliency. Harvard University and McLean Hospital. A study prepared for the Noyce Foundation.

Jacobs, J. E., Lanza, S., Osgood, D. W., Eccles, J. S., & Wigfield, A. (2002). Changes in children’s self-competence and values: Gender and domain differences across grades one through twelve. Child Development, 73, 509-527.

Menard, S. and Elliott, D. S. (1996). Prediction of adult success using stepwise logistic regression analysis. A report prepared for the MacArthur Foundation by the MacArthur Chicago-Denver Neighborhood Project.

Muris, P. (2001). A brief questionnaire for measuring self-efficacy in youths. Journal of Psychopathology and Behavioral Assessment, 23, 145-149.

Noam, G., Larson, J., Dahlmer, C., (2008). Dimensions of Success Observation Tool Program in Education, Afterschool and Resilience (PEAR). Harvard University.

Noam, G. Sneider, C. (et. al.), (2010). Common Goals, Common Assessments: Evaluation Guidelines for Youth Programs. Unpublished manuscript, Program in Education, Afterschool and Resilience (PEAR). Harvard University.

Pierce, K. M., Hamm, J. V., & Vandell, D. L. (1999). Experiences in after-school programs and children’s adjustment in first-grade class classrooms. Child Development, 70, 756-767.

Pierce, K. M., Bolt, D. M., & Vandell, D. L. (2010). Specific features of after-school program quality: Associations with children’s functioning in middle childhood. American Journal of Community Psychology, 45, 381-393.

Posner, J. K., & Vandell, D. L. (1994). Low-income children’s after-school care: Are there beneficial effects of after-school programs? Child Development, 65, 440-456.

Tyler-Wood, T., Knezek, G., & Christensen, R. (2010). Instruments for assessing interest in STEM content and careers. Journal of Technology and Teacher Education, 18(2), 341-363.


30

Vandell, D. L., M., O’Cadiz, M.P., Hall, V. (2012), California Afterschool Outcome Measures Project: Field Test of the Online Toolbox. Report to the David and Lucile Packard Foundation and the California Department of Education.

Vandell, D. L., O’Cadiz, P., Hall, V., & Westover, T. (2010). California After School Outcome Measures Project: Phase II Final Report. Report to the David and Lucile Packard Foundation.

Vandell, D. L., Reisner, E. R., Brown, B. B., Dadisman, K., Pierce, K. M., Lee, D., & Pechman, E. M. (2005). The Study of Promising After-School Programs: Examination of intermediate outcomes in year 2. Report to the Charles Stewart Mott Foundation.

Vandell, D. L., Reisner, E. R., Pierce, K. M., Brown, B. B., Lee, D., Bolt, D., & Pechman, E. M. (2006). The Study of Promising After-School Programs: Examination of longer term outcomes after two years of program experiences. Report to the Charles Stewart Mott Foundation.

Vandell, D. L., Shumow, L., & Posner, J. (2005). After-school programs for low-income children: Differences in program quality. In J. L. Mahoney, R. W. Larson, & J. S. Eccles (Eds.), Organized activities as contexts of development: Extracurricular activities, after school and community programs (pp. 437-456). Mahwah, NJ: Erlbaum.

Vandell, D. L., Warschauer, M., O’Cadiz, P., & Hall, V. (2008). Two year evaluation study of the Tiger Woods Learning Center. Report to the Tiger Woods Foundation.


31

APPENDIX A. DEMOGRAPHIC PROFILE OF YEAR 1 STUDY SITES2

REGION Agency Name DISTRICT

% FRL %

ELL

% African-

American not

Hispanic

% American Indian or Alaska Native

% Asian % Filipino

% Hispanic or Latino

% Pacific Islander

% White not Hispanic

% Two or more Races

% None Reported

(Ethnicity)

ANAHEIM ACHIEVES YMCA Anaheim City SD, Magnolia SD, Savanna SD (2 districts) 1 Ball Jr. High 80.7% 0.0% 2.1% 0.3% 8.3% 1.6% 50.4% 0.6% 36.6% 0.0% 0.0%

2 Cerritos Elem 46.3% 35.7% 4.5% 0.6% 14.5% 6.2% 36.0% 0.6% 33.4% 4.0% 0.2%

3 Disney Elem 61.1% 43.6% 2.2% 0.3% 21.4% 2.7% 54.7% 0.6% 13.6% 4.3% 0.1%

4 Gauer Elem 84.2% 57.2% 2.1% 0.0% 6.9% 2.9% 82.4% 0.3% 4.7% 0.6% 0.1%

5 Mann Elem 95.7% 61.8% 0.8% 0.3% 1.8% 0.6% 94.6% 0.0% 1.5% 0.4% 0.2%

6 Ross Elem 84.0% 58.9% 0.7% 0.1% 2.4% 1.2% 89.3% 0.6% 5.4% 0.2% 0.0%

BOYS & GIRLS CLUBS OF CENTRAL SONOMA COUNTY

7 Calmecac (K-8) 57.2% 40.7% 0.5% 1.0% 1.1% 0.0% 69.0% 0.1% 24.9% 0.0% 3.3%

8 Cloverdale Elem 68.6% 0.0% 10.9% 0.3% 2.5% 1.3% 68.7% 0.5% 14.2% 1.0% 0.6%

9 Healdsburg Elem 70.0% 59.7% 1.2% 0.7% 0.7% 0.0% 72.9% 0.3% 24.1% 0.1% 0.0%

BAY AREA COMMUNITY RESOURCES Oakland, San Rafael, San Francisco AUSD, OUSD, WCCUSD, San Rafael Unified, SFUSD (4 districts)

10 Ruby Bridges 61.7% 38.3% 28.3% 0.8% 21.0% 9.4% 16.4% 2.6% 21.3% 0.0% 0.0%

11 Wilson 77.5% 0.0% 4.7% 0.0% 0.7% 0.2% 94.2% 0.0% 0.0% 0.2% 0.0%

12 MLK Elem 71.2% 23.8% 68.2% 0.0% 8.4% 1.5% 14.2% 0.4% 4.2% 1.1% 1.9%

13 Davidson MS 58.8% 26.1% 1.9% 0.2% 4.4% 0.4% 60.9% 0.4% 29.9% 1.3% 0.7%

14 Guadalupe Elem 80.1% 57.5% 4.4% 0.4% 26.2% 13.6% 45.2% 1.7% 3.1% 2.7% 2.7%

2 Source: California Department of Education, DataQuest (2010-2011) [http://dq.cde.ca.gov/dataquest/].


32

15 San Pedro Elem 95.5% 87.6% 1.0% 0.5% 0.0% 0.2% 98.1% 0.0% 0.2% 0.0% 0.0%

BRIGHT TOMORROWS Lucia Mar Unified School District

16

Grover Beach Elem 71.8% 33.8% 1.1% 0.2% 0.8% 3.2% 57.4% 0.0% 34.2% 2.3% 0.8%

17 Nipomo Elem 73.9% 33.8% 0.9% 0.6% 1.3% 0.6% 64.0% 0.0% 31.7% 0.9% 0.0%

18 Oceano Elem 90.8% 61.2% 1.3% 0.4% 0.0% 0.7% 80.5% 0.0% 16.4% 0.7% 0.0%

BUTTE COUNTY OFFICE OF EDUCATION RURAL COUNTIES

19 Biggs Elem 84.6% 24.9% 2.0% 1.6% 1.2% 0.0% 43.4% 0.0% 50.6% 1.2% 0.0%

20 Cedarwood Elem 71.9% 0.4% 0.0% 2.1% 0.4% 0.0% 10.9% 0.0% 79.5% 7.1% 0.0%

21

Four Winds (K-12)

22 Ishi Hills MS 67.8% 1.6% 1.1% 4.8% 2.9% 0.3% 15.2% 0.0% 64.1% 11.7% 0.0%

23 Palermo MS 80.1% 10.5% 1.2% 8.3% 5.9% 0.0% 32.7% 0.2% 41.5% 10.2% 0.0%

24 Wyandotte Elem 90.6% 27.3% 10.3% 5.8% 30.3% 0.3% 15.2% 0.0% 32.4% 5.8% 0.0%

CAPS—CREATIVE BEFORE AND AFTERSCHOOL PROGRAMS FOR SUCCESS San Bernardino Unified School District

25 Cypress Elem 92.5% 0.0% 21.3% 0.7% 1.3% 1.3% 66.5% 0.9% 7.4% 0.7% 0.0%

26 Hunt Elem 95.9% 0.0% 26.5% 1.2% 0.6% 0.1% 64.0% 0.3% 6.2% 0.6% 0.6%

27 Lytle Creek Elem 97.1% 0.0% 1.6% 0.3% 0.3% 95.3% 0.0% 1.9% 0.0% 0.3% 0.0%


33

GIVE EVERY CHILD A CHANCE Central Valley Manteca Unified School District (10); San Joaquin County Office of Education (3)

28 August Knodt Elem 62.7% 26.4% 20.7% 0.5% 9.0% 9.4% 48.7% 0.9% 9.7% 1.0% 0.0%

29

Golden West (K-8) 72.9% 28.0% 5.5% 1.3% 2.8% 1.1% 62.0% 0.0% 25.8% 1.4% 0.0%

30 Banta Elem 58.4% 24.8% 2.7% 0.0% 0.7% 0.0% 57.4% 0.7% 35.9% 2.7% 0.0%

HARMONIUM SDCOE, SDUSD + San Isidro USD

31 Beyer Elem 80.9% 87.4% 1.3% 0.0% 0.0% 0.0% 98.1% 0.3% 0.3% 0.0% 0.0%

32 Perkins Elem 93.1% 0.0% 2.8% 0.0% 0.0% 0.2% 95.2% 0.0% 1.2% 0.6% 0.0%

33 Webster Elem 93.1% 0.0% 34.9% 0.0% 3.6% 0.7% 52.6% 1.2% 2.6% 3.8% 0.5%

POMONA UNIFIED SCHOOL DISTRICT

34 Alcott Elem 96.9% 56.4% 4.0% 0.3% 1.8% 0.2% 92.3% 0.0% 1.1% 0.2% 0.1%

35 Allison Elem 84.1% 36.4% 4.5% 0.0% 1.5% 0.5% 88.4% 0.0% 2.5% 2.5% 0.0%

36 Arroyo Elem 93.8% 63.0% 2.9% 0.3% 0.3% 0.1% 94.8% 0.4% 0.9% 0.3% 0.1%

37 Harrison Elem 90.0% 41.4% 11.7% 0.9% 0.9% 0.9% 79.3% 0.2% 4.3% 1.5% 0.2%

38 Madison Elem 95.2% 56.6% 2.8% 0.3% 1.9% 0.0% 94.2% 0.0% 0.4% 0.0% 0.3%

39 Vejar Elem 94.4% 46.5% 1.5% 0.1% 5.7% 0.0% 92.1% 0.2% 0.3% 0.1% 0.0%

40 Westmont Elem 88.7% 43.6% 1.6% 0.0% 2.4% 0.5% 89.3% 0.4% 5.3% 0.4% 0.2%

SAY (Social Advocates for Youth), SDCOE, San Diego USD

41 Encanto Elem 87.0% 0.0% 9.4% 0.0% 1.8% 0.3% 84.4% 0.7% 1.5% 2.0% 0.0%

42 Jones Elem 62.3% 0.0% 11.6% 0.0% 9.9% 2.9% 23.2% 0.6% 36.2% 15.7% 0.0%


34

THINK TOGETHER MORENO Valley

43 Rainbow Ridge Elem 87.1% 37.5% 15.1% 0.2% 1.0% 0.6% 74.2% 0.4% 4.5% 1.0% 3.0%

44 Butterfield Elem 88.7% 0.0% 14.1% 0.1% 1.9% 0.7% 74.8% 1.1% 5.3% 1.6% 0.4%

THINK TOGETHER TUSTIN USD (6); Santa Ana USD (15)

45 Greenville Elem 61.0% 41.3% 0.9% 0.1% 12.9% 0.8% 78.0% 0.2% 4.4% 2.7% 0.0%

46 King Elem 97.0% 85.0% 0.4% 0.1% 0.0% 0.0% 99.4% 0.0% 0.1% 0.0% 0.0%

47 Walker Elem 95.4% 83.4% 0.0% 0.0% 5.2% 0.0% 94.8% 0.0% 0.0% 0.0% 0.0%

48 Washington Elem 92.5% 76.7% 0.1% 0.1% 1.0% 0.0% 97.2% 0.6% 0.9% 0.1% 0.0%

49 Heideman Elem 88.9% 74.4% 1.6% 0.0% 0.8% 0.8% 95.6% 0.6% 0.6% 0.0% 0.0%

50 Veeh Elem 80.0% 52.7% 4.5% 0.2% 5.4% 1.0% 76.2% 0.6% 9.3% 2.3% 0.4%

THINK TOGETHER ONTARIO

51 Central (K-8) 86.2% 52.4% 2.2% 0.3% 0.0% 0.1% 93.6% 0.4% 2.7% 0.6% 0.0%

52 Danks MS 75.4% 19.1% 2.2% 0.7% 1.1% 0.7% 83.4% 0.9% 9.9% 1.1% 0.1%

TWIN RIVERS SCHOOL DISTRICT, Sacramento

53 Allison Elem 85.6% 27.6% 18.1% 0.8% 2.0% 2.0% 39.3% 1.5% 32.1% 3.8% 0.3%

54 MLK Elem 98.1% 32.5% 24.0% 1.3% 26.0% 0.6% 37.7% 2.4% 7.4% 0.6% 0.0%

PRO-YOUTH HEART VISALIA

55 Houston Elem 98.2% 58.5% 1.7% 0.2% 3.7% 0.0% 92.6% 0.0% 1.9% 0.0% 0.0% 56

Washington Elem 85.8% 34.3% 3.1% 2.2% 1.6% 0.3% 74.1% 0.3% 16.2% 1.2% 0.9%

57 Mitchell Elem 70.3% 26.6% 0.9% 2.3% 3.9% 0.5% 66.4% 0.0% 25.3% 0.8% 0.0%

58 Mountain View Elem 57.1% 8.2% 2.2% 4.3% 2.7% 0.2% 51.2% 0.0% 37.6% 1.3% 0.5%

59 Strathmore 93.5% 48.8% 0.6% 0.6% 0.4% 0.4% 87.0% 0.1% 10.5% 0.3% 0.0%


35

6 Woodville 96.8% 79.0% 0.0% 0.0% 0.0% 0.0% 87.0% 0.0% 13.0% 0.0% 0.0%

WOODCRAFT RANGERS Los Angeles USD

61 99th St. Elem 91.6% 0.0% 26.2% 0.0% 0.3% 0.0% 71.3% 0.0% 0.0% 0.0% 2.2%

62 Nevada Elem 82.0% 0.0% 5.4% 0.0% 6.5% 5.0% 76.7% 0.0% 6.3% 0.0% 0.0%

63 Nimitz MS 86.0% 0.0% 0.1% 0.2% 0.1% 0.1% 99.1% 0.0% 0.3% 0.0% 0.0%

BAY AREA AFTERSCHOOL ALL STARS SAN Jose

64 Grant Elem 75.2% 64.1% 2.0% 0.4% 5.6% 1.0% 84.5% 0.6% 3.2% 1.0% 1.8% 65

Lowell Elem 84.0% 70.2% 3.0% 0.5% 9.2% 1.0% 82.6% 0.7% 1.5% 0.7% 0.7%


36

3 Source: California Department of Education, DataQuest (2010-2011) [http://dq.cde.ca.gov/dataquest/].

APPENDIX B. COMPUTER AND INTERNET ACCESS AT YEAR 1 STUDY SITES3

REGION

Agency Name DISTRICT

# computers less than 48 mo. old

FRL

# Students per computer

# Classrooms w/ high speed internet

1 Ball Jr. High 323 3.8 42

2 Cerritos Elem 38 12.4 28

3 Disney Elem 76 8.9 35

4 Gauer Elem 70 10 27

5 Mann Middle 159 7.2 32

6 Ross Elem 215 5.1 52

7 Calmecac (K-8) 82 12.1 52

8 Cloverdale Elem 38 20.3 38

9

Healdsburg Elem 8 91.9 41

10 Ruby Bridges 0 0 20

11 Wilson 92 6.2 20

12 MLK Elem 86 3 21

13 Davidson MS 311 2.7 70

14 Guadalupe Elem 35 13.7 21

15 San Pedro Elem 125 3.3 21

16 Grover Beach Elem 140 3.4 29

17 Nipomo Elem 134 3.5 32

18 Oceano Elem 178 2.5 33

19 Biggs Elem 53 4.7 13

20 Cedarwood Elem 29 8.2 14

21 Four Winds (K-12) N/A N/A N/A

22 Ishi Hills MS 139 2.7 16


37

23 Palermo MS 123 3.3 31

24 Wyandotte Elem 59 5.6 19

25 Cypress Elem 284 2 34

26 Hunt Elem 168 4.3 36

27 Lytle Creek Elem 82 8.9 36

28 August Knodt Elem 62 13.9 49

29 Golden West (K-8) 82 7.7 41

30 Banta Elem 59 5.1 16

31 Beyer Elem 178 2.1 32

32 Perkins Elem 210 2.4 30

33 Webster Elem 246 1.7 40

34 Alcott Elem 37 25.6 40

35 Allison Elem 34 11.7 4

36 Arroyo Elem 136 5.6 34

37 Harrison Elem 50 10.6 20

38 Madison MS (Elem?) 323 2.1 31

39 Vejar Elem 175 5.6 43

40 Westmont Elem 5 110.2 29

41 Encanto Elem 158 3.9 65

42 Jones Elem 303 1.1 26

43 Rainbow Ridge Elem 63 12.7 25

44 Butterfield Elem 49 18.1 40

45 Greenville Elem 73 13.5 36

46 King Elem 40 20.7 43

47 Walker Elem 3 204.3 25


38

48 Washington Elem 220 4.6 59

49 Heideman Elem 91 5.5 16

50 Veeh Elem 73 6.6 20

51 Central (K-8) 100 6.7 34

52 Danks MS 344 3 46

53 Allison Elem 146 2.7 19

54 MLK Elem 927 0.5 51

55 Houston Elem 95 5.7 35

56 Washington Elem 37 8.7 18

57 Mitchell Elem 144 4.6 32

58 Mountain View Elem 51 12.3 27

59 Strathmore 116 6.8 40

60 Woodville 170 3.1 40

61 99th St. Elem 22 26.6 42

62 Nevada Elem 28 19.1 37

63 Nimitz Elem 486 4.3 102

64 Grant Elem 118 4.3 41

65 Lowell Elem 132 3.1 27

power of d stem year 1 (2011-12) report of ......stem programming implementers who serve as mentors...

Documents