massachusetts math and science initiative evaluation

8/2/2019 Massachusetts Math and Science Initiative Evaluation

1/28

Commissioned by Mass Insight Education

Phase 1 Evaluation Report

February 2012

MassMath+ScienceInitiative

AnevaluationoftheimpactoftheMassMath+Science

InitiativeonstudentsparticipationandsuccessonAdvancedPlacementexamsandenrollmentinpost-secondaryeducation


2/28

The Institute for Strategic Leadership and Learning (INSTLL, LLC) is an education research and consulting

firm that works with educational organizations, state education agencies, districts, and schools to promote

meaningful improvements to our system of public education.

INSTLL works to support the development and spread of innovative ideas to improve public education by

cultivating strategic leadership and learning, supporting the construction of a policy environmentconducive

to innovation and successful implementation of powerful ideas, and engaging in meaningful evaluations of

the various strategies and interventions employed to support teaching and learning.

www.instll.com

2012 Institute for Strategic Leadership and Learning. All rights reserved.

INSTLL,LLC

Acknowledgments

The author formally thanks William Guenther, President, Mass Insight Education and Morton Orlov II,

President, Mass Math + Science Initiative for commissioning this evaluation and providing considerable time

with the author answering questions, discussing preliminary findings, and contributing insights regarding the

Mass Math + Science Initiative. Many thanks also go to Wesley Chin, MMSI Operations Manager, for

providing all data sets used in the evaluation and supporting the analysis of data.

AbouttheAuthor

Brett Lane, President and Founding Member of INSTLL, LLC, an education consulting firm, has over 15

years experience working with state education agencies, districts, and schools on a wide range of research,

consultative and policy-oriented educational initiatives. As an evaluator and policy analyst, Mr. Lanes areas

of expertise include district and school improvement, state and district turnaround strategies, state

accountability systems, teacher effectiveness, high school reform, and charter schools. Mr. Lanes current

work involves researching and informing the policy conditions necessary for states and districts to effectivelysupport local district and school improvement efforts. Mr. Lane is currently working with leaders in multiple

states, including Massachusetts, to develop and implement policies and strategies that will improve public

education and close achievement gaps.

Mr. Lane has authored articles and policy briefs on a variety of education issues, including charter schools,

school turnaround, state systems of support, and district improvement. His most recent publications include

Rapid District Improvement: How districts engage in rapid and sustainable improvement efforts and

Transforming a Statewide System of Support: The Idaho Story, both published through the Center on

Innovation & Improvement.

Mass Insight Education, a 501(c)(3) non-profit organization based in Boston, MA, was founded in 1997. Itslaunch reflected the high priority that business, government, and education leaders placed at that time on the

success of Massachusetts' nascent standards-based reform drive, set in motion by the passage of the Education

Reform Act of 1993. It is the sister organization ofMass Insight Global Partnerships, which has worked since

1989 to keep Massachusetts and its businesses and institutions globally competitive. Mass Insights national

work focuses on district and state strategies to turn around low performing schools.
http://www.instll.com/http://www.instll.com/http://www.instll.com/http://www.massinsight.com/http://www.massinsight.com/http://www.instll.com/http://www.instll.com/


3/28

Abstract 1

Introduction 2

Closing Achievement Gaps and Improving College Readiness and Success 3

The MMSI Theory of Action 4

Distinguishing Features of the Mass Math + Science Initiative 5

Evaluation Overview 6

Evaluation Analysis and Findings 7

Evaluation Finding 1 9




Exploring the hypothetical impact of the MMSI in current, non-MMSI schools 14

College Readiness and Success: Preliminary observations 15

References 16

Appendix A: Summary of @Scale STEM Programs

Appendix B: Evaluation Methodology and Comparison School Selection

Appendix C: Listing of Colleges Receiving MMSI Graduates

TableofContents

A note about the College Boards Advanced Placement (AP) Program

The Mass Math + Science Initiative (MMSI) as a program and as evaluated in this report, is designed to increasestudents participation and success in AP courses, specifically Math, Science, and English courses. All references toAP courses in this report refer to the College Boards Advanced Placement program, which enables students topursue college-level studies while still in high school. The Advanced Placement program includes more than 30college-level courses, each culminating in a rigorous exam; the MMSI program supports students taking andsucceeding in Math, Science, and English AP courses. As documented by the College Board, AP courses providestudents with the opportunity to earn college credit, advanced placement or both. Each AP course is modeled upon a

comparable college course. College faculty ensure that AP courses align with college level standards by definingcurricular expectations for each course and through a review of all AP teachers syllabi.

Participation in an AP course culminates with students taking a college-level assessment that is scored by collegeand university faculty and experienced AP teachers. The College Board verifies that an AP Exam score of 5 isequivalent to a grade of A in the corresponding college course. An AP Exam score of 4 is equivalent to grades of A-,B+ and B in college, and a score of 3 is equivalent to grades of B-, C+ and C in college. Most four-year collegesand universities grant students credit based on successful AP Exam scores or 3 or greater. College Board research hassubstantiated that students who score a 3 or higher on AP Exams typically experience greater academic success incollege and are more likely to graduate on time than otherwise comparable non-AP peers.

Adapted from the The College Boards 7th Annual AP Report to the Nation


4/28

Abstract

The Mass Math + Science Initiative (MMSI) is a

statewide program, currently comprised of 53 schools in

four cohorts, that is designed to dramatically increase

students participation and performance in Advanced

Placement (AP) courses (Math, Science, and English)

and to increase the number of students matriculating to

and graduating from college. Initiated in 2007 and ledby Mass Insight Education, MMSI is part of a National

Math and Science Initiative to address the national

decline in math and science education. Building upon its

success, MMSI is in the midst of scaling up its program

by expanding to additional schools and leveraging

private and public funding to do so. Documenting and

evaluating the progress that MMSI has made in meeting

program goals is a key step in making this transition.

The evaluation is organized in two phases: Phase 1

focuses on the impact of the MMSI program on

improving students participation and success on AP

exams, as well as providing a preliminary analysis of

college readiness and attendance. Phase 2, due out in

June 2012, will assess the efficacy of the MMSI

program as it is implemented in various high school

contexts.

Since its inception in 2008-09, students in MMSI

schools have taken 13,969 AP exams and have scored 3

or better on 6,426 AP exams.

There is strong longitudinal evidence that the

participation rate (the number of exams taken per1000 students) and the success rate (the number of

exams scoring 3 or better per 1000 students) in

MMSI schools has increased significantly over time.

MMSI schools have madesignificant gains in AP

participation and success in relationship to their

own baseline and they are outperforming similarly

situated groups of non-MMSI schools, grouped

according to need, demographics, and students

income.

The impact of the MMSI program is most pronounced

among African American and Hispanic students in

high-need (e.g., urban, high-poverty schools) and

mid-need (schools between 35 and 60 percent low

income) schools.High- and mid-need MMSI schools

are successfully increasing the number of African

American and Hispanic students enrolling in AP

classes while maintaining success rates that are on

par with similar schools that have not increased

student participation.

The ability of MMSI schools to dramatically increase

African American and Hispanic students participation

in AP courses without a corresponding decline in

students success on AP exams suggests that there is

tremendous untapped capacity among students in

schools across the state that is not being accessed.

There are thousands of students who are not being given

the opportunity to take and succeed in advanced math,

science, and English courses, a crucial contributing

factor to improving college and career success.

1

Illustrative data points and observations:

In 2010, over 7,800 exams were taken in MMSI schools, out

of total eligible student population of 18,955. In contrast, only

3,685 AP exams were taken in the comparison, non-MMSI

schools, out of total eligible student population of 22,911.

Measured as a participation rate, 412 exams were taken for

every 1000 students in MMSI schools, compared to 160exams taken for every 1000 students in non-MMSI schools.

Using the growth rate observed in non-MMSI schools and

controlling for the different starting points of schools, the

following are estimates of the impact of the MMSI program,

in real numbers.

In the 2010-11 school year:

Students in all MMSI schools took approximately 3,473

additional exams that may not have been taken if the

MMSI program had not been implemented.

Students in high- and mid-need MMSI schools took

approximately 1,616 additional exams that may not have

been taken if the MMSI program had not been

implemented.

African American and Hispanic students in high-need

MMSI schools took approximately 335 additional exams

that may not have been taken if the MMSI program had

not been implemented.

If the high-need non-MMSI schools were to have had a

participation rate AND a performance success rate

similar to the MMSI high-need schools, an additional

683 exams may have scored 3 or better in 2010-11.

Additional evaluation findings and data points that merit

further investigation and will be explored in Phase 2 of the

evaluation include:

Significant differences in course-taking patterns between

high- and mid-need MMSI schools.

Wide between-school variance in AP participation and

success rates.


5/28

Introduction

The Mass Math + Science Initiative (MMSI) is a

replication program designed to dramatically increase

students participation and performance in Advanced

Placement (AP) courses (Math, Science, and English)

and to increase the number of students matriculating

and graduating from college, particularly among

traditionally underserved populations. In 2007,

Massachusetts was one of six states selected to

participate in an innovative program led by the

National Math and Science Initiative to address the

decline in math and science education in the United

States. A multi-year initiative, MMSI was organized by

Mass Insight Education in partnership with the

Commonwealth of Massachusetts and private funders

and is currently Massachusetts largest academic high

school math and science program focused on increasing

student access to, and preparation for, STEM careers.

Current Scope and Demand for MMSI

The MMSI program currently includes 53 schools from

rural, suburban and urban locales across the state and is

comprised of four cohorts of schools and nearly 8000

students participating in AP Math, Science, or English

courses. Each school enters into a formal performance

partnership with the Mass Math + Science Initiative

that describes mutual expectations and performance

goals. Cohort 1 began implementation in 2008-09 andsubsequent cohorts (Cohorts 2, 3, and 4) have entered

the program on an annual basis. An additional 24

schools are seeking access and funding to participate in

MMSI beginning in 2012-13.

Why Advanced Placement?

Research shows that students who enroll in and take AP

exams while in high school tend to perform better in

college (e.g., they have higher college GPAs and are

more likely to graduate in four years) than students that

do not take AP courses (Murphy & Dodd, 2009;

Hargrove, et al., 2008). As a state, Massachusetts has a

history of being relatively successful in having students

enroll in and perform well on AP exams (e.g., score 3 or

higher on an AP exam), compared with other states. In

2010, Massachusetts ranked 5th in the nation in AP

success (as measured by the College Board), with 23

percent of graduating seniors scoring 3 or better at least

once during high school. However, Massachusetts is

not faring nearly as well in supporting African

American, Hispanic, and low income students to

achieve success on AP exams. In 2010, Massachusetts

ranked 17th and 48th (out of 50 states and the District

of Columbia) in the percent of African American and

Hispanic students having success on an AP exam.

Improving Equity and Excellence in Massachusetts

Massachusetts has considerable room to improve in

supporting low income, African American, and

Hispanic students in gaining the requisite skills and

knowledge, particularly in STEM related courses,

necessary to succeed in college. This crucial point has

not been lost by Massachusetts leaders, as Governor

Deval Patrick and Secretary of Education Paul Reville

have set high-visibility goals for STEM success related

to increased student interest, performance, and

graduation from college (Executive Order #513.)

Illustrating the states commitment to STEM programs,

Governor Patricks supplemental FY12 budget includes$500,000 for STEM and the Governor has requested

$1.5 million in the FY13 budget and $2.4 million for

AP initiatives.

Evaluating What Works

Through its first three years of operation, the MMSI

program has provided strong evidence that it is capable

of dramatically supporting the states goals related to

STEM and reducing the achievement gaps. Schools are

eager to participate, AP enrollment in MMSI schools

has increased, and the number of students in MMSI

schools having success on AP exams has increased. As

the only comprehensive, widely applicable high school

academic program with college going impact and

metrics that is available to schools in Massachusetts, it

is important to understand exactly how, and to what

extent, MMSI is currently contributing to closing the

achievement gap and improving students readiness

for college.

Table 1. Massachusetts AP uccess Rate ranking, 2010

Massachusetts AP Success Rate

All Students 5th in nation

Hispanic Students 48th in nation

African American Students 17th in nation

Source: College Board 7th Annual AP Report to the Nation

2


6/28

The Mass Math + Science Initiative is situated at the

nexus of two critical challenges facing public education

and has taken a systems-based approach to driving

significant change and addressing these challenges. Thetwo challenges that MMSI directly addresses are: (1) the

ever-growing urgency to close achievement gaps,

especially among low income, African American, and

Hispanic students and (2) the crucial economic

imperative to fully prepare students to attend and

succeed in college, especially in STEM related courses

and career pathways. In addressing these dual

challenges, MMSI utilizes multiple levers for change and

sets precise metrics for measuring impact.

Closing the Achievement GapDespite Massachusetts consistently high scores on the

National Assessment of Education Progress (NAEP),

achievement gaps persist across a variety of performance

indicators. Table 2 provides a broad overview of the

achievement gaps in performance on the MCAS (as

measured by the Composite Performance Index) and the

percent of students enrolled in grade 8 Algebra, a widely

accepted indicator of college readiness.

Table 3 displays the stark disparity in students relative

success on AP exams, in this instance success on a Math

AP exam. In 2010-11, only 116 of the Math exams taken

by African American students and 211 of the Math

exams taken by Hispanic studentsin each case less

than 1 percentscored a 3 or better on a Math AP exam.

In contrast, over 5 percent of the exams taken by White

students scored a 3 or better on a Math AP exam. Of

particular note, the percent of exams taken by AfricanAmerican and Hispanic students in MMSI schools is

significantly higher than the rate observed in all other

non-MMSI schools across the state.

College Readiness and Success and STEM Pathways

Massachusetts has been and continues to be a leader

among states in setting goals and taking action to close

achievement gaps. In 2010, theAct Relative to the

Achievement Gap was signed into law, which provided

districts with unprecedented flexibility to change the

conditions in which schools operate. More recently (inNovember of 2011,) Governor Patrick announced four

goals and related strategies that reinforce efforts to close

achievement gaps, one of which is to ensure that students

are prepared for both college and career success.

The urgency to improve college readiness and success is

reinforced by the ever-increasing need to encourage and

prepare all students for career pathways in science,

technology, engineering and math. From the funding of

the STEM Pipeline and PreK-16 Regional Networks to

more recent efforts, such as leveraging federal Race tothe Top funding to enhance STEM teacher training,

Massachusetts has continued to develop policy, provide

funding, and support programs aimed at increasing both

the number of students preparing for and entering STEM

career pathways and the number of qualified STEM

teachers. However, improving college readiness and

success, especially in STEM, remains a pressing

challenge.

Table 2: Two measurand college readines

es of studes, 2006 and

nts acade2010

ic perf rmance

EnrollGrade 8

mentlgebra*

3rd GraE

e CPI**:A

2006 2010 2006 2010

African American 41% 36% 72 71

Hispanic 33% 28% 68 71

White 48% 47% 88 88

Low Income 34% 29% 72 73

* Percent of students en

** The Composite Perfo

extent to which a group

Source: 2011 District An

olled in Gra

mance Inde

is approachi

alysis and R

de 8 Algebr

is a 1-100 i

ng proficie

view Tool,

a

ndex sho

cy, a scor

MA ESE.

ing the

of 100.

Table 3:or be

umber andtter on a M

percent of exams tath AP exam, 2010-1

en that scored a 3, by Ethnicity

Number of exams scoringof group scoring 3

3 or better andor better

In MMSISchools In all non-MMSISchools, statewide All school (MMSIand non-MMSI)

AfricanAmerican

42/2442(1.72%)

74/11889(.62%)

116/14331(0.81%)

Hispanic72/4354(1.65%)

139/17420(.79%)

211/21774(0.96%)

White539/10916

(4.93%)5341/99115

(5.39%)5880/110,031

(5.34%)

Source: College Board, 2 10-11

Dual Challenges: Closing Achievement Gaps and

Improving College Readiness and Success

3
http://www.mass.gov/governor/pressoffice/pressreleases/2011/111109-education-summit-2011.htmlhttp://www.mass.gov/governor/pressoffice/pressreleases/2011/111109-education-summit-2011.htmlhttp://www.mass.gov/governor/pressoffice/pressreleases/2011/111109-education-summit-2011.htmlhttp://www.mass.gov/governor/pressoffice/pressreleases/2011/111109-education-summit-2011.htmlhttp://www.mass.gov/governor/pressoffice/pressreleases/2011/111109-education-summit-2011.html


7/28

The MMSI Theory of Action

The Mass Math + Science Initiative endeavors to

directly address the described challengesclosing the

achievement gap and improving college readiness and

success in STEMthrough a systems-based approach

that uses multiple levers to drive improvement.

On a programmatic level, MMSI is replicating a

research-based initiative with defined program elements

that incorporate multiple levers for changing teacher

and student behaviors and that is linked with specific

measurable goals and targets to gauge progress.

Interested schools submit a competitive proposal to

participate. Once accepted, participating schools enter

into aperformance partnership with MMSI that

articulates expectations for the partnership and sets

goals for program participation. Teachers receive 7 to 9

days of teacher training and students participate in up to

18 additional hours of study sessions during the school

year. Teacher and student incentives, as awards for

participation and success on AP exams, supplement the

teacher and student support and serve as an additional

lever for change.

At a strategic level, MMSI employs strategies designed

to ensure the scalability and sustainability of MMSIwithin and across multiple schools. In doing so, MMSI

has intentionally integrated strategies that characterize

successful reform efforts, including Massachusetts own

effort to raise academic standards.

A significant infusion of national investment, more than

$16 million through 2013, will have been used to

jumpstart the MMSI effort in Massachusetts. To

capitalize on its investment and initial success in 53

schools, MMSI is currently in the process of

transitioning to in-state public/private financing.MMSIs initial infusion of funding, combined with its

intensive and ongoing focus on goals and measuring

progress, has positioned MMSI to successfully expand

to new schools and sustain efforts in schools

demonstrating success.

The Mass Math + Science Initiative:

Provided a significant infusion of money and

investment to accelerate the implementation of the

program;

Developed mutually agreed upon goals and targets

(e.g., benchmarks) and is evaluating progress in

order to adjust and improve, as needed; and

Is transitioning to public/private financing.

Increase participation, as measured by increased

student enrollment in math science, and English AP

courses

Increase performance, as measured by an increase in

the number of qualifying scores (scores of 3, 4, or 5)on AP exams.

Increase college success, as measured by an increase

in the number of students matriculating to and

graduating from college.

Program Elements

Lead to specific goals and the

use of precise metrics

That are reinforced by strategies intended

to expand and sustain successes

4


8/28

There are a variety of educational programs and

initiatives that have been developed and are being

used to promote college readiness and success inSTEM. Each program comes with its own unique

approach, theory of action, and capacityfeatures

that distinguish one program from another. For

instance, some programs focus primarily on

building student interest in STEM, while others

focus more intensively on developing students and

teachers content knowledge, as a means of building

capacity around STEM courses and content.

To better understand the variety of STEM programs

being used in Massachusetts and to inform this

evaluation, we provide a brief summary of MMSIs

distinguishing features, based on the questions

listed below. A detailed overview of @Scale STEM

programs in Massachusetts is provided in Appendix

A.

Distinguishing Features of the Mass Math + Science Initiative

Academic or Interest Focus: Is the program

Comprehensive or Targeted with respect to

academics and content? Or is the program focused

primarily on building student interest?

Scale: Is the program statewide or located in a

geographic area, district, or school?

Goals and Metrics: Does the program have goals

that are measurable and that provide evidence of

impact or are goals and metrics difficult to

measure, or not directly linked to college success?

Incentives: Does the program include incentives

to ensure that all students have equal access and

opportunity to participate and experience success?

Questions to ask about STEM programs

5

1. MMSI s program is designed to directly improve students

academic competency and performance in STEM content

and courses, rather than focusing solely on generating

student interest in STEM career pathways.

2. MMSI is a comprehensive curriculum-based program that

includes incentives for teachers and students.

3. MMSI is one of two programs (the other being Project Leadthe Way) that are available statewide and that have goals

and metrics for measuring program outputs.

4. MMSI has specific goals related to STEM (e.g., increasing

AP enrollment in math, science and English), academic

achievementand closing achievement gaps, and increasing

college success.

MMSIs Distinguishing Features

In the growing field of STEM-

related programs, and in

particular the @Scale STEM

programsthose programs

endorsed by Governor Patricks

STEM Advisory Councilthe

MMSI program distinguishes

itself in four areas.

What are the Distinguishing Features of the Mass Math + Science Initiative in relationship to

other STEM and College Readiness programs?

Understanding the differences among STEM programsthe levers that they employ to promote college

readiness, the scale of the various programs, and the precision of the measures (e.g., metrics) used to

evaluate impactis crucial when considering the efficacy and cost benefit of various programs.


9/28

Evaluation Purposes

The purposes of this evaluation are to:

Evaluate the extent to which the Mass Math + Science Initiative is meeting its stated program goals,

focusing on increased enrollment, AP success, and college success among African American and Hispanic

students.

Explore, through interviews and data analysis, how the MMSI program elements are understood and

leveraging changes in teacher and student actions.

Provide information that can be used to inform conversations regarding policy development.

Evaluation Overview

To better understand and to document the extent of

MMSIs success, Mass Insight Education commissioned

INSTLL, LLC to conduct an external evaluation of the

MMSI program. Evaluation questions were developed in

collaboration with Mass Math + Science Initiative

leaders. Building upon MMSIs stated program goals, as

well as issues related to fidelity of implementation, the

purposes of the evaluation were articulated and key

evaluation questions identified, as listed below.

A description of the evaluation methodology, including

the process used to select the comparison schools used

in the analysis and identified in the body of this report,

is provided in Appendix B.

Evaluation Questions

1. How effective is the MMSI program in increasing

students participation and performance on AP exams,

especially among students from diverse backgrounds?

2. To what extent does participation in the MMSI

program increase students readiness for, and success

in, 2- and 4-year colleges?

3. Are there differences in MMSI program impactacross schools? What are the emerging trends and

how can Mass Insight use this information to improve

the MMSI program?

4. How are the components of the MMSI performance-

based, integrated program working together to

improve students participation, performance, and

college readiness and success?

Data Sources

College Board Data: Number of Math,

Science, and English Exams taken and Number

of Exams scoring 3 or higher.

College Board Data for MMSI schools and

Comparison schools.

National Student Clearinghouse: Verification

of MMSI students attending and graduatingfrom college.

Interviews and focus groups with school

leaders, teachers, and students.

6

Two Phase Evaluation

Phase 1, the subject of this report, focuses on

evaluation questions #1 and #2 and examines the

extent to which the MMSI program is meeting

program goals.

In Phase 2, we will conduct focus group sessions

and interviews with key stakeholders (district and

school staff, students, policymakers) to better

understand how the MMSI program is working in

particular settings, and how the program elements

are working together to achieve program goals.


10/28

Important Evaluation Terms

Performance measures are displayed as raw numbers(e.g., the number of exams taken) and as rates (e.g., 150 out of 1000 students). The two key performance measures are

Participation and Success.

AP Participation: Student participation in AP is based on the number of Math, Science, and English exams

taken in a school year and is expressed as a Participation Rate (number of exams taken for every 1000 students.)

AP Success: Student success in AP is based on the numberof Math, Science, and English exams that receive a

score of 3 or better and is expressed as a Success Rate (number of exams scoring 3 or better for every 1000

students.)

A separate AP Performance Success Rate is a measure of the rate of success among students that take exams. Listed

as a percentage, the AP Performance Success Rate is as follows:

AP Success/AP Participation = AP Performance Success Rate

Evaluation Design In Brief

A quasi-experimental design was employed to assess the impact

of MMSI on students participation and success in AP classes.

Cohort Analysis

Using data from 2006-07 to 2011-12, we compared the

participation and success of students in MMSI schools (Cohorts1, 2, and 3) with the participation and success of students in all

non-MMSI schools across the state.

Comparison Analysis

MMSI program schools represent a range of communities across

the state and serve different, and diverse, student populations. In

order to more accurately measure the impact of MMSI in

schools that serve a particular student population, we set criteria

for grouping schools based on student income and the schools

accountability status, as determined by the state. Using the

criteria for each group (defined below), we separated the MMSIschools into three distinct groups, identified groups of similarly

situated comparison schools, and then analyzed the differences

in performance between MMSI and non-MMSI comparison

schools, within each grouping.

Group A Criteria: Low Income greater than (>) 60 percent orSchool Accountability Status: Level 3 or 4

Group B Criteria: Low Income greater than (>) 35 percent andDoes not meet Group 1 Criteria

Group C Criteria: Low Income less than (


11/28

Evaluation Questions in Detail

1. How effective is the MMSI program in increasing student participation and performance, especially among

students from diverse backgrounds?

a. How effective is the MMSI program in increasing the participation of students (All students, African

American students, and Hispanic students) in Advanced Placement classes?

b. How effective is the MMSI program in increasing student performance (All students, African Americanstudents, and Hispanic students) on AP exams?

2. What is the effect of the MMSI program on students AP participation and success, in comparison with non-

MMSI schools across the state and with similarly situated (e.g., comparison) schools?

3. How are the components of the MMSI performance-based, integrated program working together to improve

students participation, performance, and college readiness? (To be addressed in Phase 2 of the evaluation)

High Need/Low Income (Group A)MMSI Program Schools

Enroll% LowIncome

High Need/Low Income (Group A)Comparison Schools

Enroll% LowIncome

Boston - Brighton High 1233 79.5 Boston - Burke High 697 76.0

Boston - East Boston High 1382 84.2 Boston - Charlestown High 942 79.7

Chelsea - Chelsea High 1353 75.1 Boston - Madison Park Voc Tech 1286 69.8

Fall River - Durfee HS 2258 70.2 Boston - The English High 777 71.7

Revere - Revere High 1474 68.8 Brockton - Brockton HS 4145 71.4

Salem - Salem High 1231 53.2 Everett - Everett High 1710 61.6

Springfield - HS of Science and Technology 1267 83.7 Fitchburg - Fitchburg High 1146 62.9

Springfield - Springfield Central HS 2046 73.5 Holyoke - Holyoke HS 1268 64.0

Worcester - Burncoat HS 1072 59.3 Lowell - Lowell HS 3403 66.0

Worcester - North HS 1149 77.1 Lynn - Classical High 1401 77.0

Worcester - South HS 1297 72.4 Lynn - Lynn English High 1739 73.5

Lynn - Lynn Voc Tech Institute 809 87.1

New Bedford - New Bedford HS 2711 63.2

Somerville - Somerville High 1344 70.8

Springfield - HS Of Commerce 1286 80.9

Springfield - Putnam Voc Tech HS 1545 79.8

Mid Need (Group B)MMSI Program Schools

Enroll% LowIncome

Mid Need (Group B)Comparison Schools

Enroll% LowIncome

Athol-Royalston - Athol High 451 49.2 Adams-Cheshire - Hoosac Valley High 668 35.9

Boston - O'Bryant Math & Science 1234 49.8 Haverhill - Haverhill High 1748 41.1

Boston Collegiate Charter 554 41.2 Holbrook - Holbrook Jr Sr High 472 37.5

Gill-Montague - Turners Fall High 294 50.7 Leominster - Leominster Center Tech 655 47.8

Greenfield - Greenfield High 483 61.1 Mohawk Trail - Mohawk Trail Reg'l High 566 32.0

Malden - Malden High 1799 61.0 North Adams - Drury High 578 52.2

Marlborough - Marlborough High 1457 36.4 North Central Essential 368 46.2

Quaboag Regional - Quaboag Regional High 582 39.7 Palmer - Palmer High 558 34.6

Randolph - Randolph High 744 55.2 Pioneer Charter Science 294 51.7

Salem Academy Charter 309 42.1 Pioneer Valley - Pioneer Valley Reg'l 536 25.4

Ware - Ware High 498 46.8 Pittsfield - Pittsfield High 976 44.1

Worcester - Worcester Tech HS 1400 61.2 Quincy - Quincy High 1441 53.9

Taunton - Taunton High 1920 39.9

Winchendon - Murdock Middle/High 755 49.5

Tables 4 and 5: Listing of High Need/Low Income (Group A) and Mid Need (Group

B) schools used in Comparison Analysis*

*Note: An Analysis of Low Need (Group C) MMSI schools is notincluded because all low need MMSI schools are in Cohort 2 or 3

8


12/28

Snapshot Analysis - MMSI Cohorts

Table 6 and Chart 1 depict the change in the number of Science AP exams taken per 1000 Juniors/Seniors for MMSI

Cohorts and for non-MMSI schools, statewide. In MMSI schools, participation rates in Science have increased at a

significantly higher rate

than in non-MMSI schools.The upward trend in

participation rates in Math

and English AP Exams in

MMSI schools mirrors the

growth in the number of

Science AP exams taken, as

displayed in Chart 1.

Statewide, overall AP

participation rates rose

slightly between 2007 and

2011, with approximately

60 to 70 exams being taken

for every 1000 students, in

Science, Math, and English.

*Data Note: The Error Bars used in Chart 1

(above) and in subsequent charts represent 3

Standard Deviations from the statewide

average, and are intended to illustrate the

scope of the increases observed in MMSI

schools.

How effective is the MMSI program in increasing the participation of students in

Advanced Placement classes? (Cohort Analysis)

Key Finding 1: MMSI Cohort Schools have significantly increased the number and percent of

Advanced Placement (AP) Math, Science, and English exams taken by students, in relation to each

Cohorts baseline year and compared with non-MMSI schools.

Data Highlight:In Cohort I schools in 2010-2011, 196 Science AP exams were taken for every 1000

students, which represents a real increase of 570 additional exams that were taken for that year (out of a

total of 918) that would not have been taken if participation rates had not increased.

Baseline Year One Year Two Year Three

Cohort 1 (8 Schools) 75 115 162 196Cohort 2 (11 Schools) 77 127 147 -

Cohort 3 (26 Schools) 51 87 - -

Statewide, Non-MMSI Schools

65 72 71 77

0"

20"

40"

60"

80"

100"

120"

140"

160"

180"

200"

Baseline" Year"One" Year"Two" Year"Three"

Numbero

fExamsTakenper1000Students

Chart'1.'All'MMSI'Cohorts:'Number'of'Science&AP'Exams'Taken''

per'1000'Juniors/Seniors,'by'Cohort'and'Year'in'Program'

Cohort"1"(8"Schools)"



Statewide,"non?MMSI"Schools*"

Table 6. Number of Science AP Exams taken per 1000 Juniors/Seniors, byCohort and Year in Program

9


13/28

Snapshot Analysis - Comparison

Schools

An analysis of participation rates among

Group A (High-need) and Group B(Mid-need) MMSI schools and their

comparison schoolsshows that MMSI

schools are improving AP participation

rates and consistently outperform

similar, non-MMSI schools. Chart 2

displays the number of exams taken per

1000 students for MMSI and non-MMSI

school. High-need schools (both MMSI

and non-MMSI) are represented in Red

and mid-need represented in Blue.

High- and mid-need MMSI schools have experienced consistent growth in AP participation rates since beginning the

program. High-need MMSI schools increased AP participation rates from a baseline of 182 exams per 1000 students

to 346 exams per 1000, in Year 3 of the program. Mid-need MMSI schools increased participation rates from a

baseline of 243 per 1000 to 612 per 1000 in Year 3. Of particular note is the fact that the AP participation rate in high-

need MMSI schoolsthose schools with the most

diverse and low income studentsconsiderably

outpaces the participation rate in mid-need non-MMSI

schools, which tend to serve a less diverse student

population.

Variance in course-taking patterns. A comparison

of course-taking patterns among high- and mid-need

MMSI schools highlights an additional piece of

information that may be important for ongoing

program improvement. As displayed in Chart 3, high-

need MMSI schools have experienced the most

success in increasing the rate of students taking

English AP exams, with smaller increases in Math

and Science. Mid-need schools, in comparison, have

seen dramatic increases in the number of Science

and Math exams being taken, as displayed in Chart4. Overall, mid-need MMSI schools are exhibiting

higher overall participation rates and more

accelerated growth than observed in high-need MMSI

schools, although each group of schools is having

success in meeting MMSI program goals and is

outperforming its comparison group.

How effective is the MMSI program in increasing the participation of students in

Advanced Placement classes? (Comparison Analysis)

10

Baseline( Year(1( Year(2( Year(3(

Math(Exams(Taken( 82.9( 117.5( 170.5( 155.5(

Science(Exams(Taken( 71.0( 119.7( 230.0( 289.0(

English(Exams(Taken( 92.8( 166.5( 182.7( 167.1(

0.0(

50.0(

100.0(

150.0(

200.0(

250.0(

300.0(

Numberper1000

Chart'4.'Number'of'Exams'Taken'per'1000'Students'by'AP'

Exam'Content'Area,'mid@need'MMSI'Schools'

0"

100"

200"

300"

400"

500"

600"

700"

Baseline" Year"1" Year"2" Year"3"

Chart&2.&Number&of&exams&taken&per&1000&students&in&All&Subjects,&MMSI&and&

NonMMSI&high&and&midneed&schools.&&

Highneed"MMSI:"Exams"Taken" Highneed"NonMMSI:"Exams"Taken"

Midneed"MMSI:"Exams"Taken" Midneed"NonMMSI:"Exams"Taken"


Math(Exams(Taken( 39.6( 62.4( 85.1( 79.0(

Science(Exams(Taken( 69.9( 75.4( 81.5( 90.1(

English(Exams(Taken( 73.5( 129.3( 141.9( 176.6(

0.0(

50.0(

100.0(

150.0(

200.0(

Number

per1000

Chart'3.'Number'of'Exams'Taken'per'1000'Students'by'

AP'Exam'Content'Area,'highneed'MMSI'Schools'


14/28

How effective is the MMSI program in increasing students performance on the

AP exam?

Key Finding 2. MMSI schools are experiencing significant growth in the real number of exams scoring 3 or

higher and in the number of exams scoring 3 or better for every 1000 students (known as a schools success

rate) in relation to each Cohorts baseline year and compared with the success rate in non-MMSI

comparison schools.

Data Highlight:If the high-need non-MMSI schools were to have had a participation rate AND a performance

success rate similar to the MMSI high-need schools, an additional 683 exams may have scored 3 or better in

2010-11.

Evidence of Latent Student Capacity: The significantly higher success rate in MMSI schools demonstrates that

MMSI schools are enrolling and supporting students that have not typically accessed AP courses. Likewise, the data

suggests that there are students in comparison non-MMSI schools who are capable of performing AP course work

but are not afforded the opportunity to do so.

Snapshot Analysis

MMSI schoolswithin cohorts and

organized according to needhave increased

students participation rate and the rate at

which students score 3 or better on AP

exams. Charts 5 and 6 display the

participation and success rates for high- and

mid-need MMSI schools and comparison

schools.

In each group of MMSI schools, we see

evidence of a rise in participation rate

coupled with a significant increase in

success rate, over time. In high-need MMSI

schools, the Year 3 success rate has doubled

since the baseline (from 52 to 115 per 1000

exams scoring 3 or better); in contrast, the

non-MMSI success rate increased from 42 to

52 per 1000. Mid-need MMSI schools have

experienced even greater improvement over

time, moving from a baseline success rate of

112 per 1000 students scoring 3 or better to a

success rate of 283 per 1000 in Year 3. As

displayed in Chart 8, the success rate in mid-

need MMSI schools has dramatically

increased to the extent that thepercent of

students passing AP exams in MMSI

schools has exceeded the percent of students

taking exams in non-MMSI schools.

11


Non0MMSI:(Exams(Taken( 97( 17( 119( 126(

Non0MMSI:(Exams(>(3( 42( 44( 44( 52(

MMSI:(Exams(Taken( 182( 274( 38( 346(

MMSI:(Exams(>(3( 52( 87( 116( 115(

(

5(

1(

15(

2(

25(

3(

35(

4(

Numberper1000

Chart'5.'Number'of'AP'Exams'taken'and'number'of'AP'Exams'

scoring'>'3'per'1000'students:'HighBneed'MMSI'and'nonBMMSI'

schools;'All'students'and'subjects'



Non0MMSI:(Exams(>(3( 87( 113( 127( 137(

MMSI:(Exams(Taken( 243( 44( 583( 612(

MMSI:(Exams(>(3( 112( 177( 261( 283(

(

1(

2(

3(

4(

5(

6(

Numberper1000

Chart'6.'Number'of'AP'Exams'taken'and'number'of'AP'Exams'

scoring'>'3'per'1000'students:'MidBneed'MMSI'and'nonB

MMSI'schools;'All'students'and'subjects'


15/28

How effective is the MMSI program in increasing the enrollment of African

American and Hispanic students in Advanced Placement classes?

Key Finding 3: MMSI Cohort Schools have significantly increased the number and percent of Advanced

Placement (AP) Math, Science, and English exams taken by African American and Hispanic students, in

relation to each Cohorts baseline year and compared with non-MMSI schools.

Data Highlights:In Year 3, 239 AP exams were taken for every 1000 African American and Hispanic students

attending high-need MMSI schools; in contrast, only 63 exams were taken for every 1000 African American and

Hispanic students in high-need non-MMSI schools.

African American and Hispanic students in high-need MMSI schools are almost 5 times as likely to take an AP

Science exam as are students attending high-need non-MMSI schools.

If the non-MMSI high-need schools were to have had a participation rate similar to the MMSI high-need schools, an

additional 1,016 AP exams would have been taken by African American and Hispanic students in 2010-11.

Snapshot Analysis

Charts 7 and 8 display the AP

participation rate for high- and mid-

need schools for all AP exams and

for Science AP exams. In addition to

the visible and significant

differences in participation rates

between MMSI and non-MMSI

schools, the data shows that MMSI

and non-MMSI schools had

different starting points, orbaseline measures of AP

participation. The difference in

initial participation rates suggest that

schools that applied for and were

selected for MMSI entered the

program with substantially higher

levels of readiness, as measured

by the baseline AP participation

rate.

Despite their already higher AP

participation rates for African

American and Hispanic students,

MMSI high- and mid-need schools

exhibited significant growth in AP

participation rate over time, in

comparison to non-MMSI high- and

mid-need schools.

12


Highneed(MMSI:(Exams(Taken( 109( 167( 170( 239(

Highneed(NonMMSI:(Exams(Taken( 42( 56( 51( 63(

Midneed(MMSI:(Exams(Taken( 159( 286( 378( 370(

Midneed(NonMMSI:(Exams(Taken( 45( 61( 72( 86(

0(

50(

100(

150(

200(

250(

300(

350(

400(

NumberofExamsTakenper1000Students

Chart'7.'Number'of'Exams'taken'per'1000'African'American'and'Hispanic'Students'in'high@'and'

mid@need'MMSI'and'Non@MMSI'schools'


Highneed(MMSI:(Exams(Taken( 50( 58( 54( 99(

Highneed(NonMMSI:(Exams(Taken( 14( 18( 14( 19(

Midneed(MMSI:(Exams(Taken( 67( 99( 146( 202(

Midneed(NonMMSI:(Exams(Taken( 10( 22( 19( 25(

0(

50(

100(

150(

200(

250(

300(

350(

400(

NumberofExamsTakenper1000Students

Chart'8.'Number'of'Science'AP'Exams'taken'per'1000'African'Amercian'students'in'high@'

and'mid@need'MMSI'and'Non@MMSI'schools'


16/28

How effective is the MMSI program in increasing the performance of African

American and Hispanic Students?

Key Finding 4. MMSI schools are dramatically and significantly increasing the number of African Americans

and Hispanics scoring 3 or better on a Math, Science, or English AP exam, in MMSI schools and relative to

schools across the state.

Data Highlights: In 2010-11, students at MMSI schools comprised 19 percent of the Hispanic and African American

student population in the state,yet accounted for 43 percent of African American and Hispanic AP exam taking

and 36 percent of exam takers scoring a 3 or better.

If the high-need non-MMSI schools were to have had a participation rate AND a performance success rate similar to

the high-need MMSI schools, an additional 206 exams taken by African American and Hispanic students may have

scored 3 or better in 2010-11.

Snapshot Analysis

Charts 9 and 10 display the growing

participation and success ratesamong African American and

Hispanic students in high- and mid-

need MMSI schools. The rate of

exams taken by African American

and Hispanic studentsscoring 3 or

higherin MMSI schools (at 60 per

1000 students in high-need schools

and 142 per 1000 in mid-need

schools) is equal to or, in the case of

mid-need schools, substantially

higher than the rate at which examsare taken, among similar students in

non-MMSI schools.

As noted in Finding #3, high- and

mid-need MMSI schools have

significantly increased the

participation rate among African

Americans and Hispanics. The

increase in the raw numbers of

exams taken by African American

and Hispanic students has

subsequently contributed to a

significant increase in the actual

numbers of exams scoring 3 or

higher and growth in the AP success

rate since the beginning of the

MMSI program.

13



Non0MMSI:(Exams(>(3( 9( 16( 31( 29(

MMSI:(Exams(Taken( 159( 286( 378( 37(

MMSI:(Exams(>(3( 48( 86( 117( 142(

(

5(

1(

15(

2(

25(

3(

35(

4(

Nubmerper1000

Chart'10.'Number'of'AP'Exams'taken'and'number'of'AP'Exams'scoring'>'3'per'1000'

students:'Midneed'MMSI'and'nonMMSI'schools;'African'American'and'Hispanic'Students''



Non0MMSI:(Exams(>(3( 8( 9( 1( 13(

MMSI:(Exams(Taken( 19( 167( 17( 239(

MMSI:(Exams(>(3( 16( 27( 44( 6(

(

5(

1(

15(

2(

25(

3(

35(

4(

Numberper1000

Chart'9.'Number'of'AP'Exams'taken'and'number'of'AP'Exams'scoring'>'3'per'1000'students:'

HighBneed'MMSI'and'nonBMMSI'schools;'African'American'and'Hispanic'Students''


17/28

What is the hypothetical impact of the MMSI in current, non-MMSI schools?

Exploring the hypothetical impact of the MMSI program.

Table 8 displays the actual numbers of eligible students (e.g., eligible juniors and seniors) in MMSI and non-MMSI

comparison schools for the 2010-2011 school year, the number of exams taken, and the number of exams scoring 3 or

better. The last two columns, Participation Rate and Performance Success Rate, provide a measure of the extent to

which schools in a particular group are successful in having students enroll in and take AP exams (the ParticipationRate) and the extent to which schools are able to support students in scoring 3 or better (the Performance Success

Rate.) The top set of rows display information forall students. The bottom set of rows focuses on the performance of

African American and Hispanic students.

A primary goal of the MMSI program is to increase enrollment and participation, especially among diverse students.

The data shows that MMSI schools are dramatically increasing participation rates. It seems logical (and in fact it was

expected) that increased participation rates would likely result in decreased performance success ratesas schools

enroll more students into AP courses (some of whom may not have been targeted for such classes), a smaller

percentage of these students would score 3 or better. However, dramatically increasing participation rates (as has been

accomplished) would theoretically overcome declining performance success rates and still contribute to increased

overall numbers of students taking AP courses and scoring 3 or higher.

All StudentsEligibleStudents

# of ExamsTaken

# of ExamsScoring >3

ParticipationRate

PerformanceSuccess Rate

Non-MMSI (All Groups) 22911 3658 2012 16.0% 55.0%

MMSI (All Groups) 18955 7812 3519 41.2% 45.0%

Non-MMSI (High- and Mid-) 15923 2444 1186 15.3% 48.5%

MMSI (High- and Mid-) 11054 4331 1621 39.2% 37.4%

Non-MMSI High-need 11687 1477 607 12.6% 41.1%

MMSI High-need 7047 2335 778 33.1% 33.3%

African American andHispanic Students

EligibleStudents

# of ExamsTaken

# of ExamsScoring >3

ParticipationRate

PerformanceSuccess Rate

Non-MMSI (All Groups) 8211 522 137 6.4% 26.2%

MMSI (All Groups) 6609 1582 421 23.9% 26.6%

Non-MMSI (High- and Mid-) 7487 486 109 6.5% 22.4%

MMSI (High- and Mid-) 5722 1380 335 24.1% 24.3%

Non-MMSI High-need 6826 429 90 6.3% 21.0%

MMSI High-need 4177 884 181 21.2% 20.5%

14

Table 8. Summary Table of 2010-11 Participation Rates and Performance Success Rates, MMSI and Non-MMSI Schools

Analysis

The numbers for all students (the top set of rows) follows the above stated logic, as MMSI groups exhibit substantiallyhigher participation rates and significantly lower performance success rates. However, our analysis of the experience

of African American and Hispanic students tells a different story and provides compelling and provocative evidence of

the real impact of the MMSI program related to efforts to close the achievement gap. In the aggregate and by need-

based group, MMSI schools are realizing dramatically higher participation rates for African American and

Hispanic Students in comparison with non-MMSI schools, while maintaining similar performance success rates.

Instead of experiencing a decline in the rate of exams scoring 3 or higher, MMSI schools are increasing the

numbers of students taking AP exams and keeping the performance success rate steady. This finding has significant

implications for the potential expansion of the MMSI program in additional schools across the state.

Note the similarPerformance Success Ratecoupled with MMSI's high

Participation Rates

Significantly higherParticipation Rates

correlated with smallerPerformance Success Rat


18/28

College Readiness and Success: Preliminary Observations

Tracking Students from High School through College

Measuring the real impact of the MMSI program on individual students requires tracking students as they graduate

from high school, enroll in a 2- or 4-year college, and subsequently graduate (or not) from college. The Mass Math +

Science Initiative team is working carefully with partner schools and with the National Student Clearinghouse to

track the college pathway of those students that take one or more AP exams while enrolled in a MMSI program

school.

Between 2008-09 and 2010-11, 5,269 MMSI students graduated from a MMSI program school, of which 4,024 have

been identified who were accepted to and attended college the first year after graduating from high school, for an

overall college attendance rateof 76 percent. According to National Student Clearinghouse data, MMSI students

(to reiterate, those students who took at least one AP exam while attending a MMSI school) are currently attending

384 different 2- and 4 -year colleges. The primary receiving colleges include the University of Massachusetts

(Amherst, Boston, Dartmouth, and Lowell campuses) and other major colleges and universities in Massachusetts, as

Table 9. Most Frequently Selected Universities/Colleges, MMSI Schools (2008-11)

University/CollegeNumber of MMSI schools*

sending MMSI students

UNIVERSITY OF MASSACHUSETTS ATAMHERST

41/44

FRAMINGHAM STATE UNIVERSITY 31/44

UNIVERSITY OF MASSACHUSETTS-BOSTON 31/44

UNIVERSITY OF MASSACHUSETTS-DARTMOUTH

31/44

BRIDGEWATER STATE UNIVERSITY 28/44

UNIVERSITY OF MASSACHUSETTS ATLOWELL

27/44

SUFFOLK UNIVERSITY 25/44

WESTFIELD STATE UNIVERSITY 25/44

NORTHEASTERN UNIVERSITY 23/44

WORCESTER POLYTECHNIC INSTITUTE 23/44

WORCESTER STATE UNIVERSITY 23/44

MASSACHUSETTS COLLEGE OF PHARMACY 22/44

WENTWORTH INSTITUTE OF TECHNOLOGY 22/44

* Data was a ailable for 44 of the 45 MMSI schools

Measuring the impact of the MMSI program on students readiness for and success in college

Measuring the actual impact of the MMSI program on increasing the number of students matriculating to and

graduating from college requires a metric that takes into account two factors that influence whether or not students

attend college and that are under the control of the MMSI program. For instance, some schools may be very good at

increasing AP enrollment, but not as attentive to supporting students matriculation to college. Other schools may

provide excellent support to students applying for college, but fail to actively enroll sufficient numbers of students in

AP courses. The success of the MMSI program is a function of (1) a schools ability to enroll students in AP courses

(AP participation rate) and (2) the capacity of the school to support students taking AP courses towards applying for

and attending college (college attendance rate.)

A College Success Report, to be produced in Summer 2012, will measure a schools success in having students

participate in AP courses and matriculate to college to quantify the impact of the MMSI program.

15


19/28

References

College Board.(2011). The 7th Annual Report to the Nation. New York, NY: The College Board

Commonwealth of Massachusetts. (2010). A Foundation for the Future: Massachusetts Plan for Excellence in

STEM Education (Science, Technology, Engineering and Mathematics),

Hargrove, L., Godin, D. & Dodd, B. (2008). College Outcomes Comparisons by AP and Non-AP High School

Experiences. New York, NY: The College Board

Murphy, D. & Dodd, B. (2009).A Comparison of College Performance of Matched AP and Non-AP Student

Groups. New York, NY: The College Board

16
http://www.nga.org/files/live/sites/NGA/files/pdf/1112STEMSUMMITCEDRONE.PDFhttp://www.nga.org/files/live/sites/NGA/files/pdf/1112STEMSUMMITCEDRONE.PDFhttp://www.nga.org/files/live/sites/NGA/files/pdf/1112STEMSUMMITCEDRONE.PDFhttp://www.nga.org/files/live/sites/NGA/files/pdf/1112STEMSUMMITCEDRONE.PDF


20/28

Appendix A

At-a-glance overview of @Scale and selected initiatives

focused on STEM and College Readiness

A - 1

Considering the Scope, Scale, and Measurable Impact of @Scale Initiatives

Dramatically increasing the numbers of low income and diverse students attending and succeeding in

college, especially in Science, Technology, Engineering, and Math (STEM) is crucialboth for

individuals and for the economic health and sustainability of the Commonwealth. There are a variety of

educational programs and initiatives that have been developed and are being used to promote college

readiness and success in STEM. Understanding the differences among these programsthe levers that

they employ to promote college readiness, the scale of the various programs, and the precision of the

measures (e.g., metrics) used to evaluate impactis crucial to consider when making decisions. To the

point, identifying those programs that use multiple levers to effect change (e.g., to contribute to system

change) and that have measurable evidence of impact and considering how to best support these

programs is part of the role of our elected officials.

Which programs are working and having success in preparing students to attend and succeed in

college?

To assist policymakers and constituents in exploring and discussing some of the distinctions amongavailable programs (in this case, the @Scale Initiatives), INSTLL, LLC and Mass Insight have prepared

this at-a-glance overview of the @Scale programs. The @Scale programs are organized according to

the following dimensions, building upon research on program implementation.

Academic or Interest Focus: Is the program Comprehensive or Targeted with respect toacademics and content? Or is the program focused primarily on building student interest?

Scale: Is the program statewide or located in a single district or school (or geographic area)? Goals and Metrics: Does the program have goals that are measurable and the provide evidence

of impact or are goals and metrics difficult to measure, or not directly linked to college success?

@Scale Programs in Brief: Focus, Goals and Metrics, and Scale

Goal and Metrics Scale

Academic Approach:

Comprehensive

Mass Math + Science InitiativeMultiple Goals

Measurable OutcomesStatewide

Project Lead the WaySingle Goal

Measurable OutcomesStatewide

Academic Approach:

Targeted

Science Transfer InitiativeMultiple Goals

Measurable OutcomesLocal

BioTeach Single GoalOutcomes Local

Interest-Based Approach

Advanced Robotics InitiativeSingle Goal

OutcomesLocal

DIGITSSingle Goal

OutcomesStatewide


21/28

Appendix A

At-a-glance overview of @Scale and selected initiatives

focused on STEM and College Readiness

A - 2

Policy Considerations and Information Display

The following pages provide a selection of questions that can be used to prompt discussion and a

comparison of @Scale programs, followed by a set of graphic organizers that provides additional

information.We offer the following topics and questions for consideration and deliberation regarding STEM related

initiatives.

Addressing Goals related to STEM, College Success, and Closing the Achievement Gap

Does the program address multiple goals (e.g., STEM state goals, College Success, andClosing the Achievement Gap) or is the program focused on a particular goal?

How does the program address the Governors goals of eliminating achievement gaps andensuring college readiness?

Program impact and the capacity of the program to measure impact

How much of an impact does the program have with low income and diverse studentpopulations? Does the program have specific measures to assess the impact of its program with respect to

improving college readiness, decreasing the achievement gap, and increasing participation and

enrollment in STEM related courses?

Using multiple levers to accelerate and sustain improvement

Does the program have the capacity to expand, and to do so with fidelity? Does the program utilize multiple levers (e.g., teacher training, improved curriculum, teacher

and student incentives) to achieve its goals?

Does the program involve a multi-year engagement, or performance agreement, with thedistrict or school?

Does the program contribute to building the capacity of the district or school?Overall cost and cost related to program scalability and effectiveness

What is the cost per student, per teacher, and per school? What would it cost to scale the program statewide? What is the cost effectiveness of the program? Specifically, what is the return on investment for

the program?


22/28

Appendix B

Evaluation Methodology

B - 1

Data Set

The data set used in the analysis contained school-level counts ofthe number of eligible students (defined as

the number of juniors and seniors in a school, and by ethnicity and gender), the number of exams taken, and

the number of exams scoring 3 or higher for MMSI schools and Comparison Schools. The data set included

separate counts for each school by ethnicity (African American, Hispanic, White, Asian) and gender for each of

the three AP content areasMathematics, Science, and Englishthat are part of the MMSI program. For

instance, the data set used in the analysis included the number of exams taken (and the number of exams scoring

3 or higher) forMath, but not separately for AB Calculus, BC Calculus, and Statistics . Also, data on low-

income students was not available for all schools and subsequently was not used in the analysis.

Calculating Participation and Success Rates

Participation Rates were computed by (1) dividing the number of exams taken (for a particular group) by the

total number of eligible students for a particular group and then (2) multiplying by 1000.

Success Rates were computed by (1) dividing the number of exams scoring 3 or higher (for a particular group)

by the total number of eligible students for a particular group and then (2) multiplying by 1000.

Cohort Analysis

The MMSI program has accepted and initiated work with a new cohort of schools on an annual basis, beginning

with Cohort 1 in 2008-09. In the analysis, the baseline for each cohort was computed by aggregating the

number of eligible students (and related number of exams taken and number of exams scoring 3 or higher) from

the two years prior to the school beginning the MMSI program. The latest year for which data is available is the

2010-11 school year.

Table B1. Computation of Baseline Data for MMSI Cohort Schools

Baseline Years in MMSI

Cohort 1 (n=8) 06-07 and 07-08 08-09, 09-10, 10-11 and 11-12

Cohort 2 (n=11) 07-08 and 08-09 09-10, 10-11 and 11-12

Cohort 3 (n=26) 08-09 and 09-10 10-11 and 11-12

Chart 1 and Table 6 (in the body of the report) present baseline and annual data for cohorts 1, 2, and 3. As

displayed in Table B1, the baselines do not come from the same years; rather, they are the baseline for the

particular cohort. The data for statewide, non-MMSI schools uses the same baseline and annual data as Cohort

1. To the point, a valid statistical comparison can be made between Cohort 1 and the non-MMSI schools, as is

presented on Chart 1. However, we also wanted to look at the relationship among cohorts, which is why we

combined the cohort data in one chart, with each cohorts baseline presented as the same baseline year.

Comparison Analysis

The District Analysis and Review Tool (DART) developed by the Massachusetts Department of Elementary and

Secondary Education was used to identify the comparison schools used in the analysis. Extensive information on

the DART, including the statistical method used by DART to identify comparable districts and schools, is

located at: http://www.doe.mass.edu/apa/dart/ .

While the DART is designed to identify comparable districts and schools for a single district (or school), the

tool is not specifically designed to identify a set of comparison schools for a group of schools. However, the

DART provides an excellent starting point for identifying a potential pool of comparison schools. The following

is the process used by INSTLL to use the DART to develop a meaningful set of comparison schools.

1. We grouped the MMSI schools into three groups, based on percentage of low-income students, each schools

accountability status, and based on discussions with MMSI program staff. As described in the body of the

report, we used the following criteria:

Group A Criteria: Low Income greater than (>) 60 percent or School Accountability Status: Level 3 or 4


23/28

Appendix B

Evaluation Methodology

B - 2

Group B Criteria: Low Income greater than (>) 35 percent and does not meet Group 1 Criteria Group C Criteria: Low Income less than (


24/28

C - 1

APPENDIX C

High school students who graduated between 2008-2011 from an high school that is or was participatingwith the Mass Math + Science Initiatives (MMSI) Advanced Placement Training and Awards Program

(APTAP), and who were enrolled in an AP math, science and/or English course(s), are or were enrolled at

the colleges/universities listed below.

The color gradient column indicates the number of MMSI APTAP high schools that the

college/university has received graduates from.

ADRIAN COLLEGE 1

ALBANY COLLEGE OF PHARMACY AND HEALTH

SCIENCES2

ALBERTUS MAGNUS COLLEGE 1ALVERNIA UNIVERSITY 1

AMERICAN INTERNATIONAL COLLEGE 7

AMERICAN UNIVERSITY 5

ANDREWS UNIVERSITY 2

ANNA MARIA COLLEGE 7

ARIZONA STATE UNIVERSITY 2

ASHLAND UNIVERSITY 1

ASNUNTUCK COMMUNITY COLLEGE 1

ASSUMPTION COLLEGE 18

ATLANTA METROPOLITAN COLLEGE 1

AUBURN UNIVERSITY 1

BARNARD COLLEGE 1

BARRY UNIVERSITY 1

BATES COLLEGE 4

BAY PATH COLLEGE 3

BAY STATE COLLEGE 5

BECKER COLLEGE 11

BENJAMIN FRANKLIN INSTITUTE OF TECHNOLOGY 1

BENNINGTON COLLEGE 1

BENTLEY UNIVERSITY 11

BERKLEE COLLEGE OF MUSIC 5

BOSTON COLLEGE 17

BOSTON UNIVERSITY 18

BOWDOIN COLLEGE 3

BRANDEIS UNIVERSITY 13

BRIDGEWATER STATE UNIVERSITY 28

BRIGHAM YOUNG UNIVERSITY 2

BRIGHAM YOUNG UNIVERSITY -IDAHO 1

WINTER/SPRING

BRISTOL COMMUNITY COLLEGE

BROWN UNIVERSITY

BRYANT UNIVERSITY 1

BRYN MAWR COLLEGE

BUCKNELL UNIVERSITY

BUFFALO STATE COLLEGE

BUNKER HILL COMMUNITY COLLEGE 1

BUTTE COMMUNITY COLLEGE

CALIFORNIA INSTITUTE OF TECHNOLOGY

CALIFORNIA STATE UNIVERSITY - FULLERTON

CAMBRIDGE COLLEGE

CAPE COD COMMUNITY COLLEGE

CAPITAL COMMUNITY COLLEGE

CARLETON COLLEGE

CARNEGIE MELLON UNIVERSITY

CASTLETON STATE COLLEGE

CENTRAL CONNECTICUT STATE UNIVERSITY

CENTRAL PIEDMONT COMMUNITY COLLEGE

CENTRE COLLEGE

CHAMPLAIN COLLEGE

CHARLESTON SOUTHERN UNIVERSITY

CHATTAHOOCHEE TECHNICAL COLLEGE

CLARK UNIVERSITYCLARKSON UNIVERSITY

CLEMSON UNIVERSITY

CLEVELAND INSTITUTE OF ART

COASTAL CAROLINA UNIVERSITY

COLBY COLLEGE

COLBY SAWYER COLLEGE

COLGATE UNIVERSITY


25/28

C - 2

COLLEGE OF CHARLESTON 2

COLLEGE OF THE HOLY CROSS 19

COLLEGE OF WILLIAM & MARY 2

COLLEGE OF WOOSTER 1

COLORADO COLLEGE 1

COLORADO STATE UNIVERSITY 1

COLUMBIA COLLEGE CHICAGO 2

COMMUNITY COLLEGE OF RHODE ISLAND 1

COMMUNITY COLLEGE OF VERMONT 1

CONNECTICUT COLLEGE 2

CORNELL UNIVERSITY 3

CULINARY INSTITUTE OF AMERICA 1

CUNY BERNARD M. BARUCH COLLEGE 1

CUNY HUNTER COLLEGE 1

CUNY QUEENS COLLEGE 1

CURRY COLLEGE 14

DANIEL WEBSTER COLLEGE 2

DARTMOUTH COLLEGE 1

DEAN COLLEGE 8

DENISON UNIVERSITY 4

DEPAUL UNIVERSITY 1

DES MOINES AREA COMMUNITY COLLEGE 1

DICKINSON COLLEGE 1

DRAKE UNIVERSITY 1

DREXEL UNIVERSITY 2

EARLHAM COLLEGE 1EASTERN CONNECTICUT STATE UNIVERSITY 3

EASTERN NAZARENE COLLEGE 8

ECKERD COLLEGE 1

ELIZABETH CITY STATE UNIVERSITY 1

ELMIRA COLLEGE 5

ELMS COLLEGE 11

ELON UNIVERSITY 4

EMBRY-RIDDLE AERONAUTICAL UNIVERSITY -

ARIZONA1

EMBRY-RIDDLE AERONAUTICAL UNIVERSITY -

DAYTONA1

EMERSON COLLEGE 16

EMMANUEL COLLEGE 18

ENDICOTT COLLEGE 10

ENTERPRISE STATE COMMUNITY COLLEGE 1

EVANGEL UNIVERSITY 1

EVEREST COLLEGE - CHELSEA 1

FAIRFIELD UNIVERSITY 3

FAIRLEIGH DICKINSON UNIVERSITY - TEANECK 2

FISHER COLLEGE-TRADITIONAL

FITCHBURG STATE UNIVERSITY 1

FLORIDA A&M UNIVERSITY

FLORIDA INSTITUTE OF TECHNOLOGY

FLORIDA INTERNATIONAL UNIVERSITY

FLORIDA STATE UNIVERSITY

FORDHAM UNIVERSITY

FRAMINGHAM STATE UNIVERSITY 3

FRANKLIN PIERCE UNIVERSITY

GEORGE MASON UNIVERSITY

GEORGE WASHINGTON UNIVERSITY

GEORGETOWN UNIVERSITY

GEORGIA HIGHLANDS COLLEGE

GEORGIA INSTITUTE OF TECHNOLOGY

GETTYSBURG COLLEGE

GORDON COLLEGE

GOUCHER COLLEGE

GREENFIELD COMMUNITY COLLEGE

GRINNELL COLLEGE

GROVE CITY COLLEGE

GUILFORD COLLEGE

HAMILTON COLLEGE

HAMPSHIRE COLLEGE

HAMPTON UNIVERSITY

HARTWICK COLLEGE

HARVARD UNIVERSITYHARVARD UNIVERSITY - CONTINUING ED 1

HAVERFORD COLLEGE

HESSER COLLEGE - SALEM

HOBART & WILLIAM SMITH COLLEGES

HOFSTRA UNIVERSITY

HOLYOKE COMMUNITY COLLEGE

HOPE COLLEGE

HOUGHTON COLLEGE

HOWARD UNIVERSITY

HUSSON COLLEGE

IONA COLLEGE

ITHACA COLLEGE

JOHNSON & WALES UNIVERSITY 1

JOHNSON STATE COLLEGE

JUNIATA COLLEGE

KEENE STATE COLLEGE 1

KENT STATE UNIVERSITY

KENYON COLLEGE


26/28

C - 3

LA SALLE UNIVERSITY 2

LAKE FOREST COLLEGE 1

LASELL COLLEGE 3

LE MOYNE COLLEGE 2

LEE UNIVERSITY 1

LEHIGH UNIVERSITY 1

LESLEY UNIVERSITY 10

LEWIS & CLARK COLLEGE OF ARTS & SCIENCES 1

LINDENWOOD UNIVERSITY 1

LIPSCOMB UNIVERSITY 1

LONG ISLAND UNIVERSITY 1

LOYOLA UNIVERSITY IN NEW ORLEANS 1

LYCOMING COLLEGE 1

LYNCHBURG COLLEGE 1

LYNDON STATE COLLEGE 2

LYNN UNIVERSITY 1

MACALESTER COLLEGE 2

MACOMB COMMUNITY COLLEGE 1

MAINE MARITIME ACADEMY 2

MARIST COLLEGE 4

MARLBORO COLLEGE 1

MARYLAND INSTITUTE, COLLEGE OF ART 1

MARYMOUNT MANHATTAN COLLEGE 2

MASSACHUSETTS BAY COMMUNITY COLLEGE 9

MASSACHUSETTS COLLEGE OF ART 11

MASSACHUSETTS COLLEGE OF LIBERAL ARTS 14MASSACHUSETTS COLLEGE OF PHARMACY 22

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 4

MASSACHUSETTS MARITIME ACADEMY 3

MASSASOIT COMMUNITY COLLEGE 8

MERCYHURST COLLEGE 1

MERRIMACK COLLEGE 12

MIAMI UNIVERSITY 1

MICHIGAN STATE UNIVERSITY 1

MIDDLEBURY COLLEGE 2

MIDDLESEX COMMUNITY COLLEGE 5

MINNESOTA STATE UNIVERSITY - MANKATO 1

MISSISSIPPI STATE UNIVERSITY 1

MONTANA STATE UNIVERSITY - BOZEMAN 1

MONTCLAIR STATE UNIVERSITY 1

MONTSERRAT COLLEGE OF ART 3

MORRIS COLLEGE 1

MOUNT HOLYOKE COLLEGE 4

MOUNT IDA COLLEGE 8

MOUNT WACHUSETT COMMUNITY COLLEGE

MUHLENBERG COLLEGE

NAZARETH COLLEGE OF ROCHESTER

NEW ENGLAND COLLEGE-SEMESTERS

NEW ENGLAND INSTITUTE OF TECHNOLOGY

NEW RIVER COMMUNITY COLLEGE

NEW YORK UNIVERSITY

NEWBURY COLLEGE

NHTI - CONCORD'S COMMUNITY COLLEGE

NICHOLS COLLEGE

NORMANDALE COMMUNITY COLLEGE

NORTH CAROLINA STATE UNIVERSITY

NORTH PARK UNIVERSITY

NORTH SHORE COMMUNITY COLLEGE

NORTHEASTERN UNIVERSITY 2

NORTHEASTERN UNIVERSITY - LAW/SPCS

NORTHERN ESSEX COMMUNITY COLLEGE

NORTHWESTERN UNIVERSITY

NORWICH UNIVERSITY

NYACK COLLEGE - ROCKLAND UG

OBERLIN COLLEGE

OZARKS TECHNICAL COMMUNITY COLLEGE

PACE UNIVERSITY

PACE UNIVERSITY - PLEASANTVILLE

PENNSYLVANIA STATE UNIVERSITY 1

PHILADELPHIA UNIVERSITYPITZER COLLEGE

PLYMOUTH STATE UNIVERSITY

POLYTECHNIC INSTITUTE OF NEW YORK

UNIVERSITY

POMONA COLLEGE

PRATT INSTITUTE

PRINCETON UNIVERSITY

PROVIDENCE COLLEGE 1

PURDUE UNIVERSITY - WEST LAFAYETTE

QUEENS UNIVERSITY OF CHARLOTTE

QUINCY COLLEGE

QUINNIPIAC UNIVERSITY 1

QUINSIGAMOND COMMUNITY COLLEGE

REGIS COLLEGE 1

RENSSELAER POLYTECHNIC INSTITUTE

RHODE ISLAND COLLEGE

RHODE ISLAND SCHOOL OF DESIGN

RICHLAND COLLEGE-DALLAS CC DISTRICT

RIVIER COLLEGE


27/28

C - 4

ROANOKE COLLEGE 3

ROCHESTER INSTITUTE OF TECHNOLOGY 4

ROGER WILLIAMS UNIVERSITY 9

ROLLINS COLLEGE 1

ROOSEVELT UNIVERSITY 1

ROXBURY COMMUNITY COLLEGE 7

SACRED HEART UNIVERSITY 4

SAINT ANSELM COLLEGE 7

SAINT JOSEPH'S COLLEGE OF MAINE 3

SAINT MICHAELS COLLEGE 6

SALEM STATE UNIVERSITY 17

SALVE REGINA UNIVERSITY 6

SANTA MONICA COLLEGE 1

SANTA ROSA JUNIOR COLLEGE 1

SARAH LAWRENCE COLLEGE 1

SAVANNAH COLLEGE OF ART & DESIGN 1

SCHOOL OF VISUAL ARTS 2

SEMINOLE STATE COLLEGE OF FLORIDA 1

SHENANDOAH UNIVERSITY 1

SIENA COLLEGE 1

SIMMONS COLLEGE 12

SIT GRADUATE INSTITUTE NON TRADITIONAL 1

SKIDMORE COLLEGE 3

SMITH COLLEGE 10

SOUTHERN MAINE COMMUNITY COLLEGE 2

SOUTHERN NEW HAMPSHIRE UNIVERSITY-

UNGRAD DAY 8

SOUTHERN UTAH UNIVERSITY 1

SOUTHERN VERMONT COLLEGE 3

SOUTHWESTERN ADVENTIST UNIVERSITY 1

SOUTHWESTERN OKLAHOMA STATE UNIVERSITY 1

SPELMAN COLLEGE 1

SPRINGFIELD COLLEGE 8

SPRINGFIELD TECHNICAL COMMUNITY COLLEGE 8

ST EDWARDS UNIVERSITY 1

ST JOHNS RIVER COMMUNITY COLLEGE 1

ST JOHNS UNIVERSITY 5

ST JOSEPH COLLEGE 1

ST LAWRENCE UNIVERSITY 2

ST MARY'S COLLEGE OF CALIFORNIA 2

ST MARYS COLLEGE OF MARYLAND 1

ST OLAF COLLEGE 1

STANFORD UNIVERSITY 1

STETSON UNIVERSITY 2

STEVENS INSTITUTE OF TECHNOLOGY 2

STONEHILL COLLEGE 1

SUFFOLK UNIVERSITY 2

SUNY COLLEGE AT FREDONIA

SUNY COLLEGE AT PURCHASE

SUNY COLLEGE PLATTSBURGH

SUNY FARMINGDALE

SUNY FASHION INSTITUTE OF TECHNOLOGY

SUNY FULTON-MONTGOMERY COMMUNITY

COLLEGE

SUNY MOHAWK VALLEY COMMUNITY COLLEGE

SUNY OSWEGO

SUSQUEHANNA UNIVERSITY

SWARTHMORE COLLEGE

SYRACUSE UNIVERSITY 1

TEMPLE UNIVERSITY

TEXAS A&M UNIVERSITY - COMMERCE

TEXAS CHRISTIAN UNIVERSITY

THE CATHOLIC UNIVERSITY OF AMERICA

THE CITADEL MILITARY COLLEGE OF SOUTH

CAROLINA

THE EVERGREEN STATE COLLEGE

THE NEW SCHOOL

THOMAS NELSON COMMUNITY COLLEGE

TRINITY UNIVERSITY

TUFTS UNIVERSITY 1

TULANE UNIVERSITY

TULSA COMMUNITY COLLEGE

UNION COLLEGE

UNIVERSITY OF ALABAMA

UNIVERSITY OF ARIZONA

UNIVERSITY OF BRIDGEPORT

UNIVERSITY OF CALIFORNIA-BERKELEY

UNIVERSITY OF CALIFORNIA-SAN DIEGO

UNIVERSITY OF CALIFORNIA-SANTA BARBARA

UNIVERSITY OF CALIFORNIA-SANTA CRUZ

UNIVERSITY OF CENTRAL FLORIDA

UNIVERSITY OF CHICAGO

UNIVERSITY OF CONNECTICUT 1

UNIVERSITY OF DAYTON

UNIVERSITY OF DELAWARE

UNIVERSITY OF FLORIDA

UNIVERSITY OF HARTFORD 1

UNIVERSITY OF HAWAII AT MANOA

UNIVERSITY OF ILLINOIS @ URBANA

UNIVERSITY OF MAINE, FARMINGTON


28/28

UNIVERSITY OF MAINE, ORONO 3

UNIVERSITY OF MARYLAND - COLLEGE PARK 3

UNIVERSITY OF MARYLAND - UNIVERSITY

COLLEGE1

UNIVERSITY OF MASSACHUSETTS AT AMHERST 41

UNIVERSITY OF MASSACHUSETTS AT LOWELL 27

UNIVERSITY OF MASSACHUSETTS BOSTON 31

UNIVERSITY OF MASSACHUSETTS-DARTMOUTH 31

UNIVERSITY OF MIAMI 4

UNIVERSITY OF NEW ENGLAND 8

UNIVERSITY OF NEW HAMPSHIRE 17

UNIVERSITY OF NEW HAVEN-SEMESTERS 9

UNIVERSITY OF NEW MEXICO 1

UNIVERSITY OF NORTH CAROLINA-PEMBROKE 1

UNIVERSITY OF NORTH CAROLINA-WILMINGTON 4

UNIVERSITY OF OREGON, MAIN CAMPUS 1

UNIVERSITY OF PENNSYLVANIA 3

UNIVERSITY OF PHOENIX 2

UNIVERSITY OF PITTSBURGH 3

UNIVERSITY OF RHODE ISLAND 16

UNIVERSITY OF ROCHESTER 5

UNIVERSITY OF SAN DIEGO 1

UNIVERSITY OF SAN FRANCISCO 2

UNIVERSITY OF SOUTH CAROLINA 1

UNIVERSITY OF SOUTH FLORIDA 3

UNIVERSITY OF SOUTHERN CALIFORNIA 2

UNIVERSITY OF SOUTHERN MAINE 3

UNIVERSITY OF TAMPA 2

UNIVERSITY OF THE ARTS 1

UNIVERSITY OF THE PACIFIC 1

UNIVERSITY OF VERMONT & STATE

AGRICULTURAL COLLEGE11

UNIVERSITY OF VIRGINIA 2

UNIVERSITY OF WISCONSIN - MADISON 1

URSINUS COLLEGE 2

UTAH STATE UNIVERSITY 1

UTICA COLLEGE 1

VASSAR COLLEGE 2

VENTURA COLLEGE

VILLANOVA UNIVERSITY

VIRGINIA COMMONWEALTH UNIVERSITY

VIRGINIA POLYTECH AND STATE UNIV

VIRGINIA STATE UNIVERSITY

WAKE FOREST UNIVERSITY

WASHINGTON ADVENTIST UNIVERSITY

WASHINGTON UNIVERSITY

WELLESLEY COLLEGE

WENTWORTH INSTITUTE OF TECHNOLOGY 2

WESLEY COLLEGE

WESLEYAN UNIVERSITY

WESTERN NEW ENGLAND UNIVERSITY 1

WESTFIELD STATE UNIVERSITY 2

WESTMINSTER COLLEGE OF SALT LAKE CITY

WHEATON COLLEGE

WHEELOCK COLLEGE 1

WILLIAMS COLLEGE

WINGATE UNIVERSITY

WOFFORD COLLEGE

WORCESTER POLYTECHNIC INSTITUTE 2

WORCESTER STATE UNIVERSITY 2

WYOTECH - BLAIRSVILLE

YALE UNIVERSITY

YESHIVA UNIVERSITY

Total Colleges/Universities: 384

massachusetts math and science initiative evaluation

Documents