fryer apollo presentation

The Impact of Apollo 20 on Student Achievement:

Evidence from Year One by

Roland G. Fryer, Jr. Robert M. Beren Professor of Economics

Harvard University EdLabs NBER

CONFIDENTIAL

Outline Apollo 20

Road-‐map for the talk: A.  The Achievement Gap in America B.  The Five Tenets: Mo=va=on and Implementa=on

1.  More Time in School 2.  Small-‐Group Tutoring and Differen=a=on 3.  Human Capital Management 4.  Data-‐Driven Instruc=on 5.  Culture and High Expecta=on

C.  Explana=on of Empirical Methodology D.  Year One Results E.  Verifica=on Checks F.  Context and Wrap-‐up

CONFIDENTIAL 3

Accoun&ng for educa&onal achievement dras&cally reduces racial and socioeconomic inequality across a wide range of important life outcomes.

28.0%

0.6% 0.0%

10.0%

20.0%

30.0%

Not Accounting for Educational Achievement

Accounting for Educational Achievement

The Black-White Wage Gap

182%

38%

0%

100%

200%

300%



The Black-White Incarceration Rate Gap

190%

90%

0%

100%

200%

300%



The Black-White Unemployment Gap

0.196

0.059

0

0.1

0.2

0.3



The Black-White Health Gap (in Standard Deviations)

Overview Educa5onal Achievement Drives Life Outcomes

CONFIDENTIAL

Among ci&es that par&cipate in NAEP, the magnitude of racial differences in educa&onal achievement is startling.

Overview The Achievement Gap

0 10 20 30 40 50 60 70 80

Math

White

Black

Hispanic

0 10 20 30 40 50 60 70 80

Reading

White

Black

Hispanic

CONFIDENTIAL

What’s more, “conven&onal wisdom” solu&ons have failed to solve the problem.

22.3 :1

20.4 :1

18.7 :1

17.9 :1 17.2 :1 17.3 :1

16.0 :1 15.6 :1

14:1

16:1

18:1

20:1

22:1

24:1

1970 1975 1980 1985 1990 1995 2000 2005

Student to Teacher Ra5o

$5,243 $6,049 $6,268

$7,347 $8,790 $8,949

$10,508 $11,438

$0

$5,000

$10,000

1970 1975 1980 1985 1990 1995 2000 2005

Total Expenditure Per Pupil (2008-‐09 $)) $12,116

23.5% 27.5% 49.6% 53.1% 56.8% 61.8%

0%

20%

40%

60%

80%

100%

1961 1971 1981 1991 2001 2006

Percentage of Teachers with a Master's Degree or Higher

76.9% 73.6% 71.4% 72.4% 73.4% 69.3% 69.8% 75.4% 74.7%

0%

25%

50%

75%

100%

1970 1975 1980 1985 1990 1995 2000 2005

HS Graduates as a ra5o of 17 year-‐old popula5on

175

200

225

250

275

300

325

1971 1975 1980 1984 1988 1990 1994 1996 1999 2004 2008

Reading and Math Achievement of 9, 13, and 17 year-‐olds, 1971-‐2008

9 year-‐olds

13 year-‐olds

Overview Conven5onal Wisdom

CONFIDENTIAL

Given the current rate of “No Excuses” charter school growth, charters will take over half a century to educate every child in America and close the achievement gap. That’s too long. We are faced with two op=ons:

1.  Create a market in which only gap closing charters can survive (great idea, tough poli=cs).

2.  Boil down charter school successes into translatable, scalable prac=ces for public schools.

Scaling Up Charter Success Apollo 20

CONFIDENTIAL

Scaling Up Charter Success Apollo 20

One of the original missions of charter schools was to experiment and incubate best prac=ces that could then be transferred to public schools. Dobbie and Fryer (2011a) examine results from 106 charter schools in New York City in order to iden=fy the prac=ces most correlated with student achievement. They find that accoun=ng for five factors explains roughly 40% of the variance in charter school performance.

CONFIDENTIAL 8

The key strategies of Apollo schools.

Scaling Up Charter Success Apollo 20 Strategies

More Time in School •  Extended day, week, and school years are all integral components of successful school

models. In the case of Harlem Children’s Zone’s Promise Academy, students have nearly doubled the amount of time on task compared to students in NYC public schools.

Small Group Tutoring and Differentiation •  In top performing schools, classroom instruction is supplemented by individualized tutoring,

both after school and during the regular school day.

Human Capital Management •  Successful charters reward teachers for performance and hold them accountable if they are

not adding value.

Data-Driven Instruction •  In the top charter schools, students are assessed frequently, and then, in small groups, re-

taught the skills they have not yet mastered.

Culture and Expectations •  In successful schools, students buy into the school’s mission and into the importance of

their education in improving their lives.

CONFIDENTIAL

Implementa5on Human Capital

In addi=on to finding nine new principals, teacher turnover spiked to 53% in Apollo schools over the summer of 2010. Value-‐added data shows that teachers who returned as Apollo teachers had a much stronger history of increasing student achievement in every subject, rela=ve to those who lea.

-‐0.7

-‐0.6

-‐0.5

-‐0.4

-‐0.3

-‐0.2

-‐0.1

0

0.1

0.2

Language Math Reading Science Social Studies

Teacher Value Added

Lea

Stayed

0%

10%

20%

30%

40%

50%

60%

2005-‐06 2006-‐07 2007-‐08 2008-‐09 2009-‐10

Teacher Departure Rates

Apollo Schools

Comparison Schools

CONFIDENTIAL

Implementa5on Increased Time in School

The school day was extended in Apollo schools during the 2010-‐11 school year: 7:45am – 4:15pm Monday through Thursday, and 7:45am – 3:15pm on Fridays. This was an average of an hour longer per school day. The school year was extended by five school days. Apollo students reported for school on August 16, 2010, while the rest of the district began on August 23, 2010.

BoXom line: The difference between instruc=onal =me in 2009-‐10 and 2010-‐11 amounts to approximately 30 school days – that’s 6 addi5onal weeks of school for students.

0

200

400

600

800

1000

1200

1400

1600

1800

Pre-‐Interven=on Typical NYC Charter Post-‐Interven=on No Excuses NYC Charter

Instruc5onal Hours per Year

CONFIDENTIAL

Implementa5on High Dosage Differen5a5on: Tutoring and Double-‐Dosing

Tutors were recruited both na=onally and locally during summer 2010. •  1158 total applicants (through August 9, 2010) •  516 applicants par=cipated in screening •  319 offered a posi=on •  257 accepted a posi=on in one of the nine schools •  173 tutors qualified for end-‐of-‐year student performance bonuses, earning an

average of $3,346. •  96 tutors are returning for the 2011-‐12 school year •  1156 6th graders and 1585 9th graders received 2:1 tutoring in the 2010-‐11 school

year. Middle school students received approximately 215 hours of tutoring/double-‐dosing High school students received approximately 189 hours of tutoring/double-‐dosing

CONFIDENTIAL

Implementa5on Data-‐Driven Instruc5on

Students were regularly assessed in order to provide teachers with accurate data on student mastery. In addi=on to the HISD 3-‐week interim assessments, the Apollo schools administered comprehensive benchmark assessments according to the schedule: • Middle school reading – December, February/March • Middle school math – December, February/March • Middle school science – December, March • Middle school social studies – December, March • High school ELA – December, January • High school math – December, March • High school science – December, March • High school social studies – December, March Aaer each assessment, teachers received student-‐level data and used this to have one-‐on-‐one goal-‐sehng conversa=ons with students.

CONFIDENTIAL

Implementa5on Culture and Expecta5ons

There was a consistent emphasis on college matricula=on at the Apollo schools, and students were supported in their efforts to apply to college.

Apollo High Schools College Applica5on and Acceptance Rates

2-‐Year 4-‐Year

Applied Accepted Applied Accepted

Jones HS 95% 95% 60% 44%

Kashmere HS 90% 90% 89% 42%

Lee HS 97% 97% 84% 47%

Sharpstown HS 94% 94% 81% 55%

“It feels like we’re actually going to school now.” – Lee HS student

“We have supports this year that we didn’t have in the past.” – Sharpstown HS student

“School is harder…but it’s good because we’ll be ready.” – Key MS student

CONFIDENTIAL

Implementa5on Culture and Expecta5ons

At the end of the 2009-‐10 school year, TNTP interviewed teachers in schools that would become Apollo schools for 2010-‐11. Teachers who returned as Apollo teachers scored higher on every type of ques=on, including their commitment to the “No Excuses” philosophy and beliefs that all students are capable of high achievement.

0

0.5

1

1.5

2

2.5

3

3.5

No Excuses Alignment with Mission

Student Achievement

Commitment to Students

Student Mo=va=on

Interview Response Scores

Lea

Stayed

CONFIDENTIAL

Given that we don’t have a randomized experiment in year one, we use several empirical methods to assess the impact of Apollo 20.

Research Design Empirical Strategies

•  Ordinary Least Squares Regression

•  Advantages: Simple and easy to interpret •  Disadvantages: Assumes linear func=onal form to allow comparisons between students

of different sta=s=cal profiles, cannot control for unobservable differences between treatment and control students

•  Nearest Neighbor Matching

•  Using the distance func=on defined above, we locate each students’ four closest sta=s=cal neighbors and compare them to the group average.

•  Advantages: No func=onal form assump=on, focuses aoen=on on students from similar backgrounds

•  Disadvantages: Cannot control for unobservable differences between treatment and control students.

igisgsi XApolloscore εγβββ ++⋅+⋅+= 210,,

)()(),( jiji XXVXXjid −ʹ′−=

CONFIDENTIAL


•  Difference in Differences

•  Advantages: By focusing on year-‐over-‐year changes in scores, we account for baseline varia=on that we do not otherwise observe.

•  While these methods have their advantages, all three will be biased if students selec=vely opt in or out of Apollo schools.

sigisgsi XApolloscore ,210,, εγβββ ++⋅+⋅+=Δ

CONFIDENTIAL

Research Design Selec5on Issues

Incoming Sixth Graders

Incoming Ninth Graders

CONFIDENTIAL

•  To account for the decline in baseline ability in Apollo schools, we need a way to iden=fy which students would have enrolled in the absence of such significant changes. This leads us to our final empirical strategy, which uses a student’s home enrollment zone to predict which students are likely to aoend Apollo schools.

•  Two-‐Stage-‐Least-‐Squares Difference in Differences

•  Advantages: Solves selec=on issue. •  Disadvantages: Possibly vulnerable to cri=ques based on the exclusion

restric=on.

igiigsi XzonedApollo εγβββ ++⋅+⋅+= 210,,


CONFIDENTIAL

Results Main Results

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

6 7 & 8 9 10 & 11

TAKS Math Results

OLS

NNM

DID

2SLS

In Math, we see posi=ve and sta=s=cally significant results in both middle and high school. The gains in grades that received high-‐dosage tutoring were drama=c.

CONFIDENTIAL

In Reading/ELA, the results are mixed. While high schools performed extremely well, there is liole evidence of success in middle school – indeed some es=mates are nega=ve.

-‐0.1

-‐0.05

0

0.05

0.1

0.15

0.2

0.25

6 7 & 8 9 10 & 11

TAKS Reading

OLS

NNM

DID

2SLS

Results Main Results

CONFIDENTIAL

Verifica5on Alternate Test

The results presented so far are extremely promising. However, some have argued that the gains on state exams can be driven by test-‐specific preparatory ac=vi=es at the expense of general student learning (Jacob 2005, Heilig and Darling-‐Hammond 2008). To provide some evidence on this maoer, we also looked at Stanford 10 scores. This test is na=onally normed and not =ed to any accountability measure, so the incen=ves to teach to the test or otherwise manipulate results are minimal

CONFIDENTIAL

Stanford 10 tests show a similar paoern to TAKS, sugges=ng that our results are not driven by teaching to the test.

-‐0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

6 7 & 8 9 10 & 11

Stanford Math

OLS

NNM

DID

2SLS

-‐0.2

-‐0.15

-‐0.1

-‐0.05

0

0.05

0.1

0.15

0.2

6 7 & 8 9 10 & 11

Stanford Reading

OLS

NNM

DID

2SLS

Verifica5on Alternate Test

CONFIDENTIAL

A second poten=al threat to our results is aori=on from our sample. Our main results use the sample of students who enroll in an Apollo school or a comparison school at the beginning of the 2010-‐11 school year and for whom we have test score data in the spring of 2011. If Apollo students are more likely to miss the Spring 2011 TAKS examina=on, this could bias our es=mates. Given the major and sudden changes that occurred in Apollo schools, it is plausible that parents might choose to move their children to a private school or a charter school like KIPP or YES. In these instances, we would not observe their test scores. To examine this issue, we es=mate the impact of Apollo on the probability of entering our analysis sample.

Verifica5on AXri5on

CONFIDENTIAL

Verifica5on AXri5on

We es=mate that Apollo students are 0.6% more likely to miss an exam than comparable students in comparison schools, though this es=mate is not sta=s=cally significant. Dropping the top 0.6% of Apollo students from the analysis does not change our findings. For the difference-‐in-‐difference es=mates, we must also be able to match students to their 2009-‐10 score to calculate a trend. We find that Apollo students are roughly 4% more likely to be missing baseline scores. Since we are able to replicate our DID results with other models that do not require baseline scores, this is not problema=c.

Outcome Marginal Effect Outcome Marginal Effect

Missing 2011 Math .006 Missing 2010 Math .040***

(.005) (.017)

Missing 2011 Rdg. .006 Missing 2010 Rdg. .038

(.005) (.017)

Notes: Standard Errors reported in parentheses.

CONFIDENTIAL

Apollo schools were also subjected to unprecedented levels of scru=ny and accountability – the nine principals, in par=cular, earned bonuses for mee=ng performance goals and could be terminated if they fell short. One might worry that such a high-‐stakes environment might induce school officials to cheat when repor=ng scores. Accordingly, we examined student responses on the 2010-‐11 TAKS and ranked every grade in every school according to four measures designed to detect suspicious paoerns in student answers:

1.  Unusual blocks of consecu=ve iden=cal answers given by mul=ple test-‐takers 2.  Unlikely correla=on in answer responses within grades 3.  Unusually high variance in these correla=ons 4.  Unlikely combina=ons of correct answers – for instance, gehng easy

ques=ons wrong and hard ques=ons correct

Verifica5on Chea5ng Analysis

CONFIDENTIAL

Verifica5on Chea5ng Analysis

Aaer adding the ranks of all schools and grades on each of these four measures, we find that the average Apollo school is in the 43.9 percen=le in math and 43.1 percen=le in reading.

CONFIDENTIAL

Results Subcomponents

0

0.1

0.2

0.3

0.4

0.5

0.6

6 9

Tutoring Effec5veness

Above-‐Grade Comparison

Reading Comparison

-‐0.1

-‐0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

7 8 10 11

Double-‐Dosing Effec5veness

Math

Reading

Unsurprisingly, our triple-‐difference analysis shows that tutoring was highly effec=ve in increasing achievement. The double dosing results, however, are all sta=s=cally zero except for a large posi=ve effect in eighth grade math.

CONFIDENTIAL

Results Context

The Apollo results are strikingly similar to the effects of aoending the Harlem Children’s Zone Promise Academy Middle School, KIPP Lynn Middle School, and a group of “No Excuses” Boston-‐area charters.

Apollo 20 Harlem

Children’s Zone

KIPP Lynn Boston Charter Schools

Oversubscribed All Urban

6th Grd. Math .484 .249

MS Math .234 .229 .346 .359 .148

MS Reading -‐.014 .047 .120 .198 .098

9th Grd. Math .739

HS Math .368 .265 .122

HS Reading .189 .364 .169

Sources: Dobbie and Fryer (2010), Dobbie and Fryer (2011b), Abdulkadiroglu et al (2011), Angrist et al. (2010)

CONFIDENTIAL

Results Return on Investment

Implemen=ng Apollo cost roughly $2,042 per student. A back-‐of-‐the-‐envelope calcula=on shows that that internal rate of return from this investment is 20.16%. Compared to other educa=onal interven=ons, this return is large.

Ini5a5ve IRR

Apollo 20 20.16 %

“No Excuses” Charter School 18.50 %

Early Childhood Educa=on 7.99 %

Reduced Class Size 6.20 %

Sources: Curto and Fryer (2011), Heckman (2010), Krueger (2003)

CONFIDENTIAL

Lessons Learned

The main takeaway from Apollo Program in 2010-‐11 is that a fully loaded Apollo program in which all five tenets are implemented with fidelity produces large, significant gains in student performance. We must think hard about how to scale-‐up tutoring in a manner that maximized student performance while controlling the costs of the program.

CONFIDENTIAL

References

Abdulkadiroglu, A=la, Joshua Angrist, Susan Dynarski, Thomas J. Kane, and Parag Pathak (2011), “Accountability in Public Schools: Evidence from Boston’s Charters and Pilots”, forthcoming in Quarterly Journal of Economics. Angrist, Joshua D., Susan M. Dynarski, Thomas J. Kane, Parag A. Pathak, and Christopher R. Walters. 2010. "Inputs and Impacts in Charter Schools: KIPP Lynn." American Economic Review Papers and Proceedings, 100(2): 239–43. Dobbie, Will and Fryer, Roland G. (2011a), “Are High Quality Schools Enough to Increase Achievement Among the Poor? Evidence From the Harlem Children’s Zone”, Forthcoming in American Economic Journal: Applied Economics. Dobbie, Will and Fryer, Roland G. (2011a), “Are High Quality Schools Enough to Increase Achievement Among the Poor? Evidence From the Harlem Children’s Zone”, Working Paper No. 15473 (NBER, Cambridge, MA) Dobbie, Will and Fryer, Roland G. (2011b), “School Characteris=cs and Student Achievement: Evidence from NYC Charter Schools”, Unpublished Manuscript (Harvard University). Heckman, James J., Seong Hyeok Moon, Rodrigo Pinto, Peter A. Savelyev, and Adam Yavitz, “The Rate of the Return to the HighScope Perry Preschool Program”, Journal of Public Economics, 94: 114–128, (2010). Heilig, Julian V. and Linda Darling-‐Hammand (2008), “Accountability Texas Style: The Progress and Learning of Urban Minority Students in a High-‐Stakes Tes=ng Context”, Educa&onal Evalua&on and Policy Analysis, 30(2): 75-‐110. Jacob, Brian A. (2005), “Accountability, incen=ves and behavior: the impact of high-‐stakes tes=ng in the Chicago Public Schools”, Journal of Public Economics 89: 761-‐796. Krueger, Alan B. (2003) , “Economic Considera=ons and Class Size,” The Economic Journal, 113, F34—F63.

fryer apollo presentation

Documents