technical report the effects of the new orleans post … › preferen › you must read this ›...
TRANSCRIPT
Douglas N. Harris, Tulane UniversityMatthew F. Larsen, Lafayette College
July 15, 2018
Education Research Alliance NOLA.org
Technical Report
THE EFFECTS OF THE NEW ORLEANS POST-KATRINA MARKET-BASED SCHOOL
REFORMS ON STUDENT ACHIEVEMENT, HIGH SCHOOL GRADUATION, AND COLLEGE
OUTCOMES
1
The Effects of the New Orleans Post-Katrina Market-Based School Reforms on Student Achievement, High School Graduation, and
College Outcomes
Douglas N. Harris Matthew F. Larsen
July 15, 2018
Abstract: The school reforms put in place in New Orleans after Hurricane Katrina represent the most intensive market-based school accountability system ever created in the United States. Almost all public schools were taken over by the state, which turned over management to autonomous non-profit charter management organizations working under performance contracts; collective bargaining and tenure were ended, yielding flexible human capital management; and traditional school attendance zones were eliminated, expanding choice for families. More than a decade later, this study provides the first examination of the effects of this package of reforms on students’ short-term and long-term outcomes, using matched difference-in-differences. We find that the package of reforms improved the quantity, quality, and equity of schooling in the city on almost every available measure, increasing average test scores by 0.28-0.40 standard deviations, high school graduation by 3-9 percentage points, college attendance by 8-15 percentage points, college persistence by 4-7 percentage points, and college graduation by 3-5 percentage points. These effects translate to 10-67 percent increases over baseline levels. The reforms also apparently reduced educational inequality by race and income on most measures. Our estimation procedures address potential threats to identification, including, for example, changes in the population. The reforms highlight the potential of market-based school reforms, though we also identify reasons why effects of this large size and range may not be expected in other locations and circumstances. Acknowledgements: This study was conducted at the Education Research Alliance for New Orleans at Tulane University. The authors wish to thank the organization’s funders: the John and Laura Arnold Foundation, William T. Grant Foundation, the Spencer Foundation and, at Tulane, the Department of Economics, Murphy Institute and School of Liberal Arts. We also thank Joshua Angrist, Robert Bifulco, Deven Carlson, Joshua Cowen, John Easton, David Figlio, Adam Gamoran, Dan Goldhaber, Helen Ladd, Jane Arnold Lincove, Susanna Loeb, Andrew McEachin, Parag Pathak, Ross Rubenstein, Robert Santillano, Amy Schwartz, John Yinger, Lindsay Bell Weixler, and seminar participants at the National Bureau of Economic Research, Syracuse University, Texas A&M University, Tulane University, the University of Arkansas, and the University of Virginia. For other important contributions, including data collection and analysis, we thank Alica Gerry and Nathan Barrett. Author Information: Douglas N. Harris (corresponding author) is Professor of Economics, the Schleider Foundation Chair in Public Education, Director of the Education Research Alliance for New Orleans, and Director of the National Center for Research on Education Access and Choice (REACH) at Tulane University ([email protected]). Matthew F. Larsen is an Assistant Professor of Economics at Lafayette College and a Non-Resident Research Fellow at ERA-New Orleans ([email protected]).
2
Introduction For the past century, America’s publicly funded schools have been almost
universally operated by local government agencies that assign students to schools based
on their neighborhoods. This could be efficient with school districts competing against
one another (Tiebout, 1956; Hoxby, 2000), though political forces (Kollman, Miller, &
Page, 1997), labor unions (Hoxby, 1996; Strunk & Grissom, 2010; Lovenheim & Willen,
2016), and other factors may make public sector education production inefficient (Chubb
& Moe, 1990; Hill, Pierce, & Guthrie, 1997; Hanushek, 2002). To address these
problems, Friedman (1962) argued for reform of the American schooling system so that
families are “free to choose” where their children attend school, government subsidies
follow the student to induce more direct competition among schools, and non-
governmental organizations supply schooling subject to minimal regulation.
The school reforms put in place in New Orleans after the tragedy of Hurricane
Katrina offer arguably the first direct test of the effects of these two alternative models.
Prior to Katrina, the New Orleans school system was well aligned with almost every
other city in the United States. In addition to neighborhood-based assignment of students
to schools, the vast majority of schools were governed and operated by the Orleans Parish
School Board (OPSB). The city’s teachers worked under a union contract that established
a single salary schedule and extensive work rules. The union contract, along with state
tenure protections, also provided teachers with considerable job security.
3
After Hurricane Katrina struck New Orleans on August 29, 2005, all the
hallmarks of the traditional school district were eliminated (Harris, 2015).1 The state
government took over the school system, moving oversight of almost all the city’s public
schools from the local OPSB to the statewide Louisiana Recovery School District (RSD).
Many OPSB schools were quickly turned into charter schools and, over time, so too were
all RSD schools. Attendance zones were essentially eliminated, creating open school
choice for families. All educators were fired. The OPSB allowed its teacher union
contract to expire and never replaced it. By virtue of becoming charter schools, tenure
protections were also eliminated. Local and state agencies still had a role, especially in
funding schools, but control was mostly limited to passing funds on to schools (on a per-
pupil basis) and negotiating and enforcing contracts with their charter schools. They no
longer made decisions about school personnel, compensation, curriculum, instruction, or
other most other aspects of school management. Over just a few years, the government
role was dramatically altered, from operating schools to overseeing them. Consistent with
Friedman’s (1962) call, the district-based “one best system” of U.S. public education
(Tyack, 1974) was eliminated for the first time since it took shape a century earlier.
As sudden as these changes were in New Orleans, the new policies themselves
reflected a two-decade shift toward test-based and market-based accountability
throughout the United States. Induced by evidence of possibly inefficient resource use
(Hanushek, 1996), poor showings on international assessments (National Commission on
Educational Excellence, 1983; Goldin & Katz, 2008), and flat test score trends
(Hanushek & Woessman, 2010), the 1994 re-authorization of the federal Elementary and
1 Hurricane Rita struck just one month later on September 24, 2005. For simplicity, however, we simply refer to “the hurricane” going forward.
4
Secondary Education Act (ESEA) began requiring standardized testing and school report
cards. The frequency of testing and stakes attached to them increased under No Child Left
Behind (NCLB) in 2001 (Dee & Jacob, 2011). These new incentives shifted the
traditional public school system in the U.S., albeit only slightly, toward performance-
based contracting that is common in other parts of the public sector.
The New Orleans reforms also followed the longer national trend toward market
accountability through parental school choice and opening up the supply side through
non-governmental organizations, such as charter schools (e.g., Angrist et al., 2010;
Abdulkadiroğlu et al., 2011; CREDO, 2013a), private school vouchers (Rouse, 1998;
Krueger & Zhu, 2004; Abdulkadiroğlu et al., 2015; Mills & Wolf, 2017), and intra- and
inter-district choice among traditional public schools (Berry, Jacob, & Levitt, 2005;
Harris, Witte, & Valant, 2017). With both performance-based contracts and market
pressure from families, as well as autonomy from district and union rules, to meet these
demands, the theory is that leaders would have stronger incentives for efficient operation
and the autonomy to respond to these incentives by creating better schools.
Though the word accountability has been commonly used in the national debate,
the actual incentives facing schools, and the degree of autonomy they have to respond to
them, have changed relatively little. Before Hurricane Katrina, some districts around the
country had experimented with school-level autonomy (Ravitch, 2000) and mayoral and
state takeovers of schools and districts (Wong & Shen, 2006; Gill et al., 2006); but most
of these efforts were short-lived, and local school board politics, unions, and school
attendance zones remained as core components of school operations (Ravitch, 2000).
NCLB increased the volume of testing, but only a small fraction of the schools slated for
5
corrective action under NCLB experienced significant intervention, such as closure or
takeover (GAO, 2007).2 This may be why researchers have found the NCLB effects to be
small at best (Dee & Jacob, 2011).
Market accountability, too, seems to have had a limited influence. At about the
time Katrina made landfall, only two percent of U.S. students attended charter schools,
and 13 percent of U.S. students attended a non-assigned publicly funded school (Harris,
Witte, & Valant, 2017).3 Only seven districts had more than 20 percent of their students
in charter schools, and none had more than 50 percent (National Alliance for Public
Charter Schools, 2013). New Orleans was typical in having only four charter schools (out
of 127 schools in total). While research increasingly shows positive effects of charter
schools (e.g., Angrist et al., 2010, 2016; Abdulkadiroğlu et al., 2011, Booker et al., 2011;
CREDO, 2013a) and they seem to create competition that improves outcomes in nearby
traditional public schools (Gill & Booker, 2008; Epple, Romano & Zimmer, 2015), their
market share has been too small to have much effect on outcomes across entire cities or
regions.4 For these reasons, advocates for accountability and school autonomy have
argued that policymakers have not gone far enough with these reforms (Hill & Lake,
2004, Evers, 2014; Peterson, 2014; Walberg, 2014). President Trump and his Secretary of
Education, Betsy DeVos, seem to agree as they have advocated for large state and
national school choice programs.
2 A synthesis of evidence from studies of pre-NCLB accountability found more positive cumulative effects on test scores averaging 0.08 standard deviations (Lee, 2008). Also, see Carnoy & Loeb (2003). 3 We include in “non-assigned publicly funded schools” students who attend schools labeled charter, magnet, and intra-and inter-district schools of choice (Harris, Witte, & Valant, 2017). The percentage of students in charter schools has since increased to almost five percent (NCES, 2015). 4 The majority of studies on competitive effects from choice suggest positive effects, and almost none show negative effects (many are null) (Gill & Booker, 2008). Other studies have examined the effects of competition within the traditional public schooling market (e.g., Hoxby, 2000; Belfield & Levin, 2003; Rothstein, 2007).
6
In New Orleans, policymakers went much further with school accountability and
autonomy than perhaps any district or state ever had. There have also been some signs of
success. New Orleans statewide’ ranking on the percentage of students who are proficient
has moved from the 67th ranked district to the 39th (of 68) ranked districts since the
reforms (Louisiana Department of Education, 2015).5 Averaging across all subjects and
grades, the test score gap between New Orleans and the rest of the state decreased by
0.37-0.44 student-level standard deviations (s.d.) from 2005 to 2014 (see Table 1). New
Orleans’ high school graduation rates, any college attendance, and college graduation
increased by 17-20 percentage points, 10 percentage points, and 2.1 percentage points,
respectively. These positive trends, combined with other evidence (CREDO, 2013b;
Abdulkadiroğlu et al., 2016), suggest the New Orleans reform effects may have been
positive. As a result, the New Orleans system has been widely hailed among reform
advocates and national political leaders with otherwise divergent views, ranging from
Democratic President Obama (2010) and his Secretary of Education Arne Duncan
(Dreilinger, 2014) to President George W. Bush’s Director of the Institute for Education
Sciences (Whitehurst, 2012) and Republican Louisiana Governor Bobby Jindal (America
Next, 2015). Reflecting this broad political support, at least 27 districts nationally are
following New Orleans’ lead with similar, albeit somewhat less aggressive, reforms (Hill
& Campbell, 2011).
There have been reasons to be skeptical, however. Prior studies have focused
narrowly on high-stakes test scores, which are prone to cheating and strategic behavior
(Jacob, 2005). More generally, prior studies have not been designed to estimate causal
5 For comparability, the post-Katrina New Orleans “district” ranking is based on a weighted average of the New Orleans RSD and OPSB schools.
7
effects of the city’s reforms.6 Here, we provide some of the first evidence about the
effects of the New Orleans reform package on students’ long-term academic outcomes
using several difference-in-differences strategies that compare the pre- and post-reform
periods in New Orleans to matched comparison groups. We find that the effects on
achievement and college graduation closely mirror the strong positive direction of the
descriptive trends. The reforms increased student achievement by 0.28-0.40 s.d., high
school graduation by 3-9 percentage points (10-14 percent from baseline levels), college
attendance by 8-15 percentage points (15-67 percent), college persistence by 4-7
percentage points (25-28 percent), and college graduation by 3-5 percentage points (30-
45 percent).
With additional analysis, we are able to largely rule out several threats to
identification, including population change, strategic behavior from performance-based
accountability, trauma and disruption from the hurricane, and the effectiveness of the
interim schools that evacuated students temporarily attended. The treatment effects
appear to be an order of magnitude larger than the potential biases, and some biases
partially cancel out. The main potential alternative explanation for the effects, beyond the
reforms themselves, is that the charter-based system gained a considerable financial
advantage over the comparison groups (Buerger & Harris, 2016), though these alone
cannot explain the overall effects.
6 Abdulkadiroğlu et al. (2016) studied the effectiveness of charter schools as compared with the schools run directly by the OPSB after the reforms started. The Center for Research on Education Outcomes (CREDO, 2015) compared annual student growth in post-Katrina New Orleans’ charter schools with growth of similar students (“virtual twins”) in traditional public schools mostly in other districts. These first two important studies focused only on the post-Katrina period and did not consider school quality prior to the reforms, and therefore were not intended or designed to study the effects of the reforms. Sacerdote (2012) found that New Orleans evacuees experienced larger increases in school quality than evacuees from other Louisiana parish/districts, which confirms the low performance of pre-treatment New Orleans schools but, again, does not address their post-Katrina improvement within New Orleans.
8
There are many ways to reform schools and a variety of roles for markets and
governments. New Orleans is an important case because it shares Friedman and other
market-focused school reformers’ focus on school choice for students and autonomy for
educators, but the city diverges by maintaining an active role for government in
regulation and performance-based contracting. The next section describes our methods
and how they address threats to identification. This is followed by discussion of data,
results, and conclusions.
Model and Identification
Estimation Strategy
We attempt to identify causal effects using a combination of matching and
difference-in-differences (DD) analysis. Specifically, we estimate the effects of the New
Orleans school reform package starting with standard two-period DD estimation (Angrist
& Pischke, 2009):
𝐴!"# = 𝛾! + 𝑋!"#𝛽 + 𝜆𝑑! + 𝛿 𝑁𝑂𝐿𝐴! ∙ 𝑑! + 𝜀!"# (1) where 𝐴!"# is the achievement of student i in school district j at time t, 𝛾! is a vector of
school district fixed effects, 𝑋!"# is a vector of student covariates7, 𝑑! indicates whether
the outcomes pertain to a single pre-treatment period or a single post-treatment period,
and 𝑁𝑂𝐿𝐴! is an indicator set to unity for New Orleans and zero for students in the
matched comparison group districts. No other district in Louisiana experienced the
reforms, and these therefore serve as a useful comparison group. Under certain
assumptions discussed below, especially that student outcomes would have moved in
7 These include race, free/reduced price lunch status, special education status, limited English proficiency, and grade repetition. In addition, we include bin indicators for each stratum in the matching process.
9
parallel absent the treatment, ordinary least squares estimation of 𝛿 provides an unbiased
estimate of the average treatment effect (ATE).8
Our first estimates are based on equation (1) using only the most recent pre-
reform year possible, and the most recent post-reform period available in the data, which
varies by method (see below).9 However, the effects of the reforms may have emerged
gradually over time. In creating an entirely new system of schooling, New Orleans
leaders not only had to create new schools, but an entirely new governance structure and
new institutions to develop charter school operators (e.g., New Schools for New Orleans),
new avenues for recruiting teachers to the city (e.g., Teach for America, which had some
presence in the city before the reforms), and new ways to provide information to parents
to help them choose schools (e.g., New Orleans Parents Guide). The state RSD existed
prior to Katrina but had just a handful of staff and had not been designed to carry out its
new responsibilities.
To estimate these dynamic effects and avoid imposing restrictive assumptions of
two-period DD and related types of models,10 we instead rely mostly on Granger/event
study estimates (Granger, 1969; Autor, 2003; Angrist & Pischke, 2009) as follows:
𝐴!"# = 𝛾! + 𝜆! + 𝑋!"#𝛽 + 𝛿!! 𝑁𝑂𝐿𝐴! ∙ 𝑑!,!!!!!! + 𝛿!! 𝑁𝑂𝐿𝐴! ∙ 𝑑!,!!
!!!! + 𝜀!"#(2)
where 𝜆! is a vector of year indicators, m is the number of years in the data prior to
treatment, and q is the number of years after treatment. This implies that 𝛿!! is the
(adjusted) difference in outcomes of the control and treatment groups 𝜏 periods before
8 Athey and Imbens (2002) discuss additional linearity assumptions used in DD estimation. 9 We refer to the spring of the school year throughout the remainder of the study, since this is when students take tests. So, 2005 means the 2004-05 school year and so on. 10 When there are more than two periods of data, it is sometimes recommended to add group-specific time trends as follows: 𝐴!"# = 𝛾!! + 𝛾!!𝑡 + 𝜆! + 𝑋!"#𝛽 + 𝛿 𝑁𝑂𝐿𝐴 ∙ 𝑑! + 𝜀!"#where t is a continuous time period variable and 𝛾!! is the slope (Angrist & Pischke, 2009). This specification yields biased estimates, however, when there are dynamic effects (Pischke, 2005). Equation (2) avoids this problem.
10
treatment. This provides a test of parallel trends, and if the assumption holds, then it is
reasonable to interpret 𝛿!! as causal effects of the reforms.11 The estimation of (2) also
shows how the effects increase (or decrease) toward the longer-term effects from the
estimation of (1).
We use two general strategies to estimate equations (1) and (2): (a) panel analysis
using only that portion of the pre-hurricane student population that returned to their pre-
hurricane district for at least one post-treatment year; and (b) pooled cross-sections of
student cohorts who were in the same grades pre- and post-treatment (e.g., comparing
achievement for the 2005 cohort of 4th graders with the 2014 cohort of 4th graders). With
the panel approach, we are able to study a fixed group of individuals and thereby account
for unobserved differences directly; however, the returning group is a small, non-random
subsample of the original population, which limits statistical power and generalizability.
Also, eventually, the outcomes of the pre-treatment students cannot be measured because
they move on to non-tested grades or graduate, making it impossible to study the longer-
term reform effects that are of primary interest. With pooled cross sections, the sample is
much larger as almost all students who were in New Orleans schools pre- or post-Katrina
contribute to the estimation, but we have to rely on observable demographic information
to account for population change.
Our methods vary somewhat depending on the dependent variable. With test
scores, we can use both panel and pooled analyses, at least for certain years. However,
with one-time events like high school graduation and college attendance, we have to rely
mainly on the pooled approach because, by their nature, these long-term outcomes cannot 11 While these pre-treatment 𝛿!! terms can be interpreted as parallel trends tests in the event studies, we report a separate parallel trends test that assumes simple linear pre- and post-trends for control and treatment.
11
be measured for individuals annually across time and therefore cannot be used to track
individuals pre- and post-treatment to account for unobservable differences. This limits
the usefulness of the panel approach with one-time events.
Where possible, we designed the analyses to have enough pre-treatment data to
allow parallel trends tests. We also account for potential endogeneity using a variety of
the methods discussed by Bertrand, Duflo, and Mullainathan (2004): graphing the
dynamics of the effects (see model (2)), adding treatment-specific time trends that vary
pre- and post-reform, and looking for an effect prior to intervention (placebo tests). The
use of both panel and pooled analysis, one additional identification strategy, and various
direct tests for specific forms of bias all reinforce our results.
We report results estimated at the student-level with Generalized Estimating
Equation (GEE) clustering at the district level (Liang and Zeger, 1986). However, the
GEE approach rests on asymptotic assumptions about the number of clusters. Inference is
generally only valid with at least 30-50 clusters (Kezdi, 2004; Cameron, Gelbach, and
Miller, 2008; Angrist & Pishke, 2009), and our preferred estimates include only 6-8
districts. To address this, we also report estimates using almost all of the more than 60
districts in the state. The standard errors are smaller in that case, but not enough to
change the conclusions. We also carried out some estimation with data aggregated to the
district-by-year level, which generally yields conservative standard errors (Bertrand et al.,
2004; Angrist & Pischke, 2009).12 The results are generally robust to these and many
other variations in methods.
12 A third common alternative, the wild bootstrap (Cameron, Gelbach, & Miller, 2008) is infeasible in this case because there is only one treatment cluster.
12
Threats to Identification
There are many general threats to identification with natural experiments,
especially endogenous policy adoption and the simultaneous (unobserved) policy changes
that also affect the outcome of interest. In the case of the New Orleans school reforms,
there are five specific threats that serve as alternative potential causes of the city’s
positive test score trends.
First, the population of the city changed (The Data Center, 2014; Vigdor, 2008).
In the process of rebuilding, city leaders decided to shut down and eventually replace
most of the major public housing projects. For this and other reasons, low-income
residents may not have returned, and more socio-economically advantaged families may
have replaced them, increasing outcomes for reasons unrelated to the reforms.
Second, when Louisiana families evacuated, they generally placed their children
in schools near their new and temporary residences. New Orleans evacuees experienced
larger gains in school quality in these “interim schools” relative to non-New Orleans
evacuees (Sacerdote, 2012), and if these gains did not fade out, then some of the later
increases in achievement might reflect the performance of these interim schools rather
than the New Orleans reforms.13
13 A related possibility is that the students who evacuated benefited from being in better housing and/or neighborhoods. It is possible that, as in some past disasters, New Orleans’ housing quickly surpassed the quality that had preceded it and that this drove positive educational effects for those who did return (Hornbeck & Keniston, 2017). This seems unlikely, however, as the quality of the New Orleans’ housing stock was unchanged for non-flooded homes for sale and remained considerably worse for flooded homes one full year after Katrina (McKenzie & Levendis, 2010). Most homeowners lacked flood insurance, and state and federal programs averaged about $40,000 per home, or about 17 percent of their prior values. This likely understates the percentage of the housing value covered for low-income homeowners, but not enough to generate an overall improvement in the housing stock for low-income families. It is also worth considering whether there was a general improvement in neighborhood quality after the storms, in which case the effects might be interpreted more in line with the well known Moving to Opportunity (MTO) program (Ludwig et al., 2013). However, this, and other evidence presented later about poverty, suggests
13
Third, prior research has shown that accountability induces some schools to
manipulate high-stakes measures and/or reallocate resources in ways that reduce
unobserved outcomes that are lower-stakes (Jacob, 2005; Figlio, 2006; Koretz, 2009).
Such strategic behavior may be especially important in New Orleans where schools are
closed or taken over based substantially on the test scores and graduation rates that
represent some of our main dependent variables (Bross, Harris, & Liu, 2016) and
therefore where the stakes tied to test scores and graduation rates are very high.
Fourth, NCLB had been adopted a few years prior to Katrina, and the law’s key
provisions were just being implemented when the storm and reforms hit. Since low-
performing schools are the focus of NCLB sanctions, and New Orleans had a
disproportionate share of such schools, the post-Katrina improvements in New Orleans’
outcomes might have occurred anyway. While the effects of NCLB have apparently been
modest on a national level (Dee & Jacob, 2011), the effects were larger in some places
than others, and larger effects are especially likely in cities like New Orleans where a
large share of schools would have been affected.
Fifth, our difference-in-differences analysis suggests that average spending per
pupil increased by roughly $1,358 per pupil (13 percent). A growing body of research
suggests that school spending increases student outcomes (e.g., Jackson, Johnson, &
Persico, 2016; Lafortune, Rothstein, & Schanzenbach, 2016). These additional resources
may have increased outcomes even if the market-based reforms had not been adopted.
While these first five threats to identification suggest the trends would tend to bias
estimated effects upwards, the direction of a sixth threat could have the opposite
that the improvements in student outcomes we observe cannot plausibly be explained by neighborhood effects either. We thank David Figlio for mentioning this potential role for housing.
14
influence: Hurricane Katrina was one of the worst disasters in American history14 and
created persistent trauma and anxiety for residents (DeSalvo et al., 2007; Elliott & Pais,
2006; Weems et al., 2010). Some of these psychological effects were driven by poor
post-storm labor market outcomes among those who had lived in the most heavily
flooded areas (Groen & Polivka, 2008). Those with worse post-hurricane housing and
labor market outcomes also experienced worse Post-Traumatic Stress Disorder (PTSD)
(Elliott & Pais, 2006). While most of the psychological evidence pertains to adults, there
is also evidence of trauma and disruption among children more than two years after the
hurricane (Brown et al., 2011)15, and this apparently reduced academic learning at least in
the short term (Pane et al., 2006, 2008; Sacerdote, 2012).
Data and Matching
Dependent Variables
The Louisiana Department of Education (LDOE) provided student-level
longitudinally linked data for essentially all public school students in the state for each
year 2001-2014. Key variables include student test scores, exit codes (for high school
graduation), college outcomes, student demographics, grade levels, and schools attended.
Pre- and post-Katrina, students took state standardized tests in grades 3-8.16
14 As many as 1,900 people died as a result of the storm, and the city experienced at least $80 billion dollars in damage to physical infrastructure (Pane et al., 2008). 15 One sample of students reported thoughts of the following common disaster-related events 30 months after the hurricanes: “having thoughts someone might die (79%), having clothes or toys ruined (78%), having their home badly damaged or destroyed (65%), witnessing others hurt during the storm (45%), having a pet hurt or die (41%), thinking they might die during the storm (38%), having trouble getting food and water (20%)” (Brown et al., 2011, p.576). 16 The high school testing data is omitted because, like many states, Louisiana switched to End-of-Course (EOC) exams in high school after Katrina, which created issues of comparability. Also, students can take the high school tests in different grades, depending on when courses are available, creating an additional source of endogeneity. For these reasons, we focus on scores in grades 3-8, where the testing regime is more stable, along with high school graduation and college outcomes.
15
High school graduation is a stronger predictor of long-term life outcomes than
student test scores and serves essentially as a prerequisite to college (Levin et al., 2006;
Heckman, Lochner, & Todd, 2008). The design of state data systems, however,
complicates the measurement of high school graduation because exit codes are self-
reported by schools. This means, first, that some high school credentials will be higher
quality than others. Also, since Louisiana high schools are held accountable for
graduation, they may misreport the data in ways that inflate their graduation rates, similar
to cheating on high-stakes standardized tests (e.g., Jacob, 2005). Manual audits by the
state using samples of the hardest-to-verify codes (transfer to private schools, out-of-state
public schools, and home schools) have been unable to confirm the school data
(Louisiana Department of Education, n.d.).
To address these problems, we use three different graduation rates. Variable
Grad1 counts only regular diplomas as graduates and defines the denominator (zeros and
missing exits) in ways that approximate the state-defined graduation rate (hard-to-verify
exit codes are coded as missing); Grad2 is the same but counts hard-to-verify exit codes
as zeros (non-graduation); and Grad3 uses the Grad1 definition of the denominator but
broadens the definition of the numerator to include alternative completion such as
GEDs.17 Unless otherwise specified, we include both on-time and delayed high school
graduation together, though the restriction to on-time graduation has a minimal effect.18
17 Since the analysis is at the district level, all three high school graduation rate definitions ignore transfers between schools within districts; transfers between districts are counted as missing in all three definitions because they are neither positive nor negative outcomes, and depending on the grade, these transfers are removed from the cohort of the sending school for graduation purposes. 18 An additional potential problem with the high school data is that there was no formal validation process prior to 2007 (when graduation was added into school performance measures for accountability purposes). Therefore, we are also assuming that any trend in the measurement error due to this lack of validation is orthogonal to treatment.
16
An additional methodological challenge emerges because state and federal
definitions of graduation rates are based on the share of first-time 9th graders who
graduate on time with regular diplomas.19 This means we need five years of pre-treatment
data to calculate a single pre-treatment on-time graduation rate (one year for identifying
students who are first-time 9th graders plus four more years of high school). In order to
carry out the parallel trends test we also need two pre-treatment cohorts and therefore six
years of pre-treatment data. Given that we only have five years of pre-treatment data, we
report pooled results for first-time 9th graders without parallel trends test and for first-
time 10th graders with parallel trends tests.
We also study college attendance (on-time and any enrollment), persistence, and
graduation. For these outcomes, we restrict to first-time 12th graders, rather than high
school graduates) because of concerns that high schools might not let some students
graduate if they are performing poorly or not planning to attend college. We also restrict
persistence and graduation to that which occurs within 5 years of the end of students’ first
year in 12th grade, so that the time period is the same before and after the reforms.
The underlying sources of the college data provided by LDOE are the Louisiana
Board of Regents (BOR; 2001-2011) and the National Student Clearinghouse (NSC;
2005-2016), both of which cover two-year and four-year colleges. The BOR includes
only in-state public colleges and universities (two-year and four-year) and some private
colleges, but only includes information about on-time college enrollment (the year
immediately after high school graduation), omitting persistence and graduation. The NSC
19 In both cases, we use students who are in the given grade for the first time. It is common for students who are doing poorly academically to be held back and remain in the same grade for multiple years. Some of these first-time 9th and 10th graders may not be promoted to higher grades, and this shows up in our results through either dropout or delayed graduation.
17
data, in contrast, cover above 90 percent of all colleges, public and private across the
nation (Dynarski, Hemelt, & Hyman, 2013). In Louisiana, 93 percent of higher education
institutions are covered in the NSC; this under-states enrollment coverage as the excluded
colleges are very small in size. The NSC data also include on-time college enrollment but
go further to incorporate any enrollment, persistence, and graduation. This has
implications for the matching process with college outcomes, as discussed below.
An implicit assumption throughout the analysis is that measurement error is
orthogonal to treatment. This could be violated if there were either pre-to-post treatment
changes in college missing data rates or changes in the composition of colleges attended
by students from treatment and comparison districts. The former could occur, for
example, because of the shift from BOR to NSC, each of which could have different
types of measurement error; and the latter could arise if certain colleges were affected by
the hurricanes, or if there were treatment effects on the types of colleges students
attended that were correlated with the missing data rate.20 Our additional analysis largely
rules this out and seems consistent with orthogonal measurement error.21
20 We cannot distinguish missing observations from non-enrollment in our data in the case of high school graduation and college outcomes. As is standard in this form of analysis, when a student is “missing,” we assume the student is not enrolled. 21 To test whether measurement error is orthogonal, we assumed that the BOR data were complete for those institutions included in it, and then compared the BOR with the NSC to obtain the NSC missing data rate for every college for the years the two data sources overlap (2005-2011). (As noted above, only a handful of Louisiana colleges did not participate in the NSC, and these receive a missing data rate of 100 percent.) Next, we identified the percentage of students in each district who attended each college to calculate a college missing data rate by district. We found no effect on the missing data rate, suggesting that the data satisfy the above measurement error assumption. (Since we are assuming the BOR data are accurate, all districts had a 0 percent missing data rate in the pre-treatment period. For this reason, we also did not estimate the effect via DD analysis. Instead, we only calculated the first differences in the post-treatment period.) Since the NSC also covers out-of-state colleges, we also considered whether this might introduce bias; however, 95 percent of Louisiana college-goers attend in-state, so this has a minimal influence.
18
Matching
Having a within-state comparison group allows us to account for the differences
in the test scales and state data collection methods across grades and years, as well as
changes in state policy that are unrelated to the New Orleans’ school reforms. Our
preferred specifications also narrow the comparison group further to just hurricane-
affected districts. If the trauma/disruption and interim school effects were the same in
New Orleans and other hurricane-affected districts, then this sample restriction would
eliminate it as a source of bias.22 As we will show, the estimates are robust to statewide
versus hurricane-affected matched samples with test scores and college entry. With high
school graduation, the effects are also consistently positive and significant across
samples, but differ in magnitude.
Panel Matching. In the panel analysis with test scores, our first preferred
matching method involves the following steps: (a) restrict to hurricane-affected school
districts; (b) from those affected districts, drop students who never returned to their pre-
treatment district; and (c) among the returning students, use Mahalanobis matching to
identify comparison students with similar composite test score levels in both of the two
most recent pre-treatment years (2004 and 2005), stratifying by year of return. To
account for grade repetition, step (c) is further stratified so that students who ever-
repeated (never-repeated) a grade pre-treatment are exact-matched to other students in
22 The hurricanes apparently affected New Orleans more than all but perhaps two districts. According to Pane et al. (2006), 81 percent of the displaced students came from Orleans, Jefferson, and Calcasieu Parishes. Five additional parishes account for nearly all of the remaining displaced students: St. Tammany, St. Bernard, Plaquemines, Vermilion, and Cameron. Pane et al. (2006) define “displaced” as any student who exited the school system because of the hurricane, as determined by the state government and parishes. We consider all eight parishes to be hurricane-affected in what follows. Pane et al. (2008) show that New Orleans accounted for more than half the students in the entire state who left their home districts for a long enough period that they enrolled in another Louisiana district or left the state and did not return.
19
each respective category pre-treatment.23 Step (b) helps ensure that the comparison group
is similar to New Orleans schools24 in both the observed and unobserved factors
associated with return to the original district.25 In other estimates, we added
demographics to the list of matching variables, along with pre-treatment dependent
variables; but the results were robust, and we therefore report only the outcome-matched
results. Panel matching requires implausible assumptions in the case of high school
graduation and college outcomes because we cannot match, at the student level, on
dependent variables that are one-time events. Therefore, we employ the panel method
only with test scores.
Pooled Matching. Since it is not possible to match on pre-treatment dependent
variables for individual students in the pooled method, our preferred strategy is to match
whole schools using their pre-treatment school-level dependent variables and then
assume that changes in the unobserved factors affecting student outcomes are
(conditionally) independent of treatment. We show below that this assumption is
plausible, suggesting that the pooled estimates have limited bias.
In our preferred pooled analysis, we match the post-reform cohorts on pre-reform
measures as follows: (a) restrict to hurricane-affected districts; (b) identify potential 23 In Louisiana, students were retained in grades 4 and 8 if they do not reach the Basic level on one or more tests. The number of tests for which Basic is required has changed over time, as has enforcement of the policy, and it is common for schools to seek waivers to allow students to be promoted regardless of their scores. 24 When we say “New Orleans schools,” we mean all publicly funded and governed schools in the city, including those authorized by both the RSD and OPSB. 25 For example, parents who were unemployed prior to the hurricane might have evacuated with their children to other districts and found jobs there, reducing the probability of returning to the original district. Since we cannot observe unemployment, and we would expect unemployment to influence student learning, this family characteristic would introduce bias in the absence of matching. The matched comparison group allows us to account for it, to the degree that the factors determining return were the same across districts. There may also have been unobserved factors associated with the neighborhoods from which families moved. People tend to live near others with similar incomes; if families in some neighborhoods returned sooner than others, then this should mean that the ability to return depended on (unobserved) income, which would affect returnees and non-returnees in similar ways, ceteris paribus. Matching based on year of return helps account for these potentially important differences in sample characteristics.
20
match schools, i.e., those that exist in 2002-2005 and all available pre- and post-treatment
years and have at least 10 students in each tested subject and grade;26 (c) drop districts
that have fewer than 3-4 potential school matches (depending on the outcome measure);
and (d) among remaining schools, Mahalanobis match treatment schools to the
comparison group based on the specific pre-treatment outcome levels. The entire
matching process is carried out separately by district, so that each district is matched to
the distribution of New Orleans.27 Steps (a)-(d) are carried out separately for each student
outcome measure, so that the matched comparison schools are somewhat different across
outcomes.
With college entry, the matching process is essentially the same as high school
graduation, matching on the dependent variable. The situation is more complex, however,
with longer-term college outcomes. We only have the NSC data for a single pre-reform
year (2005), which precludes the usual parallel trends test when studying college
persistence and graduation. Instead, we take the school weights from the matching
process that uses the 2005 values for the lagged dependent variable and apply these
weights to a test that uses multiple years of college attendance, which we can measure in
multiple pre-treatment years. This process still allows us both to match on the specific
dependent variable of interest (in levels) and to test for parallel trends at the same time,
even if the data do not allow the usual test with parallel trends on the dependent variable
26 We matched on a different number of pre-reform years in the panel and pooled analyses because different methods yielded parallel trends (though the post-trends look very similar regardless of matching). 27 In the panel method, we match at the student level so that the weights create comparison districts of the same size as New Orleans. In the pooled method, the matching is at the school level which means that each comparison district has the same number of schools as New Orleans. This, combined with the fact that we are estimating at the student-level, means that the weighted number of students in the pooled sample is dependent on the size of the matched schools. Later, we refer to robustness checks where we aggregated the data up to the school and then district level so that each district has the same weight regardless of size or number of students.
21
itself (e.g., when college graduation is the dependent variable, the parallel trends test is
based on college attendance).
These pooled analyses are essentially the only feasible method with high school
graduation and college outcomes. They also have the advantage of allowing us to study
effects over a much longer period of time, in contrast to the panel method. A final
advantage of the pooled approach relates to the implementation of NCLB, which might
have increased scores in New Orleans even in the absence of the city’s larger reform
effort and probably done so more than in other districts because of the city’s
disproportionate share of low-performing schools. Since NCLB places pressure on whole
schools, matching at the school level, as we do in the pooled method, may better address
observed and unobserved differences in how NCLB affected student outcomes in the
comparison and treatment groups. In any event, we expect such bias to be relatively small
given prior research on NCLB (Dee & Jacob, 2011).
Descriptive Statistics
Table 1 Panel A provides descriptive statistics for New Orleans’ student
demographics and outcomes pre-treatment (2005). In addition to reinforcing the
improvement over time in New Orleans’ student outcomes, the data show that the New
Orleans public school student population was extremely socio-economically
disadvantaged in the pre-reform period with 83 percent eligible for free and reduced price
lunch (FRPL); almost all the students are racial/ethnic minorities, and 94 percent are
black (depending on the year). The differences between 2005 and 2014 also provide a
first indication that the demographics of the New Orleans public school population did
not change significantly or in a clear direction after the hurricane. The percentage of
22
students eligible for FRPL increased after the reforms from 83 to 88 percent, and the
percent of black students dropped from 94 to 88 percent. We provide additional analysis
of population change using Census data later.
Panel B of Table 1 compares the characteristics of New Orleans and the panel
matched comparison group in the pre-reform period only. The matching succeeded in
improving the baseline match between New Orleans and the comparison group for the
panel analysis. Without matching, as we have seen, New Orleans was 0.55 s.d. below the
rest of the state (averaging across subjects). With matching, this is reduced. For example,
the table shows that the panel comparison group is only 0.04-0.10 s.d. higher than New
Orleans in pre-reform test levels (averaging across grades). The fact that we can match
only at the school level in the pooled analysis clearly makes the match less successful,
however. The pooled matching method yields a difference between New Orleans and the
comparison group of 0.35 s.d. Table 1 also shows similar gaps in baseline outcome levels
for high school graduation and college outcomes. While the focus here is on outcome
levels, we are most concerned with the parallel trends tests shown later.
Recent research suggests that the DD method, especially with matching on lagged
dependent variables, tends to replicate the results from randomized control trials (St.
Clair, Hallberg, & Cook, 2016). Also, the specific methods proposed here at least partly
address all of the main threats to validity. The panel DD avoids the issue of population
change. The restriction to hurricane-affected districts addresses interim school effects and
trauma/disruption. Matching on test scores helps address the threat posed by NCLB.
Below, we discuss additional methods for addressing population change in the pooled
analysis as well as strategic behavior from test-based accountability.
23
Population Change
One of the main threats to identification in the pooled analysis is that the
unobserved characteristics of the population may have changed disproportionately in
New Orleans relative to the comparison group. As noted earlier, the New Orleans
population had even higher rates of FRPL eligibility after the reforms than before (Table
1). However, FRPL is problematic because it cannot capture the difference between
students just below the poverty line and those in extreme poverty, and because the FRPL
reporting rates depend on how schools administer the FRPL program.28 We therefore
provide additional evidence on this issue.
Table 2 Panel A provides data on pre-treatment 3rd graders, including all pre-
treatment students and only those who returned (returnees), for New Orleans and other
hurricane-affected districts. By 2010, New Orleans returnees had somewhat lower pre-
treatment scores than the overall pre-treatment New Orleans population, while, in the
other districts, the returnee scores were higher than the pre-treatment population. The DD
therefore favors the comparison districts by 0.043 s.d. In other words, the change in the
population actually reduced post-Katrina New Orleans scores by a small amount, leading
to a possible downward bias in our pooled effect estimates.
Since the analysis of the panel sample is somewhat limited (e.g., they only include
returnees, and the pooled analysis includes all post-Katrina students), we commissioned
the U.S. Census Bureau to provide detailed demographics for households with students in
28 More recently, the federal government has also instituted Community Eligibility, which reduces validity of school-level FRPL percentages as indicators of poverty. However, this change occurred after the years in the present analysis.
24
public schools for each district in the state.29 Table 2 Panel B provides difference-in-
differences analysis of these Census data, showing that some socio-economic changes
favor New Orleans and others favor the hurricane-affected districts. For example, median
household income dropped by $736 in New Orleans, but increased in the comparison
districts by $1,750, for a simple DD of -$2,486 (2012 dollars).30 The percentage of the
population with a BA or higher, however, increased by five percentage points in New
Orleans but by only three percentage points in the comparison group, for a DD of two
percentage points favoring New Orleans.
To identify the potential influence of these Census-based demographic shifts on
student learning, we used data from the USDOE’s Early Childhood Longitudinal Study
(ECLS) to estimate the partial correlation between achievement levels and each of the
demographic measures.31 With the resulting regression coefficients (shown in Table 2
Panel C), we then carried out an out-of-sample prediction of the achievement
levels/growth change expected from the changes in Census demographic measures.32 The
results are shown in Table 2 Panel D. The simulated cumulative effect across five years
in the reformed school system (our estimate of the “dosage”33), averaged across the
demographic measures, is 0.012 s.d. with a range of -0.012 (favoring the comparison
29 The Census could only provide these data for the three parishes/districts with more than 100,000 residents (Calcasieu, Jefferson, and St. Tammany). 30 The absolute decline in socio-economic characteristics in New Orleans is corroborated by Vigdor (2008). 31 In each regression, the ECLS test score (in levels and growth, respectively) is regressed on one demographic measure and a vector of school fixed effects. 32 We estimate the models separately for achievement levels and achievement growth so that the cumulative predicted effect reflects both. See table notes for details on the different cumulative measures. 33 For students who were enrolled in 2006, we found an average of 5.5 years, but this is an over-estimate because some students would have (re-)entered after 2006, and these students would have lower dosages. Given that these data include 2006-2014 (eight years), we might have expected a higher number, but note that dosages are truncated for students who were very young or near the end of their high school careers in 2006. Also, some students switch between the public and private schools and/or between districts.
25
districts) to 0.044 s.d. (favoring New Orleans).34 The latter number suggests a possible
upward bias in the treatment effects estimates in the pooled analysis, but all of the
estimates are small in magnitude relative to the average treatment effects we show later.
Overall, it appears that the elimination of public housing and the disproportionate
impact of flooding on low-income neighborhoods had a minimal effect on the relative
demographics or test scores of the public school population in the years after the
hurricanes. This is partly because the hurricane affected 80 percent of the city, so that all
demographic groups were affected. For example, the black middle class, whose children
also attended public schools in large numbers, saw a large population drop (Plyer,
Shrinath, & Mack, 2015). In addition, the increase in the number of federal Section 8
public housing vouchers was much larger than the drop in public housing units, so more
low-income families, and their children, were apparently able to return to the city than
appears at first glance.35 This evidence suggests that population change is not a major
threat to identification in the pooled analysis, especially after controlling for measureable
demographic changes.
34 The results in Table 3 are based on reading test scores only and for the entire population. We therefore also re-estimated the Panel C models for low-income ECLS students, which increases the predicted achievement effects, and re-estimated for ECLS math, which reduces the effects; thus the reported effects on reading for the whole population represent a middle ground. We thank Jane Lincove for suggesting these checks. 35 According to Seicshnaydre and Albright (2015), the number of housing vouchers used changed from 4,763 in 2000, to 8,400 in 2005 (which includes some post-Katrina months), to 17,437 in 2010, for an increase of at least 10,000 units. In contrast, public housing units dropped by about 5,000 units.
26
Results
Average Treatment Effects on Achievement
Panel Estimates for Achievement
Table 3 reports results from the panel analysis estimation of average treatment
effects (ATEs) based on equation (1) for 4th and 5th graders (separately) for 2006
returnees. Again, these are from models with matching on the lagged dependent variable,
controls for student demographics and grade repetition, and bin indicators.36 The column
(1) sample includes almost all Louisiana students who have data pre- and post-hurricane
(without matching)37; column (2) includes the entire state matched on test score levels.
We follow the same pattern in columns (3) and (4), showing unmatched and matched
samples with the hurricane-affected districts, the latter being our preferred specification.
Since our test scores end in grade 8, we can follow pre-treatment 4th (5th) graders only
through 2009 (2008).
A bit more than half of the 32 estimates are positive and significant, and only
three have negative signs (all are insignificant). The estimates are similar between the
state and hurricane-affected districts, though the unmatched estimates often fail the
parallel trends tests (see the coefficient and significance level under the standard error for
each estimate in the table). Matching largely solves the parallel trends problem, and the
point estimates remain mostly positive.
36 The year of return is based on the year the tests were taken, so a 2007 returnee likely returned in the fall of 2006. However, 2006 returnees almost all returned in spring of 2006 because all the schools were closed through fall of 2005. The vast majority of students who returned and who have post-Katrina data in grades 3-8 had returned by 2007. Also, there are very few returnees in other hurricane-affected districts to match with after 2007. 37 We excluded only those students who did not return to their 2005 district for at least one year and students who took alternative assessments. These same exclusions apply to both New Orleans and the comparison districts.
27
Focusing just on our preferred specification where students are matched to those
in other hurricane-affected districts, the point estimates average about 0.10 s.d. through
2009 for pre-treatment 4th and 5th graders (Table 3 Panel A). Since the matching is based
on test score levels, the sensitivity to matching may mean that NCLB, other statewide
policies or a change in the test scale had particular influence on low-performing students
and schools in the matched comparison group.
To leverage the entire panel, and not just two time points, we also estimate
equation (2) (i.e., Granger/event studies). Especially in math and ELA, the effects in later
years emerged from a combination of an initial dip in scores in the first year of return
followed by a positive upward trajectory. The negative effects in the first year of return
could reflect either low-performance of schools in the early years (followed by
improvement) or the trauma of returnees in New Orleans the first few years after the
storm that faded out. It is difficult to empirically distinguish between these alternative
theories, though the pooled results suggest that schools steadily improved after 2009 (see
later results), which suggests that trauma/disruption partly explains the initial dip.
Other robustness checks reinforce the above results. In addition to 2006 returnees,
we considered 2007 returnees. The 2007 returnees have both a smaller dosage and greater
potential for conflating the effects of the reforms with interim school quality; therefore, it
is not surprising that the 2007 returnees display less pronounced initial dips in scores and
that the estimates for the last available year are more positive than for the 2006 returnees
(results available upon request). We also re-estimated using Mahalanobis matching on
28
both test scores and year of return (instead of exact matching on year of return). The
results were quite similar (available upon request).38
While these estimates suggest positive effects of the New Orleans school reforms,
a key disadvantage of the panel analysis is that it stops in 2009 and prevents us from
testing whether the upward trajectory continued. Three years (2006-2009) might be
considered a short span of time to implement an entirely new type of schooling system
and recruit and select new schools and educators. In 2009, most schools were still being
operated directly by the RSD, and the majority of teachers were still those from the pre-
treatment period (Barrett & Harris, 2015). Only three schools had been closed or turned
over to other operators in this time frame, compared with 45 schools between 2008 and
2015. Also, even if the system had reached equilibrium, the dosage was limited to a
maximum of 3.5 grades for spring 2006 returnees and less for later returnees. Finally, the
initial dip in scores upon return suggests that trauma/disruption effects may have been
larger for New Orleans students and pulled down the measured effects in the short term.
If the objective is to estimate the long-term cumulative effects of the program, then these
panel analysis limitations imply that the panel estimates in Figure 3 and Table 3 are
biased downwards. The analysis that follows avoids these limitations.
Pooled Estimates for Achievement
The pooled results, shown in Table 4, are positive for every specification and
statistically significant in each of the 16 cases. Focusing just on the hurricane-affected
matched sample, the estimates are all positive and statistically significant, in the range of
0.35-0.43 s.d. across subjects. Figure 3 also suggests that the positive effects are the
38 Specifically, we re-estimated using Mahalanobis matching on test scores and year of return (which reduces extremely poor matches on test levels while sacrificing similarity on year of return).
29
result of steady improvement leading up to 2013, with an apparent plateau in 2014. The
unmatched estimates still often fail the parallel trends tests, but this problem mostly goes
away with matching. (The event study results in Figure 3 suggest even larger point
estimates than those in Table 4 for 2014, apparently due to the covariance between the
student demographics and the year indicators. However, we focus on the more
conservative estimates in Table 4.)
One of the main potential threats to identification in the pooled analysis is the
change in population. However, recall that our various estimates in Table 2 suggest very
small population changes. Also, the trends in achievement effects in Figure 3 are
inconsistent with those of population change. We find an initial positive spike in
socioeconomic status in New Orleans right after the hurricane, which dissipated in the
ensuing few years. If population change was the driving force behind the estimated
effects, then we would have expected a large initial achievement effect followed by a flat
or declining effect trend. This is almost the opposite of the actual trend in Figure 4, which
displays negative initial spikes and positive trajectories, reinforcing the idea that
population change does not bias the pooled estimates.
Switchers, IV, and Other Identification Strategies
A third identification strategy, aside from the panel and pooled approaches above,
involves only students who switch into or out of New Orleans (“in-switchers” and “out-
switchers,” respectively) and who remain in their new districts for (at least) one academic
year within either a pre-reform or a post-reform period. The basic logic is that these
switches should affect student outcomes in proportion to the change in school quality.
Therefore, if New Orleans school quality improved, then those who switched into New
30
Orleans before Katrina, what we call pre-Katrina in-switchers, should see less outcome
improvement than post-Katrina in-switchers (opposite for out-switchers). In the simplest
Switcher Method 1 (M1), we take the one-year difference in achievement for individual
students just before and after the switch and compare this growth before and after the
reforms (with regression adjustments). We also estimate a Switcher Method 2 (M2) that
accounts for changes in statewide trends in cross-district switching (i.e., district
mobility).39
One advantage of the switcher method, like the pooled analysis, is that it allows
us to test for effects many years later. We specifically use data from 2001-2005 and
2009-2013. The identifying assumption of the Switcher-M1 approach is that the
unobserved factors affecting both students’ district of enrollment and achievement are
time-constant (i.e., the types of students moving into New Orleans did not change over
time). With Switcher-M2, the assumption is weaker: that the unobserved factors
associated with cross-district mobility only follow the same trend in New Orleans and the
rest of the state.
If New Orleans schools improved because of the reforms, then we would expect,
at a minimum, that the in-switcher estimates are more positive than the out-switcher
39 The general model for the switcher strategy is: 𝐴!"# = 𝜆𝐴!",!!! + 𝜃! + 𝛽!𝑑! + 𝛽!𝐼𝑛𝑆𝑤𝑖𝑡𝑐ℎ!" + 𝛽!(𝐼𝑛𝑆𝑤𝑖𝑡𝑐ℎ!"×𝑑!) + 𝜀!" where the dependent variable 𝐴!"# is achievement in the receiving school district k. The Switcher-M1 model includes only lagged achievement of student i in time t in sending district j (𝐴!",!!!), a vector of grade fixed effects (𝜃!), and an indicator for the post-Katrina period (𝑑!) where the analysis is limited to students who switch districts. In this model, we are interested in 𝛽!which simply compares achievement growth from switches that occur before and after the reforms. Switcher-M2 is essentially a differences-in-differences analysis and accounts for the possibility that the types of students who switch districts changed over time by using switches throughout the state as a comparison group. This involves adding 𝐼𝑛𝑆𝑤𝑖𝑡𝑐ℎ!" as an indicator for whether the switch was specifically into New Orleans (𝐼𝑛𝑆𝑤𝑖𝑡𝑐ℎ!" = 0 for cross-district switches where New Orleans is neither the sender nor the receiver). Under Switcher-M2, we are primarily interested in 𝛽!. We then carry out the same estimation replacing 𝐼𝑛𝑆𝑤𝑖𝑡𝑐ℎ!" with 𝑂𝑢𝑡𝑆𝑤𝑖𝑡𝑐ℎ!". Unlike the pooled and panel strategies, there is no matching involved. We thank Andrew McEachin for suggesting this general approach.
31
estimates; that is, switching into New Orleans should have become relatively beneficial
post-reform and switching out should come at a relative achievement cost. This is largely
what we find. The in-switcher estimates for are 0.10 and 0.07 s.d. larger than the out-
switcher estimates, for the M1 and M2 methods, respectively. These estimates are not
directly comparable to the earlier pooled estimates because the switcher estimates are, by
their nature, annualized effects, while the main pooled estimates are cumulative across
years. To compare them, we re-estimated the models from Table 3 with annual
achievement gains as the dependent variable, instead of achievement levels.40 These
estimates are similar on average to the switcher analyses in Table 4. For this reason, and
because the switcher analysis involves so few students,41 we rely mainly on the pooled
version of equation (1) (e.g., Table 3) throughout the rest of the analysis.
We also considered carrying out an instrumental variables (IV) strategy akin to
Abdulkadiroğlu et al. (2014) who estimate the effects of charter schools by leveraging
plausibly exogenous takeovers of public schools by charter schools. Specifically, they
estimate the effect of attending a charter school using the pre-takeover attendance in the
takeover public school as an instrument for subsequent charter attendance. Since students
were guaranteed slots in the new charter schools (and tend to stay in the same school
buildings in any event), they create a “grandfathering” instrument. This instrument has a
strong first stage and plausibly satisfies the exclusion restriction because students could
40 Specifically, we estimated the first difference as 3rd-to-4th grade growth for the 2010-11 cohort of 3rd graders minus 3rd-to-4th grade cohort in the 2003-04 cohort of 3rd graders. Thus, there are two dimensions of changes over time in this case: within student over time and across cohorts over time. This can provide additional protection against violations of the parallel trends assumption as in a typical triple difference (DDD) model, although our preferred DD method described in the main text seems to satisfy the parallel trends assumption. Nevertheless, while the DDD increases measurement error in the dependent variable, two of the four DDD estimates are statistically significant (science and social studies), and the average point estimate is 0.07 standard deviations in annual growth, slightly below the in-switcher results. 41 The switcher strategy requires restricting the sample to just 10 percent of New Orleans students, a small and possibly unusual sample.
32
not have easily known the public schools would be turned into charter schools. The main
assumption is that attendance in pre-Katrina schools did not (conditionally) affect the
achievement trajectory after the schools became charters.
The analogous approach here would be to use attendance in pre-reform New
Orleans schools as the grandfathering instrument for attending post-reform New Orleans
schools. However, the exclusion restriction is problematic in the present setting. First,
attendance in New Orleans public schools pre-treatment could influence post-Katrina
outcomes (even after conditioning on covariates) because of trauma/disruption, the same
problem we have in the panel analysis. Second, as Abdulkadiroğlu et al. (2014) indicate,
the assumption is much more plausible with outcome gains, which is infeasible for the
key one-time events that are of interest here (high school graduation and college entry).
These problems in the present context do not arise in the one considered by
Abdulkadiroğlu et al. (2014).42 Additionally, note that the earlier switcher analyses utilize
a similar type of exogenous variation (switching districts), but without the problematic
exclusion restriction.
Our objective in this section has been to estimate the ATEs of the New Orleans
school reforms through 2014. The results are consistently positive and arguably large in
magnitude across all of the following: panel versus pooled, state versus hurricane
districts, DD versus switcher, achievement levels versus gains specifications, and across
42 We also considered using synthetic cohort analysis, though this approach does not have good statistical properties in this situation. Synthetic cohort analysis is designed for situations where there is a single treatment unit (e.g., school district) and there are multiple candidate comparison groups, some of which are more similar to the treatment group at baseline. In this case, we do have a single treatment unit (New Orleans), but almost all the variance is between schools within school districts. More generally, synthetic cohort analysis is not as useful when: (a) there is a common support problem at the level of the treatment unit (i.e., New Orleans has lower pre-treatment scores than all the districts); and (b) there are smaller units (schools) nested within the treatment unit and most of the variance in outcomes is between these smaller units, rather than the more aggregated units (i.e., districts). Under these conditions, Mahalanobis matching at the lower-level unit of aggregation appears more effective in identifying a reasonable comparison group.
33
all four academic subjects. In additional analyses (available upon request), we also show
that the results are robust across student-level versus district-level data aggregation.
Moreover, as noted above, there is no evidence of bias from missing data, or attrition.
In summary, our preferred pooled estimates, with hurricane-affected districts and
matching, suggest that the reforms increased student achievement by 0.40 s.d. (range
across subjects: 0.35-0.43 s.d.) A linear extrapolation of the panel estimates yields a very
similar value of 0.39 s.d. Alternatively, if we assume the difference between the panel
and pooled estimates in 2009 persists into the future, this yields a projected effect of 0.28
s.d. The fact that the panel estimates could reflect upward bias in the pooled analysis
(e.g., due to population change) or downward bias (e.g., due to the more severe trauma
and disruption effects for New Orleans students) implies a range of 0.28-0.40 s.d.
Average Treatment Effects for High School Graduation
We focus on pooled analysis for high school graduation, with school-level
matching based on pre-treatment graduation rates and estimation of equation (2) for
cohorts of first-time 9th and 10th graders. The tables and figures report the years the
students started these earlier grades (the cohort years), e.g., for 10th graders, the last
cohort where we can identify an effect is the one that was in that grade in 2011-12, for
which the on-time graduation year would be 2014. We focus on any graduation but
discuss on-time graduation as well.
The pooled results, shown in Table 6, are uniformly positive and usually
statistically significant, in the range of 3-13 percentage points, across 9th and 10th graders.
As in the analysis of test scores, the estimates are least positive and most likely to be
insignificant in the matched analyses. In this case, the estimates are also smaller in the
34
hurricane-affected districts, regardless of matching. Therefore, our preferred range is 3-9
percentage points. From Table 1, the baseline graduation rate was 50-60 percent, so this
effect translates into a 10-14 percent improvement in the high school graduation rate.
Our preferred estimates are positive but insignificant when we use cohorts of 10th
graders. This could be related to the fact that a large share of high school dropout occurs
between 9th and 10th grade. However, given that we cannot test for parallel trends with 9th
graders, we cannot rule out that the results are driven by non-parallel trends.
The time trend seems to suggest a more immediate effect than the gradual
improvement we saw with test scores (Figure 4). This may be because the first effect on
graduation is for the 2008 cohort of 10th graders who experienced the reforms into 2010.
That is, compared with the test score analyses, the first graduation effect reflects both a
larger dosage in total, and a larger share of dosage that occurred after the reforms had a
chance to affect schools.
We find no evidence that the high school graduation results are driven by
accountability-based strategic behavior, i.e., schools taking advantage of the fact that
some exit codes are difficult for the state to verify. The estimates are nearly identical
across the graduation rates definitions (this can be seen by looking down each column of
Table 6 within the 9th and 10th grade groups). That is, even if we treat the less reliable exit
codes as dropouts, the reform effects on graduation are very similar.
The results are consistently more positive with delayed compared to on-time
completion (available upon request). This may reflect a combination of increased
academic expectations and increased effort by educators to keep students in school, so
that students persist even after they have fallen a grade level behind. The reforms also
35
may have led New Orleans schools to increase use of “credit recovery” programs that
allow students to gain academic credit through possibly less rigorous and less time-
consuming online programs (Harris, Liu, & Barrett, 2017). While this is difficult to test,
conversations with educators in multiple districts indicate that credit recovery programs
are being used in most districts and are not limited to New Orleans.
Our preferred ATE estimate of 3-9 percentage point effect is much smaller than
the descriptive improvement in New Orleans of 17-20 percentage points (Perry, Harris, &
Buerger, 2015). This is mostly because high school graduation rates were increasing
statewide, due in part to the addition of federally mandated high-stakes accountability for
graduation rates that started around 2007 (Harris, Liu, & Barrett, 2017). Still, a 3-9
percentage point effect is economically large, as the later cost-benefit analysis shows
more clearly.
Average Treatment Effects on College Attendance and Graduation
College attendance and college graduation are especially important outcomes,
given: (a) how strongly they predict long-term life outcomes (Kane & Rouse, 1995;
Goldin & Katz, 2008; Heckman, Humphries, & Veramendi, 2016); and (b) that they are
less prone to strategic behavior than high school measures because they are collected
completely outside of schools and without school accountability.
The ATEs for on-time college attendance and any attendance are 15 and 8
percentage points, respectively (Table 7).43 The effects are uniformly positive and
statistically significant across samples and matching. The reported estimates all pass the 43 On-time means that students attended college immediately after graduating high school. One reason for using this approach is that this is how college entry is defined in the BOR data that we used in the matching process. The college persistence measure discussed below does not make this restriction. The generally positive estimates across these various definitions suggest that our results are robust to how we define college outcomes.
36
parallel trends test for our preferred specifications with matching.44 Given the baseline
college attendance rate for 12th graders of 53.4 percent (any attendance) and 22.5 percent
(on-time; see Table 1), these represent a 15-67 percent improvement over the baseline.
The only estimates with negative signs are those for attendance in two-year colleges.
Given the overall positive effect on attendance, it therefore appears that some students
likely shifted from two- to four-year colleges, and others, who would not have attended
any college, attended a two-year college. This shift to four-year college is indicative of a
larger economic return (e.g., Kane & Rouse, 1995). As an additional check, we re-
estimated the effects on college attendance using only the BOR data, and the results are
qualitatively similar.
We measure persistence by comparing the percentage attending college with the
percentage attending college for two or four years in total. (Note that this refers to the
total number of years in any college and does not distinguish two-year colleges from
four-year colleges.) We see positive effects of 4-7 percentage points for the two
persistence measures (25-28 percent increases from baseline).
We also estimated effects on college completion within five years of expected
high school graduation, among students who reached 12th grade. The effect magnitudes
are positive though smaller than the others, at 3-5 percentage points (30-45 percent
effect). Figure 5A shows the event study results for college attendance and graduation,
allowing us to compare the effects for specific cohorts. This reinforces that those students
who do attend college as a result of the reforms are usually benefiting beyond just the
first semester or year.
44 Recall that the parallel trends tests use only on-time college attendance, given our inability to track persistence and graduation before 2005.
37
The smaller absolute effects on college graduation apparently reflect three factors.
First, the college graduation rate is 10 percent at baseline (among 12th graders) as shown
in Table 1, so the 3-5 percentage point effect is larger, in percentage terms.45 Second,
college graduation is under-counted in the NSC. If the under-counting is orthogonal to
treatment, or higher in the treatment group, then this biases the effect estimates
downward. Third, we can only estimate graduation effects for the first three cohorts of
12th graders (through 2009) who had experienced more disruption and lower dosages in
New Orleans schools during the early stages of the reform’s development.46 As with the
parallel trends tests shown for college attendance in Table 7, we do not reject the null of
non-parallel trends in any of the college persistence and graduation levels.47
Like the high school graduation results, the event studies suggest an immediate
effect on college outcomes. One possible explanation is that the effect was driven by an
increase in educators’ expectations for students to attend college, which could occur
quickly without significant academic improvement.48 The average treatment effects for
all student outcomes, as well as the magnitudes of potential biases, are summarized in
Table 8.
45 According to Bailey and Dynarski, the national college graduation rate is 25 percent (conditional on college entry) among low-income students. This cannot be directly compared with the graduation rate for 12th graders. 46 Research on school closure, for example, suggests that high school students have more difficulty bouncing back from school disruptions (Larsen, 2017). 47 As discussed earlier, we cannot measure college persistence and graduation in the pre-reform years, therefore we modified the parallel trends test, using the weights from matching on a single lagged value of the dependent variable, but using only college attendance for the parallel trends test. The range of these modified parallel trends tests is -0.001 to +0.019, where positive point estimates could suggest upward bias, though the standard errors are not close to reaching statistical significance. These are not shown in Table 7 to avoid confusion between the modified and unmodified versions of the parallel trends tests. 48 Also, note that, compared with elementary schools, a large share of high schools (and high school seats) remained under the school district control after Katrina. This is because the only schools not taken over by the state were those that were high performing. Some pre-Katrina high schools were high performing because they were selective admissions. However, most OPSB schools were also turned into charter schools, and all were affected by the move to school choice and the elimination of the union contract.
38
Effect Heterogeneity
One of the most common critiques of the New Orleans school reforms is that they
have been inequitable and even harmful to disadvantaged students. Numerous media
reports and lawsuits have alleged denied admission, disproportionate numbers of
suspensions and expulsions, and insufficient services among certain disadvantaged
students under the city’s reforms (P.B. v. Pastorek, 2010). Charter school leaders in New
Orleans have reported advising challenging students to move to other schools (Jabbar,
2015), and there are some other signs students have done so (Ferguson, 2017). This
problem was especially pronounced with school discipline, albeit only in the early years
of the reform (Hernandez, forthcoming). While these practices cannot plausibly bias our
ATE estimates,49 they may affect the distribution of benefits across groups.
Given that the vast majority of New Orleans students are black and/or low-
income, the ATEs reported earlier clearly suggest that these disadvantaged groups
benefited with higher outcomes. However, it could still be the case that the reforms
exacerbated education gaps across groups within the district. To test this, we carried out
the same estimation methods as above, but separately by FRPL and race/ethnicity.50 The
49 If low-performing students are pushed out of post-reform schools, they almost all end up in some other publicly funded school in the city and therefore still show up in our analyses regardless. 50 Identification of effects for English Language Learners (ELLs) and special education students is left for future research due to several additional methodological issues. The ELL population in New Orleans was small before the storm and grew considerably afterwards. Also, there are extremely few ELL students in the comparison districts with which to match. The empirical challenges with special education are a bit different. After the storm, many special education students began taking new types of alternative assessments. There is no crosswalk between these and the regular state tests, and the percentage of students taking the alternative assessments changed widely over time. Moreover, there are good reasons to believe that selection into special education worked differently before and after the reforms, which limits us to panel analysis over just the first few years. For these reasons, we leave the analysis of this important topic to a separate study.
39
matching process is similar, except for the additional stratification by subgroup.51 The
Granger/event study results by race and FRPL and for each of the main outcomes can be
found in Figures 6 and 7.
The results are much more positive for disadvantaged students, compared with
other students, with regard to high school graduation and college-going. This is not true
with test scores, however. In none of the models or years did black or FRPL students see
larger effects than their white or non-FRPL counterparts, in terms of achievement. In the
later years (pooled analysis), the effects for black and FRPL students converge to those of
their counterparts, suggesting that all the various groups benefited in similar ways in the
long run. With high school graduation, in contrast, black and low-income students seem
to have benefited slightly more, and from the very beginning of the reforms.
With college attendance, essentially the entire citywide effect of the reforms
seems to have come from improvement among black and FRPL students. The results for
black students violate parallel trends, though in a way that suggests our results may
under-state the disproportionate effects for black students. In additional analysis, we re-
ran the results unmatched and see similar results in the post-reform period (but still with
non-parallel trends in the pre-reform period).
With FRPL, we also see more positive effects for lower-income students initially,
followed by convergence. These unstable trends with regard to FRPL, especially in the
test score and college attendance results, likely reflect that FRPL is not an especially
51 In the pooled subgroup matching, we also restricted the comparison group to schools that had at least 10 students in the given subgroup (e.g., 10 in FRPL and 10 non-FRPL), so that the estimates for each pair of subgroups reflect the same comparison schools. Also, we matched on the test scores of each pair of subgroups simultaneously; for example, for each New Orleans school, we looked for a comparison school where FRPL students had similar test scores to the FRPL students in the New Orleans school and where the non-FRPL students in the potential comparison also had scores similar to the non-FRPL students in the New Orleans school.
40
reliable indicator of poverty, especially in this setting.52 Given that the race indicators do
not suffer from the same flaws and that there is a strong positive correlation between
income and race, the effect heterogeneity by race is probably more valid.
In both the panel and pooled cases, we also carried out many of the same
robustness and bias checks for each subgroup. In general, the sub-group analyses pass
these tests and are robust with alternative specifications. The fact that the results for black
students mirror those of the ATEs also reinforces the validity of the latter, showing that
the results are the same even with a different method of matching.53
Spending and Costs
To interpret these positive effects on achievement, high school graduation, and
college outcomes, we also have to consider the increase in school spending that
accompanied the market-based reforms (Buerger & Harris, 2016). First, the spending
increase can be viewed as an alternative to the market-based reforms as an explanation
for the above effects on student outcomes. Second, it can be viewed as an investment in
the reforms that calls for a cost-benefit-analysis. We consider the cost side from both of
these perspectives.
52 There are two issues with FRPL: the administration of the program generally and the rules that apply under natural disasters. To the latter point, after Katrina, almost all New Orleans’ public school students could be considered “homeless” when they first returned, and this automatically made them eligible for FRPL. This is because, under FRPL rules, a student is considered homeless if “s/he is identified as lacking a fixed, regular and adequate nighttime residence by the LEA homeless liaison, or by the director of a homeless shelter” (USDA, 2014). Many students were living with relatives or in homes that were still heavily damaged. Thus, even some students who are otherwise socio-economically advantaged could be considered homeless and eligible for FRPL. Since FRPL students are only compared with other FRPL students, this likely led to what appear to be large achievement effects at first and then smaller effects. Further, this pattern would not appear in the panel analysis because FRPL eligibility in that case is based entirely on pre-treatment FRPL eligibility. We thank Lindsay Bell Weixler for pointing out this issue with the FRPL homeless designation. 53 The high school graduation effects, however, look noticeably larger for blacks than in the ATEs in Figure 5. The reason this is possible is that we always weighted the districts to match the size of New Orleans, and these other districts often had very small black populations. Therefore, the comparison districts used in estimating the ATEs represent different demographic populations than those in the effect heterogeneity analyses.
41
The counterfactual in this analysis is what would have happened to student
outcomes if New Orleans schools had returned to business as usual, only with more
funding. Jackson, Johnson, & Persico (2016) found that a $1,000 increase in school
spending, caused by state school funding lawsuits, increased high school graduation rates
by roughly 10 percentage points. Also, Lafortune, Rothstein, and Schanzenbach (2016)
found that state funding adequacy lawsuits increased relative spending in low-income
districts by about $700 per pupil and reduced the NAEP achievement gap with high-
income districts by about 0.1 s.d. Taken at face value, these high-end effect estimates
suggest that the increased spending could explain a substantial share of our estimated
effects.
It is questionable, however, whether the results from these studies provide a valid
indication of the counterfactual in this case. First, the corruption and dysfunction in the
Orleans Parish School Board prior to the storm implies that the additional resources
would not have been used to generate better outcomes to the extent that the average
district did in the above school funding studies. Second, the city’s spending increase,
which came mainly from local funding and philanthropists, may have been partly caused
by the reforms. The same inefficiencies that led to public disenchantment with the local
OPSB pre-Katrina led to a widespread perception in the city that the reforms improved
schools (Cowen Institute, 2016). This increased public support likely contributed to
political support for local property tax levies and the backing of philanthropists that
produced the spending increase. Any effect of spending on student outcomes, in this
sense, may not be just an alternative explanation, but rather an indirect effect of the
42
reforms. Therefore, while spending almost certainly contributed to the overall effect, it is
unclear whether it was a substantial cause.
We also provide a cost-benefit analysis in Table 8 based on the prior frameworks
of Krueger & Whitmore (2001), using the above $1,358 per student estimate of the
reform costs,54 combined with evidence from other studies on the labor market returns to
cognitive skill and years of education (Murnane, Willett, & Levy, 1995; Krueger &
Whitmore, 2001). As shown at the bottom of Table 8, the New Orleans reforms easily
pass a simple cost-benefit test. More importantly, the benefit-cost ratios (and internal
rates of return) are in the same range as the Perry Preschool experiments (Heckman et al.,
2010) and are larger than the Tennessee STAR experiment and almost all the other
rigorously studied programs.55 The notes to Table 8 provide additional details about the
assumptions of this analysis.
Conclusion
Critics of American schooling have long advocated for a radically different
system than the government-driven school district model that still predominates
throughout the country. New Orleans is the first U.S. school system to overturn that
system and replace it with a market-driven one. We find that that the reform package put
in place after Hurricane Katrina had large positive effects on both the quality and quantity
of education New Orleans students received. The reforms increased student achievement
by 0.28-0.40 s.d., high school graduation by 3-9 percentage points (10-14 percent),
college attendance by 8-15 percentage points (15-67 percent), college persistence by 4-7
54 The $1.8 billion investment in buildings was slow to yield actual improvements in buildings and could not have had a significant influence on these results. 55 See Harris (2009) for comparative cost-effectiveness of these and other programs.
43
percentage points (25-28 percent), and college graduation by 3-5 percentage points (30-
45 percent). Moreover, the reforms reduced most education gaps between this
socioeconomically challenged district and the rest of the state, and between advantaged
and disadvantaged groups within the district.
The results are robust across multiple identification strategies and dozens of
robustness checks, as well as various additional tests we conducted to test the threats to
identification. None of our three types of analysis suggests that population change could
explain more than 10 percent of our preferred estimates. The net effects of interim
schools and trauma/disruption also seem small and temporary, and these tend to work in
the other direction of the other biases, thus partly cancelling them out. The fact that we
see effects across all academic subjects and across all outcomes, regardless of the stakes
attached to those measures, also reinforces that the improvement was not the result of
strategic behavior.
Our results are also consistent with specific types of operational changes that
occurred in the city as a result of the reforms: (a) with attendance zones almost entirely
eliminated, families became more active choosers and travelled much further to get to
higher quality schools (Harris & Larsen, 2015); (b) schools are differentiated in the types
of programs they provide, making good matches with family preferences more likely
(Arce-Trigatti, Lincove, Harris & Jabbar 2015); (c) the state RSD ended the contracts of
schools that were low-performing (Bross, Harris, & Liu 2016); and (d) the teacher
workforce changed significantly and in ways plausibly consistent with achievement
growth (Barnett & Harris, 2015). Moreover, some of the potential unintended
consequences did not emerge. Additional DD analyses suggest, for example, that
44
voluntary student mobility (Maroulis, Santillano, Jabbar, & Harris, 2015) and segregation
(Barrett, Weixler & Harris, 2015) have been largely unaffected by the reforms.
Nevertheless, the fact that the New Orleans reforms seem to have been beneficial
on average, and for key subgroups, does not mean these benefits would extend to other
cities. In general, external validity considerations rest on the types of people served, the
intensity and quality of policy implementation, and the basis of comparison
(counterfactual). Market-based school reforms seem consistently effective with black and
low-income students in urban areas, perhaps because they have such test scores and
greater geographic density in which competition can develop (Harris, 2017).
The New Orleans reforms were also implemented with an unusual, and perhaps
unusually large and high-quality, supply of educators. A national out-pouring of support
from across the nation occurred in response to Hurricane Katrina. The city also became
an epicenter for school reform and a magnet for ambitious, talented, young educators
from around the country (Harris, 2015). These distinctive features in New Orleans were
almost certainly important given the importance of teachers in generating student
outcomes. More broadly, the results had everything going for them in this case.
While the reforms were implemented in an entire school district, taking the policy
to a larger scale could prove more challenging. Teacher quality again comes into play
because the supply of educators from Teach for America and other alternative preparation
programs is limited. New Orleans is also a relatively small urban district and requires
relatively few teachers. Taking New Orleans-style reforms to larger districts, or simply
more districts, would require larger shifts in teacher supply.
45
Finally, the counterfactual in this difference-in-differences analysis is a pre-
treatment school system that, by just about any measure, was failing badly. Corruption,
mismanagement, and rapid turnover of superintendents resulted in extremely poor student
outcomes (Council of Great City Schools, 2001; Buerger & Harris, 2015, Cowen
Institute, 2015; Perry, Harris, & Buerger, 2015). Even some of the strongest critics of the
reforms agree that major changes were in order (Ferguson, 2017). New Orleans had
nowhere to go but up.
While the generalizability of the findings are, as always, a bit unclear, there is
much to be learned here. More than a decade ago, Hoxby (2000) speculated on how
difficult it might be to ever observe the effects of a massive reform in a U.S. school
system and that it would take 10 years to see a radical departure from the traditional
school district reach equilibrium.56 The conditions she described are quite similar to what
we see in New Orleans. The successes documented here force educators and
policymakers to question assumptions about how an education system can and should be
designed and operated. It shows that, at least under certain circumstances, intensive
system-wide school reform, based on principles of accountability and school autonomy,
have the potential to produce large effects on student learning. The question now is
whether such large gains can be achieved at scale in other cities, through these or other
means, without a tragedy like Hurricane Katrina.
56 Hoxby (2000, p.1209) writes that the “Tiebout process . . . is still the most powerful force in American schooling. It will be years before any reform could have the pervasive effects that Tiebout choice has had on American schools. Moreover, the short-term effects of reforms [would be] misleading because … the supply response to a reform--the entry or expansion of successful schools and the shrinking or exit of unsuccessful schools--may take a decade or more to fully evince itself.”
46
References Abdulkadiroğlu, A., Angrist, J.D., Dynarski, S., Kane, T.J., & Pathak, P. (2011)
Accountability and Flexibility in Public Schools: Evidence from Boston’s Charters and Pilots. The Quarterly Journal of Economics 126: 699–748.
Abdulkadiroğlu, A., Angrist, J.D., Hull, P.D., & Pathak, P.A. (2015). School vouchers
and student achievement: First-year evidence from the Louisiana Scholarship Program. Working Paper 2015.06. School Effective and Inequality Initiative. Cambridge, MA: Massachusetts Institute of Technology.
Abdulkadiroğlu, A., Angrist, J.D., Hull, P.D., & Pathak, P.A. (2016). Charters without
lotteries: Testing takeovers in New Orleans and Boston. American Economic Review 106(7): 1878-1920.
America Next (2015). K-12 Education Reform: A Roadmap. Downloaded April 27, 2015
from: http://americanext.org/wp-content/uploads/2015/02/America-Next-K-12-Education-Reform.pdf.
Angrist, J.D., Cohodes, S.R., Dynarski, S.M., Pathak, P.A., & Walters, C.R. (2016).
Stand and Deliver: Effects of Boston’s Charter High Schools on College Preparation, Entry, and Choice. Journal of Labor Economics 34(2): 275-318.
Angrist, J.D., Dynarski, S.M., Kane, T.J., Pathak, P.A., & Walters, C.R. (2010). Inputs
and Impacts in Charter Schools: KIPP Lynn. American Economic Review 100(2): 239-43.
Angrist, J.D., Pathak, P., & Walters, C.R. (2013). Explaining charter school
effectiveness. American Economic Journal: Applied 5(4): 1-27. Angrist, J. & Pischke J-S. (2009). Mostly Harmless Econometrics. Princeton, NJ:
Princeton University Press. Arce-Trigatti, P., Harris, D., Lincove, J., & Jabbar, H. (2015). Many options in New
Orleans public schools. Education Next 15(4): 25-33. Athey, S. & Imbens, G. (2003). Identification and inference in nonlinear
difference-in-differences models. Econometrica 74(2): 431-497. Barrett, N. & Harris, D. (2015). Significant Changes in the New Orleans Teacher
Workforce. New Orleans, LA: Tulane University, Education Research Alliance for New Orleans.
Berry, J. Jacob, B.A., & Levitt, S.D. (2005). The impact of school choice on student
outcomes: an analysis of the Chicago Public Schools. Journal of Public Economics 89 5: 729-760.
Belfield, C.R. & Levin, H.M. (2003). The effects of competition on educational
outcomes: a review of US evidence. Review of Educational Research 72(2): 279-341.
47
Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How much should we trust differences-in-differences estimates? Quarterly Journal of Economics 119(1): 249-275.
Booker, K., Sass, T., Gill, B., & Zimmer, R. (2011). The Effects of Charter High Schools
on Educational Attainment. Journal of Labor Economics, 29(2), 377-415. Brown, T.H., Mellman, T.A., Alfano, C.A., & Weems, D.F. (2011). Sleep fears, sleep
disturbance, and PTSD symptoms in minority youth exposed to Hurricane Katrina. Journal of Traumatic Stress 24(5): 575–580.
Buerger, C., & Harris, D., (2015). How can decentralized systems solve system-level
problems? An analysis of market-driven New Orleans school reforms. American Behavioral Scientist 59(10): 1246–1262.
Carnoy, M. & Loeb, S. (2003). Does external accountability affect student outcomes? A
cross-state analysis. Educational Evaluation and Policy Analysis 24(4): 305-331. Center for Research on Education Outcomes (2013a). National Charter School Study.
Palo Alto, CA: Stanford University. Center for Research on Education Outcomes (2013b). Charter School Performance in
Louisiana. Palo Alto, CA: Stanford University. Chubb, J.E., & Moe, T.M. (1990). Politics, Markets, and Schools. Washington, DC:
Brookings Institution. Council of Great City Schools (2001). Rebuilding Human Resources in New Orleans
Public Schools. Washington, DC. Cowen Institute for Public Education Initiatives (2015). State of Public Education in New
Orleans. New Orleans, LA: Tulane University. Cowen Institute for Public Education Initiatives (2016). What Happens Next? Voters’
Perceptions of K-12 Public Education in New Orleans. New Orleans, LA: Tulane University.
The Data Center (2014). Who Lives in New Orleans and Metro Parishes Now? New
Orleans, LA. Dee, T. & Jacob, B. (2011). The impact of No Child Left Behind on student achievement,
Journal of Policy Analysis and Management 30(3): 418-446. DeSalvo, K. B., Hyre, A.D., Ompad, D.C., Menke, A., Tynes, L.L., & Muntner, P.
(2007). Symptoms of posttraumatic stress disorder in a New Orleans workforce following Hurricane Katrina. Journal of Urban Health 84(2): 142-152.
Dreilinger, D. (2014). Arne Duncan: New Orleans education community is 'leading the
nation where we have to. Times-Picayune. Downloaded January 1, 2016 from: go'http://www.nola.com/education/index.ssf/2014/12/arne_duncan_new_orleans_educat.html.
48
Dynarski, S.M., Hemelt, S.W. & Hyman, J.M. (2013). The Missing Manual: Using National Student Clearinghouse Data to Track Postsecondary Outcomes. NBER Working Paper No. 19552. Cambridge, MA: National Bureau of Economic Research.
Elliott, J.R. & Pais, J. (2006), Race, class, and Hurricane Katrina: Social differences in
human responses to disaster. Social Science Research 35: 295–321. Epple, D., Romano, R., & Zimmer, R. (2015). Charter schools: A survey of research on
their characteristics and effectiveness. NBER Working Paper 21256. Cambridge, MA: National Bureau of Economic Research.
Evers, W. (2014) Implementing standards and testing. In Chester E. Finn and Richard
Sousa, What Lies Ahead for America’s Children and Their Schools. Stanford, CA: Hoover Institution Press.
Ferguson, B. (2017). Outcomes of the State Takeover of New Orleans Schools. Dorrance
Publishing Company. Figlio, D. (2006). Testing, crime and punishment. Journal of Public Economics 90(4):
837-851. Friedman, M. (1962). Capitalism and Freedom Chicago: University of Chicago Press. Fryer, R.G. (2014). Injecting charter school best practices into traditional public schools:
Evidence from field experiments. Quarterly Journal of Economics 129(3):1355-1407.
Gill, B. & Booker, K. (2008). School competition and student outcomes. In Helen F.
Ladd and Edward B. Fiske (Eds) Handbook of Research in Education Finance and Policy (pp.183-202). New York: Routledge.
Goldin, C. & Katz, L. (2008). The Race Between Education and Technology. Cambridge,
MA: Harvard University Press. Groen, J. & Polivka, A. (2008). The effect of Hurricane Katrina on the labor market
outcomes of evacuees. American Economic Review 98(2): 43–48. Hanushek, E.A. (1996). A more complete picture of school resource policies.
Review of Educational Research 66(3): 397-409. Hanushek, E.A. (2002). Publicly Provided Education. NBER Working Paper 8799. Cambridge, MA: National Bureau of Economic Research. Hanushek, E.A. & Woessman, L. (2010). The High Cost of Low Educational
Performance: The Long Run Impact of Improving PISA Outcomes. Paris: Organization for Economic Development and Cooperation.
Harris, D.N. (2009). Toward policy-relevant benchmarks for interpreting effect sizes:
Combining effects with costs. Educational Evaluation and Policy Analysis 31(1): 3-29.
49
Harris, D.N. (2017). Why Managed Competition is Better than a Free Market for Schooling. Washington, DC: Brookings Institution.
Harris, D.N., Liu, L., & Barrett, N. (2017). Is the Rise in High School Graduation Rates
Real? High-Stakes School Accountability and Strategic Behavior in Louisiana. Paper presented at the annual meeting of the Association for Public Policy and Management (APPAM). Washington, DC.
Harris, D.N. & Larsen, M. (2015). What Schools Do Families Parents Want (and Why)?
Academic Quality, Extracurricular Activities, and Indirect Costs in New Orleans Post-Katrina School Reforms. New Orleans, LA: Education Research Alliance for New Orleans, Tulane University.
Harris, D.N., Witte, J. F., & Valant, J. (2017). The market for schooling. Shaping
Education Policy: Power and Process (2nd edition). New York, NY: Routledge. Heckman, J.J., Humphries, J.E., & Veramendi, G. (2016). Returns to education: The
causal effects of education on earnings, health, and smoking. NBER Working Paper 22291. Cambridge, MA: National Bureau of Economic Research.
Heckman, J.J., Lochner, L.J., & Todd, P.E. (2008). Earnings Functions and Rates of
Return, Journal of Human Capital 2(1): 1–31. Heckman, J.J., Moon, S.H., Pinto, R., Savelyev, P. & Yavitz, A. (2010). A New cost-
benefit and rate of return analysis for the Perry Preschool program: A summary. NBER Working Paper 16180. Cambridge, MA: National Bureau of Economic Research.
Hill, P. & Campbell, C. (2011). Growing Number of Districts Seek Bold Change
With Portfolio Strategy. University of Washington: Center for Reinventing Public Education.
Hill, P. & Lake, R. (2004). Charter Schools and Accountability in Public Education.
Washington, DC: Brookings Institution Press. Hill, P. L. Pierce, J. Guthrie (1997). Reinventing Public Education: How
Contracting Can Transform America’s Schools. Chicago: University of Chicago Press.
Hornbeck, R. & Keniston, D. (2017). Creative Destruction: Barriers to Urban Growth and
the Great Boston Fire of 1872. American Economic Review, 107 (6): 1365-98. Hoxby, C.M. (1996). How teachers' unions affect education production. Quarterly
Journal of Economics 111(3), 671-718. Hoxby, C. (2000). Competition among Public Schools Benefit Students and Taxpayers?
American Economic Review 90(5): 1209-1238. Hoxby, C. (2002). School choice and school productivity (or could school choice be a
tide that lifts all boats). NBER Working Paper 8873. Cambridge, MA: National Bureau of Economic Research.
Jackson, K.C, Johnson, RC., & Persico, C. (2016). The Effects of School Spending on
Educational and Economic Outcomes: Evidence from School Finance Reforms.
50
The Quarterly Journal of Economics 131(1): 157–218. Jacob, B.A. (2005), Accountability, incentives and behavior: The impact of high-stakes
testing in the Chicago Public Schools. Journal of Public Economics 89: 761-796. Kane, T., & Rouse, C. (1995). Labor-Market Returns to Two- and Four-Year College.
American Economic Review 85(3): 600-614. Kollman, K., Miller, J.H., & Page, S.E. (1997). Political institutions and sorting in a
Tiebout model. American Economic Review 87: 977-92. Koretz, D. (2009). Measuring Up: What Educational Testing Really Tells Us. Cambridge,
MA: Cambridge University Press. Krueger, A. B., &Whitmore, D. M. (2001). The effect of attending a small class in the
early grades on college-test taking and middle school test results: Evidence from Project STAR. Economic Journal 111, 1–28.
Krueger, A.B. & Zhu, P. (2004). Another look at the New York City voucher experiment.
American Behavioral Scientist 47(5): 658-98 Lafortune, J., Rothstein, J., & Schanzenbach, D.W. (2016). School Finance Reform and
the Distribution of Student Achievement. NBER Working Paper No. 22011. Cambridge, MA: National Bureau of Economic Research.
Lee, J. (2008). Test-driven external accountability effective? Synthesizing the evidence
from cross-state causal-comparative and correlational studies. Review of Educational Research 78(3): 608-644.
Levin, H., Belfield, C., Muennig, P., & Rouse, C. (2006). The Costs and Benefits of an
Excellent Education for America's Children. Liang, K-Y. & Zeger, S.L. (1986). Longitudinal data analysis using Generalized Linear
Models. Biometrika 73(1): 13-22. Louisiana Department of Education (n.d.). Graduation Exit Code Pre-Reviews.
Downloaded May 13, 2018 from: https://www.louisianabelieves.com/docs/default-source/data-management/final-exit-code-pre-reviews.pdf?sfvrsn=2.
Louisiana Department of Education. (2015). High School Performance. Retrieved from
http://www.louisianabelieves.com/docs/default-source/katrina/final-louisana-believes-v5-high-school-performance.pdf?sfvrsn=2.
Lovenheim, M. & Willen, A. (2016). The Long-Run Effects of Teacher Collective
Bargaining. CESifo Working Paper Series No. 5977. Available at SSRN: https://ssrn.com/abstract=2815374.
Ludwig, J., Duncan, G.J., Gennetian, L.A., Katz, L.F., Kessler, R.C., Kling, J.R.,
Sanbonmatsu, L. (2013). Long-Term Neighborhood Effects on Low-Income Families: Evidence from Moving to Opportunity. American Economic Review 103(3): 226-31.
51
Maroulis, S., Santillano, R., Jabba, H., & Harris, D. (2015). The Push and Pull of School Performance: Evidence from Student Mobility in New Orleans. Unpublished manuscript.
McKenzie, R. & Levendis, J. (2010). Flood Hazards and Urban Housing Markets: The
Effects of Katrina on New Orleans. The Journal of Real Estate Finance and Economics 40(1): 62–76.
McLaughlin, K.A., Berglund, P., Gruber, M.J., Kessler, R.C., Sampson, N.A., &
Zaslavsky, A.M. (2011). Recovery from PTSD after Hurricane Katrina. Depression and Anxiety 28: 439–446.
Mills, J. & Wolf, P. (2017). Vouchers in the Bayou: The Effects of the Louisiana
Scholarship Program on Student Achievement After 2 Years. Educational Evaluation and Policy Analysis 39(3): 464-484.
National Alliance for Public Charter Schools (2013). A Growing Movement:
America’s Largest Charter School Communities. Downloaded July 3, 2015 from: http://www.publiccharters.org/press/students-32-school-districts-attend-public-charter-schools-market-share-report/.
National Commission on Excellence in Education (1983). A Nation at Risk. Washington,
DC: U.S. Government Printing Office. Obama, B. (2010). Remarks by the President on the Fifth Anniversary of Hurricane
Katrina in New Orleans, Louisiana. August 29, 2010. Retrieved April 6, 2013 from www.whitehouse.gov.
Pane, J.F., McCaffrey, D.F., Tharp-Taylor, S., & Asmus, G.J., Stokes, B.R. (2006).
Student Displacement in Louisiana After the Hurricanes of 2005 Experiences of Public Schools and Their Students. Santa Monica, CA: Rand Corporation.
Pane, J.F., McCaffrey, D.F., Kalra, N. & Zhou, A.J. (2008) Effects of student
displacement in Louisiana during the first academic year after the hurricanes of 2005. Journal of Education for Students Placed at Risk 13(2-3): 168-211.
Paxson, C. & Rouse, C.R. (2008). Returning to New Orleans after Hurricane Katrina.
American Economic Review 98(2): 38-42. P.B. v. Pastorek (2010). No. 2:10-cv-04049. The U.S. District Court of the Eastern
District of Louisiana. October 25, 2010. Perry, A., Harris, D., Buerger, C., & Mack, V. (2015). The Transformation of New
Orleans Public Schools: Addressing System-Level Problems Without a System. New Orleans, LA: The Data Center.
Peterson, P. (2014). Holding students to account. In What Lies Ahead for America’s
Children and Their Schools, Chester E. Finn and Richard Sousa (Eds). Stanford, CA: Hoover Institution Press.
52
Pischke, J-S. (2005). Empirical Methods in Applied Economics: Lecture Notes. Downloaded July 24, 2015 from: http://econ.lse.ac.uk/staff/spischke/ec524/evaluation3.pdf.
Plyer, A., Shrinath, N., & Mack, V. (2015). The New Orleans Index at Ten. New Orleans
LA: The Data Center. Ravitch, D. (2000). The Great School Wars: A History of the New York City Public
Schools. Baltimore, MD: Johns Hopkins University Press. Rothstein, J. (2007). Does competition among public schools benefit students and
taxpayers? A comment on Hoxby (2000). American Economic Review 97(5): 2026-2037.
Rouse, C. (1998). Private school vouchers and student achievement: An evaluation of the
Milwaukee Parental Choice Program. Quarterly Journal of Economics 113(2): 553-602.
Rouse, C. E., Hannaway, J., Goldhaber, D., & Figlio, D. (2013). Feeling the Florida heat?
How low-performing schools respond to voucher and accountability pressure. American Economic Journal: Economic Policy 5(2): 251-81.
Sacerdote, B. (2012). When the saints come marching in: Effects of Katrina evacuees on
schools, student performance and crime. American Economic Journal: Applied 4(1): 109-135.
Sastry, N. & Gregory, J. (2013). The effect of Hurricane Katrina on the prevalence of
health impairments and disability among adults in New Orleans: Differences by age, race, and sex. Social Science & Medicine 80: 212-129.
Seicshnaydre, S. & Albright, R.C. (2015). Expanding Choice and Opportunity in the
Housing Choice Voucher Program. New Orleans: The Data Center. St. Clair, T., Hallberg, K., & Cook, T.D. (2016). The validity and precision of the
comparative interrupted time-series design: Three within-study comparisons. Journal of Educational and Behavioral Statistics 41(3): 269-299.
Strunk, K.O. & Grissom, J.A. (2010). Do strong unions shape district policies? Collective
bargaining, teacher contract restrictiveness, and the political power of teachers’ unions. Educational Evaluation and Policy Analysis 32(3): 389-406.
Tiebout, C. (1956). A pure theory of local expenditures. Journal of Political Economy
64(5): 416-424. Tyack, D. (1974). The One Best System: A History of American Urban Education.
Cambridge, MA: Harvard University Press. United States Department of Agriculture (2014). Eligibility Manual for School Meals.
Washington, DC. Vigdor, J. (2008). The Economic Aftermath of Hurricane Katrina. Journal of Economic
Perspectives 22(4), 135–154.
53
Walberg, H. (2014). Expanding the options. In What Lies Ahead for America’s Children and Their Schools, Chester E. Finn and Richard Sousa (Eds). Stanford, CA: Hoover Institution Press.
Weems, C. F., Taylor, L. K., Cannon, M. F.,Marino, R. C., Romano, D.M., Scott,
B. G., & Triplett, V. (2010). Post traumatic stress, context, and the lingering effects of the Hurricane Katrina disaster among ethnic minority youth. Journal of Abnormal Child Psychology 38: 49–56.
Whitehurst, R. (2012). The Education Choice and Competition Index: Background and
Results 2012. Washington, DC: Brookings Institution. Wong, K. & Shen, F. (2006). Mayors Improving Student Achievement: Evidence from a
National Achievement and Governance Database. Paper prepared for the 2006 annual meeting of the Midwestern Political Science Association, April 20-23, 2006.
54
Figure 1A: Trends in New Orleans Student Achievement Levels
Figure 1B: Trends in New Orleans High School Graduation Rates
Figure 1C: Trends in New Orleans College Entry
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Test
Sco
re S
td. D
ev.
Math ELA Science Social Studies
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Gra
d R
ate
Grad1 Grad2 Grad3
0
0.1
0.2
0.3
0.4
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Col
lege
Ent
ry R
ate
Any College 2-Year 4-Year
Notes: The break in the middle of the figures reflects the landfall of Hurricane Katrina and the beginning of the school reforms. The break is longer with high school graduation and college outcomes because more years of data are required to calculate a single rate in these cases. The years in the x-axis, for high school graduation and college attendance, reflect the cohort year (when students were on-time 10th and 12th graders, respectively). More details are in the main text.
55
Figures 2: Test Score Average Treatment Effects from Panel Estimation
Panel A: 2005 4th Graders Who Returned in 2006
Panel B: 2005 4th Graders Who Returned in 2007 Notes: Results are based on panel estimation of equation (2) with the matched hurricane-affected comparison group of districts (without covariate adjustment in the effect estimation). See additional detail in Table 3. Dashed grey lines indicate 95% confidence intervals.
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5
2004 2005 2006 2007 2008 2009
Test
Sco
re S
td. D
ev.
Years
Math
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5
2004 2005 2006 2007 2008 2009
Test
Sco
re S
td. D
ev.
Years
ELA
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5
2004 2005 2006 2007 2008 2009
Test
Sco
re S
td. D
ev.
Years
Social Studies
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5
2004 2005 2006 2007 2008 2009
Test
Sco
re S
td. D
ev.
Years
Science
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5
2004 2005 2006 2007 2008 2009
Test
Sco
re S
td. D
ev.
Years
Math
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5
2004 2005 2006 2007 2008 2009
Test
Sco
re S
td. D
ev.
Years
ELA
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5
2004 2005 2006 2007 2008 2009
Test
Sco
re S
td. D
ev.
Years
Science
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5
2004 2005 2006 2007 2008 2009
Test
Sco
re S
td. D
ev.
Years
Social Studies
56
Figure 3: Test Score Average Treatment Effects from Pooled Estimation
Notes: Estimates are based on equation (2) with the matched hurricane sample, averaged across grade levels. See Table 4 provides the equivalent estimates based on equation (1). Dashed grey lines indicate 95% confidence intervals.
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Test
Sco
re S
td. D
ev.
Math Scores
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Test
Sco
re S
td. D
ev.
ELA Scores
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Test
Sco
re S
td. D
ev.
Science Scores
-0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Test
Sco
re S
td. D
ev.
Social Studies Scores
57
Figure 4: High School Graduation Average Treatment Effects from Pooled Estimation
Panel A: First-time 10th graders
Panel B: First-time 9th graders
Notes: Graduation is defined here in a way that most closely approximates the typical state-defined measure (Grad1). Estimates are based on equation (2) for the matched hurricane sample. The omitted reference year is 2003 for 10th graders. The dot to the left of Panel B shows that 2002 is the reference point for 9th graders and we cannot test parallel trends in that case due to data limitations. Dashed grey lines indicate 95% confidence intervals.
-0.2
-0.1
0
0.1
0.2
0.3
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Gra
d R
ate
-0.2
-0.1
0
0.1
0.2
0.3
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Gra
d R
ate
58
Figure 5: College Outcome Average Treatment Effects from Pooled Estimation
Panel A: College Attendance
Panel B: College Graduation
Notes: Estimates are based on equation (2) for the matched hurricane sample. Years on the x-axis indicate the year that students were 12th graders, and we use a five-year college graduation, looking forward from that year. College entry is based on “on-time” college entry the fall after a student’s 12th grade year. The last cohort where this calculation is feasible is therefore 2009 (soon after the reforms began). Dashed grey lines indicate 95% confidence intervals.
-0.1
0
0.1
0.2
0.3
2004 2005 2006 2007 2008 2009
Col
lege
Gra
d R
ate
-0.1
0
0.1
0.2
0.3
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Col
lege
Ent
ry R
ate
59
Figures 6A: Effect Heterogeneity – Math Scores (by Race)
Effects by Family Income (FRPL)
Notes: The panel results are for 2005 4th graders who returned by 2006. (We omit results for 2006 scores for this group because of problems with the test administration that year, in the wake of hurricanes.) Dashed grey lines indicate 95% confidence intervals.
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
2004 2005 2006 2007 2008 2009
Test
Sco
re S
td. D
ev.
Panel Estimation
Black White
-0.4
-0.2
0
0.2
0.4
0.6
0.8
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Effe
ct (S
td. D
ev.)
Pooled Estimation
Black White
60
Figure 6B: Effect Heterogeneity – Math Scores (by FRPL)
Notes: See notes on panel and pooled analyses in Figure 6A and Tables 3 and 4.
-0.4 -0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
2004 2005 2006 2007 2008 2009
Effe
ct (S
td. D
ev.)
Panel Estimation
FRPL Non-FRPL
-0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1 1.1
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Effe
ct (S
td. D
ev.)
Pooled Estimation
FRPL Non-FRPL
61
Figure 6C: Effect Heterogeneity – High School Graduation (by Race and FRPL; Pooled Estimation)
Notes: As with the average treatment effects for high school graduation in Table 6, these figures show results for pooled analysis of first-time 10th graders who had returned to New Orleans by 2007 (and who therefore could be identified as first-time 10th graders in 2008). The results with 9th grade cohorts are similar (available upon request). Dashed grey lines indicate 95% confidence intervals. See additional notes in Figure 6A.
-0.4
-0.2
0
0.2
0.4
0.6
0.8
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Effe
ct (G
rad
Rat
e)
Black White
-0.4
-0.2
0
0.2
0.4
0.6
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Effe
ct (G
rad
Rat
e)
FRPL Non-FRPL
62
Figure 6D: Effect Heterogeneity – College Entry (by Race and FRPL; Pooled Estimation)
Notes: As with the average treatment effects for college outcomes, these effect heterogeneity estimates are based on pooled estimation with cohorts of 12th graders. Dashed grey lines indicate 95% confidence intervals.
-0.2
-0.1
0
0.1
0.2
0.3
0.4
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Effe
ct (E
ntry
Rat
e)
Black White
-0.2
-0.1
0
0.1
0.2
0.3
0.4
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Effe
ct (E
ntry
Rat
e)
FRPL Non-FRPL
63
Table 1A: Descriptive Statistics for New Orleans Before and After Katrina
Notes: Table 1A includes New Orleans students in the spring testing file for the given year, excluding students who took alternative assessments. The distribution of individual student scores is normalized to statewide 𝜇 = 0 and 𝜎=1 for the statewide population within year, grade, and subject. The mean differences in the far right-hand column indicate changes before and after the reforms in the New Orleans sample.
MeanN Mean s.d. Min Max N Mean s.d. Min Max Diff.
DemographicsAfrican-American 30,251 0.935 0.247 0 1 18,417 0.877 0.328 0 1 -0.057Hispanic 30,251 0.012 0.109 0 1 18,417 0.038 0.191 0 1 0.026Other 30,251 0.020 0.140 0 1 18,417 0.026 0.158 0 1 0.006White 30,251 0.033 0.179 0 1 18,417 0.059 0.236 0 1 0.026FRL 30,240 0.832 0.374 0 1 18,416 0.875 0.331 0 1 0.043Special Education 30,252 0.113 0.317 0 1 18,417 0.070 0.255 0 1 -0.044ELL 30,252 0.018 0.133 0 1 18,417 0.026 0.158 0 1 0.008
Test ScoresMath 30,068 -0.505 1.032 -4.5121 3.249 18,329 -0.093 1.032 -4.4327 3.173 0.413ELA 29,767 -0.539 1.011 -4.4943 3.313 18,309 -0.136 1.056 -4.9827 3.806 0.402Science 29,478 -0.624 0.931 -4.2155 3.682 18,342 -0.207 1.025 -4.9275 4.162 0.417Social Studies 29,449 -0.539 1.027 -4.2205 3.802 18,321 -0.097 1.050 -5.2573 4.406 0.443
Graduation 9th Grade (2002 vs 2011)Grad 1 4,287 0.524 0.499 0 1 2,899 0.726 0.446 0 1 0.202Grad 2 4,486 0.501 0.500 0 1 3,166 0.665 0.472 0 1 0.164Grad 3 4,293 0.610 0.488 0 1 2,902 0.785 0.411 0 1 0.175
College Attendance (on-time) 12th Grade (2004 vs 2012)Any Attendance 3,878 0.225 0.418 0 1 2,426 0.328 0.469 0 1 0.1032-Year Attendance 3,878 0.067 0.250 0 1 2,426 0.070 0.255 0 1 0.0034-Year Attendance 3,878 0.158 0.365 0 1 2,426 0.258 0.438 0 1 0.100
College Attendance (any) 12th Grade (2004 vs 2009)Any Attendance 3,878 0.534 0.499 0 1 2,306 0.655 0.475 0 1 0.1212-Year Attendance 3,878 0.287 0.452 0 1 2,306 0.411 0.492 0 1 0.1244-Year Attendance 3,878 0.372 0.483 0 1 2,306 0.393 0.489 0 1 0.021
College Persistence 12th Grade (2004 vs 2009)2 Full Years 3,878 0.278 0.476 0 1 2,306 0.374 0.484 0 1 0.0964 Full Years 3,878 0.155 0.418 0 1 2,306 0.214 0.410 0 1 0.059Years of College 3,878 1.099 2.172 0 5 2,306 1.394 1.394 0 5 0.295Grad-Rate 3,878 0.100 0.300 0 1 2,306 0.121 0.121 0 1 0.021
2004-05 2013-14 Panel A: Demographics(New Orleans)
64
Table 1B: Descriptive Statistics for New Orleans Relative to Comparison Groups (Pre-Katrina)
Notes: The pooled results use all grades while the panel results use only 4th graders who returned to their original district in 2006. High school graduation rates are based on cohorts of 9th graders. College outcomes are for cohorts of 12th graders. College entry reflects immediate enrollment after high school graduation. Most of the differences in the right-hand column are statistically significant at p<0.05 and almost are significant at p<0.10.
Panel Pool Panel Pool Panel Pool
DemographicsAfrican-American 0.920 0.935 0.419 0.730 0.500 0.204Hispanic 0.011 0.012 0.020 0.021 -0.009 -0.009Other 0.030 0.020 0.041 0.026 -0.011 -0.006White 0.040 0.033 0.520 0.222 -0.480 -0.189FRL 0.859 0.832 0.730 0.802 0.129 0.030Special Education 0.109 0.113 0.264 0.158 -0.155 -0.044ELL 0.028 0.018 0.008 0.013 0.019 0.005
Test ScoresMath -0.287 -0.505 -0.208 -0.205 -0.078 -0.300ELA -0.293 -0.539 -0.253 -0.250 -0.040 -0.289Science -0.517 -0.624 -0.423 -0.202 -0.093 -0.422Social Studies -0.467 -0.539 -0.359 -0.160 -0.108 -0.380
Graduation 9th Grade (2002)Grad 1 0.524 0.565 -0.041Grad 2 0.501 0.514 -0.013Grad 3 0.610 0.765 -0.155
College (2004)Attendance (on-time) 0.225 0.352 -0.1272-Year Attendance (on-time) 0.067 0.040 0.0274-Year Attendance (on-time) 0.158 0.298 -0.140Attendance (any) 0.534 0.571 -0.0372-Year Attendance (any) 0.287 0.217 0.0704-Year Attendance (any) 0.372 0.459 -0.0872 Full Years 0.278 0.365 -0.0884 Full Years 0.155 0.226 -0.071Years of College 1.099 1.959 -0.861Grad-Rate 0.100 0.137 -0.0374-Year Grad Rate 0.091 0.125 -0.034
Panel B: Pre-Reform Mean Differences (2004-05)
New OrleansOther Hurricane Districts
(Matched)New Orleans Minus
Comparison
65
Table 2: Effects of Population Change
Notes: Panel A shows difference-in-differences (DD) of demographics and test scores (from LDOE administrative data) between all public school students in 2005 in the respective districts and the returnees in those same districts. Panel B shows DD in district-wide demographics based on Census data (public school students only). Panel C reports regression coefficients based on the ECLS, using the same demographics as in the Census; we regressed reading score levels (and gains, separately) on the variable in the left column plus a vector of school fixed effects; each reported coefficient is from a different regression with standard errors are in parentheses. Panel D provides simulated effects of demographic change; specifically, we carried out an out-of-sample prediction, inserting the Census-based DD changes from Panel B into the regression model in Panel C. The “Cumulative” effects come from adding the effect on 3rd grade test levels to the 5th grade gains multiplied by the dosage through 2012 to obtain the total predicted effect of demographic change in student test scores. Standard errors of prediction are available upon request.
Panel A: Population Change (Average Pre-Katrina Characteristics of 3rd Graders)
All Pre-Katrina Students Returnees Diff
All Pre-Katrina Students Returnees Diff Diff-in-Diff
FRL 0.866 0.874 0.008 0.610 0.606 -0.004 0.012Special Ed 0.101 0.103 0.002 0.164 0.171 0.007 -0.005ELL 0.017 0.016 0.000 0.034 0.032 -0.001 0.001Reading Scores -0.665 -0.683 -0.018 0.118 0.143 0.025 -0.043
Panel B. Census Demographic Changes (Public School Students Only)
1999 2013 Change 1999 2013 Change Diff-in-DiffIncome (2013 $) $43,189 $42,453 -$736 $69,659 $71,408 $1,749 -$2,485Prop. BA+ 0.10 0.15 0.05 0.16 0.19 0.03 0.02Prop. Child Poverty 0.57 0.58 0.01 0.30 0.32 0.02 -0.01Prop. < H.S. 0.33 0.20 -0.13 0.23 0.16 -0.07 -0.06
Panel C. Partial Correlations Between Demographics and Test Scores (from ECLS)
Grade 3 Grade 5 Grade 8 Grade 5 Grade 8Income (thous., 2013 $) 0.003 0.003 0.003 0.0004 0.0009
(0.0002) (0.0002) (0.0003) (0.0001) (0.0002)BA+ 0.139 0.253 0.229 0.046 0.092
(0.021) (0.023) (0.03) (0.013) (0.022)Child Poverty -0.437 -0.423 -0.402 -0.082 -0.101
(0.028) (0.035) (0.051) (0.022) (0.038)<H.S. -0.369 -0.366 -0.405 -0.08 -0.076
(0.044) (0.048) (0.065) (0.029) (0.054)
Panel D. Predicted Effects of Census Demographic Change on Student Test Scores (Using Panels B and C)
Grade 3 Grade 5 Grade 8 Grade 5 Grade 8 CumulativeIncome (thous., 2013 $) -0.007 -0.007 -0.007 -0.001 -0.002 -0.012BA+ 0.003 0.005 0.005 0.001 0.002 0.007Child Poverty 0.004 0.004 0.004 0.001 0.001 0.008<H.S. 0.022 0.022 0.024 0.005 0.005 0.044
Average 0.005 0.006 0.006 0.001 0.001 0.012
Dep Var: Test Levels Dep Var: Test Gains
Test Levels Test Gains
New Orleans Hurricane-Affected Districts
New Orleans Hurricane-Affected Districts
66
Table 3: Average Treatment Effects on Test Scores based on Panel Estimation (2006 Returnees)
Notes: Each cell represents a separate regression with estimation at the student level and controls for race, free-reduced price lunch, special education status, and English proficiency in 2005 are included. Columns 2 and 4 are weighted by the number of times a student is matched using a Mahalanobis matching process on 2004 and 2005 test score levels. The first number in each cell is the point estimate for 𝛿 in equation (1). The second number (in parentheses) is the GEE cluster standard error. The third number, in brackets, is the coefficient on a parallel trend taken from a separate comparative interrupted time series estimate where the pre-2006 linear trend is interacted with a New Orleans indicator variable. See text for discussion of the matching process. Significance levels: *** p<.01, ** p<.05, * p<.10.
Entire State Entire State w/ Student
Matching
Hurricane Districts Only
Hurricane Districts w/
Student Matching
MathPost x NOLA 0.222*** 0.190*** 0.181*** 0.173** s.e. (0.055) (0.058) (0.057) (0.071) Parallel Trends Test [0.102**] [-0.002] [0.181***] [-0.011]ELAPost x NOLA 0.123** 0.121** 0.135** 0.084
(0.057) (0.060) (0.058) (0.073)[0.239***] [0.009] [0.206***] [0.013]
SciencePost x NOLA 0.223*** 0.102* 0.204*** 0.057
(0.056) (0.059) (0.057) (0.077)[-0.008] [-0.020] [-0.008] [-0.032]
Social StudiesPost x NOLA 0.249*** 0.093 0.259*** 0.094
(0.060) (0.063) (0.061) (0.080)[-0.022] [-0.025] [-0.041] [-0.057]
Number of Districts 68 68 8 8
Math 0.160*** 0.061 0.162*** 0.060Post x NOLA (0.058) (0.061) (0.060) (0.074)
[-0.069] [0.001] [-0.103**] [0.005]ELAPost x NOLA 0.220*** 0.036 0.179*** -0.005
(0.058) (0.061) (0.059) (0.075)[-0.249***] [-0.009] [-0.214***] [-0.001]
SciencePost x NOLA 0.082 -0.023 0.082 -0.097
(0.055) (0.059) (0.057) (0.072)[-0.048] [0.025] [-0.087*] [0.030]
Social StudiesPost x NOLA 0.225*** 0.083 0.213*** 0.087
(0.055) (0.058) (0.057) (0.073)[-0.066] [0.009] [-0.055] [0.016]
Number of Districts 68 68 8 8
Panel A: 2005 4th Grade Cohort 2005 vs 2009 Diff-in-Diff
Panel B: 2005 5th Grade Cohort 2005 vs 2008 Diff-in-Diff
67
Table 4: Test Score Average Treatment Effects from Pooled Estimation(2005 to 2014)
Notes: Each cell represents a separate regression with estimation at the student level, and controls for race, free-reduced price lunch, special education status, and English proficiency in 2005 are included, using data from 2005 and 2014 only. Columns 2 and 4 are weighted by the number of times a student’s school is matched using a Mahalanobis matching process on 2002 test score levels. The first number in each cell is the point estimate for 𝛿 in equation (1). The second number (in parentheses) is the GEE cluster standard error. The third number is the coefficient on a parallel trend taken from a separate comparative interrupted time series estimate where the pre-2006 linear trend is interacted with a New Orleans indicator variable. See text for discussion of the matching process. Significance levels: *** p<.01, ** p<.05, * p<.10.
Entire StateEntire State w/
School MatchingHurricane Districts
Hurricane Districts w/ School Matching
Math 0.402*** 0.387*** 0.362*** 0.426** s.e. (0.019) (0.023) (0.052) (0.119) Parallel Trends Test [0.037***] [0.007] [0.050***] [0.031**]
ELA 0.363*** 0.360*** 0.322*** 0.345***(0.015) (0.023) (0.025) (0.070)[0.014***] [-0.008] [0.023***] [0.029*]
Science 0.350*** 0.331*** 0.318*** 0.398***(0.015) (0.023) (0.040) (0.062)[0.005**] [-0.013***] [0.012**] [-0.013]
Social Studies 0.381*** 0.361*** 0.347*** 0.425***(0.016) (0.023) (0.027) (0.062)[0.018***] [-0.006] [0.024***] [-0.012]
Number of Districts 68 53 8 6
68
Table 5: Test Score Average Treatment Effects from Students Switching Districts (Annualized Effects)
Notes: Coefficients are based on student-level regressions of achievement on lagged achievement, grade fixed effects, and an indicator for whether the switch occurred before or after Katrina (Post-Katrina). Method 1 (M1) focuses only on switchers either in or out of New Orleans. Method 2 (M2) uses all possible switchers in the state and interacts the post-Katrina variables with the type of switch being made. M1 has a range of 5,066-6,761 observations, while Method 2 has a range of 81,290-82,364. Pre-Katrina district switches are included for 2002-2005, and the post-Katrina years are 2009-2013. Standard errors are GEE clustered at the sending district level for in-switchers and the receiving district levels for out-switchers. See text and earlier footnotes for more details on the model. By design of the identification strategy, the coefficients reflect annual changes in achievement, and therefore the magnitudes cannot be directly compared with the panel and pooled estimated reported elsewhere. Significance levels: *** p<.01, ** p<.05, * p<.10.
M1 M2 M1 M2
MathPost-Katrina 0.105*** -0.072*** 0.029 -0.074***
(0.031) (0.015) (0.047) (0.016)
Switch Type -0.089*** -0.136***(0.023) (0.017)
Switch Type*Post-Katrina 0.175*** 0.089**(0.035) (0.044)
ELAPost-Katrina 0.104*** -0.055*** 0.006 -0.056***
(0.016) (0.012) (0.022) (0.014)
Switch Type -0.120*** -0.114***(0.024) (0.023)
Switch Type*Post-Katrina 0.156*** 0.058**(0.020) (0.024)
SciencePost-Katrina 0.095** -0.053*** 0.033 -0.060***
(0.043) (0.018) (0.037) (0.015)
Switch Type -0.189*** -0.187***(0.024) (0.024)
Switch Type*Post-Katrina 0.145*** 0.082**(0.046) (0.039)
Social StudiesPost-Katrina 0.115*** -0.041* -0.051*** 0.077
(0.028) (0.021) (0.016) (0.048)
Switch Type -0.151*** -0.218***(0.035) (0.023)
Switch Type*Post-Katrina 0.151*** 0.119***(0.034) (0.044)
Switch in to New Orleans Switch out of New Orleans
69
Table 6: High School Graduation Average Treatment Effects from Pooled Estimation
Notes: Each cell is from a separate pooled regression estimation of equation (1). Graduation measure 1 (Grad1) only counts graduates who receive a regular diploma from their school and includes students who move out of the public school system in the denominator (see text for more detail). The second measure (Grad2) uses the same definition of graduation as Grad1, but excludes students who left the public school system from the calculation, while Grad3 uses the same total pool of students as Grad1, but allows for alternative degrees. Columns 2 & 4 use school level match weights from Mahalanobis matching of graduation rates in 2002 for the 9th grade cohorts and both 2002 and 2003 for the 10th grade cohorts. The first number in each cell is the point estimate for 𝛿 in equation (1). The second number (in parentheses) is the GEE cluster standard error. The third number, in brackets, is the parallel trends test. See text for discussion of the matching process. Significance levels: *** p<.01, ** p<.05, * p<.10.
Entire StateEntire State w/
School MatchingHurricane Districts
Hurricane Districts w/
School Matching
2012 10th Grade Grad 1 0.120*** 0.096*** 0.069* 0.031 s.e. (0.013) (0.018) (0.030) (0.060) Parallel Trends Test [-0.027***] [-0.016***] [-0.022*] [-0.015]
Grad 2 0.102*** 0.088*** 0.064* 0.055(0.011) (0.015) (0.033) (0.051)
[-0.025***] [-0.012**] [-0.021] [-0.008]
Grad 3 0.126*** 0.097*** 0.079*** 0.039(0.011) (0.016) (0.021) (0.038)
[-0.060***] [-0.044***] [-0.057***] [-0.046***]2011 9th Grader Grad 1 0.119*** 0.126*** 0.079* 0.075* s.e. (0.012) (0.011) (0.034) (0.032)
Grad 2 0.100*** 0.110*** 0.064 0.091*(0.012) (0.009) (0.041) (0.042)
Grad 3 0.126*** 0.123*** 0.090*** 0.081**(0.009) (0.011) (0.018) (0.023)
Number of Districts 68 42 8 5
70
Table 7: College Outcome Average Treatment Effects from Pooled Estimation
Note: Each cell is from a separate pooled regression estimation of equation (1). Estimation is restricted to first-time 12th graders. Columns 2 and 4 use school level match weights from Mahalanobis matching on pre-treatment values of the dependent variables. On-time attendance measures compare 2005 rates to 2012, all other outcomes compare 2005 to 2009 rates. The first number in each cell is the point estimate for 𝛿 in equation (1). The second number (in parentheses) is the GEE cluster standard error. The third number, where shown, is the parallel trend test based on on-time enrollment from the BOR data. See text and footnotes for details of the modified parallel trends tests for college persistence and graduation. Significance levels: *** p<.01, ** p<.05, * p<.10.
Entire StateEntire State w/
School MatchingHurricane Districts
Hurricane Districts w/
School Matching
Attendance (on-time)Any College Attendance 0.103*** 0.114*** 0.095*** 0.150*** s.e. (0.010) (0.014) (0.019) (0.026) Parallel Trends Test [0.016***] [0.003] [0.011] [-0.001]
2-Year Attendance -0.019*** -0.029*** -0.010 -0.020*(0.005) (0.009) (0.007) (0.008)[0.002] [0.003] [0.002] [0.002]
4-Year Attendance 0.122*** 0.138*** 0.105*** 0.161***(0.010) (0.012) (0.016) (0.025)
[0.013***] [-0.006] [0.010] [-0.006]
Attendance (any)Any College Attendance 0.067*** 0.066*** 0.079** 0.078** s.e. (0.010) (0.012) (0.025) (0.025)
2-Year Attendance -0.020 -0.029 0.003 -0.008(0.014) (0.014) (0.012) (0.022)
4-Year Attendance 0.059*** 0.068*** 0.064*** 0.090***(0.007) (0.008) (0.012) (0.014)
Persistence2 Full Years in College 0.068*** 0.060*** 0.071** 0.070** s.e. (0.008) (0.007) (0.025) (0.025)
4 Full Years in College 0.042*** 0.034*** 0.042* 0.044(0.007) (0.007) (0.022) (0.024)
Years of College 0.243*** 0.198*** 0.239** 0.205*(0.027) (0.028) (0.083) (0.096)
GraduationAny Graduation 0.036*** 0.021*** 0.035* 0.032*** s.e. (0.005) (0.005) (0.016) (0.006)
4-Year Graduation 0.047*** 0.033*** 0.048** 0.045**(0.004) (0.004) (0.016) (0.013)
Number of Districts 68 44 8 6
71
Table 8: Effect Summary and Cost-Benefit Analysis Notes: Panels A and B summarize results reported elsewhere (see column 1 descriptions). The two Panel DD estimates for 2014 are shown in brackets because they are extrapolations of the panel effect (see text discussion). The estimate for “Years of Education” combines the effects on high school graduation and college. Panel C provides the cost-benefit analysis. The present discounted value of costs comes from multiplying the dosage by the annual additional costs and applying the discount rate ((δ=0.035). The benefits come from adding the returns to quality (test scores from Panel A) and the returns to quantity (years of education from Panel B), adjusting for productivity growth and discounting. The ratio of these easily exceeds unity, indcating that the reforms pass a cost-benefit test.
Effect Category 2007 2008 2009 2011/12 2014Panel A: Test Score Summary Total NOLA improvement (rel. to 2005) (Table 1) 0.10 0.15 0.25 0.35 0.42 Threats to Identification Population Change Pre-Kartina Scores of Returnees 0.10 0.06 0.04 -0.06 Census/USDOE Simulation 0.01 Interim Schools/Trauma (Pane et al. 2008) -0.06 Effects from DD Panel DD - Low (Figure 3) -0.03 0.05 0.11 {0.28} {0.28} Panel DD - High (Figure 3) {0.39} Pooled DD (Table 4) 0.00 0.11 0.23 0.40 0.40
Panel B: Years of Education Summary Total NOLA improvement (rel. to 2005) HS graduation (Table 1; NOLA only) 0.18 College attend. (Table 1; NOLA only) 0.10 Effects from DD HS graduation (Figure 5A) 0.07 0.11 0.06 College attend. (Figure 5B) 0.10 0.10 0.08 0.15 0.13 Effects on years of education 0.42Panel C: Benefit-Cost Analysis Key Parameters Dosage (post-reform rears in NOLA) 4.50 Annual cost/pupil (from DD) $1,300 Discount rate 0.035 Productivity growth rate 0.010 Return to year of educ (from prior res.) 0.04-0.08 Return to 1 s.d. inc. in test scores (from prior res.) 0.05-0.08
NOLA Adj. Benefit-Cost Ratio (from Panels A and B) 5.66-9.96 BCR: Perry Preschool 7.1-12.2 BCR: Class Size (STAR) 2.83