Do Accountability and Voucher Threats Improve Low-Performing Schools?

David N. Figlio and Cecilia Elena RouseNBER Working Paper No. 11597August 2005JEL No. I20, I21

Aim of the paper

Study the effects of the threat of vouchers and school stigma in Florida on the performance of low-performing schools

General Framework 1/3

• Florida’s program utilizes stigma by grading schools “A” (excellent) through “F” (failing)

• Students in “F” schools are eligible for taxpayer-founded vouchers (Opportunity Scholarship Program) cheques gave to low income families in order to help them in the school choice. The families can choose among private or higher-rated schools

• OSP has two rationales: – Fairness to reduce social stratification – more competition into the provision of education (in fact, based

on the economic theory, an increased competition should force schools to improve)

General Framework 2/3• 1996 Florida began rating schools based on their

aggregate norm –referenced test performance introduction of the Critically Low Performing Schools List (four school rating groups)

• 1998 introduction of the new Florida Comprehensive Assessment Test (FCAT) based on the SSS in grades 4 reading ,5 math, 8 and10 both in reading and math

• 1999 introduction of the A+Plan. It divided school in five rating groups and it establish private school vouchers assigned in event of chronic low performance. Two types of tests: nationally norm referenced test FCAT-NRT and the curriculum based assessmentFCAT-SSS

the two tests assess different set of skills. The FCAT-SSS tests a narrower set of skills but in greater depth than do the broader nationally NRT

General Framework: A + Plan 3/3

• In the early years of the A+ Plan, schools were identified as low performing:

• if fewer than 60% attain level 2 in reading, fewer than 60% attain level 2 in math, and fewer than 50% attain level 3 in writing.

• In addition, schools that improved by more than one grade level or retained an “A” were awarded an additional $90-$100 per student.

Evidence of test score improvements

0.3

Improvements in students learning 1/3

• Real gains or statistical artifact?• If real gains due to : voucher threats or

other elements of the accountability system such as grading stigma?

There are several alternative hypotheses that might explain real gains:

1) mean-reverting measurement error2) changing student characteristics3) opportunistic teachers’ behaviours

Improvements in students learning 2/3

1) Mean – reverting measurement error• The gains obtained by “F” rated-schools were due to mean

reversion. Indeed the gains obtained by these low rated schools in the following year had been larger than the average gains. This could be due to the “floor effect” and/or “ceiling effect”: Schools that scored highly on the test in one year are disproportionately likely to experience lower than average gains the following year

• The school rating is based on test’s result of only one year, thereby it’s possible that many of the F-rated schools had only a transitory low grade due to low test scores. Their score would have increased in subsequent years even in absence of the A+ Plan

higher score obtained

in one year

lower than average gains in the following year

Improvements in students learning 3/3

2) Changing students’characteristics districts attempted to redraw school attendance area boundaries in order to improve the student characteristics of low rated-schools.

3) Opportunistic schools’ behaviours schools only focus on high-stakes grades test, teaching to the students the test’s matter.

Empirical framework 1/2

T ist is student i’s test score in school s in year tF is dummy variable indicating whether or not the school attended by student i received a failing grade of “F” in 1999. YEARt is a vector of year effects

POSTt dummy variable indicating if the year is after the implementation of the A+ Plan (in these data it is equal to one if the year is 2000 and zero otherwise)

GRADEist is a vector of dummy variables indicating the student’s grade in school

s school-level time trends predicted from the pre-1999 data

s is a vector of school-fixed effects

ist is a normally distributed error term

key coefficient of interest. It measures the change in a student’s test score resulting from her school having received an “F.”

Empirical framework 2/2

• Utilizing a difference-in-differences framework • The Tist –Tist-1 is the outcome observed for two groups of test

score in school s for two time periods. One of the groups is exposed to a treatment (A+Plan) in the second period but not in the first period. The second group is not exposed to the treatment during either period.

• The dummy variable POSTt captures possible differences between the treatment and control groups prior to the policy change. F is captures aggregate factors that would cause changes in outcome even in the absence of a policy change. multiplies the interaction term, F is and POSTt

• This particular specification of the education production function allows us to control for the student’s prior academic achievement (by controlling for Tist-1) and this allows to have the right estimates of the effect of the A+ Plan.

Data• Student-level data from a set of participating school districts from

1995-2000.• These data include scores on:

– FCAT-SSS Florida Comprehensive Assessment Sunshine State Standards

– Norm-referenced NRT – Basic student demographic attributes, including information on

student race, ethnicity, poverty status, limited English proficiency status, and disability status.

• Data on third, fourth, and fifth grade NRT scores and FCAT-SSS scores for grade 4 in reading and grade 5 in math. These data include over 1500 schools

• Focus on elementary grades (from 1 grade to 6 grade) because they comprised the vast majority of “F” schools

• Dependent variable is the normal curve equivalent of the test scores• The mean is 50, the individual-level standard deviation is 21.06

Dependent variable

Dummy variables

0.07

Ss at 10%

Ability

•Study changes in academic performance that resulted from the earlier policy of placing schools on a critically low performing list

Conclusion1/2• Overall, they find that students showed significant improvement on

the high-stakes exam (the FCAT-SSS) in both reading and math. • However, once they examine the effect of the accountability system

on a nationally norm referenced test and control for student characteristics (by including the lagged test score), they find that there was no large-scale relative improvement in average NRT reading scores due to the A+ Plan among students in the lowest performing schools.

• There were modest relative improvement in average NRT math scores, although this improvement appears largely concentrated in the high stakes grade.

• And while they find some evidence that sanctioned schools put additional resources into low-performing students in math, more generally they find little evidence that these schools treated students differentially than low-performing schools that did not receive an F.

Conclusion 2/2

• A+plan is seen from Florida as a good tool to improve the low performing schools in this way all children can have a good quality of education.

• Florida with the institution of this provision is sure that the improvement of the low performing schools conducts to an improvement in children’ learning. This would be logical

• But as the authors show the low performing schools in order to avoid the stigma to receive an F (that is the main reason why the school improve and not for the voucher threat) they behave in an opportunistic manner for example focusing on the low performing children and leave aside the other subgroups of students, teaching the high-stake test

contradiction with the two rationales of the OSP: fairness and competition