how are california’s charter schools performing? · 2014-09-24 · charters often differ in grade...

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.

R E P O R TM A Y 2 0 0 5

EdSource® is a not-for-profit 501(c)(3) organization established in California in 1977.

Independent and impartial, EdSource strives to advance the common good by developing and widely distributing trustworthy, usefulinformation that clarifies complex K–12 education issues and promotes thoughtful decisions about California’s public school system.

ith the Charter Schools Actof 1992, California began

allowing for the creation of anew type of public school. The school’sorganizers would operate more or lessindependently under a charter—orperformance agreement—negotiated witha school district or county office of educa-tion. Groups could start new schools fromscratch or convert existing schools to char-ter status. A charter school would besubject to fewer restrictions but be moreaccountable for student achievement. Theschools were to meet student performancegoals as specified in their charter.

Today about 500 charter schoolsoperate throughout California. Somehave been around for more than 10 years,while others received their charters onlyrecently. While most operate independ-ently, a few are under the auspices oflarger organizations, some of which arefor profit.

In recent years, a number of studieshave attempted to assess the effectivenessof the charter experiment, particularly itsimpact on students’ academic achieve-ment. These studies come to verydifferent conclusions depending on theschools, the timeframe, and the perform-ance measures they analyze. Charter

advocates and opponents alike watchthese studies closely, with both sideslooking for findings to bolster their case.Advocates—who see charter schools as acatalyst for broad reform—want to showthat school-level decision making andfreedom from most Education Coderequirements allow charters to excel.Detractors, on the other hand, seeksupport for their argument that site-levelcontrol, combined with freedom fromthe regulations and formal oversight of apublic bureaucracy, do not necessarilylead to better instruction and studentperformance and may even hurt educa-tional quality.

This report begins with a discussion ofwhy performance comparisons betweencharter and noncharter schools can becomplicated. It also summarizes the mostrecent data from California’s assessmentand accountability systems, comparingtypes of charter schools and looking atthem within the context of the largersystem. An overview of the often contra-dictory findings from prominent researchstudies illustrate how different expertsaddress the comparison of academicperformance among charter schoolstudents versus their peers in the regularsystem. This report also adds to that ongo-

ing discussion with a new EdSource analy-sis that approaches the issue somewhatdifferently. Taken together, the informa-tion in this report raises questionsCalifornians should be asking about thestate’s charter schools and their perform-ance, and it illuminates some reasons whyclear answers are so elusive.

In California, the academic progress ofstudents in individual charter schools can beexamined using the same test-based meas-ures used for noncharters, howeverimperfect those may be. According to statestatute, charter schools are required toparticipate in the state’s standardized testingprograms. They are also evaluated using thesame accountability systems the state andfederal governments require of other publicschools. (See the box on page 2.) Finding

How Are California’s Charter Schools Performing?

W

On page 19 you will find a summary of themajor conclusions included in this reportregarding charter school performance in California.

Key Findings

Exploring performance among charterschools is a challenge

testing and accountability scores for indi-vidual schools is thus an easy matter.

Generalizing about school perform-ance and comparing groups of schoolspresents a much more complex challenge,particularly if one is interested incomparing charters to other publicschools. This is because of the constantturnover and increase in the number ofschools and because the universe of char-ter schools is—in a variety of ways—quitedifferent from noncharters. Within thatuniverse, charter schools are also markedlydifferent from each other. In addition, forvarious reasons they are substantially lesslikely to have full data sets.

Constant change makes comparisons difficultWhile some schools open and close eachyear in California, the more than 9,000public schools are—as a group—rela-tively stable. The total number of schoolschanges little from year to year, and themix of grade levels is on the whole fairly

constant and predictable, as is the numberof students these schools serve.

Compare that to the charter schoolsegment. Between 1993–94 and2004–05, the number of charterschools went from 31 to 518. Duringthat same time, another 40 schools hadtheir charters revoked and 95 othersclosed, according to California Depart-ment of Education (CDE) records. (SeeFigure 1.)

The constantly growing number ofcharter schools makes generalizationsabout charter school improvement overtime difficult. A simple calculation—suchas the percentage of high schools that mettheir state-determined growth targets onthe Academic Performance Index (API)over five years—is only possible for a verysmall number of schools. (For an explana-tion of how the API works, see the box onpage 4.) Presumably, as the number oflong-lived charter schools increases,assessments of their improvement overtime will become more meaningful.

Charters often differ in grade configurationsCharters are also more varied in theirgrade-level configurations. Most Cali-fornia charter schools, like most regularpublic schools, are configured as elemen-tary, middle, or high schools; but asubstantial portion depart from thatmodel. In 2003–04, a full 13% servedall of grades kindergarten through 12,and another 15% could be categorizedas nontraditional because they eitheroffered only one grade, or they coveredan unusual grade span that crossed overthe traditional categories (e.g., K–9 or1–11). Unconventional grade configu-rations are more common amongcharters, perhaps in part because somestart-up charters add a grade each year asthey become more established.

For calculating API scores and rank-ing charter schools along with others, thestate groups all these various configura-tions into three categories—elementary,

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.

2 ● How Are California’s Charter Schools Performing? ● May 2005

Inside This ReportExploring performance among charter schools is a challenge ....................1

A comparison based on types of charters ....................................6

Comparisons by grade levelElementary schools..............................9Middle schools ................................11High schools ......................................13

Research findings conflict ........................15

EdSource tries another approach ............16

EdSource finds noteworthy patterns ........19

Summaries of national and California studies ..............................21

This report was written by:Brian EdwardsMary Perry

Research by:Brian EdwardsNoli Brazil

EdSource thanks the wide group of experts—of variedperspectives and professional backgrounds—whoprovided review and comment on this report.

E D S O U R C E R E P O R T

State tests and accountability measures that apply to charter schools

Charter schools are required to administer the following tests to their students and report theresults to the state:

● The Standardized Testing and Reporting (STAR) program, which includes the California Standards Tests(CSTs) in grades 2–11 plus the nationally referenced CAT/6 survey test (which replaced the SAT-9 in2003). Starting in 2005, only 3rd and 7th graders will take the CAT/6.

● The California High School Exit Exam (CAHSEE).

● The California English Language Development Test (CELDT) for English learners.

In California, parents have a right to opt out of testing for their child.

For purposes of school accountability, charter schools are ranked and/or evaluated based on thefollowing:

● The Academic Performance Index (API), which incorporates the results of STAR and CAHSEE to gener-ate Base and Growth API scores.

● Adequate Yearly Progress (AYP), per federal requirements, based on the percent of students proficientin math and English on the CSTs, or the CAHSEE for high school students, plus additional indicators.


May 2005 ● How Are California’s Charter Schools Performing? ● 3

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.

middle, and high schools. (See the box onpage 5.) While this approach makes iteasier to compare large groups of schools,it may also create some misleadingcomparisons when school configurationsdo not fit the typical pattern.

Charter schools’ student demographicsappear somewhat differentComparing charters as a group to allnoncharters can also be misleadingbecause of student characteristics.Charter schools as a whole enroll somewhat different students thannoncharters. Charter schools are lesslikely to serve Hispanic students,English learners, or students from low-income families. Elementary schoolsdiffer the most in this regard, while highschools differ the least. (It is importantto note that the percentage of low-income students in charter schools maybe undercounted because it is oftenbased on eligibility for free/reduced-priced meal programs, which manycharter schools do not run.)

When it comes to total school enroll-ments, however, the difference in sizebetween charter and noncharter schools ismost dramatic at the high school level.Elementary schools are more similar.

Some research indicates that smallschools are more effective, at least withsome students. That represents anothercomplicating factor because it is unclearto what extent school size, as opposed tocharter status, may account for somedifferences in academic performance.

Later in this report, analyses ofelementary (page 9), middle (page 11),and high schools (page 13) provide detailedcomparisons of student demographics,school size, and teacher characteristics.

Some authorities cite a less tangibledifference between students in nonchar-ter versus charter schools. Demographicdata cannot reveal the qualitative differ-ences in student attitude, motivation, orfrustration with traditional school struc-tures that might differentiate charterstudents. Those differences could make amaterial difference in their performance.

Status and operations differentiate charters from each other As stated earlier, the community of char-ter schools is quite diverse. Charters canbe categorized in ways that help describehow they differ and illuminate someimportant distinctions vis-a-vis tradi-tional public schools. This reportdistinguishes among charters according tohow they were established, how long theyhave been operating as a charter school,and whether they offer their instructionprimarily in classroom settings.

The first distinction is between conver-sion and start-up charters. Conversioncharters were once regular public schools.Start-ups, on the other hand, began as char-ter schools. Presumably a conversion schooldecided to break away from state and/ordistrict office policies on curriculum,finance, the academic calendar, or othermajor issues. This new charter status wasoverlaid onto an existing school culture andongoing staff and community relation-ships. By contrast, a start-up oftenrepresents one person’s or group’s brain-child that exists independent of a districtcontext. It presumably begins as a newenterprise in which roles, relationships, andprocesses are all created. (These generaliza-tions may not fit every school officiallycategorized as a conversion or start-up.)

Start-ups, which make up about 70%of all charter schools, vary more dramat-ically from noncharter public schoolsthan conversions do. For example, themedian noncharter school has 33% whitestudents, the median conversion schoolhas 45%, and the median start-up 55%.The relationships are similar for otherstudent and school characteristics, withthe charter schools generally havingsmaller student bodies and lower percent-ages of English learners, Hispanicstudents, and experienced and fullycredentialed teachers.

Second, there are established and newcharters. It is reasonable to expect that acharter’s performance in its beginning

figure 1

3160

83110 125

159

245

305

363

418454

518**

0

100

200

300

400

500

600

93–94 94–95 95–96 96–97 97–98 98–99 99–00 00–01 01–02 02–03 03–04 04–05Year

Enrollment†10,761 23,228 30,977 39,624 48,101 68,685 99,048 113,956 132,643 158,942 167,764 181,734

Num

ber

of C

hart

er S

choo

ls

Data: California Department of Education (CDE) EdSource 5/05

The number of charter schools* has increased dramatically in California

*To be counted among the schools open in a given year, a school must operate at a minimum from November 1 through February.**None of the achievement data in this report correspond to the 2004–05 school year.† Enrollment data is not available for a few schools each year.



stage is not necessarily indicative of itsfull potential. For this report, EdSourcesomewhat arbitrarily defines conversioncharters as “established” if they have been

open for a full school year. Start-ups aredeemed “established” if they have beenopen for two full years. Conversions andstart-ups are treated differently because it

is assumed that conversions, though theyexperience changes in governance andfinance when they convert, generally stillenjoy some continuity in students, staff,

How the Academic Performance Index (API) works

Since 1999, California has evaluated school performance using its Acade-mic Performance Index (API) system. The API score is a one-numbersummary of a school’s scores on various tests. The school also receives anAPI for each “numerically significant” subgroup of pupils categorized byethnicity and poverty.

The API system is organized into two-year cycles. Some time between Januaryand March, each school receives a “Base API” score, based on its students’performance on tests given in the spring of the prior year. It also receivesgrowth targets for improvement. Students take state tests again in the latespring, and those scores are used to calculate a “Growth API” that schoolsreceive in the fall, completing the two-year cycle.

Student scores on the state’s standardized tests are used to create theindex. The combination of tests used in the index has evolved along with thestate testing system. Beginning with the 2002 Base APIs, the California stan-dards tests—which assess students’ mastery of the state’s academic contentstandards—have played a dominant role in the API. Student performance onthese tests is reported as meeting one of five performance levels: far belowbasic, below basic, basic, proficient, and advanced.

An API score is based on the distribution of scores among the five performancelevels, with various subjects and tests receiving differing weights in that calcu-lation. API scores can range from 200 to 1000. If all students score in the topperformance band on all subtests, the API score will be 1000. The state set800 as the target score for all schools.

The elements that make up the API have changed almost every year, which isone of several reasons why some criticize it. However, in each API cycle, theGrowth API is calculated using the same elements as the Base API, ensuringthat the Base/Growth results are comparable. For high schools, the CaliforniaHigh School Exit Exam, which assesses mastery of standards in Englishlanguage arts and math, was incorporated beginning with the 2002 Base API.

Base API scores are used to compare or rank schools and to set growthtargets. Schools are ranked with schools of the same type—elementary,middle, and high.* First, they are ranked against all their counterparts in thestate, and next they are ranked against the 100 most similar schools, basedlargely on student demographics. For both types of rankings, they are clusteredinto 10 groups of roughly equal size known as “deciles.” The bottom 10% ofeach school type belongs to Decile 1, the second lowest 10% to Decile 2, andso on. A shorthand has developed around these rankings: a “7/5” school, for

example, would be a school that received a state decile ranking of 7 and a“similar schools” ranking of 5.

To achieve growth in its API score, a school (or subgroup) needs to decreasethe percentage of pupils who score in the lower performance bands andincrease the percentage who score in the higher bands. The API formularewards growth from the bottom of the performance distribution more heavily.This creates an incentive for a school to work with its lowest-performingstudents.

For a school with a Base API score below 800, the annual growth target is 5%of the difference between 800 and its Base score. For example, a school witha Base score of 500 would have a growth target of 15, which is 5% of the differ-ence between 500 and 800. Schools with Base scores of 800 and above areexpected to maintain scores at that level. Subgroup growth targets are gener-ally 80% of the school’s target.

The Growth API is one way to measure improvement from year to year. TheAPI does not reflect individual students’ test-score growth. It summarizes aschool’s performance in one year (Base) and compares it to the school’sperformance in the following year (Growth), but the groups of students at eachgrade level are different. For example, in a K–5 elementary school, the BaseAPI includes scores of fifth graders who would no longer attend the schoolduring the Growth API year. This diverges from the “gold standard” for assess-ing schools’ academic performance, which would be to follow the test-scoregrowth of individual students over time.

Before 2002, some schools were not assigned APIs. Originally, California didnot assign API scores to every school every year, and many charter schoolswere among those excluded. When the system was first created, a school didnot receive an API if it had fewer than 100 test-takers or if its student body waspredominantly at-risk students (e.g., continuation high schools, court schools,or other alternative schools). These schools instead qualified for the “Alterna-tive Schools Accountability Model.” Further, schools with irregularities in thetest administration, or that experienced a significant demographic changefrom the previous year, did not get an API score in the second year.

(For a further explanation of the various components used to calculate the API, see www.edsource.org/edu_acc_api.cfm.)

*“Small” schools—those with fewer than 100 test-takers—technically are a part of the AlternativeAccountability system.Their API scores come with an asterisk to indicate that the scores may not beas accurate an indicator of the schools’ “true” performance as it is for larger schools. Small schoolsdo not affect the rankings, but they are given the ranking associated with their API scores.

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.

and facilities. In 2003–04, about 65% ofcharter schools would have been consid-ered “established” under EdSource’sdefinition. Established and new schoolswere similar in terms of student ethnicityand parent education levels, but the newerschools tended to have less experiencedand credentialed teaching staffs and to besmaller. For example, the mediannoncharter had 620 students, the medianestablished charter had 297, and themedian new charter had 169.

Finally, there are classroom-based andnonclassroom-based charters. In 2001 theLegislature established a definition of anonclassroom-based charter school as onethat does not require its pupils to be onsiteunder the direct supervision of a teacherfor at least 80% of the instructionaltime—four school days out of five, forexample. Schools that provide a substan-tial portion of their instruction throughdistance learning or independent studygenerally fit that definition. Charters witha traditional classroom system, whichmake up almost 70% of all charterschools, are considered classroom-based.

Charters deemed nonclassroom-based were very similar to theirclassroom-based counterparts in totalenrollments but had relatively largepercentages of white students and lowproportions of English learners. Theyalso had a high rate of student “mobility”(median of 55% versus 15% in regularpublic schools), indicated by a relativelyhigh percentage of students in their firstyear at the school (lowest grade excluded).There are many plausible explanations forthis. For example, it could indicate that adifferent type of student is attracted tothese schools. Perhaps students havingtrouble in conventional school settingsuse nonclassroom-based schools as atransition between more traditionalschools. Such differences in the type ofstudents served, which are not readilymeasured and reported to the state,should be kept in mind when comparing

the achievement of these schools to othercharters and mainstream noncharters.

Variations in funding and other policiescomplicate comparisonsCharter schools generally receive moremoney per pupil than districts receive ingeneral purpose funds (revenue limitfunding) but less than the total mostdistricts receive when special purposefunds are considered. But wide variationsin funding per district leave even thissimple statement open to debate.

It is clear that many charters foregoextra funding they could receive becausethey want to avoid the administrativerequirements of some state and federalprograms. In a July 2003 evaluation ofCalifornia’s charter schools, RANDfound that charters were much lesslikely than comparable noncharters toparticipate in eight relatively largeprograms, including the federal Title Iprogram for low-income students, K–3Class Size Reduction, and Supplemen-tal Instruction.

Many charter schools, especiallystart-ups, also spend at least a portion of

their operating funds to cover the cost offacilities, a necessity few regular publicschools face. Although charter schoolshave in recent years been given moreavenues for securing facilities—or fund-ing for them—some are unable to takeadvantage of those options. For example,some of these funding avenues requirethat the school or at least its operatorshave a successful track record or that theyserve primarily low-income students.

On the other side of the equation,some charter schools enjoy substantialamounts of support from private sources.These funds usually come with fewer ofthe requirements attached to governmentfunds. RAND reported that charterschools on average received $433 perstudent from these sources while compa-rable regular schools received $83.

Within the charter community,schools differ in other important ways.Charters can choose to operate as finan-cially independent entities, getting theirfunds straight from the state. Or theycan be financially dependent, with theirchartering school district passing statefunds through to them and helping



© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.

How schools are classified for the Academic Performance Index (API)

For API purposes, most California public schools are ranked among schools of the same type—elemen-tary, middle, and high. However, schools have a great variety of grade spans, which necessitates rules forclassifying schools into those three categories.

The California Department of Education (CDE) generally classifies schools based on the largest numberof grades a school has in “core” grade spans of K–5, 7–8, and 9–12. (Sixth grade is left out of the coregrade spans because some view it as an elementary grade while others see it as a middle grade.) A K–8school, for example, would be considered an elementary school because it has six grades in the coreelementary grades versus only two grades in the core middle grades. For schools with an equal numberof grades in two or three core spans, the CDE assigns schools to a type based on the largest enrollmentin a core grade span. This means that a hypothetical 4–10 school with 130 students in grades 4–5, 190students in grades 7–8, and 120 students in grades 9–10 would be deemed a middle school for purposesof the API because the middle grade span has the most students. Schools are assigned the same type forboth years in a two-year API Base/Growth cycle. From one cycle to the next, however, their type maychange based on the students that they serve.

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



them with some management, adminis-trative, and/or operational functions.

Charter schools also vary in theiremployment agreements with teachers andother staff. At one end of the spectrumare those whose teachers are representedby the same union as their counterpartsthroughout their chartering district. Atthe other end are schools where teacherschoose to forego union representationwith its employment protections andcollective bargaining rights.

(For more details on the differences infunding, see EdSource’s June 2004report, Charter Schools in California: AnExperiment Coming of Age.)

A substantial portion of charter schools donot have API dataA final factor that makes assessments ofcharter school achievement complex isthe substantial portion of schools with-out API data. Consider the charterschool universe in 2003–04. That year,454 charter schools were open, but 82did not have 2004 Base API scores. Of

those 82 schools: 12 were not in theCalifornia Department of Education’sAPI file; 23 were in the alternativeschools accountability model, whichdoes not use API scores to monitorperformance; and 47 were listed in theAPI file but did not have scores for vari-ous reasons (explained below). Those 47schools constituted 11% of the charterschools listed.

In contrast, of the state’s 8,766noncharter public schools, 7,813 were inthe API system and 3% lacked scores.Another 953 schools were in the alterna-tive schools accountability model (ASAM).

Among charter schools, 353 werelisted in the 2004 Growth API file,which reports growth between the 2003Base and 2004 Growth APIs. Of those,80 schools (or 23%) lacked scores. Incontrast, 6% of noncharters in the 2004Growth API file did not have data. Figure2 illustrates the wide variations in APIdata available based on school types.Note that almost half of nonclassroom-based schools are missing data.

The California Department of Educa-tion lists a variety of reasons why schoolsdo not receive a score, but there are twomain reasons that charter schools are miss-ing data. One is that the schools did not testa sufficiently large group of students tohave statistical meaning and preserve indi-vidual students’ privacy. The other is thatthe students tested were not sufficientlyrepresentative of the school as a whole,often due to a large percentage of parents’excusing their children from STAR tests,upon which API scores are largely based.

In addition, new, start-up charterschools do not receive their first APIscore until much of their second year ofoperation has passed because they do notparticipate in the state testing systemuntil the end of their first year, and ittakes about nine months for Base scoresto come out. Effectively then, it is notuntil a charter’s third year that its firstGrowth API score, and thus its successunder the state’s accountability system,has been reported. Of course, this is truefor new noncharter schools as well.

A comparison based on types of charters

When test data is used as the measure ofschool performance, it can be done severalways. The most accurate approach is tocompare the performance of a particulargroup of students over time. In Californiasuch a comparison cannot be done at thestate level because individual student datais not available. California’s API systemmakes it possible, however, to do threeother types of comparisons.

Absolute comparisons ignore differ-ences in student characteristics as Californiadoes with its API rankings. These arereleased each year along with schools’ BaseAPI scores. The state system only makesabsolute comparisons among all schoolsthat serve similarly-aged students, i.e.,elementary, middle, and high school.

Similar school comparisons look atschools whose students share similar char-acteristics. This approach acknowledgesthat school performance is highly corre-lated to students’ backgrounds, mostnotably the education level of their parents.Schools in California receive a “similarschools” ranking that serves this purpose.

Comparisons over time examine asingle school’s improvement in perform-ance. In California this involves acomparison of Base and Growth APIs in atwo-year cycle. The state assigns eachschool a growth target that is a benchmarkfor its success and reports on whether eachschool met its target. (This kind of meas-ure is less meaningful when a school ishigh performing from the start).

The following sections use thesethree types of comparisons to look atcharter school performance. They exam-ine charter school performance by type,by grade level, and finally based on thestate’s newly established performancecriteria for charter renewal. The result isa rich tapestry of information that notonly illuminates charter school perform-ance, but also makes clear how manyquestions remain.

In California’s public schools, thestate-adopted academic content stan-dards represent official expectations forthe skills and knowledge students willmaster in school. The California stan-dards tests (CSTs) are used to measurestudents’ mastery of the standards.

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



Students’ scores on the CSTs and othertests are summarized into AcademicPerformance Index (API) scores for theschool as a whole and subgroups ofstudents. (For a more detailed explana-tion of the API and its limitations, seethe box on page 4.)

Not everyone supports the state’sacademic content standards or thinks thatthe CSTs accurately measure students’mastery of those standards. Some see theAPI as an unreliable and arbitrary meas-ure of schools’ and subgroups’achievement. But meeting API growthtargets represents the state’s goal for allschools, including charters. As such, it isas fair and sound a measure of charterstudents’ achievement as it is for nonchar-ters. However, all charter schools are notalike, and the ways they differ seem tohave some relationship to their ability tosucceed based on the API measure.

Missing data clouds comparisons ofimprovement by charter typeFigure 2 shows the percentage of charterschools in the various categories thatimproved over time, meeting both school-wide and subgroup API growth targets inthe 2003 Base/2004 Growth cycle. Italso shows the number of schools heldaccountable in the API system and thenumber with actual API scores. Acomparison of the two numbers showsthat a substantial portion of some typesof charter schools did not have API data,making it more difficult to compareperformance in a meaningful way.

Based on these data, conversionschools narrowly outperformed start-upschools, though it must be noted that thenumber of start-ups missing data wasclose to the total number of conversionschools. Classroom-based schools didsubstantially better in meeting growthtargets than nonclassroom- based schoolsthat had growth data, but nearly half ofthe nonclassroom-based schools lackedgrowth scores. Somewhat surprisingly,

figure 2 Percent of schools meeting 2004 API Growth Targets

School Type Number of Number of Percent of Schools with(Pie charts: Percent 2003-04 Schools in Schools with API Data Meeting 2004with/without API data) Enrollment API System API Data API Growth Targets

Yes No

All Noncharter Schools 5,931,819 7,671 7,237 48% 52%

All Charter Schools 143,264 353 273 60% 40%

Charter Schools By Type

Conversion 48,508 79 75 61% 39%

Start-up 94,756 274 198 59% 41%

Established 106,116 229 188 59% 41%

New 27,148 124 85 61% 39%

Classroom-based 94,288 252 218 64% 36%

Nonclassroom-based 48,976 101 55 44% 56%

5%95%

28%72%

6%94%

23%77%

18%82%

31%69%

13%87%

46%54%


Schools with API data Schools without API data

new schools beat out established charters by a small margin. (To be consid-ered “established,” a conversion must have been open by 2001–02, and a start-up must have been open by 2000–01.)

Classroom-based schools consistentlyperform betterComparing the performance of charterschools by category is likely to interestpolicymakers who want to foster thecreation of successful charter schools anddiscourage the less successful types.However, policy decisions based just ondata for these broad categories couldoverreach. For example, conversionschools are outperforming start-ups, butare the conversion schools relying mostlyon classroom-based instruction whilestart-ups are mostly nonclassroom-based?Cross-tabulating the various categoriesmakes it possible to see beneath the

surface comparison to address this typeof question.

In Figure 3, schools are first catego-rized as established or new, thensubdivided into classroom-based or notand—within that—as conversions orstart-ups. The table shows the percentageof each subcategory that improved overtime, meeting its growth targets in 2004.The number of schools in each groupshould also be considered when compar-ing performance.

From this analysis, it appears that themost successful charters in meeting 2004API growth targets were newer, classroom-based, conversion schools. But while allschools in that group met their targets, thegroup included only five schools.

Classroom-based charters in othersubcategories also did well, however. Thethree runner-up subcategories all offeredclassroom-based instruction, and all didbetter than noncharters, 48% of which

met their growth targets. The leastsuccessful charters were established,nonclassroom-based, conversion schools.Only 40% met their targets, but three of13 were missing data.

Given that classroom-based charterstended to perform well—and that verysimilar percentages of established andnew charter schools offered classroom-based instruction—the similarity inperformance between established andnew charters does not seem as surpris-ing as it otherwise might. Similarly, thestronger performance of conversionschools as compared to start-ups maybe partially explained by the fact thatconversions were more likely than start-ups to offer classroom-basedinstruction (87% of schools with APIscores versus 77%).

Another factor affecting perform-ance is the grade levels that a schoolserves. For example, elementary schoolshave typically been more successful onthe API, both in terms of overall scoresand in the percentage of schools meetinggrowth targets. Therefore, comparisonsof different types of charters, andbetween charters and noncharters, musttake the grades served into considera-tion. The following section examineshow elementary, middle, and highschool charters, as defined in the APIsystem, have performed. This looks atimprovement over time, absolutecomparisons, and similar schoolscomparisons. Before turning toperformance per se, it is helpful tounderstand how charter schools at eachgrade level compare to noncharters.Thus each of the grade-level sectionsthat follows provides that data. Theperformance of various subcategories ofcharters within those grade-level types isalso discussed.

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



figure 3 Percent of schools meeting 2004 API growth targets,further disaggregated

Established Classroom-based vs. Conversion Number of Percent with vs. New Nonclassroom-based vs. Start-up Schools Growth Data

Meeting Growth Target

In API With Yes NoSystem Growth

Data

Noncharter Schools 7,671 7,237 48% 52%Charter Schools 353 273 60% 40%

Conversion 61 60 62% 38%Start-up 100 90 64% 36%Conversion 13 10 40% 60%Start-up 55 28 43% 57%Conversion 5 5 100% 0%Start-up 86 63 62% 38%Conversion 0 0 N/A N/AStart-up 33 17 47% 53%


EstablishedCharters

New Charters

Classroom-based

Nonclassroom-based

Classroom-based

Nonclassroom-based

The data indicate that elementary char-ter school students are less likely tocome from the backgrounds mostcorrelated with low student achieve-ment than their nonchartercounterparts. The “typical” (median)elementary charter has about half asmany English learners and studentseligible for free/reduced-price meals aswell as a third as many students whoseparents do not have a high schooldiploma. These charter schools are alsoabout 50% smaller but have morestudents new to the school. They alsohave a much greater percentage ofteachers who are less than fully creden-tialed and less experienced.

Charter elementary schools surpassnoncharters meeting growth targets for first time in 2004As with charter schools generally, thenumber of charter elementary schoolshas grown substantially, making the taskof looking at their progress over timesomewhat complex. In addressing thequestion of whether they are meeting thestate’s expectations for improvement, thefollowing statistics are important to keepin mind:● 202 charter elementary schools were in

the API system of accountability inthe 2003 Base/2004 Growth cycle.

● 175 schools—87%—have the datanecessary to determine whether theymet their growth targets in the mostrecent API cycle, compared to just 49schools in the 1999/2000 cycle.

● Of those 175 schools, 58 were conver-sions and 117 were start-ups; 145were classroom-based and 30 werenonclassroom-based.

● These 175 schools had 70,198students.Figure 4 compares elementary charter

schools to noncharters. The data show

their ability to meet the state’s expecta-tions for improvement over time based onAPI growth targets for the school as awhole and for all significant studentsubgroups. (See the box on page 4 forfurther explanation).

In the last two cycles—since the statestandards have played a predominant rolein the index—elementary charterperformance has improved. With the2004 Growth API, charters exceedednoncharters on this measure for the first



© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.

Key Characteristics of California Elementary Schools† in 2003–04

Charter Noncharter Schools (n=231) Schools (n=5,427)

Student Characteristics Median Median(Statewide Average) Range for Middle Half Range for Middle Half

English Learners (29%) 11%** 24%*3%–50% 9%–46%

Hispanic (43%) 17% 39%8%–52% 16%–70%

White (36%) 54% 30%8%–76% 8%–60%

African American (8%) 4% 3%1%–11% 1%–9%

Asian (8%) 1% 3%0%–4% 1%–9%

Free/Reduced Price Meals (52%) 26%* 53%*2%–63% 21%–78%

Parents Not High School Grads (21%) 4%* 15%*1%–18% 4%–32%

One+ Parent a College Grad (18%) 26%* 16%*14%–36% 8%–28%

Student Mobility (20%) 24%* 17%*14%–45% 12%–22%

School Characteristics

Enrollment (average size 566) 222 555123–422 402–729

Teachers Not Fully Credentialed (6%) 13% 3%0%–29% 0%–8%

Teachers in First Two Years (11%) 20% 8%10%–38% 3%–14%

Data: California Department of Education (CDE)–CBEDS EdSource 5/05†Includes only schools in the API system not those in the Alternative Schools Accountability Model.*Data missing for up to 20% of schools.

**Data missing for at least 20% of schools.

Statewide average: shows the percentage of elementary students or teachers in each category statewide.

Median: shows the percentage for a “typical” school—one at the 50th percentile for a given characteristic. Equal numbers of schoolshave higher and lower percentages.

Range for middle half: shows the percentages for the schools at the 25th and 75th percentiles. It provides a sense of how schoolswithin each group vary and how charters and noncharters differ as well.

Elementary charter school performance is improving

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



time. On average, these schools gainedeight points while noncharter elementaryschools gained four. This greater growthallowed 57% of elementary charters tomeet their targets, while 46% ofnoncharters hit their marks.

Based on absolute rankings, the elementary charter top performers were classroom-basedThe spring 2004 test results were alsoused to create new Base API scores. Basedon those scores, schools were rankedamong others serving the same gradelevels and, separately, among 100 schoolsfacing comparable challenges.

For absolute comparisons, schools areranked in 10 bands, called deciles, witheach decile representing 10% of schools.The figures below reflect the percentagesof charter elementary schools that placedin the highest, middle, and lowest range ofdeciles. It includes the 206 charters listedin the 2004 Base API that had a decilerank. To put the percentages in perspective,keep in mind that about 30% of schoolsoverall occupy the top three deciles, 40%fall in deciles 4–7, and the remaining 30%are in the lower three deciles.

● High-performing, deciles 8–10:26% (53 schools)

● Mid-performing, deciles 4–7: 39% (80 schools)

● Low-performing, deciles 1–3: 35% (73 schools)The charter schools in the top three

deciles were predominantly establishedschools with classroom-based programs.Of the 13 charter elementary schools inDecile 10, all but one fit that description.Similarly, the Decile 8 and Decile 9 char-ter schools—of which there were40—included 32 schools that were estab-lished and classroom-based.

Interestingly, the charter elementaryschools in the low-performing decilesrepresent a much wider range of configu-rations. Many classroom-based schoolsare among the lower deciles, but notdisproportionately. New start-ups areover-represented in the lower perform-ance bands, however.

Based on similar schools rankings, manyelementary charters are not doing wellThe state uses the “similar schools”rankings to indicate how well a school isdoing compared to the set of 100

schools most like it in terms of studentdemographics, teacher qualifications,and a few other factors. A school with asimilar schools rank of “1” scored in thebottom 10% of its comparable schools,and a school with a “10” scored in thetop 10%. When looking across all ofCalifornia’s public schools, each similarschools rank contains roughly 10% ofschools. Therefore, if a large group ofcharter schools is performing on parwith noncharters, roughly 30% willscore in deciles 8–10, 40% will have arank of 4–7, and the remaining 30% willscore in deciles 1–3. Because schoolswith fewer than 100 test scores do notreceive such rankings, the number ofcharter schools with such ranks in the2004 Base API is 142 (versus 206 withstatewide decile ranks). The elementarycharters with similar schools ranks scoredas follows:● High-performing compared to similar

schools, ranks 8–10:23% (33 schools)

● Mid-performing compared to similarschools, ranks 4–7: 23% (32 schools)

● Low-performing compared to similarschools, ranks 1–3: 54% (77 schools)Here again, conversion schools did

much better than start-ups, and class-room-based charters substantiallyoutperformed schools that were notclassroom-based. Those that did verywell relative to their comparisonschools, scoring an “8” or above, wereall classroom-based. Nonclassroom-based charter schools, on the otherhand, consistently performed worsethan the schools to which they werecompared. Of the 26 nonclassroom-based schools, 18 received a similarschool ranking of 1, and none scoredabove a 6.

figure 4 Elementary schools that made API growth targets

Two-year API Base/Growth Cycle

1999/00 2000/01 2001/02 2002/03 2003/04

Number of charters in API 56 73 96 130 202accountability system

Number of charters with growth data 49 69 89 120 175

Percent of charters with growth data 69% 49% 52% 75% 57%making growth targets

Percent of noncharters with growth 77% 64% 60% 83% 46%data making growth targets

● It is more meaningful to compare percentages within individual API cycles because of the growing number of charter elementaryschools and the changing index used to measure school performance (the API).

● The bottom two rows reflect the schools with API data in both years of a given two-year API cycle.


As with elementary schools, chartermiddle schools tended to serve studentsfrom more advantaged backgrounds butwith more uncredentialed and inexperi-enced teachers. The median chartermiddle school had about one-third fewerEnglish learners, one-third fewer low-income students, and students with muchbetter-educated parents. The typical char-ter middle school had three times as manyteachers who were not fully credentialedand twice as many teachers in their firsttwo years in the profession. The charterschools were also dramatically smaller,and the mobility of their student popula-tions was comparable to noncharters.

The small group of charter middle schoolsshowed impressive improvement from2003 to 2004Using the API to track over time theprogress of charter middle schools as agroup is a particular challenge because ofthe very small number of schools todayand the even smaller number in existencein 1999. In the 2003 Base/2004 GrowthAPI cycle:● 31 charter middle schools were in the

API system of accountability.● 27 schools—or 87%—had the data

necessary to determine whether theymet their growth targets, compared tojust 11 schools in the 1999/2000 cycle.

● Of those 27 schools, nine were conver-sions and 18 were start-ups. All wereclassroom-based.

● These 27 schools had 13,572students.Figure 5 compares charter schools to

noncharters in terms of their ability toachieve their API growth targets and meetthe state’s expectations for improvement.After a sharp drop between the first andsecond cycles, charter middle schoolshave shown steady improvement in theirability to meet their targets. By compari-

son, the percentage of successfulnoncharters has fluctuated more.

The percentage of charter middleschools that have reached their targets hasincreased in each of the last three cycles.The last two cycles, when the state’s chal-lenging standards figured predominantly

in the index, the percentages were animpressive 74% and 81%. The extent ofimprovement is also notable. Chartermiddle schools gained an average of 25points between the 2003 Base and 2004Growth APIs. Noncharters, on the otherhand, averaged 11 points of improvement.

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



Charter middle schools are small in number but perform well

Key Characteristics of California Middle Schools† in 2003–04




Hispanic (41%) 22% 36%12%–62% 17%–63%

White (37%) 43% 37%4%–64% 11%–62%


Asian (8%) 3% 3%0%–7% 1%–10%

Free/Reduced Price Meals (41%) 26%* 40%3%–68% 14%–65%

Parents Not High School Grads (21%) 5%* 16%*2%–22% 6%–32%

One+ Parent a College Grad (20%) 29%* 18%*14%–42% 10%–29%

Student Mobility (19%) 15%* 13%*10%–52% 10%–19%


Enrollment (average size 956) 213 914104–393 660–914





Statewide average: shows the percentage of middle students or teachers in each category statewide.




Charter middle schools shine in similarschools rankings in 2004 An absolute comparison between charterand noncharter middle schools is limitedin its significance because of the smallnumber of charters. The 2004 Base APIstatewide rankings include just 42 char-ter middle schools. That said, the

performance of charter middle schoolsbased on this indicator presents a morepositive picture than is the case withelementary schools. Once again, the high-performing schools were disproportion-ately established conversion schools. Newstart-ups were over-represented amongthe lower deciles.

The 2004 Base API rankings for the42 charter middle schools follow:● High-performing, deciles 8–10:

38% (16 schools)● Mid-performing, deciles 4–7:

38% (16 schools)● Low-performing, deciles 1–3:

24% (10 schools)When compared to schools with simi-

lar students and similarly credentialedteachers, charter middle schools alsoperformed well. In the similar schools rank-ings, where each school is compared to 100schools with similar challenge levels, 53%of schools did quite well. Start-ups, espe-cially those that had been open for at leasttwo years, were particularly noteworthy. The32 middle schools that received similarschools rankings performed as follows:● High-performing compared to similar

schools, ranks 8–10: 53% (17 schools)● Mid-performing compared to similar

schools, ranks 4–7: 31% (10 schools)● Low-performing compared to similar

schools, ranks 1–3: 16% (5 schools)

figure 5 Middle schools that made API growth targets


1999/00 2000/01 2001/02 2002/03 2003/04




Percent of noncharters making 61% 51% 38% 68% 54%growth targets

● It is more meaningful to compare percentages within individual API cycles because of the growing number of charter middle schoolsand the changing index used to measure school performance (the API).



Algebra I and California High School Exit Exam (CAHSEE) test scores are important indicators

API scores do not tell the whole story of academicperformance.

Charter middle schools are not doing as wellteaching algebraIn the late 1990s, the state adopted math stan-dards that called for eighth graders to take AlgebraI. As a result, middle schools have faced growingpressure to instruct their students in this subject.Both charter and noncharter middle schools haveroughly 40% of eighth graders enrolled in algebraand taking the California standards test. But on anabsolute comparison, the charter studentsperformed worse than their counterparts in 2004.In charter schools, 21% of eighth-grade algebrastudents scored “proficient” or “advanced.” Asubstantially higher 35% did so in regular publicschools. It is likely that neither charters nor regularpublic schools are entirely satisfied with theirperformance, but the noncharters definitely outper-form charters in this key area.

Charter 10th graders did not fare as well on theCAHSEE in 2004In the 2003–04 school year, 10th graders in nonchar-ter high schools outperformed their counterparts incharter high schools on the exit exam.School-level dataregarding students who passed both the math andEnglish language arts portions of the test are not avail-able. On the math section alone, 74% of noncharterstudents passed while 61% of charter students passed.The difference in passing rates in English language artswas much smaller: 75% of noncharter 10th graderspassed, while 71% of charter students did.

As with meeting API growth targets, performanceamong different categories of charters also varied.As the table indicates, conversion schools beatstart-ups and classroom-based charters outper-formed their nonclassroom-based counterparts by ahealthy margin in both cases. Established chartershad slightly higher percentages of students passingthe CAHSEE than did new charters.

Percent of charter and noncharter 10thgraders who passed the CAHSEE in 2004

Percent Passing

Math English

Noncharters 74% 75%

All Charters 61% 71%

Comparisons Among Types of Charters

Conversion 80% 84%

Start-up 56% 67%

Classroom-based 69% 76%

Nonclassroom-based 52% 65%

Established 62% 73%

New 58% 67%

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.


As was true at the elementary andmiddle-grade levels, the students in char-ter high schools tended to be moreadvantaged than those in regular highschools, but the differences were muchless pronounced. When it came to theteaching force, however, charter highschool teachers tended to lack credentialsand experience by even wider marginsthan was true at the other two levels. Thecharter schools were also dramaticallysmaller. The high mobility rate, measuredby the percentage of students who firstattended the school in the present year(students in the school’s lowest grade areexcluded from the calculation), mayreflect how many new charter highschools came into operation during the2003–04 school year.

Charter high schools do well meeting APIgrowth targetsFor the 2003 Base/2004 Growth API cycle:● 120 charter high schools were in the

API system of accountability.● 71 schools (59%) had the data neces-

sary to determine whether they mettheir growth targets, compared to just10 schools in the 1999/2000 cycle.

● Of those 71 schools, eight wereconversions and 63 were start-ups; 46were classroom-based and 25 werenonclassroom-based.

● These 71 schools had 29,032students.The number of charter high schools

in the state’s main accountability systemhas nearly quintupled since it began in1999. From the 25 schools that were inthe system in the first year, the numberhas increased to 120 in 2004. Of thoseschools that were open in 1999, 19 werestill operating as charters in 2004. But alarge group of newer schools has enteredthe scene in recent years. As a result, any

comparison between charter highschools in the two time periods is less anindication of improvement than a snap-shot of two very different populations’performance.

Figure 6 indicates the number of char-ters and noncharters that have met their

API growth targets over time (see the APIbox on page 4 for further explanation).Except in the 2002/2003 cycle—whencharters did not do quite as well as tradi-tional high schools—the charter schoolswith API data have outperformednoncharters by healthy margins. In the



© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.

Key Characteristics of California High Schools† in 2003–04




Hispanic (34%) 24% 28%9%–48% 13%–52%

White (44%) 46% 44%13%–72% 19%–70%


Asian (7%) 1% 3%0%–3% 1%–9%

Free/Reduced Price Meals (28%) 16%* 20%*0%–49% 7%–46%

Parents Not High School Grads (18%) 10%** 14%*3%–23% 5%–26%

One+ Parent a College Grad (22%) 22%** 21%*16%–34% 14%–30%

Student Mobility (22%) 48%** 11%*20%–66% 7%–16%


Enrollment (average size 1,347) 181 1,501101–413 306–2,294





Statewide average: shows the percentage of high school students or teachers in each category statewide.



Charter high school performance is mixed

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



2003/2004 cycle, 58% of charter highschools met their growth targets comparedto 49% of traditional high schools.

It is also notable that charter highschools were able to post an average gainof 17 points in this API cycle, whichcompared favorably to noncharters’ aver-age of 13 points. One factor that couldexplain some of this difference is that, atleast in the 2003 state rankings, 20% ofcharter high schools were in the bottomdecile. Those charters in particular hadquite a bit of room to grow, achieving anaverage of 35 points of growth in the2004 Growth API.

Absolute comparisons based on 2004Base API ranks paint a negative picture for high schools In the 2004 Base API, 124 charter highschools achieved the following rankings:● High-performing, deciles 8–10:

22% (27 schools)● Mid-performing, deciles 4–7:

27% (33 schools)● Low-performing, deciles 1–3:

52% (64 schools)Unfortunately, 27 charter high

schools were missing data and are left outof any analysis.

Start-ups formed the vast majority ofthe 124 schools, and they were spread

among the decile ranges. The nine conver-sion schools with decile ranks, however,were over-represented among the higherdeciles. Another noticeable trend was thatthe 45 nonclassroom-based schools weregenerally found in the lower deciles, withonly four schools in the upper range.Also, an additional 19 nonclassroom-based schools did not have rankings.Finally, the bulk of the 27 charter highschools in the high-performing category(deciles 8–10) were start-up schools thatprovide classroom-based instruction.

Charter high schools tended to be at the extremes in the 2004 similarschools rankings Only 76 charters were included in therankings—meaning that they had morethan 100 test scores. On this measure,charters tended to be at the extremes,doing quite well or not well, with rela-tively few in between. ● High-performing compared to similar

schools, ranks 8–10:42% (32 schools)

● Mid-performing compared to similarschools, ranks 4–7: 16% (12 schools)

● Low-performing compared to similarschools, ranks 1–3: 42% (32 schools)

As was true with statewide ranks,nonclassroom-based schools tended to earnlow marks, with 19 of 31 schools in thebottom 10%. (Here it is important toremember that this type of charter oftenserves students who are there because of alack of fit with mainstream schools and thatnonclassroom-based charters have quitehigh mobility rates.) In contrast, classroom-based charter high schools generally did wellcompared to similar schools, with morethan 60% reaching the higher decile range.Newer schools also tended to rank highamong comparable schools.

In 2005 charter renewal requires a newminimum performance standard California’s initial charter legislationincluded the specific expectation thatcharter schools would be accountable forstudent performance when they appliedfor charter renewal every five years. At thetime, however, the state did not have thecomprehensive testing and accountabilitysystems it has since developed. The defi-nition of acceptable student performancewas left to school organizers and theirchartering districts to determine. Thischanged in 2003 when lawmakers setminimum expectations for charter schoolperformance based on the API.

Beginning in January 2005, a charterschool requesting renewal must either:● meet its API growth targets (in the

prior year, in two of the last threeyears, or in the aggregate for the priorthree years); or

● rank in at least Decile 4 on its BaseAPI (using either state or similarschools rankings in the prior year or intwo of the last three years); or

● qualify for the Alternative SchoolsAccountability Model.Between January and August 2005,

the 80 charter schools up for renewal willbe the first to be affected by the new law.Among those schools, 78% have satisfiedat least one of the renewal criteria. Theremaining 18 schools lack the relevant

figure 6 High schools that made API growth targets


1999/00 2000/01 2001/02 2002/03 2003/04




Percent of noncharters making 41% 27% 29% 67% 49%growth targets

● It is more meaningful to compare percentages within individual API cycles because of the growing number of charter high schoolsand the changing index used to measure school performance (the API).



© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



data. Under Assembly Bill (AB) 1137, ifa charter school cannot show that it metone of the performance criteria describedabove, the charter-granting authoritymust determine that the academicperformance of the charter school is atleast equal to that of the public schoolsthe charter school pupils would otherwisehave attended and the schools in the localdistrict that have similar racial composi-tions. The California Department ofEducation reviews the chartering agency’sanalysis and makes a recommendation tothe agency regarding the school’s renewal.

Another 72 charter schools will be upfor renewal between September 2005 andAugust 2006. Based just on their APIinformation from 2002 to 2004, 71%have already qualified for renewal. Eightschools have not yet met the criteria, and13 lack the relevant data.

A number of state and national studieshave attempted to address whether theacademic performance of students incharter schools is better, worse, orcomparable to those who attend nonchar-ter schools. Invariably proponents andopponents of charters gather around eachnew study, mining it for evidence tosupport their arguments. All of thesestudies, however, are subject to criticismbecause of the many challenges previouslydescribed in this report. They also all usedifferent methodologies. It may not besurprising, then, that they all seem tocome to somewhat different conclusions.

National studies are particularly complexNational studies face a particularly diffi-cult challenge because researchers mustfirst somehow choose among the variedtypes of test data available.

Much of the national research hasdepended on the National Assessment ofEducational Progress (NAEP) as its

measure of performance. NAEP is theonly assessment taken by students inevery state, but it has certain limitationsbecause only a portion of students take itand each student takes only a portion ofthe test items for a given subject. A 2004report by the American Federation ofTeachers (AFT) generated headlines andsubsequent controversy by claiming thatfourth and eighth grade students in char-ter schools were scoring lower on NAEPthan their counterparts in noncharterschools. Entitled Charter School Achievementon the 2003 National Assessment of EducationalProgress, the report compared NAEPreading and math scores in a nationallyrepresentative sample of charter andnoncharter public schools. Critics say thereport looked at schools serving just 3%of charter school students nationwide.(See page 21 for more about this study.)

Other researchers have used state test-ing system results as their measure ofperformance. One such study pronouncedthat charters were doing better than regu-lar schools. Caroline Hoxby of HarvardUniversity and the National Bureau ofEconomic Research authored A Straightfor-ward Comparison of Charter Schools and RegularPublic Schools in the United States in 2004, onthe heels of the AFT report. Shecompared the reading and mathematicsproficiency of elementary charter schoolstudents to noncharter students based onthe specific tests given in their respectivestates. She found that charter schoolsoutscored the nearest noncharter publicschools. Hoxby also looked specifically atCalifornia schools. (See page 21 for moreabout this study.)

Again, these two national studies—andmany others—have both supporters anddetractors. The same is true of the Califor-nia-based studies that have been done.

California studies differ in their methodologiesand their conclusions Researchers who have compared theperformance of charters to noncharters

in California have also used different datasets and analytical methods, and they havereported different results. Studies consid-ering only API scores as issued by thestate concluded that charter schoolsperformed somewhat better. However,researchers with access to student-level—as opposed to only school-levelscores—found that regular publicschools did slightly better, though notalways by a statistically significant margin.

One of the most substantial of thesestudies was done by RAND and overseenby the Legislative Analyst’s Office. ThisJuly 2003 study, Charter School Operationsand Performance: Evidence from California,compared the performance of chartersand noncharters and also looked at howperformance varies among different kindsof charter schools. The authors also didanalyses using both the API and SAT-9(Stanford Achievement Test, 9th Edition)test scores. Their conclusions were slightlymixed with respect to comparisonsbetween charters and regular schools andamong various types of charters. (See page22 for more about this study.)

David Rogosa, an education professorand statistician at Stanford University,has been vocal in charter school research,sometimes critiquing other researchers’work. In 2003 Rogosa used his unusualexpertise on the API to develop andpublish an analysis that depended onstudent-level data to simulate API scoresfor the years 1999 to 2002. These mockAPIs were constructed entirely withscores from the nationally normed SAT-9. His analysis showed that theperformance of charter school studentsvaried depending on students’ grade leveland socioeconomic status. (See page 23for more about this study.)

These California studies—andothers that have been done—primarilyreflect performance on the SAT-9, thestate’s basic skills test used from 1998through 2002. To a great degree, theypre-date the California standards tests

Research findings conflict in charter/noncharter comparisons

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



(CSTs), which assess students’ profi-ciency on the state’s adopted contentstandards and which have been incorpo-rated into accountability measures morerecently. This is because the studies weredone before multiple years of CST datawere available. The emphasis on SAT-9data may make the performance assess-ments somewhat less meaningful intoday’s standards-focused context. Theyalso contrast with the general findingnoted previously that charter schoolperformance on the API has improvedsince 2003 when the standards-basedtests took on more weight in the index.

EdSource tries another approachAlong with the controversies, each succes-sive analysis directly comparing theacademic performance of charter andnoncharter schools has helped to showhow complicated such comparisons can

be and how the results can vary. EdSourceused that growing knowledge base toguide yet another attempt to answer themost commonly asked question: Are char-ter schools more successful, less successful, orsimply comparable to noncharters in their abilityto improve student performance?

Prior research highlights shortcomingsof the APIThe earlier studies mentioned above—particularly the California studies—influenced the way that EdSourceapproached this question. The firstconsideration was to make sure the meas-ure of performance looked atimprovement over time and did so in athoughtful way. Second, the schoolsbeing compared were selected carefully toensure that they face similar challengeswith respect to student demographicsand that their starting points, in terms of

performance, were similar. Comparingcharters and noncharters that are compa-rable in both respects makes it possible tofocus more closely on the direct impact ofcharter status on improvement in studentachievement.

To meet these criteria, this analysisfocuses on a small number of well-estab-lished charter and noncharter schools andon discrete performance measures. Assuch, the findings cannot be used togeneralize about all areas of studentperformance or all charter schools. Inparticular, it could be argued that itdistorts charter school achievement byfocusing only on the more stable charters.The 89 charter schools selected serve atotal of 53,473 students in 2003–04.

EdSource’s analysis is noteworthybecause it uses scores on the CSTs ratherthan the state’s API or SAT-9 scores. TheCST data provide more detailed informa-tion than API scores, and they are salientin California’s new accountability climateunder the federal No Child Left BehindAct (NCLB).

The experience of the RANDresearchers influenced this decision. In itsevaluation of California charter schools,the RAND team used API scores for avariety of reasons. They found, however,that because the index aggregates all testscores in a school into one number, itobscures important information.Further, the weight given to variouscomponents of the index is based on apolicy decision by the State Board ofEducation. For example, in elementaryschools, English scores count for 60% ofthe API and math scores count for 40%.If one wants to know how students in aparticular grade or subject did—or ifone disagrees with the board’s weightingof various subjects in the index—APIscores are not satisfactory.

In addition, the API system dividesschools into three broad school types—elementary, middle, and high. Becausegrade configurations within each type

California standards tests provide a reliable though imperfectbasis for comparison

Evaluations of academic performance would ideally consider more than just standardized test scores.However, in California the STAR tests are the only academic measures with the qualities necessary tomake a valid comparison between schools, including the comparison of charters and noncharters.

Using STAR tests to compare schools has several advantages

● All schools arguably have access to the material covered on the STAR tests.

● The assessments have been screened for ethnic bias, for being a reasonable gauge of what they arepurported to measure, and for being reliable—meaning that if two highly similar students took the test,they would get similar scores.

● The tests are secure: teachers and students do not know in advance what the exact questions will be.

● Virtually all students in both types of schools take the STAR tests. As a result, it is unlikely that, forexample, only the stronger or weaker students are taking the test in one type of school or the other.

The STAR program does not directly measure the “value added” by individual schools The data only provide annual snapshots of how each group of students perform on a particular test. Giventhe state’s still-developing data system, the performance of last year’s third graders can be compared tothis year’s third graders, but the progress of one group of third graders over time is not measured. Werelast year’s third graders to be compared to this year’s fourth graders, other data problems could ariserelated to the connection between third and fourth grade tests and changes in the groups of studentstested due to student mobility.



vary, and score growth varies within eachgrade, school-level API scores do not offerprecise “apples to apples” comparisons.

As an alternative to the API, RANDand other researchers used SAT-9 testscores for analyses of charter perform-ance. Now, however, schools and districtsare focused on increasing the percentageof students scoring “proficient” or aboveon the CSTs. And multiple years of dataon these tests are now available. Alongwith new weighting in the API, accounta-bility for “adequate yearly progress”underNCLB makes these tests more important.

EdSource comparison uses CST scoresEdSource compared growth in thepercent of students scoring proficient orabove on math and English CSTs in char-ters and noncharters between 2002 and2004. The analysis includes only scoresfrom grades 3, 7, and 10 in English andonly grades 3 and 7 in math. (Studentstake different math CSTs in grades 8–11based on the course they are taking. As aresult, it is not sound to compare schools’scores on math CSTs beyond grade 7.The exit exam, a uniform statewide meas-ure, is more appropriate. Those scores arediscussed earlier in this report.)

NCLB has encouraged Californiaschools to focus on increasing thepercentage of students scoring proficientor above on English and math CSTs.However, a substantial portion ofstudents still score at the lowest profi-ciency level, “far below basic.” Helpingthem improve is also an important goal.The EdSource analysis therefore alsoconsiders schools’ ability to decrease thepercentage of students scoring at thislowest performance level.

The results show strong performanceamong established charter schoolsUsing a carefully selected subset ofschools, EdSource made two compar-isons. One compared the amount ofimprovement that a group of charter

Creating the EdSource dataset

To set up a comparison of charter and tradi-tional public schools, EdSource combined datafrom three electronic files provided on the Cali-fornia Department of Education website: 1) CSTscores in English language arts and math, 2)student demographics, and 3) the “school char-acteristics index” (SCI) scores for similarschools rankings. The data pertain to the2001–02 through 2003–04 school years.

Schools had to fit specific criteria to be used inthe analysis, including: ● Any school that experienced a significant

demographic change in its student body wasremoved from the analysis. A “significantdemographic change” is defined as a changeof more than 10 percentage points in morethan one of the following categories: Englishlearners, students eligible for free/reduced-price meals, ethnic groups, and parentaleducation levels (e.g., percent of studentswith at least one college-educated parent).

● Only schools that had been open since atleast 2000–01 were included. Given thatmany start-up charter schools face cash flowand facilities challenges in their early years,only start-ups that had been open since1999–2000 were included.

● To measure challenge, EdSource used theschool characteristics index (SCI). The CDEcomputes the SCI each year as a first step tocreating the “similar schools” rankings in theBase API. Elementary, middle, and highschools have different SCIs. Thus an elemen-tary school with an SCI of 150 has a differentchallenge level than a middle school with thesame SCI. Because so many charter schoolshave nontraditional grade configurations(e.g., some charters with 3rd graders areclassified as middle schools), EdSourcecould not simply use charter schools’ SCIscores to find comparable noncharterschools. Instead, EdSource ranked schools ofsimilar type (elementary/middle/high)according to their SCI and assigned each

school a percentile ranking. With thosepercentile rankings, EdSource could comparetwo schools serving a common grade basedon their rankings, regardless of whether theywere elementary, middle, or high schools. Forexample, an elementary school at the 30thpercentile in its SCI rankings and servingthird graders could be compared to a middleschool at the 30th percentile in its SCI rank-ings and serving third graders. For anoncharter to be compared to a charter, itsSCI percentile ranking had to be within fivepercentile points of the charter’s SCIpercentile ranking.

● For two schools to be compared, they alsohad to be similar in their starting points vis-a-vis student performance. In 2002, theschools were within five percentage points ofeach other in their percentage of studentsscoring proficient and above, or in thosescoring far below basic. For example, anelementary charter school at the 65thpercentile on the SCI with 35% of its thirdgraders scoring proficient and above couldbe compared to any school ranking from the60th to 70th percentile on the SCI and with30% to 40% of its third graders scoring profi-cient and above.

In the end, 89 charter schools and 5,762 tradi-tional public schools met all the criteria.Schools serving more than one of the relevantthree grades—3, 7, and 10—were used in morethan one grade-level analysis. For example, if aschool served grades K–8, its grade 3 scoreswere used in the analysis of grade 3 perform-ance, and its grade 7 scores were used in thegrade 7 performance analysis. A handful of thecharter schools that performed considerablybetter or worse than schools facing similar chal-lenges did not have a comparison group andwere not included in the school-level match-ups. Noncharters that were similar to multiplecharter schools were assigned to multiplecomparison groups.

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.


schools made to the improvement of agroup of similar noncharter schools. Theother looked at the percentage of charterschools that showed more improvementthan their comparison schools.

Charters outperformed nonchartersin both cases. The first analysis found thatcharter students showed slightly greaterimprovement than those in noncharterson the English and math CSTs from 2002to 2004. The second analysis showedcharter schools performing better in amajority of direct comparisons between acharter school and a group of similarnoncharters. (For more details on howEdSource narrowed the set of schools forits analysis, see the box on page 17.)

Each combination of grade, subject,and performance level—for example grade3, math, proficient and above—wasconsidered separately. For each school, thepercentage scoring at a particular perform-ance level in 2002 was subtracted from the2004 percentage. That improvement in percent-ages was the important variable for both analyses.(“Improvement” was counted two ways: asan increase in the percent of students scor-ing proficient and above, and as a decreasein the percent scoring far below basic.) Itis important to note the improvement isbetween two different groups ofstudents—for example the third graders of2002 and the third graders of 2004.

CST scores for charter studentsimproved by greater marginsIn the first computation, each school’simprovement level was weighted by thenumber of test-takers. The averages werethen calculated for the charter and nonchar-ter groups. Taking a weighted average creates(figuratively speaking) one big charterschool and one big noncharter school.

Figure 7 shows the weighted averageimprovement for students in charterschools and in the comparison nonchar-ter schools in each grade, subject, andproficiency level. On all 10 comparisons,the students in charter schools improved

their scores more than noncharters. Thismeans that the charters studied werebetter than their comparison schools atincreasing the percentage of high-scoringstudents and reducing the percentage oflow-scoring students. Although thedifferences appear small—ranging from0.6 percentage points to 3.8 points—they are statistically significant, meaningthat they fall outside the typical range ofrandom variation. The smallest gapbetween charters and noncharters was inthird grade English for high-scoringstudents. Both groups actually declined inthat percentage from 2002 to 2004, butcharters did so by slightly less (-1.8 vs. -2.4percentage points).

The greatest gap was also in thirdgrade, but on math scores. There, chartersdecreased the far below basic group by 7percentage points versus 3.2 points fornoncharters. The differences in improve-ment in 10th-grade English scores werealso relatively large.

Even the largest differences betweencharters and noncharters, however, arenot as large as the variation within eachtype of school. For example, in grade 7,English, proficient and above, one charterschool declined by 16 percentage pointswhile another improved by 29 points.That 45-point difference within the char-ter group compares to a difference of less

than one percentage point between chartersand comparison noncharters.

School-level comparisons also showgreater improvement among chartersFor the second analysis, EdSource calcu-lated the percentage of charters thatshowed more improvement than theircomparison schools did. Each charterschool’s improvement was compared tothe weighted average improvement of itsnoncharter comparison schools.

Figure 8 shows the number andpercent of charter schools that outper-formed their comparison schools.Charter schools generally did better thantheir comparison groups. Again, thecomparisons look both at schools wheremore students improved into the “profi-cient and advanced” performance bands,and at those where more studentsimproved out of the “far below basic”band. In eight of the 10 separate compar-isons—based on grade levels, subjects,and performance bands—a majority ofcharters outperformed their nonchartercomparison group.

The comparison in which charterswere least successful, with just 43%“winning,” was in Grade 3 math in theproficient-and-above performance band.On the other hand, 73% of charterschools outperformed their comparison


figure 7

Proficient and Above Far Below Basic

Charter Noncharter Charter Noncharter

English Language ArtsGrade 3 -1.8 -2.4 -0.8 0.9Grade 7 5.1 4.3 -1.8 -0.9Grade 10 6.6 3.0 -2.5 -0.3

MathGrade 3 12.5 11.1 -7.0 -3.2Grade 7 5.6 4.3 -1.6 0.2

Scores for charter school students improved more than those for comparable noncharter students from 2002 to 2004

Data: Adapted from California Department of Education (CDE)–STAR research files EdSource 5/05

Average percentage-point change in students scoring…

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



schools in decreasing the percentage of7th graders scoring in the far-below-basicband on the math CST.

The results also reinforced a familiarpattern in charter school achievement:conversion schools outperformed start-ups (with seventh-grade English the oneexception). Consider the example ofthird-grade English, in which 45% ofcharters prevailed based on the percent-age of students scoring proficient andabove. Among conversion schools, 50%outperformed their comparison schoolswhile 32% of start-ups beat theircomparison schools.

This analysis also varied in an impor-tant way from the API comparisonsprovided earlier in this report. Class-room-based charters did not alwaysoutperform their nonclassroom-basedcounterparts. For example, in 7th gradeEnglish, 10th grade English, and 7thgrade math, nonclassroom-based schoolsoutperformed their comparison nonchar-ter schools more often than theclassroom-based charters beat out theirs.

On the other hand, subdividingcharter schools by type revealed thatcharters that were both conversion and

classroom-based fared better thanschools that were both start-up andnonclassroom-based. (Schools are notsubdivided into “established” and“new” in this analysis because all schoolsincluded in the dataset met EdSource’sdefinition of “established.”)

Many studies of charter school perform-ance focus on the viability of the charterapproach. Often, the explicit or impliedconclusion is that this educational exper-iment either should or should notcontinue. But as the analyses presented inthis report make clear, there is not apreponderance of evidence that makes an“up or down” vote on the charter schoolconcept obvious or even desirable.

More than 500 charter schools are inoperation today in California and someof them have been open for more than adecade. It behooves educators, policy-makers and the public to learn moreabout the schools themselves, theirsuccesses, and their failures. There are alsomany questions to be explored in regard

to the best role they can play in the publiceducation system and what conditionsmake them most successful.

Key findings provide a fresh and revealinglook at charter performanceUsing state testing data, EdSource foundseveral patterns of charter schoolperformance that are particularly note-worthy. Collectively they present somereasons for optimism and some causes forconcern. They also raise a number ofinteresting questions that should beexplored more thoroughly.

Missing test data from a large percent-age of the state’s charter schools underminesthe ability to assess their academic perform-ance and thus their effectiveness as analternative learning environment for somestudents. For the 2004 Growth AcademicPerformance Index, about 23% of the char-ter schools in the state’s API system lackedsufficient test data to even receive APIscores. That compares to just 6% of thenoncharter schools in the API system. Only13% of classroom-based charter schoolswere missing data, while 46% of nonclass-room-based charter schools did notadminister the state’s standards based teststo enough students to yield official APIresults. (See page 6.)

Among charter schools with data,particularly classroom-based charters,improvement meeting academic growthtargets is particularly notable. The relativeperformance of charters started toimprove in 2003 when API scores beganemphasizing the California standards testsinstead of off-the-shelf norm-referencedtests. In 2004, 64% of classroom-basedcharters (which constitutes 60% of allcharters) met their 2004 API growthtargets, outperforming both noncharterschools (48%) and the nonclassroom-based charter schools (44%) for whichdata is available. (See page 8.)

Elementary charters lag behind non-charters on Base API scores, but haverecently done better meeting growth

figure 8

Proficient and Above Far Below BasicNumber of Number (and Number of Number (and

Comparisons percent) of Times Comparisons percent) of TimesCharter Outperformed Charter Outperformed

Comparison Group Comparison GroupEnglish Language Arts

Grade 3 65 29 (45%) 67 35 (52%)Grade 7 46 26 (57%) 47 26 (55%)Grade 10 19 12 (63%) 19 11 (58%)

MathGrade 3 68 29 (43%) 66 34 (52%)Grade 7 48 29 (60%) 48 35 (73%)

Number and percentage of charter schools that improved morethan their comparison schools

Data: Adapted from California Department of Education (CDE)–STAR research files EdSource 5/05

Change in percentage of students scoring at specified levels on California standards tests from 2002 to 2004

EdSource finds noteworthy patterns,but more study is needed

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



targets. In an absolute comparison betweenall elementary charters and noncharters,the charter schools on average have slightlylower 2004 Base API scores. This is inspite of the fact that elementary charterschools serve a smaller proportion ofEnglish learners and low-income studentsthan their noncharter counterparts.However, charter elementary schools haveimproved the rate at which they meet theirAPI growth targets. In 2004 the group ofelementary charters was the first to outper-form noncharters, with 57% of themmeeting their targets compared to 46% ofnoncharters. (See page 9.)

The small group of charter middleschools performed quite well. Just 31 char-ter middle schools were included in thestate’s accountability system in2003/2004 and that small numbertempers any assertions about the academicperformance of charters serving this agegroup. That said, among the 27 of thosecharter middle schools that had 2004Growth API scores, 81% met their growthtargets (compared to 54% of nonchartermiddle schools). (See page 11.)

California’s middle grades charterschools on average serve fewer disadvan-taged students than their noncharter peerschools. While it is therefore not surpris-ing that they ranked relatively high ontheir 2004 Base API scores, they alsocompared favorably to noncharter middleschools with similar students based onthe state’s similar schools rankings.

Firm conclusions about charter highschool achievement are problematic.Charter high schools increased in numbervery quickly, going from 34 schools withAPI growth data in 2003 to 120 in 2004.Most of these new charters are start-upschools that tend to differ substantiallyfrom noncharters in the students thatthey serve and the teachers that theyemploy. (See page 13.)

More than half of charter highschools ranked in Deciles 1–3 (thebottom 30%) based on the 2004 Base

API. But comparing them to nonchartersserving similar students nets more mixedresults, with equal proportions comparingwell and poorly. In terms of improvement,missing data is again a challenge. Just 71high schools had growth data for the2003/2004 cycle. Of those schools,58% met their growth targets comparedto 49% of regular high schools.

In an EdSource study that matched 97well-established charters against a largegroup of comparable noncharters, thecharter schools’ test scores generallyimproved more than those of noncharters.Based on an analysis of California stan-dards tests (CSTs) for grades 3, 7, and 10in English and grades 3 and 7 in math,charters did slightly better at increasingthe percentages of students scoring“proficient and above” and decreasing thepercentage scoring “far below basic” from2002 to 2004. The differences appearslight, but they are statistically signifi-cant—meaning outside the normal rangeof random variation. (See page 16.)

In eight out of 10 sets of school-levelcomparisons between charters andnoncharters, a majority of chartersshowed more improvement on the mathand English CSTs from 2002 to 2004than their comparison noncharter schools:● Charters serving grade 3 students

outperformed their comparisonnoncharter schools in improving theirpercentages scoring “far below basic”butnot those scoring “proficient and above.”

● A majority of charters serving grade 7students outperformed their compari-son schools in both subjects and atboth score levels.

● In 10th grade English, a majority ofcharters showed greater improvementthan their matched noncharters atboth score levels.

Continued study of California charterschool performance is importantState and federal policymakers are eagerto find solutions for the state’s chroni-

cally low-performing schools. Under theNo Child Left Behind Act, turningthem into charter schools is one option,but is conversion to charter status aneffective solution?

Not necessarily, but it might beunder certain circumstances. Just as inthe regular public school system, thecharter community has examples ofboth low-performing and exemplaryschools. Before state and local policy-makers advocate the use of public taxdollars to convert low-performingpublic schools to charter status, theyshould closely examine the types ofcharter schools that are succeeding, forwhat kinds of students, under whatconditions.

That said there are real success storiesamong charter schools. Many of them areproviding exciting learning environments.And many are successful serving studentswho for various reasons have not found agood fit in a regular public school setting,including many low-income children ofcolor. Of particular note is the burgeon-ing use of charters as a strategy foraddressing the stubborn challenges ofhigh school reform, particularly in disad-vantaged communities.

In addition—based on state testingdata—charter schools have recentlystarted to make impressive gains, improv-ing performance and meeting academicgrowth targets faster than noncharters inmany cases. The charter API growth datafor 2004 is definitely promising, but oneyear does not yet make a trend.

California’s charter school experi-ment can be a lightning rod forconflicting political views. But studentsand the state would be better served ifcharter schools were instead seen aslaboratories of school improvement thatadd to the information researchers,educators, and policymakers can use tomake sure that every child has an oppor-tunity for academic success in the state’spublic schools.

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



CONTROVERSIAL REPORT CLAIMS CHARTERS NATIONALLY FALL SHORT OF EXPECTATIONSIn 2004, in a national report titled Charter School Achievement on the 2003National Assessment of Educational Progress (NAEP), the American Federationof Teachers (AFT) compared NAEP reading and math scores in a nationally repre-sentative sample of charter and noncharter public schools.The paper reportedlylooked at schools serving about 3% of charter school students nationwide.

The NAEP reports include the percent of students scoring at four proficiencylevels and an average “scale score” that reflects the difficulty of the questionsanswered correctly. In each state, only a sample of students take the test, andstudents take different portions of the test.

Charter students in grades 4 and 8 score below noncharter students on NAEPAFT found that fourth graders in charter schools scored an average of six pointslower on the scale (228 vs. 234) in math and seven points lower (210 vs. 217)in reading. Charter students in eighth grade scored five points lower in mathand two points lower in reading. AFT stated that, with the exception of eighthgrade reading, the results were statistically significant. (Statistical significanceis reached when differences are outside the typical range of random variation.)AFT estimated that the differences between charter and noncharter perform-ance translate to about half a school year.

AFT also disaggregated the data by poverty status, locale, and ethnicity, findingsimilar differences for low-income students. However, when students werecompared only to those of the same ethnicity, or from similar locales, the differ-ences were much less. An exception was in central cities, where fourth gradecharter students trailed by seven points in math.

A group of researchers questions the AFT analysisA group of researchers from across the country faulted the report for relying onlimited student background information and considering only one backgroundfactor at a time rather than considering all available factors simultaneously.(For example, focusing on race does not take into account differences in aver-age income among races. Looking at all background factors together allows aresearcher to isolate the impact of each factor.) In addition, they criticized thereport for ignoring important student background information and for onlyexamining test scores for one point in time.

For a copy of the AFT report, go to www.aft.org/pubs-reports/downloads/teachers/NAEPCharterSchoolReport.pdf. It includes a subset of data for Cali-fornia students.

2003 PILOT STUDY USING NAEP RESULTS ALSO AVAILABLEThis study looked at fourth grade math and English performance at 150 randomly selected charter schools. For a copy go to:http://nces.ed.gov/nationsreportcard/pdf/studies/2005456.pdf

ECONOMIST COMPARES WITHIN STATES, FINDS MORE POSITIVE RESULTS,AND ALSO RECEIVES CRITICISMIn a study of charter school performance entitled A Straightforward Compari-son of Charter Schools and Regular Public Schools in the United States,Harvard economist Caroline Hoxby compared the reading and mathematicsproficiency of charter school students to noncharter students based on state-specific tests. She generally dealt with fourth graders’ scores but used grades3 or 5 in cases where grade 4 data were unavailable.

The charter schools were compared to the schools that their students wouldmost likely otherwise attend—the nearest noncharter school—and also to thenearest noncharter with a “similar racial composition.” In 92% of the cases, thenoncharter that had the similar racial composition was also the closest school.Hoxby’s report states that she covered about 99% of the nation’s fourth gradecharter school students. (Presumably she meant 99% of charter students whowere tested, not all students.)

Fourth grade score comparisons yield good news for charters Based on both reading and math test scores, Hoxby found that charter schoolsoutscored the nearest noncharter public schools.The percent of students profi-cient in reading was four percentage points higher, and in math the advantagewas two points. When compared to the nearest regular public school with ademographically similar student body, charter schools showed a five percent-age point advantage in reading and a three point lead in math.

Hoxby reported data for those states with statistically meaningful populationsof charter school students, generally more than 200. That included California.

To get a copy of Hoxby’s study, including the Calfornia results, go to:www.wacharterschools.org/learn/studies/HoxbyCharters_Dec2004.pdf

Hoxby’s methods have been questionedIn a paper titled Advantage None: Re-Examining Hoxby’s Finding of CharterSchool Benefits, the Economic Policy Institute (EPI) asserts that the professor’sanalysis does not adequately control for student backgrounds. When EPIanalyzed Hoxby’s dataset and controlled for race and income, it found only onestate—California—where reading scores were higher to a statistically significantdegree in charters as opposed to noncharters. It found no states where char-ters had a statistically significant edge in math.

To get the EPI critique, go to: http://epinet.org/content.cfm/bp158

SRI INTERNATIONAL REPORT LOOKS AT FIVE OTHER STATESThis analysis of charter schools’ability to meet performance standards in five states—Texas, Colorado, Illinois, Massachusetts, and North Carolina—is available at:www.sri.com:8000/policy/cep/choice/PCSP_FinalReport_2004_OPA_approved.pdf

National and California studies of charter school performance

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.



RAND STUDY LOOKS AT BOTH API AND SAT-9 SCORES IN CALIFORNIA,WITH MIXED RESULTS Called for in legislation and overseen by the Legislative Analyst’s Office (LAO),the RAND report—Charter School Operations and Performance: Evidence fromCalifornia—compared the performance of charters and noncharters and lookedat how performance varies among different kinds of charter schools. The latteranalysis distinguished between charter schools based on the way they werefounded (as a conversion of an existing public school or a wholly new start-up),whether their instruction follows a traditional classroom-based approach ornot, and whether the school had been operating under a charter for at leastthree years.

Based on API, charters and noncharters show no significant differences in student performanceAfter adjusting for changes in demographics over time in each school, RANDcompared annual growth in API scores among charters and conventionalschools in the 1999–2000 through 2001–02 API growth cycles.The study foundno statistically significant difference between charters and noncharters. Thetype of charter school did not matter either.

Analysis of SAT-9 test scores showed slightly better performance for nonchartersRAND did a separate comparison using individual students’ test scores, adjust-ing for differences in ethnicity, parent education, English learner status, gender,and whether a given year was the student’s first at the school. All of thesestudent characteristics are correlated with student performance. Controlling forthem is a way for researchers to attempt to isolate the effect the school per sehas on student performance.

With these adjustments, they found some statistically significant differences inthe average percentile scores of students in charter versus noncharter schools.For example:

● In math: noncharters scored higher in the elementary (49.3 vs. 47.8) andsecondary (47.7 vs. 45.4) levels. (“Secondary” schools were defined asmiddle and high schools.)

● In reading: noncharters scored higher at the secondary (42.1 vs. 40.7) level,but scores at the elementary level were nearly identical.

But classroom-based charter schools outperform noncharters RAND also looked at performance differences based on the type of charter school.Classroom-based charters generally scored slightly higher than regular publicschools. However, the average student in schools offering nonclassroom-basedinstruction had scores that were between five and nine points lower. Clearly,nonclassroom-based schools bring down the average scores of charter schools.

In discussing this finding, the RAND report notes that nonclassroom-basedschools may differ from others in ways not easily captured in statistics. Forexample, if nonclassroom-based schools are designed to help students whohave left conventional public schools because of problems they have in tradi-tional settings, then it is not appropriate to compare nonclassroom-basedcharters to conventional schools. Such differences in students would not showup in statistical data about whom a given school serves.

RAND also found differences between the performance of new and estab-lished charters. One interesting finding was that nonclassroom-based charterschools differed depending on whether they were less than three years old.The newer nonclassroom-based charter schools performed substantiallybetter—from four to seven percentile points higher on the SAT-9 dependingon the grade and subject. It is unclear, however, what accounted for thedifference.

To see the RAND report, go to: www.rand.org/publications/MR/MR1700/.For the official analysis and report to the Legislature by the LAO, go to:www.lao.ca.gov/2004/charter_schools/012004_charter_schools.pdf

STUDY BY RESEARCHERS AT CAL STATE L.A. IS GOOD NEWS FOR CALIFORNIA CHARTERS, BUT RESULTS ARE QUESTIONEDThree researchers from California State University, Los Angeles (CSULA)—Simeon Slovacek, Antony Kunnan, and Hae-Jin Kim—used API data to concludethat charter schools did a better job of improving the performance of low-income students than regular public schools did.

In their March 2002 report—California Charter Schools Serving Low-SESStudents: An Analysis of the Academic Performance Index—the researchersused Base API scores from 1999 through 2001 to compare the performanceof charters and noncharters with a majority of students eligible forfree/reduced-price meals. Their analysis of the dataset they createdrevealed that, in schools with at least 50% low-income students, the APIscores of charters went up by 22.6% from 1999 to 2001 while the scores ofnoncharters improved 19.4%. The difference was more pronounced inschools with at least 75% low-income students—28.1% improvement incharter schools versus 23.8% for noncharters. The CSULA report alsoconcluded that the type of instruction that a charter offered, its length oftime open, and whether it was a start-up or conversion did not seem to makea difference in performance.

This analysis was criticized based on the quality of its dataset and the fact thatit lumped all the charter schools together as one group without regard towhether the percentages of elementary, middle, and high schools are the sameas in the universe of noncharter schools.

To see a copy of this study go to:www.calstatela.edu/academic/ccoe/c_perc/rpt1.pdf

FOR ADDITIONAL STUDIES OF CALIFORNIA CHARTER SCHOOL ACADEMICPERFORMANCE, SEE:

● The Performance of California Charter Schools by Margaret E. Raymond ofCREDO at the Hoover Institution at Stanford University. Go to:http://credo.stanford.edu/downloads/ca_chart_sch.pdf

● Catching the Wave: Lessons from California’s Charter Schools by NelsonSmith of the Progressive Policy Institute. Go to: www.ppionline.org/documents/CA_Charters_0703.pdf



ROGOSA’S STUDENT-LEVEL ANALYSIS SHOWS MIXED RESULTS IN CALIFORNIA THROUGH 2002In an analysis published in June 2003, Stanford Professor David Rogosa workedwith Academic Performance Index (API) scores to compare student perform-ance in charter and noncharter schools. An expert on the API calculation anddata, Rogosa developed simulated API scores for the years 1999 to 2002.Thesemock APIs were constructed entirely with scores from the nationally normedSAT-9 (Stanford Achievement Test, 9th Edition).

The dataset consisted of student test scores from 93 charter schools thatreceived API scores from 1999 through 2002 and from 6,584 noncharters. (Asmall percentage of each type of school was missing one year’s API score.)

Charter students in most grades show less improvement than those innonchartersFor his comparisons, Rogosa grouped the test scores from all charter studentsacross the state by grade and computed API scores. For example, he groupedthe scores of all second grade students from all of California’s charter schoolsand computed an API score as if they attended one large charter school. He didso to avoid comparing, for example, charter schools with grades 3–6 to regularschools with grades 2–5 because different grades have shown different levelsof score-growth. Rogosa was able to create these mock API scores because hehad access to student-level data.

Rogosa showed results that varied by grade level and by students’ socioeco-nomic status. As the table below shows, at four grade levels (2, 4, 10, 11)charter students showed greater improvement (3–11 API points) from 1999 to2002, but in the other six grades, noncharter students demonstrated bettergains (2–33 points). The number of charter schools contributing scores from10th- and 11th-grade students was small, as was the number of scores.

Grade-by-Grade Score Improvement on a “Mock API” from 1999 to 2002

Results for disadvantaged students also show noncharters doing betterRogosa also compared the performance of socioeconomically disadvantagedstudents based on the concentration of poverty in their schools. He found thatthose attending noncharters tended to show greater gains. Whether the disad-vantaged students were in the minority or majority at their school, they tendedto do better in noncharter schools—quite substantially in some grades. Taking

all schools into consideration, disadvantaged noncharter students showedmore improvement than charter students in eight of 10 grades tested (3–39more points).

Among schools in which at least half of students were disadvantaged, nonchar-ter schools showed more improvement with their disadvantaged students inseven of 10 grades (3–51 more points).

Charter schools look better in simulation of schoolwide APIs Rogosa also simulated schoolwide APIs by aggregating grade-level scoresinto groupings of grades 2–6, 2–8, 9–11, and 2–11. With this level of analy-sis, the performance gap between charters and noncharters narrowed—andin some cases reversed. For example, when test scores for all charter schoolstudents in grades 2–6 are combined to produce API scores for 1999through 2002, the students gained 76 API points over the three years versus74 points for noncharter students. However, students in charter high schoolsshowed 15 points of growth while noncharter high school students gained20 points.

A copy of this study is available at: www-stat.stanford.edu/~rag/api/charter9902.pdf

Charter studentsNoncharter students

Average increase in mock API for socioeconomically disadvantaged students in all schools

114

101

113

75

58

18

0.3

13 16 15

104110 107

88

71

53

3930

19 22

0

20

40

60

80

100

140

120

2 3 4 5 6 7 8 9 10 11Grade

Gai

n in

moc

k A

PI s

core

, 199

9-20

02

Average increase in mock API for all students & schools

8982

92

6052

8 6 5

1924

788484

68

55

41

2924

16 17

0102030405060708090100

2 3 4 5 6 7 8 9 10 11Grade


Gai

n in

moc

k A

PI s

core

, 199

9-20

02

Average increase in mock API for socioeconomically disadvantaged students in predominantly poor schools

120111

124

88

73

29

-12 2 0.611

110116 113

92

73

53

3931

20 25

-20

0

20

40

60

80

100

120

140

2 3 4 5 6 7 8 9 10 11Grade


Gai

n in

moc

k A

PI s

core

, 199

9-20

02

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.


Ray Bacchetti, PresidentScholar in Residence, Carnegie Foundation for the Advancement of Teaching

Peter Schrag, Vice PresidentContributing Editor, Sacramento Bee

Lawrence O. Picus, Fiscal OfficerProfessor, Rossier School of Education,University of Southern California

M. Magdalena Carrillo Mejia, SecretarySuperintendent, Sacramento City Unified School District

Davis CampbellPresident, California School Boards Association Governance Institute

Kathleen ChaPresident, American Association of University Women–California

Christopher CrossChairman, Cross & Joftus, LLC

Brenda DavisPresident-Elect, California State PTA

Gerald C. HaywardPartner, Management, Analysis & Planning, Inc.

Jacqueline JacobbergerPresident, League of Women Voters of California

Kelvin K. LeeSuperintendent, Dry Creek Joint Elementary School District

Paul J. MarkowitzTeacher, Las Virgenes Unified School District

John B. MocklerPresident, John Mockler & Associates

Ted OlssonRetired Manager, Corporate Community Relations,IBM Corporation

Amado M. PadillaProfessor of Education, Stanford University

Krys WulffDirector-at-Large, American Association of University Women

2004–05 EdSource Board of Directors

Trish WilliamsEdSource Executive Director

4151 Middlefield Road, Suite 100 Palo Alto, CA 94303-4743 • 650/857-9604 • Fax: 650/857-9618

[email protected] • www.edsource.org • www.ed-data.k12.ca.us

© C

opyr

ight

200

5 by

EdS

ourc

e, I

nc.

Studies of charter school performanceVarious national and California studies are described on pages 21 to 23 of this report. Websiteaddresses indicate where copies of these studies are available.

Data about individual charter schools in CaliforniaThe Ed-Data Partnership website—at www.ed-data.k12.ca.us—provides a wealth of data aboutevery charter school in California, including student background, staffing information, andsummary AYP and API reports. Data as far back as 1992–93 is available on the site.

In addition, a new “Compare Schools” feature on Ed-Data enables you to develop customizedreports comparing schools you select. You can also use the “Highest/Lowest” feature to createlists of California charter schools you would like to see. For example, you can create a list of the20 charter high schools with the highest enrollments, or all of the elementary charters that have100% fully credentialed teachers.

Information on charter school laws and policies● The charter school section of EdSource Online, which includes an overview, relevant data, and a

list of EdSource publications related to charter schools is at: www.edsource.org/edu_chart.cfm

● The California Charter Schools Association can be found at: www.charterassociation.org

● The Charter Schools Development Center’s website is at: www.cacharterschools.org

To Learn More

EdSource thanks Reed Hastings for his investment in our work on charter school issues.

how are california’s charter schools performing? · 2014-09-24 · charters often differ in grade...

Documents