julian r. betts and y. emily tang, university of california, san diego

THE EFFECT OF CHARTER SCHOOLS ON STUDENT ACHIEVEMENT: A META-ANALYSIS

OF THE LITERATURE

CAMPBELL COLLOQUIUM EDUCATION PANEL, MAY 2012

Julian R. Betts and Y. Emily Tang,University of California, San Diego

([email protected], [email protected])We are grateful to the Center on Reinventing Public

Education, University of Washington, Bothell, for funding this research

mailto:[email protected]

mailto:[email protected]

2

Introduction and Motivation Selecting Studies to Include Assessment of Alternative Methods of

Evaluating the Impact of Charter Schools

Challenges in Study Collection/Review Process

Description of Methods Used in Review Results Future Research and Policy Implications

OUTLINE

3

SOME BACKGROUND ON US EDUCATION

Persistent concern over the performance of US public schools at the elementary and secondary levels

Elementary Grades K-5 (ages 5-11)

Secondary Middle: Grades 6-8 (ages 11-14) High: Grades 9-12 (ages 14-18)

4

THE US SPENDS A LOT (PER PRIMARY SCHOOL PUPIL) ON EDUCATION, OBTAINS AVERAGE EDUCATIONAL OUTCOMES

Source: Gruber (2010)

5

THE US SPENDS ABOUT AVERAGE (% OF GDP) ON EDUCATION, OBTAINS AVERAGE EDUCATIONAL OUTCOMES

Source: OECD (2011)

6

IN THE US THE SCHOOL THAT A STUDENT ATTENDS IS PRIMARILY DETERMINED BY WHERE HE/SHE LIVES

San Diego Unified School District Elementary School Boundaries 2011-12

7

WHAT IS A CHARTER SCHOOL? Charter schools are a relatively new

alternative to traditional neighborhood public schools ~ 20 years, substantial growth in the 2000s

A succession of U.S. presidents has named charter schools as important agents of school reform

8

APPROXIMATELY 5% OF PUBLIC SCHOOLS ARE CHARTER SCHOOLS, THIS NUMBER IS GROWING

Source: Lake and Gross (2011)

9

WHAT IS A CHARTER SCHOOL? Charter schools are publicly funded,

governed by organization under contract with the state

Charter schools are exempted from parts of the state education code, freeing them to innovate with respect to curriculum, pedagogy and hiring of teachers

10

CHARTER SCHOOLS ARE DIFFERENT FROM EACH OTHER, EXAMPLES FROM SAN DIEGO

Albert Einstein Academy: “independent charter school that would have

a dual instructional focus of German-English immersion within the context of a rigorous academic instructional model”

Charter School of San Diego: initially developed from a state bill

“designed to reduce the dropout rate by recovering students who had been out of school for more than 45 days”

11

SELECTING STUDIES FOR THIS LITERATURE REVIEW

Scope: Include studies of US elementary and secondary charter school performance US public K-12 education is decentralized Most data on student performance are collected at

the level of a US state, or the level of a school district (smaller than a US state)

Outcomes: Include studies that use student performance on math and reading standardized tests as an outcome measure

Methods: Include studies that use credible approaches to address selection bias

12

SELECTION BIAS: MAIN CONCERNS WITH ALTERNATIVE APPROACHES LEADING TO EXCLUSION

Snapshots of average student achievement at one point in time can be misleading as they do not account for self-selection into schools US school attendance based largely

on geographic residence. Students choosing to attend charter

schools are likely different in observable and unobservable ways

13

UNOBSERVED CHARACTERISTICS CORRELATED WITH CHARTER SCHOOL ATTENDANCE

Negative selection (downward bias) Example: An underprivileged, disadvantaged

student without family support is at high risk of dropping out of school. She is advised by her high school counseling staff to transfer to a charter school, and she chooses to transfer.

Problem: Underprivileged, disadvantaged students without family support are not likely to obtain high test scores in any school, traditional or charter.

The estimate of charter school effectiveness based on comparison of charter school student performance and traditional school student performance would be biased downwards.

14

UNOBSERVED CHARACTERISTICS CORRELATED WITH CHARTER SCHOOL ATTENDANCE

Positive selection (upward bias) Example: An active, concerned, involved parent

is dissatisfied with the traditional public school in his/her neighborhood. The parent decides to opt-out of the traditional school and enroll his/her child in a charter school.

Problem: Students with active, concerned, involved parents are likely to obtain high test scores in any school, traditional or charter.

Implication: The estimate of charter school effectiveness based on comparison of charter school student performance and traditional school student performance would be upwardly biased.

15

SELECTING STUDIES FOR THIS LITERATURE REVIEW

National Charter School Research Project issued a White Paper (drafters: Betts and Hill, 2006) arguing that lottery-based studies and student-level longitudinal “value-added” studies were the two most credible approaches

These methods more convincing than other methods.

16

METHODS MATTER

Source: Hill (2006)

17

4 COMMONLY USED METHODS OF ANALYSIS IN THE INCLUDED STUDIES

In the set of studies we include, there are four approaches used

1) Lottery-based studies 2) Fixed-effect studies, that compare a

student’s gains in achievement in years attended a charter to his or her gains in years attended a traditional public school

3) Propensity score matching 4) Other types of matching (e.g. CREDO)

18

LOTTERY-BASED ANALYSIS

Source: Waiting for Superman movie (2010)

19

LOTTERY-BASED ANALYSIS Obvious benefit: expected outcomes

identical for lottery winners and losers if lottery conducted fairly

But several weaknesses to this “gold standard”

External validity Most charter schools not oversubscribed

Mathematica study of charter middle schools: only 130/492 oversubscribed

Could be bias from attrition

20

PROPENSITY SCORE MATCHING Assumes “selection on observables” If students in charter schools have unobserved

variations in ability or motivation, will be biased

Two major studies of KIPP (Knowledge is Power Program) schools have used this approach

CREDO at Stanford has produced string of influential state-level studies. Uses a unique matching process. Not propensity score but has similar issue with “selection on observables”

21

STUDENT FIXED-EFFECTS Benefit: Avoids need to compare one student with another,

instead comparing individual students’ trajectories in charter schools and traditional public schools

But many elementary students never switch between the two types of schools – external validity issue Zimmer et al (2009) compare test-score gains of charter “stayers”

and switchers and do not get clear-cut result. But in some cases “stayers” have higher test-score gains

Suggests downward bias from using this method Zimmer et al (2009) also raise concerns about reversibility –

are the effects of attending a charter dependent on the order in which a student attends the charter and the traditional public school? Find some evidence that this is the case.

Unobserved heterogeneity may change over time. Fixed effects cannot solve

22

INCLUDED STUDIES 40 reports now available, with just under

100 estimates of effects for each of math and English Language Arts (reading)

Lottery-based studies still quite rare: still only 8 papers that use lotteries, covering 90 charter schools

We exclude studies using less rigorous methods, specifically, those that do not use student-level test score gains as outcomes.

23

CHALLENGES IN STUDY COLLECTION/REVIEW PROCESS

Handling large weight (large number of students and large number of schools) studies Solution: Analyze with and without large weight studies

Handling the different methods used in different studies Solution: Investigate whether method of analysis matters

Some reports omit important information, e.g. number of schools in the sample Solution: Email exchange with authors

24

Introduction and Motivation Assessment of Alternative Methods of


Selecting Studies to Include Challenges in Study Collection/Review

Process Description of Methods Used in

Review Results Future Research and Policy Implications

25

OUR METHODS OF ANALYSIS Fisher test – Is there evidence that no study finds

negative effects; conversely, evidence of no positive effects?

Formal meta-analysis provides overall estimated effect, its statistical significance and measures of how much true underlying variation there is across studies

Histograms Show variability and the influence of weighting of studies

Vote-counting as a way of assessing variation in results

26

HETEROGENEITY IS AN UNDERLYING THEME

Look for variations in effect by: Subject area tested (math vs. reading) Grade span (E, M, H) Geographic location KIPP vs. non-KIPP Is there a systematic difference in results

based on the method researchers use?

27

METHODS USED IN REVIEW Testing Whether Charter Schools in

Any Study Increase or Decrease Achievement Relative to Traditional Public Schools

Meta-Analysis of Effect Size Histograms and Vote Counting as

Measures of Variation

28

METHOD #1: EVIDENCE OF NO POSITIVE EFFECTS, OR NO NEGATIVE EFFECTS?

Fisher’s combined test

S is distributed with df=2k Null hypothesis: No positive effects Null hypothesis: No negative effects€

S = −2 ln(pi)i=1

k

∑

€

χ2

29

METHOD #1: EVIDENCE OF NO POSITIVE EFFECTS, OR NO NEGATIVE EFFECTS?

We conduct this analysis 12 times: 6 ways of combining grades, and two subjects (math and ELA)

First sign of heterogeneous effects of charter schools: in 9/12 cases there is clear evidence of BOTH negative and positive effects

Three exceptions with evidence of positive effects but no evidence of negative effects: elementary and middle school ELA scores, and middle school math scores

PROBABILITY OF NO POSITIVE EFFECTS IN ANY OF THE STUDIES: ALMOST ZERO

Grade-Span Reading Tests Math TestsElementary <0.001 <0.001

Middle <0.001 <0.001High <0.001 0.001

El’y, Middle, and Combined El’y/Middle

<0.001 <0.001

All <0.001 <0.001

Studies of All Grades or Largest Grade Span(s) If An

All-Grade Study Not Available

<0.001 <0.001

30

PROBABILITY OF NO NEGATIVE EFFECTS IN ANY OF THE STUDIES: ALMOST ZERO IN MOST CASES, AND QUITE HIGH IN 3 CASES

Grade-Span Reading Tests Math TestsElementary 0.987 <0.001

Middle 0.994 0.978High <0.001 0.001

El’y, Middle, and Combined El’y/Middle

<0.001 <0.001

All <0.001 <0.001

Studies of All Grades or Largest Grade Span(s) If An

All-Grade Study Not Available

<0.001 <0.001

31

32

METHODS USED IN REVIEW Testing Whether Charter Schools in Any

Study Increase or Decrease Achievement Relative to Traditional Public Schools



33

METHOD #2: FORMAL META-ANALYSIS Assume charter school estimates are

randomly distributed Therefore it is important to estimate both the

mean and the variation Underlying “true” variation across studies is

the extent to which variation cannot be explaining by sampling error (“uncertainty”) in individual estimates

Omitted many studies of individual KIPP schools as they would have disproportionate influence Include KIPP schools in subsidiary analysis

34

THE MEAN EFFECT IS A WEIGHTED AVERAGE

In a random effects meta-analysis, we take a weighted average of the effect sizes across studies. If Yi is the effect size for the ith of k studies, and Wi is the weight for each study, our overall estimated effect size M is :

(1)

1

1

k

i iik

ii

WYM

W

35

WEIGHTS DEPEND ON WITHIN-STUDY VARIANCE AND ESTIMATED ACROSS-STUDY (TRUE) VARIANCE The weight for each study is the inverse of the sum of the

within-study variance (based on the standard error) and an estimate of the true between-study variance, T2:

(2)

T2 based on a method of moments estimate of the variance of true effect sizes.

Note that as T2 becomes large relative to the average within-study variance estimate, then we will tend toward equal weighting across studies; whereas as T2 becomes relatively small, the weights can become highly unequal with heavier weight given to studies with the lowest sampling variance.

2

1

i

iY

WV T

36

AN ESTIMATE OF WHAT % OF THE VARIANCE ACROSS STUDIES IS TRUE

Use the I2 statistic (Higgins et al., 2003) Provides estimate of the percentage of

variation across studies that reflects true underlying variation

37

SAMPLE OF OUR RESULTS ON EFFECT SIZES

* Indicates statistically significant (5% level)

Grade Span Reading Tests Math Tests

E (Elementary) 0.022* (9-7), 77.7%

0.049* (10-8), 94.7%

38

SAMPLE OF OUR RESULTS



E (Elementary) 0.022* (9-7), 77.7%

0.049* (10-8), 94.7%

“On average, attending a charter school is associated with an increase in test scores in reading equal to 0.022 of a standard deviation per year.”

39

SAMPLE OF OUR RESULTS



E (Elementary) 0.022* (9-7), 77.7%

0.049* (10-8), 94.7%

Nine studies covering 7 geographic areas77.7% of the variation across studies

represents true variation in charter school effects, rather than “noise”

40

OVERALL EFFECT SIZE ESTIMATESGrade Span Reading Tests Math Tests

E (Elementary) 0.022* (9-7), 77.7%

0.049* (10-8), 94.7%

M (Middle) 0.011 (9-7), 85.7%

0.055* (10-8), 92.0%

H (High) 0.054 (7-5), 98.3%

-0.015 (8-6), 98.6%

Combined E/M -0.009 (15-12), 93.4%

-0.012 (15-12), 97.9%

E, M, and Combined E/M

0.002 (31-17), 90.3%

0.020* (33-18), 96.8%

All 0.008 (17-14), 98.4%

0.014 (18-15), 97.7%

41

ELEMENTARY/MIDDLE SCHOOL MATH EFFECTS: MEANINGFUL BUT NOT HUGE

Enough to move a student at the 50th percentile to the 52nd percentile after attending a charter for one year

Elementary school reading impact is smaller: enough to boost a student from 50th to about percentile 50.8

42

ELEMENTARY SCHOOL READING EFFECT SIZES

NOTE: Weights are from random effects analysis

Overall (I-squared = 77.7%, p = 0.000)

San Diego

Chicago

San Diego

National

California

NYC

Delaware

Study

NYC

Boston

ID

0.02 (0.01, 0.04)

0.04 (-0.01, 0.09)

0.10 (0.03, 0.18)

-0.08 (-0.17, 0.01)

0.01 (0.01, 0.01)

-0.00 (-0.01, 0.00)

0.19 (0.02, 0.35)

0.03 (0.00, 0.07)

0.04 (0.01, 0.07)

0.06 (0.01, 0.10)

ES (95% CI)

100.00

6.80

3.70

2.61

27.01

25.00

0.88

12.45

%

12.83

8.73

Weight

0.02 (0.01, 0.04)

0.04 (-0.01, 0.09)

0.10 (0.03, 0.18)

-0.08 (-0.17, 0.01)

0.01 (0.01, 0.01)

-0.00 (-0.01, 0.00)

0.19 (0.02, 0.35)

0.03 (0.00, 0.07)

0.04 (0.01, 0.07)

0.06 (0.01, 0.10)

ES (95% CI)

100.00

6.80

3.70

2.61

27.01

25.00

0.88

12.45

%

12.83

8.73

Weight

0-.3 -.2 -.1 .1 .2 .3

43

ELEMENTARY SCHOOL MATH EFFECT SIZES



San Diego

San Diego

Chicago

Idaho

ID

Boston

NYC

NYC

National

Delaware

California

Study

0.05 (0.02, 0.08)

0.29 (0.22, 0.37)

-0.19 (-0.30, -0.08)

0.12 (0.04, 0.19)

0.33 (0.03, 0.63)

ES (95% CI)

0.02 (-0.03, 0.07)

0.19 (0.02, 0.36)

0.09 (0.06, 0.12)

-0.00 (-0.00, 0.00)

0.04 (0.01, 0.07)

-0.03 (-0.04, -0.02)

100.00

8.88

5.89

8.51

1.10

Weight

11.44

2.89

14.29

16.50

14.25

16.26

%

0.05 (0.02, 0.08)

0.29 (0.22, 0.37)

-0.19 (-0.30, -0.08)

0.12 (0.04, 0.19)

0.33 (0.03, 0.63)

ES (95% CI)

0.02 (-0.03, 0.07)

0.19 (0.02, 0.36)

0.09 (0.06, 0.12)

-0.00 (-0.00, 0.00)

0.04 (0.01, 0.07)

-0.03 (-0.04, -0.02)

100.00

8.88

5.89

8.51

1.10

Weight

11.44

2.89

14.29

16.50

14.25

16.26

%

0-.3 -.2 -.1 .1 .2 .3 .4

44

MIDDLE SCHOOL READING EFFECT SIZES



Chicago

NYC

ID

San Diego

Delaware

Texas

National

Boston

National

San Diego

Study

0.01 (-0.02, 0.04)

-0.06 (-0.14, 0.01)

0.04 (-0.02, 0.10)

ES (95% CI)

-0.08 (-0.12, -0.04)

0.08 (0.04, 0.12)

0.01 (-0.01, 0.04)

-0.10 (-0.23, 0.03)

0.17 (0.07, 0.27)

0.02 (0.02, 0.02)

0.01 (-0.04, 0.06)

100.00

8.49

10.00

Weight

13.69

13.58

16.06

4.21

5.65

17.37

10.94

%

0.01 (-0.02, 0.04)

-0.06 (-0.14, 0.01)

0.04 (-0.02, 0.10)

ES (95% CI)

-0.08 (-0.12, -0.04)

0.08 (0.04, 0.12)

0.01 (-0.01, 0.04)

-0.10 (-0.23, 0.03)

0.17 (0.07, 0.27)

0.02 (0.02, 0.02)

0.01 (-0.04, 0.06)

100.00

8.49

10.00

Weight

13.69

13.58

16.06

4.21

5.65

17.37

10.94

%

0-.3 -.2 -.1 .1 .2 .3

45

MIDDLE SCHOOL MATH EFFECT SIZES



ID

Boston

Texas

San Diego

Chicago

National

National

Idaho

Delaware

Study

NYC

San Diego

0.05 (0.01, 0.10)

ES (95% CI)

0.54 (0.39, 0.69)

-0.00 (-0.02, 0.02)

0.01 (-0.09, 0.11)

-0.09 (-0.16, -0.02)

-0.08 (-0.20, 0.04)

0.02 (0.02, 0.02)

-0.05 (-0.18, 0.08)

0.09 (0.05, 0.13)

0.24 (0.16, 0.31)

0.06 (0.03, 0.10)

100.00

Weight

4.81

14.17

7.90

10.32

6.31

14.66

5.88

13.10

%

9.70

13.15

0.05 (0.01, 0.10)

ES (95% CI)

0.54 (0.39, 0.69)

-0.00 (-0.02, 0.02)

0.01 (-0.09, 0.11)

-0.09 (-0.16, -0.02)

-0.08 (-0.20, 0.04)

0.02 (0.02, 0.02)

-0.05 (-0.18, 0.08)

0.09 (0.05, 0.13)

0.24 (0.16, 0.31)

0.06 (0.03, 0.10)

100.00

Weight

4.81

14.17

7.90

10.32

6.31

14.66

5.88

13.10

%

9.70

13.15

0-.3 -.2 -.1 .1 .2 .3

46

HIGH SCHOOL READING EFFECT SIZES



National

San Diego

San Diego

Texas

San Diego

Study

ID

Boston

Delaware

0.05 (-0.03, 0.14)

-0.02 (-0.02, -0.02)

0.15 (0.10, 0.20)

0.04 (-0.24, 0.33)

-0.16 (-0.18, -0.14)

0.03 (-0.01, 0.07)

ES (95% CI)

0.16 (0.02, 0.31)

0.21 (0.16, 0.26)

100.00

16.98

15.97

6.09

16.84

16.33

%

Weight

11.60

16.18

0.05 (-0.03, 0.14)

-0.02 (-0.02, -0.02)

0.15 (0.10, 0.20)

0.04 (-0.24, 0.33)

-0.16 (-0.18, -0.14)

0.03 (-0.01, 0.07)

ES (95% CI)

0.16 (0.02, 0.31)

0.21 (0.16, 0.26)

100.00

16.98

15.97

6.09

16.84

16.33

%

Weight

11.60

16.18

0-.3 -.2 -.1 .1 .2 .3

47

HIGH SCHOOL MATH EFFECT SIZES



National

San Diego

San Diego

Texas

San Diego

Study

ID

Boston

Delaware

0.05 (-0.03, 0.14)

-0.02 (-0.02, -0.02)

0.15 (0.10, 0.20)

0.04 (-0.24, 0.33)

-0.16 (-0.18, -0.14)

0.03 (-0.01, 0.07)

ES (95% CI)

0.16 (0.02, 0.31)

0.21 (0.16, 0.26)

100.00

16.98

15.97

6.09

16.84

16.33

%

Weight

11.60

16.18

0.05 (-0.03, 0.14)

-0.02 (-0.02, -0.02)

0.15 (0.10, 0.20)

0.04 (-0.24, 0.33)

-0.16 (-0.18, -0.14)

0.03 (-0.01, 0.07)

ES (95% CI)

0.16 (0.02, 0.31)

0.21 (0.16, 0.26)

100.00

16.98

15.97

6.09

16.84

16.33

%

Weight

11.60

16.18

0-.3 -.2 -.1 .1 .2 .3

48

READING EFFECT SIZES FOR STUDIES THAT COMBINE ELEMENTARY AND MIDDLE SCHOOLS



ID

Texas

DC

NYC

Ohio

Chicago

Chicago

Texas

North Carolina

Massachusetts

Georgia

Missouri

Minnesota

Ohio

Arkansas

Arizona

Study

-0.01 (-0.02, 0.00)

ES (95% CI)

-0.08 (-0.10, -0.06)

-0.01 (-0.02, 0.01)

0.09 (0.02, 0.16)

-0.08 (-0.12, -0.04)

0.00 (-0.01, 0.01)

-0.04 (-0.06, -0.02)

0.09 (0.06, 0.12)

-0.09 (-0.12, -0.07)

0.00 (-0.01, 0.02)

0.01 (-0.00, 0.01)

0.03 (0.01, 0.05)

-0.02 (-0.03, -0.01)

-0.00 (-0.01, 0.00)

0.02 (0.00, 0.04)

-0.01 (-0.02, -0.01)

100.00

Weight

6.85

7.10

2.57

4.74

7.64

6.85

5.61

6.00

7.50

7.86

7.05

7.46

7.80

7.00

7.96

%

-0.01 (-0.02, 0.00)

ES (95% CI)

-0.08 (-0.10, -0.06)

-0.01 (-0.02, 0.01)

0.09 (0.02, 0.16)

-0.08 (-0.12, -0.04)

0.00 (-0.01, 0.01)

-0.04 (-0.06, -0.02)

0.09 (0.06, 0.12)

-0.09 (-0.12, -0.07)

0.00 (-0.01, 0.02)

0.01 (-0.00, 0.01)

0.03 (0.01, 0.05)

-0.02 (-0.03, -0.01)

-0.00 (-0.01, 0.00)

0.02 (0.00, 0.04)

-0.01 (-0.02, -0.01)

100.00

Weight

6.85

7.10

2.57

4.74

7.64

6.85

5.61

6.00

7.50

7.86

7.05

7.46

7.80

7.00

7.96

%

0-.3 -.2 -.1 .1 .2 .3

49

MATH EFFECT SIZES FOR STUDIES THAT COMBINE ELEMENTARY AND MIDDLE SCHOOLS



Texas

Missouri

ID

Ohio

North Carolina

NYC

Minnesota

Chicago

Georgia

Study

Arkansas

DC

Massachusetts

Chicago

Texas

Arizona

Ohio

-0.01 (-0.03, 0.01)

-0.12 (-0.16, -0.08)

0.03 (0.01, 0.04)

ES (95% CI)

-0.06 (-0.07, -0.05)

-0.16 (-0.20, -0.12)

0.12 (0.03, 0.21)

-0.03 (-0.04, -0.02)

0.02 (-0.02, 0.06)

-0.01 (-0.02, -0.00)

0.05 (0.03, 0.07)

0.01 (-0.00, 0.03)

0.06 (0.05, 0.07)

0.02 (0.01, 0.03)

0.08 (0.06, 0.11)

-0.04 (-0.05, -0.04)

-0.18 (-0.26, -0.10)

100.00

6.23

7.24

Weight

7.56

6.11

3.43

7.44

6.23

7.59

%

7.20

7.36

7.48

7.53

7.00

7.60

4.01

-0.01 (-0.03, 0.01)

-0.12 (-0.16, -0.08)

0.03 (0.01, 0.04)

ES (95% CI)

-0.06 (-0.07, -0.05)

-0.16 (-0.20, -0.12)

0.12 (0.03, 0.21)

-0.03 (-0.04, -0.02)

0.02 (-0.02, 0.06)

-0.01 (-0.02, -0.00)

0.05 (0.03, 0.07)

0.01 (-0.00, 0.03)

0.06 (0.05, 0.07)

0.02 (0.01, 0.03)

0.08 (0.06, 0.11)

-0.04 (-0.05, -0.04)

-0.18 (-0.26, -0.10)

100.00

6.23

7.24

Weight

7.56

6.11

3.43

7.44

6.23

7.59

%

7.20

7.36

7.48

7.53

7.00

7.60

4.01

0-.3 -.2 -.1 .1 .2 .3

50

METHODS USED IN REVIEW Testing Whether Charter Schools in Any

Study Increase or Decrease Achievement Relative to Traditional Public Schools



51

METHOD #3: HISTOGRAMS Another way of displaying the variation

across studies Tried weighting each study equally and

weighting studies by number of observations Latter approach gives heavy weight to CREDO

studies Our formal meta-analysis is closer to

weighting studies equally than weighting by observation

54

METHOD #4: VOTE COUNTING Categorize studies by sign of effect and whether

statistically significant Method is problematic because it ignores fact that

many statistically insignificant results all in the same direction may, taken together, suggest a statistically significant result

We use mostly to highlight the heterogeneity Typically find that for most grade spans >50% of

studies show positive effects, but this weakens and sometimes reverses if we weight studies by number of observations

55

RESULTS VARY BY METHOD Lottery results yielded the most positive

results, followed closely by propensity score matching.

These were followed by fixed effects and other matching methods (which are fairly similar with mixed positive and negative results)

But it may not be the method that matters quite so much as the specific schools studied Example: Propensity score results are large but

focus on KIPP schools

56

RESULTS VARY BY METHOD

57

RESULTS VARY BY METHOD

58

REPLICATION OF RESULTS USING DIFFERENT METHODS

There are 3 studies/pairs of studies that replicate lottery results using more traditional “value-added” methods

They generally suggest that value-added models can get close to the lottery results (but in a few cases estimates slightly to meaningfully lower): Boston (Abulkadiroglu et al.) New York (Hoxby et al. and CREDO) San Diego Preuss School (McLure et al., Betts, Tang

and Zau)

59

REPLICATION OF RESULTS USING DIFFERENT METHODS

There are 3 studies/pairs of studies that replicate lottery results using more traditional “value-added” methods

They generally suggest that value-added models can get close to the lottery results (but in a few cases estimates slightly to meaningfully lower): Boston (Abulkadiroglu et al.) New York (Hoxby et al. and CREDO) San Diego Preuss School (McLure et al., Betts, Tang

and Zau)

60

Introduction and Motivation Selecting Studies to Include Assessment of Alternative Methods of


Challenges in Study Collection/Review Process

Description of Methods Used in Review Results Future Research and Policy

Implications

61

IMPLICATIONS FOR RESEARCH

Evaluate individual schools Charters are meant to innovate; unlikely

that all charters will have the same impact

Charters should obtain permission from applicants to gather student records

States and chartering authorities should regularly receive lottery data

62


Focus on successful schools to identify characteristics that may be working E.g. longer day/time, student population targeted,

discipline policies, teacher management

Obtain more details about charter school heterogeneity and study them

Obtain more details about charter school closures

63


Probably important to examine more than results on math and ELA achievement.

A handful of studies point to positive charter effects on graduation, college attendance and behavior. Expand focus to include outcomes other

than math/reading test scores

64

WHAT WORKS CLEARINGHOUSE (WWC) FOR CHARTER SCHOOL RESULTS

In the long run it would be good to have a non-partisan group that collected and interpreted school-level charter results.

65

IMPLICATIONS FOR POLICY

Status as a charter school vs. traditional public school unlikely to be (on its own) meaningful Promoting charter schools for sake of charter

schools probably not productive path to comprehensive reform

Continue expansion (no particular reason not to) Still only ~5% of traditional public school sector Renew focus on traditional public school reform

Exploit flexibility of charter schools by using them as laboratories to learn what works

66

THANK YOU! Published version available at: http://www.crpe.org/cs/crpe/view/csr_pubs/467 Executive summary at: http://www.crpe.org/cs/crpe/view/csr_pubs/468

http://www.crpe.org/cs/crpe/view/csr_pubs/467

http://www.crpe.org/cs/crpe/view/csr_pubs/468

67

SUPPLEMENTARY SLIDES

68

ADDING KIPP STUDIES BACK IN HAS BIG EFFECT


Including KIPP Schools

M 0.070* (38-33), 88.3%

0.180* (39-34), 96.8%


0.034* (60-43), 90.8%

0.105* (62-44), 98.6%

Results Including Only KIPP Estimates

M 0.096* (29-unknown),

82.7%

0.223* (29-unknown),

93.7%

69

SENSITIVITY TO EXCLUSION OF CREDO STUDIES

CREDO (Stanford) has produced impressive string of mostly state-wide longitudinal student studies. Match each charter student to an average of several similar demographics and test scores

Many charter students are matched based on their test scores AFTER they enter charter schools potential bias

Hoxby (2009) has concerns about measurement error that may bias charter coefficient down CREDO offers partial rebuttal

70

RESULTS SOMEWHAT STRONGER IF OMIT CREDO STUDIES


E 0.034* (8-6), 79.5%

0.072* (9-7), 95.2%

M 0.010 (8-7), 87.2%

0.068* (9-8), 92.8%

H 0.072 (6-4), 98.5%

-0.002 (7-5), 97.5%

Combined E/M -0.023 (6-5), 95.5%

-0.041 (6-5), 96.9%


0.008 (22-10), 92.0%

0.038* (24-11), 95.0%

All 0.016 (10-9), 86.6%

0.041* (11-10), 67.7%

71

EFFECTS FOR URBAN DISTRICTS AND SCHOOLS LARGER THAN FOR ALL DISTRICTS


E 0.046* (6-4), 61.8%

0.085 (6-4), 92.2%

M 0.009 (5-4), 87.0%

0.139 (5-4), 94.8%

H 0.101* (4-2), 78.2%

0.019 (4-2), 42.7%

Combined E/M -0.003 (4-3), 86.2%

0.021* (4-3), 47.7%


0.016 (15-5), 84.1%

0.077* (15-5), 92.4%

All 0.008 (8-6), 63.2%

0.045* (8-6), 74.8%

72

VARIATIONS BY RACE/ETHNICITY Surprisingly few studies test for variation

by race/ethnicity. CREDO studies an important exception

Patterns not uniform, but overall, charter effects decline in the following order: black > Hispanic > Native American > white Results for whites typically negative, not always

significant. Biggest exception is high school reading, with a positive and significant effect

73

VARIATIONS BY ENGLISH LEARNER, SPECIAL EDUCATION, MEAL ASSISTANCE

Effects often insignificant, perhaps due to smaller sample sizes

But some evidence of positive effects of charter schools on EL and special education students in both math and reading from studies of all grades and studies of middle schools

julian r. betts and y. emily tang, university of california, san diego

Documents

state charter schools

effect of charter schools

independent charter

percentage of public

teachers99charter schools

public school prices

school district smaller

primary school pupil