![Page 1: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/1.jpg)
Magnet Schools and Peers: Effects on Student Achievement
Dale Ballou
Vanderbilt University
November, 2007
Thanks to Steve Rivkin, Julie Berry Cullen, Adam Gamoran, Ellen Goldring and Keke Liu.
![Page 2: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/2.jpg)
Research Questions
• Has attending a magnet school caused an increase in mathematics achievement?
• How large is the influence of peers on mathematics achievement?
• How much of the magnet school effect remains after controlling for the influence of peers?
![Page 3: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/3.jpg)
Study Setting
• Middle Schools in a Large Southern District
1 selective academic magnet
4 non-selective magnets
5 student cohorts
6 years: 1998-99 through 2003-04
Grades 5 & 6
![Page 4: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/4.jpg)
Admissions Lotteries
• Oversubscribed magnets conduct lotteries
• Students may enter multiple lotteries
• Students who are not outright winners are placed on wait lists
• Wait-listed students accepted until the first week of school
![Page 5: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/5.jpg)
Non-lottery Admissions
• Sibling preferences• Promotion from a feeder school• Geographic priority zone
These students are not included in the study sample (though they do enter the calculation of peer characteristics).
![Page 6: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/6.jpg)
Research Design
• Lotteries assign students randomly to school type and to peers (magnet school peers vs. non-magnet peers).
• Randomized design circumvents biases arising from self-selection of schools and peers.
![Page 7: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/7.jpg)
Limitations of Design
• Results may not generalize beyond lottery participants.
• Effects are relative (magnet schools vs. mix of non-magnet schools attended by lottery losers).
![Page 8: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/8.jpg)
Lottery Participation Academic Composite
Non-Academic
Applicants 2315 2594
Outright Winners
883 1450
Delayed Winners
223 756
Losers,This Lottery
1209 388
Losers, All Lotteries
539 199
![Page 9: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/9.jpg)
Grade 5 Enrollments
Academic Composite Non-Academic
This Magnet 758 1061
Other Magnets 287 339
Non-Magnets 834 846
Left system or Not Tested
436 346
![Page 10: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/10.jpg)
• Substantial non-compliance, especially among winners of non-academic lotteries, attenuates estimated treatment & peer effects based on comparison of winners and losers.
Remedy: Use lottery outcomes as instruments to predict probability of attending magnet school, outcomes interacted with peer variables at magnet & zoned schools as instruments for peer characteristics.
Potential Pitfalls
![Page 11: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/11.jpg)
• High rates of attrition from district can introduce systematic differences between treatment and control groups.
Remedies:Control for student characteristics (race, income, ESL, special ed, gender, prior achievement).Analyze attrition patterns for evidence of differences between winners and losers.
![Page 12: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/12.jpg)
• Participating in multiple lotteries increases chances of winning. “Multiple participants” may differ in ways related to achievement.
Remedy: Control for the combination of lotteries each student entered. Winners are compared to losers who entered the same combination.
![Page 13: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/13.jpg)
• Lotteries randomly assign students to magnet school peers or peers in their neighborhood (zoned) school, but lotteries do not determine the characteristics of the latter—residential decisions do.
Remedy: Control for characteristics of the peers in the zoned school.
![Page 14: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/14.jpg)
Peer Characteristics
• Percentages black, low income (free & reduced-price lunch program), special ed, ESL, female
• Absenteeism rate
• Disciplinary incidents (rate per student)
• Intra-year mobility
• Prior achievement in math and reading
![Page 15: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/15.jpg)
Model (Summary)
• Two treatment variables (academic magnet, composite non-academic magnet)
• Variation in peers resulting from lottery outcomes
• Other controls (student characteristics, peers at the zoned school, lottery participation indicators, year by grade effects)
![Page 16: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/16.jpg)
Findings
• When model does not include peer characteristics
- Academic magnet, + 18% in grade 5, drops to +10% in grade 6 (% of normal year growth)
- Non-academic magnet, no grade 5 effect, +54% in grade 6
![Page 17: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/17.jpg)
• When models include peer characteristics- Reducing percent black from 75% to 25% increases scores by 60% of normal year growth.- Effect of percent low income is about half that large.- Other peer characteristics have no statistically significant effect.
![Page 18: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/18.jpg)
- Controlling for either percent black or percent low income, the effect of the academic magnet disappears.
- The large 6th grade effect in the non-academic magnets remains substantially undiminished.
![Page 19: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/19.jpg)
Checking Alternative Interpretations
• Are peers a proxy for heterogeneous response to treatment?
Check: Interact magnet treatment indicators with all observed student characteristics.
Finding: Peer effects are undiminished.
![Page 20: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/20.jpg)
• Are peers a proxy for teacher quality?
Check: Control for teacher quality by including teacher fixed effects.
Finding: Peer effects are undiminished.
![Page 21: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/21.jpg)
Attrition, Academic Magnet
Lottery Participants
Left System After Grade: Winners Losers
4 13% 21%
5 8% 14%
6 9% 11%
7 6% 9%
![Page 22: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/22.jpg)
Attrition, Composite Non-Academic Magnet
Lottery Participants
Left System After Grade: Winners Losers
4 8% 12%
5 12% 9%
6 10% 4%
7 12% 16%
![Page 23: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/23.jpg)
Potential Attrition Biases
• Lottery losers are more likely to leave the system than winners.
• Losers are also more likely to leave when they can afford private schooling. These tend to be higher-achieving students.
Result: Losers who remain in the system have lower achievement than winners who remain.
![Page 24: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/24.jpg)
• Unfavorable peers at zoned school make losers more likely to leave system.
• Effect greatest among those who can afford private schooling.
Result: Quality of peers positively correlated with losers’ achievement. Estimated peer effects appear too strong.
![Page 25: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/25.jpg)
Checking Attrition Bias
• Are rates of attrition correlated with variables that predict individual achievement (race, income, prior achievement)?
• Yes, but not differently for winners and losers.
![Page 26: Magnet Schools and Peers: Effects on Student Achievement Dale Ballou Vanderbilt University November, 2007 Thanks to Steve Rivkin, Julie Berry Cullen, Adam](https://reader034.vdocuments.us/reader034/viewer/2022042821/56649d555503460f94a3280b/html5/thumbnails/26.jpg)
Conclusions
• For at least some students in some places, magnet schools have a positive effect on academic achievement.
• There are very strong peer effects on middle school achievement. Do not appear to operate through behaviors readily quantified with administrative data (attendance, disruptions, mobility).