report cards: the impact of providing school and child test-scores on educational markets jishnu das...
TRANSCRIPT
Report Cards: The Impact of Providing School and Child Test-scores on Educational Markets
Jishnu Das (World Bank)
With:Tahir Andrabi (Pomona College)Asim Ijaz Khwaja (HKS, Harvard)
The Context
Private Schools have expanded dramatically since the 1990s in South Asia
61.6
34.9
2.7
.9
6334
.91.
21
66.2
28.3
3.1
2.4
87.5
8.1
3.9
.5
63.5
332.
31.
3
0
20
40
60
80
Pe
rce
nt
En
rolm
en
t
Punjab Sindh NWFP Balochistan Total
Govt
Privat
e
Other
Madra
ssa
Govt
Privat
e
Madra
ssa
Other
Govt
Privat
e
Other
Madra
ssa
Govt
Privat
e
Madra
ssa
Other
Govt
Privat
e
Other
Madra
ssa
Calculations by Nobuo Yoshida and Tomoyuki Sho (The World Bank)
PSLM (2005-06)
Enrolment in Pakistani Educational Institutions
Pakistan Rural India
The Context: Its not what you think!
Although much discussion about madrassas, this is not where the action is!
The number of new Madrassas added every year tapered off after 2000
0
2000
4000
6000
8000
1960 1970 1980 1990 2000 2010Year
Public Private Madrassa
By Year of Formation
New Educational Institutions in Pakistan
The Context: The “New” Village Environment
In our sample of 112 villages, there were 812 schools 50% of rural Pakistan’s most populous province—Punjab—
live in villages like one of these two
A village in Central Punjab A village in North Punjab
The Questions
Can better information about school performance take advantage of the market structure to improve educational outcomes?
What is the equilibrium impact of information on educational markets? Quality Price Quantity
Is there a strong case for the provision of better information?
Address these issues using an experimental design in Punjab province, Pakistan
Precursors
Information can lead to a number of different types of results! Positive: Information leads to greater accountability/verifiability, competition
(Bjorkman & Svenson (2008) – community-based health reporting/monitoring in Uganda; Jin & Leslie (2003) – Restaurant Hygiene cards in LA Rokoff and Turner (2008); Chiang (2008) – public school accountability in US; Hastings
(2007)
Nothing: Information may be known, not understood/credible/believed; Banerjee et al. (2007) - no learning improvements from information dissemination (Indian
state)
Negative: Information may lead to greater cream-skimming/sorting (winner takes all) Education : Chile (Urquiola and Mizala 2007) Health: Dranove et al. (2003) – hospital outcomes in NY Direct Manipulation: US – cheating teachers (Jacob & Levitt, 2003)
Gaps All market-level studies are observational (Dranove and others, Urquiola and Mizala,
Jin and Leslie. Experimental work thus far either in cases where markets are sparse or market
reactions not examined
First experimental equilibrium results on impact of information in education
Remainder of talk
A note on private schools The data The experiment The Results A note on the results
A note on private Schools
All unaided, very sparsely regulated, co-educational, mostly small “mom & pop” operations
Better learning than public schools
Learning
450
500
550
600
3 4 5 3 4 5 3 4 5
English
Math
Urdu
Kno
wle
dge
Sco
res
Government School Children in Classes 3, 4 & 5
Learning in government and private schools
Priv
ate
Sch
ools
in C
lass
3
Priv
ate
Sch
ools
in C
lass
3
Priv
ate
Sch
ools
in C
lass
3
450
500
550
600
3 4 5 3 4 5 3 4 5
English
Math
Urdu
Kno
wle
dge
Sco
res
Government School Children in Classes 3, 4 & 5
Learning in government and private schools
Priv
ate
Sch
ools
in C
lass
3
Priv
ate
Sch
ools
in C
lass
3
Priv
ate
Sch
ools
in C
lass
3
Probably causal differences
A note on Private Schools (II)
And cheaper, too!
260
184
259
75 6882
0
50
100
150
200
250
Co
st f
or
Eve
ry P
erc
en
t C
orr
ect
(in
Rs.
)
Government Private
Cost of Schooling
English Math Urdu
Data (I)
Sample: 112 villages from 3 districts in the Punjab, Pakistan Randomly selected from list of villages with at least one private
school in 2000 (3rd of villages & 50% of pop); somewhat bigger/richer than average village
Defining Schooling Markets: Goal – capture parents/children complete choice set & schools’
potential market Feasible:
92 % of children attend village school (HH census) Large distance effect – most primary-school children go within 15
minutes Create 15 minute (30m for RYK) boundaries around village HHs
(include some schools right outside village boundary) (Figure 3)
LEAPs Project - www.leapsproject.org : Baseline HH census (80,000) Four Rounds (2004-2007):
School (823) Questionnaires: General School Questionnaire Class Teacher Questionnaire Head Teacher Questionnaire
Educational Performance: Child-Tests (Follow 12,000 plus children over 4
years) in English, Urdu and Mathematics
Household-Level Information Detailed Household Interviews for randomly
selected HHs (1,800) Short school-based Child questionnaire (randomly
select 10 in each school)
Child
Administration Teacher
School
Family
Attainment
Data (II)
Data (III)
Survey Instruments & Timeline: HH census (80,000 hhs) – 2003 Round 1 (Baseline):
School-Based (Jan-Feb 04): (i) 823 primary schools + class 3 teachers; (ii) 800+ Class 3 teachers; (iii) 6,000 class 3 kids (brief info)
HH-Based (March-Apr 04): Detailed HH surveys (1,800); part matched on class 3 children
Child-Tests (Jan-Feb 04): 12,000+ Class 3 children – Norm-referenced test to maximize variation – Use Item Response Theory to get at underlying child knowledge; we administer (minimize cheating etc.)
Report Card Intervention – Sept/Oct Round 2 (2005): Report all of Round 1 Surveys/Tests
(96% children tracked)
The Experiment After baseline, villages within each of 3 districts divided with equal probability into
treatment and control
Report Card provided to each Class 3 kid parent in school-meeting – explain scores
Parent Card 1: Child Info
In all 3 subjects (Maths, Urdu & English): Child score and quintile Child’s School score &
quintile Child’s village score and
quintile
Quintile described as “needing a lot of work” to “very good”
The Report Card Intervention
Parent Card 2: Village Schools Info
For all Primary schools in villages give : School Name Tested Children School scores and
quintiles in all 3 subjects
“Bundled-Impact”: Information (child,
schools) Increase precision,
verifiability Meeting effect?
A note on the information
This is not value-added information Why?
Feasible intervention Theoretical considerations (who can back out
VA better?) Empirically doesn’t look too bad
Nevertheless, combination of selection and measurement error may lead to erroneous inference by parents
A further note on the information Reliability vs. Measurement
Error (Kane & Staiger) Information is fairly reliable
Low measurement error of test
Large variation across schools - see Figure
Selection (into schools) Value-added estimates? Selection Not as severe (see
learning gaps Figure) Need to have Information be
clear and understandable Policy feasible/relevant
Households may be better able to back out value-added
-4-2
02
sch
scor
e
0 5 10 15School ID
village w/ 15 schools; test-score & (2) standard-error bands (computed using IRT)
Family Rich-Poor
Father Literate-Illiterate
Mother Literate-Illiterate
Child Female-Male
Private-Public
0 50 100 150 0 50 100 150 0 50 100 150
English Math Urdu
Size of Adjusted Gap
Learning Gaps
What should we expect? 3 Broad Classes of models Symmetric information
Some unobservable components of quality for both schools and households
Asymmetric information: Price signals quality Asymmetric information: Price does not signal
quality
In Model 1 price declines for all schools; depending on structure of demand can get heterogeneous declines by initial quality; quality weakly improves
In Model 2 price declines more for initially higher performing schools; quality weakly improves
In Model 3 price/quality movements are ambiguous
Results: Quality
Village-level Average
Child-Level Average
Child-Level Average (No switchers)
Report Card Villages
0.114
(0.045)
0.095
(0.038)
0.102
(0.038)
Good Private School
0.039
(0.516)
Bad Private School
0.347
(0.019)
Government Schools
0.088
(0.053)
Notes
Learning:
Similar across subjects; holds 2 yrs after
Attrition:
Unlikely concern: no difference in baseline scores for attritors between treatment and control samples
Switching/Dropouts:
Results entirely driven by children who stayed in same school:
Few Switch Schools (5%); Gains similar if restrict to non-switching children
For gains to be attributable to switchers, need switchers to have gained 1.7sd, given numbers---highly unlikely!
Results: Quantity
Notes
Enrolment
Large increase in RC villages (almost 5 percent)
Entirely from Government schools, entry into Grade i
Switching
No evidence of increased churning
But evidence of differential churning
School Closures
Significant among initially low performing private schools
Enrolment Change
Probability that child switches
Probability that child drops-out
School Closure
Report Card Villages
33.78
(13.75)**
0.012
(0.009)
0.006
(0.004)
--
Good Private School
-2.079
(0.592)
0.039
(0.516)
0.037
(0.200)
Bad Private School
-4.738
(0.248)
0.347
(0.019)
0.117
(0.030)
Government Schools
6.536
(0.006)
0.088
(0.053)
NA
Results: Price (Private Schools Only)
Notes
Fees
Large Declines—24 percent across the board
Larger in initially higher performing private schools
These are reported by the school; we obtain identical results using reports from households
In Rs. ($1=Rs.60 in 2003)
In Log fees
Report Card Villages
-217.96
(65.090)***
-0.24
(0.087)***
Good Private School
-241.841
(0.000)
-0.257
(0.001)
Bad Private School
-139.171
(0.237)
-0.126
(0.286)
Results: Schools or Households?
Notes
Household
Little evidence of any big changes (consistent with Das and others 2009)
Children in bad private schools are now playing less Sleeping more
Spending more time in school
Schools
Private schools increase teacher eduation, textbooks
More time on task—fewer breaks
(5) (6) (7) (8)
Parental Time Spent on Education
with Kids
Kids' Time Spent on School Work Outside
of School
Parental Spending on Education Not
Including School FeesChild Time
Spent PlayingSUBGROUP POINT ESTIMATES, F-TEST p-VALUES IN PARENTHESESBad private school -1.490 6.530 -375.684 -85.07***
(0.234) (0.788) (0.174) (0.010)Good private school -0.890 27.012* -32.930 -11.430
(0.207) (0.052) (0.844) (0.519)Government school -0.149 1.563 -172.899* -11.520
(0.625) (0.873) (0.055) (0.108)
Table 8: School and Household Input Changes
Household Inputs (Household Level)
(1) (2) (3) (4)
Class Teachers Improve to Matric
Percent Change in Matric Teachers
Textbook Probit Break Time
SUBGROUP POINT ESTIMATES, F-TEST p-VALUES IN PARENTHESESBad private school 0.165 0.035 0.405 -22.789
(0.29) (0.56) (0.09) (0.01)Good private school 0.183 0.037 -0.009 1.609
(0.06) (0.08) (0.95) (0.65)Government school 0.064 0.025 0.046 1.57
(0.39) (0.87) (0.77) (0.47)
Table 8: School and Household Input Changes
School Inputs (School Level)
Conclusion
RC increase learning and/or fees drop – equity and efficiency both increase? Results depend on pre-existing market conditions (parental demand; eductaional production
function - school type vs. effort)
Cost of Intervention ~ fee drop RC exercise cost $1 per child (testing, grading & dissemination) Cost savings ~ $3/child in private schools (1/3rd of all children enrolled in private
schools)
Welfare calculations? Tricky: Typical Cost-Benefit calculations in LIC ignore welfare costs for providers
→ learning gains free of cost BUT: complete welfare analysis has to factor in provider welfare loss – transfer
from school to parents & decline in rents – need more structural approach
Policy Questions & Caveats: Public vs (socially cheaper) Private sector Intervention simultaneously improve private sector (equity, efficiency) and public
sector – State’s role as information provider (rather direct regulator)
Further Notes or how I began to worry that this may actually lead to policy
Theory: Outlined 3 classes of theory; there are others Question: Why changes in provider behavior, but no increased
churning? Alternative Question: (Hastings): Why switching but no change
in provider behavior? Answer: We don’t know the dynamic equilibrium process in
control villages (ratcheting?) Empirics: Is this a big effect?
CANNOT compare SD increases across tests Can simulate changes from 0.05sd to 1.3sd by changing the test! Have to link to some cardinal change (see Heckman)
Trying to calibrate to TIMMS using identical questions Can answer: how big is this change relative to the world distribution
Longer-term effects (up to 5 years later) How do we treat utility of providers in welfare computations?
Feasibility This is proof of concept; mainstreaming is a different issue