moving from evaluation to implementation
Post on 04-Dec-2021
2 Views
Preview:
TRANSCRIPT
Ms Marie-Louise Samuels and
Dr Stephen Taylor
28 APRIL 2015
Constitutional Hill
MOVING FROM EVALUATION TO
IMPLEMENTATION
Presentation at the DPME/DSD Roundtable on emerging evidence on impact of programmes on wellbeing
of young children
OVERVIEW
• Introduction to the Grade R programme
• Impact Evaluation of Grade R on learning
outcomes
• Data, methodology and results
• Improvement plan
• Reflections on the whole evaluation
process
• Progress to date
1994 1999 2004 2009 Pre-1994 2014
• Racial discrimination of ECD provisioning
• 75% fee-based
• 80% private and 20% public
• Audit of Early Childhood Development Policies
• Nationwide Audit of ECD services
• Education White Paper 5 on ECE
• Conditional Grant
• Children’s Act
• Regulations
• National Integrated Programme of Action
• Outcome 1
• NELDS
• NDP
• Education White Paper 1 on Transformation in Education
• Interim Policy on Early Childhood Education
• National Pilot Project
• Equitable share (Grade R)
• National Integrated Plan for ECD
• Tshwaragano Ka Bana
• ECD International Conference
• Expanded Public Works Programme
• ECD Policy and Programmes
• National Curriculum Framework for Children from Birth to Four
• Outcome 13
LOOKING BACK, LOOKING AHEAD
THE IMPACT EVALUATION
• Department of Planning, Monitoring and
Evaluation (DPME) in the Presidency works
together with other government departments to
evaluate key programmes and policies
• In 2011, The Grade R programme was selected
as one of the first evaluations on the NEP.
• A team of researchers from the University of
Stellenbosch was contracted to conduct an
impact evaluation.
• This impact evaluation of the Grade R
programme has now been completed, has been
presented to Cabinet, and is publicly available.
THREE KEY FACTS EMERGE
7
1. A person’s life trajectory is
established very early
2. Family wealth matters for young
children
3. It is possible to intervene effectively &
to improve the trajectories of young
children, but the later the remediation,
the less effective it is
DATA SET
• Dataset of 18102 schools
• Obtained by merging SNAP data on learners registered for each grade, test data from Annual National Assessments (ANA) of 2011 and 2012, and EMIS Masterlist
• ANA provides data on performance in maths and home language for Grades 1 to 6; – Converted to normalised score with mean 0 and standard deviation of 1 for
each grade, to make scores comparable (in relative terms) across grades
• EMIS provides school quintile and school fees – School fees in 2007 provide a measure of affluence and resources
• Large datasets allows more precise estimation of effect sizes
8
MEASURING TREATMENT
Treatment measure: % of learners in a given cohort that attended Gr R
Limitations:
• School may provide Grade R to a wider catchment area, i.e. learners move to other schools after Gr R
• Ratio exceeds 100% in some cases; we top censor these measures to be at most 100%
• Some Gr X learners may have attended Gr R at another educational facility
• Where data for Grade R are missing, we assume no treatment
9
% OF LEARNERS WITH GRADE R BY YEAR & SCHOOL QUINTILE
11
School quintile 2005 2006 2007 2008 2009 2010 2011 2012
1 43.9 51.4 63.1 72.0 79.2 83.0 85.4 86.1
2 45.3 53.6 65.1 74.5 82.3 87.0 89.2 90.3
3 50.7 59.8 67.7 73.8 80.3 85.2 87.3 89.2
4 54.9 60.8 66.4 71.3 76.4 79.6 82.3 84.2
5 57.3 60.5 64.0 66.0 71.5 75.7 77.3 78.9
All 48.0 55.1 64.2 71.7 78.5 83.1 85.3 86.6
DETERMINING CAUSAL IMPACT
• Other factors may also influence outcomes – Some we can control for (e.g. SES) – 0bservables – Others we cannot – Unobservables
• Example: Factors that could affect both treatment and learning, e.g.: – Better managed schools may more easily introduce Gr
R, and would usually have better learning outcomes – Departments may encourage weaker schools more to
introduce Gr R
14
TREATMENT AND RESULTS ACROSS SCHOOLS AND GRADES
15
Re
lati
ve p
erf
orm
ance
Treatment (attending Gr R)
TREATMENT AND RESULTS ACROSS SCHOOLS AND GRADES
16
Re
lati
ve p
erf
orm
ance
Treatment (attending Gr R)
FIXED EFFECTS: EACH SCHOOL TREATED SEPARATELY TO ELIMINATE UNOBSERVABLES
17
Re
lati
ve p
erf
orm
ance
Treatment (attending Gr R)
School A
School B
School C
School D
School E
FIXED EFFECTS: EACH SCHOOL TREATED SEPARATELY TO ELIMINATE UNOBSERVABLES
18
Re
lati
ve p
erf
orm
ance
Treatment (attending Gr R)
School A
School B
School C
School D
School E
INTERPRETING EFFECT SIZES
• Treatment effects are measured in standard deviations (SD) of test scores
– Coefficients on the treatment variable reflect the effect of a change from zero treatment to full treatment, i.e. from having no children attending Grade R to having all children attending
• International literature assumes that a year’s learning improves test scores by ± 40% to 50% SD
– But NSES data indicate SA learning might only be 33% SD in a year (less than 18% in weakest 3 provinces)
19
EFFECT OF TREATMENT (FIXED EFFECTS MODEL)
• Maths score +2.5% of a standard deviation (SD) for 2012 sample • Home language score +10.2% SD
– Assuming 40% SD to be equivalent to one grade level in school, this is equivalent to
• 6% of a year of learning in maths, or what average learner should learn in 12 days (if a school year is 200 days of instruction)
• 25% for home language, or 50 days of learning
– No clear evidence of fade-out
• Quite small effects: – A review of programmes in USA found average effects on cognitive
outcomes of 42% SD – Oklahoma’s high quality universal preschool programme saw
• an 80% SD gain in pre-reading and reading skills • a 65% SD gain in pre-writing and spelling skills, and • a 38% SD gain in early math reasoning and problem-solving
– In Argentina, one year of pre-primary increased average third grade test marks in maths and Spanish by 23% SD
20
21
OLS regression
Pooled 2011 2012
Dependent variable: Standardised Mathematics test score
Treatment 0.159** 0.199** 0.145**
School fixed effects No No No
Grade fixed effects Yes Yes Yes
Observations 47694 14954 32740
R-squared 0.230 0.267 0.223
Dependent variable: Standardised Home Language test score
Treatment 0.151** 0.153** 0.165**
School fixed effects No No No
Grade fixed effects Yes Yes Yes
Observations 47696 14957 32739
R-squared 0.315 0.306 0.338
Ordinary least squares and school fixed effects regression results
Note: * p<0.05, ** p<0.01; robust standard errors in parentheses.
22
OLS regression School fixed effects
Pooled 2011 2012 Pooled 2011 2012
Dependent variable: Standardised Mathematics test score
Treatment 0.159** 0.199** 0.145** 0.053** 0.074** 0.025*
School fixed effects No No No Yes Yes Yes
Grade fixed effects Yes Yes Yes Yes Yes Yes
Observations 47694 14954 32740 129410 41451 87959
R-squared 0.230 0.267 0.223 0.001 0.001 0.000
Dependent variable: Standardised Home Language test score
Treatment 0.151** 0.153** 0.165** 0.093** 0.060** 0.102**
School fixed effects No No No Yes Yes Yes
Grade fixed effects Yes Yes Yes Yes Yes Yes
Observations 47696 14957 32739 129419 41461 87958
R-squared 0.315 0.306 0.338 0.001 0.001 0.001
Ordinary least squares and school fixed effects regression results
Note: * p<0.05, ** p<0.01; robust standard errors in parentheses.
SA effect sizes in comparison (in % SD)
2.5 20.3
10.1
-0.8
0.0
1.5
10.2 19.4
11.5
-0.2
7.7
1.7
40 42
80
65
38
23
-10 0 10 20 30 40 50 60 70 80
Maths: All
Maths: Q5
Maths: Q4
Maths: Q3
Maths: Q2
Maths: Q1
Home Language: All
Home Language: Q5
Home Language: Q4
Home Language:Q3
Home Language: Q2
Home Language: Q1
One year of learning
US average preschool
Oklahoma pre-writing & spelling
Oklahoma pre-reading
Oklahoma early maths reasoning
Argentina Gr 3 Maths & Spanish
CONCLUSIONS ABOUT GRADE R IMPACT
1. Dataset enables estimation of effects with great
accuracy
2. Fixed effects control for unobserved
heterogeneity (endogeneity), thus causal effects
can be estimated
3. Grade R has had a positive impact on learning
Effects may be lasting: little sign of fade-out (decay)
in higher grades
Channels not clear, however (e.g. role of nutrition/
school feeding)
Negligible effects in bottom quintiles
25
CONCLUSIONS ABOUT GRADE R IMPACT
4. Measured effects are small for full sample Maths: overall effect less than 1 month worth of learning
2.5% SD for 2012
Home Language: ± 2 months 10.2% SD
5. Effects stronger for better performing
provinces & for higher quintiles: they share
characteristics in programme delivery that have a
positive impact
6. Thus programme quality needs to be
prioritised
26
RECOMMENDATIONS
1. Grade R underfunded according to DBE’s own
criteria (30% rather than 70% of other learners),
with large inter-provincial differences
Quality requires threshold levels of funding of both
personnel and LTSM – need to ensure this
Provinces must ensure Grade R is not crowded out by
other spending
2. Funding levels obscured by not disaggregating
funding within schools, using different categories
Needs greater attention at school and provincial level
27
RECOMMENDATIONS CONT.
3. A quality year of preschool is critical to effectively manage the transition to Grade 1 Close monitoring of teaching and learning in Gr R justified and
possible; requires dedicated personnel Develop common tools to assess language, emergent literacy, maths
development
Establish quality criteria for schools & ECD centres to self-assess & for M&E
4. More specific developmental research needed to inform educational strategies
5. Grade R curriculum has key role to play in closing the gaps Indicators which reflect elements of quality Grade R need to recognise the
critical importance of mediated language enrichment
Provide on-going structured curriculum support for teachers in implementing CAPS, with practical ideas on ‘how’ to achieve learning outcomes
Increase opportunities for in-service training focused on providing teachers with practical strategies for supporting early learning & opportunities to see & practice best teaching
Develop/evaluate evidence-based learning programmes & resources designed for the local context & appropriate for children from poor backgrounds
KEY ELEMENTS OF THE IMPROVEMENT PLAN
ELEMENT PROGRESS
Interim Grade R policy The recommendations on the amendment of legislation were presented to the Ministerial Task Team for consideration.
The development of a human resource development strategy.
A Training and Curriculum committee consisting of key stakeholders has been established as a sub-committee of the National Interdepartmental Committee. The HRD plan is part of the committee’s responsibilities. 3, 860 teachers have been enrolled in a course towards a Level 6 qualification
KEY ELEMENTS OF THE IMPROVEMENT PLAN
ELEMENT PROGRESS
The development of a programme to support curriculum implementation in all Grade R classes
Established a Foundation Phase Subject Committee. Aligned the Grade R resource pack to the National Curriculum Statement. Delivered Grade R workbooks Training of Grade R practitioners towards the Level 6 qualification.
The development of a monitoring and evaluation system integrated with the other objectives developed at all levels, including Grade R in the Annual National Assessment (ANA) system.
Activity has been included in the 2015/16 plan for the DBE.
top related