second cross-country workshop of the africa program for education impact evaluation dakar, december...
TRANSCRIPT
![Page 1: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/1.jpg)
Second Cross-country Workshop of theAfrica Program for EducationImpact EvaluationDakar, December 2008
Data Collection Top to Bottom
(Skipping some stuff in the middle)
Moussa Blimpo and David EvansAfrica Impact Evaluation Initiative (AIM),
World Bank
![Page 2: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/2.jpg)
For impact evaluation, you want Baseline Follow up (1, 2…)
When could you use existing data? Regular surveys (reliable, regular EMIS) Regular testing (annual achievement
tests)
Most of our countries, insufficient administrative data
Why do a survey?
![Page 3: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/3.jpg)
Collecting your own data
Who should collect? Bureau of Statistics: has capacity & good
place to build capacity Other firm: could be better (but might not
be) University: maybe cheaper, often less
existing structure & so more work, more monitoring
Allow time to contract Procurement, disbursement all take time
![Page 4: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/4.jpg)
How much is this going to cost?[hint: quite a bit]
320 primary schools
10 student questionnaires
test of 40 students
1,000 primary schools
head teacher questionnaire
teacher questionnaire
parent questionnaire
![Page 5: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/5.jpg)
How much is this going to cost?[hint: quite a bit]
USD $35,000 USD $200,000
320 primary schools
10 student questionnaires
test of 40 students
![Page 6: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/6.jpg)
How much is this going to cost?[hint: quite a bit]
USD $50,000 USD $60,000 1,000 primary
schools head teacher
questionnaire teacher
questionnaire parent questionnaire
![Page 7: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/7.jpg)
How much is this going to cost?[hint: quite a bit]
USD $50,000 USD $200,000
![Page 8: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/8.jpg)
Large variations in cost
320 schools USD $50,000
1,000 schools USD $200,000
Big differences! Scale of survey Size of country Labor costs Difficulty in getting to schools
Household surveys: More expensive
![Page 9: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/9.jpg)
How long will it take?
15 working days for 275 schools 10 working days for 320 schools 40 working days for 2,000 households
Factors: Number of teams, size of teams Small group: longer time, better monitoring Big group: shorter time, harder to monitor
Enumerator training: ~ one week Pre-testing: 1-2 weeks Design: ~ one month (or more, ideally)
![Page 10: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/10.jpg)
Not cheap and not fast, but…
Maximize value by delivering multiple products Baseline / follow-up for impact
evaluation Social issues: Corporal punishment
(home & school) in the Gambia Performance: Student achievement in
Sierra Leone for previously unmeasured grades
![Page 11: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/11.jpg)
Questionnaire design
1. Identify relevant issues• Make sure consider intermediate outcomes
• School grants: measure school materials, school infrastructure
• SBM: activity of SMC, frequency of meetings, visits to school
• And final outcomes like student achievement
• Gather info to help understand mechanisms• Gambia – school-management training deals with
a range of areas• Measures from each can suggest which elements
are working and which are not
![Page 12: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/12.jpg)
2. Draft questionnaire
a. Harmonize • Lots of instruments already available:
using them allows easy comparison• Cross-country: Making the Grade,
classroom observation• Within-country: EMIS• Existing APEIE: Sierra Leone, the
Gambia, Rwanda, Madagascar, Senegal
b. Write questions
![Page 13: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/13.jpg)
Question pointer: Specific questions
Does your teacher assign homework?
Did your teacher assign homework yesterday?
Do you have regular PTA meetings?Can you recall the dates of the last
two PTA meetings?
![Page 14: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/14.jpg)
Question pointer: Verify when possible
![Page 15: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/15.jpg)
Survey pointer: Leave room
Leave space for enumerators to note aberrations
• Gambia: 43% head teacher absenteeismYikes!
• but all off administering a national test
![Page 16: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/16.jpg)
b. Distribute draft• Get input from sectoral experts • Get input from technical experts
c. Finalize draft
![Page 17: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/17.jpg)
3. Field test
But I’m an expert: Why do I need to field test?
Literacy in Sierra Leone▪ Testing Class 4▪ Passage from Class 1 book
![Page 18: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/18.jpg)
3. Field test
a. Train real enumerators b. Go to real schoolsc. Test on a small sample of studentsd. Analyze
• Test again• Analyze
• Field test again (as needed)• …
e. Finalize questionnaires
![Page 19: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/19.jpg)
Sampling
With school-based impact evaluation, often survey all schools involved in the intervention Needed for statistical power
If sampling students within school to interview, identify a random process
![Page 20: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/20.jpg)
Sélection et formation des enumérateurs.
1. Les enumérateurs sont des acteurs très importants dans le processus: ils collecteront les données qui déterminent les résultats des interventions
2. Ils doivent alors être formes quelque peu sur les élément de base de l’évaluation d’impact:• Afin qu’ils soient alertes sur les sujets tel que la
contamination.• Afin qu’ils soient alertes sur les aspects qui
pourrait compromettre l’évaluation d’impact.
![Page 21: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/21.jpg)
Composition des équipes Un superviseur Des enumérateurs Autre expertise en fonction du contexte
Antropometriste Examinateurs (Gambie: Enumérateurs)
▪ Instructions du test de lecture non suivi Chauffeur Autres Matériels et outils nécessaires
![Page 22: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/22.jpg)
La Sélection: Gambie Un test avec un volet de connaissances
élémentaires en statistique et un volet spécifique aux questions relatives a la collecte de données.
![Page 23: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/23.jpg)
La Sélection II
D’abord, sélection initiale de 40 sur 151 candidats
Ensuite, entrainement des 40 présélectionnés pendant une semaine sur les instruments et l’EI.
Enfin, donner ensuite un test pour retenir les 30 enumérateurs.
Avantage:▪ La formation est prise au sérieux par les enumérateurs
car le job n’est pas garantie.▪ On aura 10 enumérateurs de réserve forme et prêt a
travailler au besoin.
![Page 24: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/24.jpg)
Challenges: Gambie Beaucoup de réticences a cette
procédure. Sélection traditionnellement basée sur les relations et l’expérience.
L’entrainement est limite uniquement a la maitrise des instruments
Nombre insuffisant de formateurs des enumérateurs
Ecart de 2 semaines entre la fin de la formation des enumérateurs et le départ sur le terrain.
![Page 25: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/25.jpg)
Equipe typique en Gambie
Pour la Gambie on a eu besoin de 10 équipes de 3 enumérateursPendant 3 semaines dont 15 jours effectif de travail. En moyenne 2 écoles enquêtées par jour et par équipe.
![Page 26: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/26.jpg)
B. Suivi sur le terrain0. Importance du suivi des le départ pour corriger les
erreurs. Une page manquantes dans quelques questionnaires Doit pas supposer que la supervision sera bien faite par
l’agence de collecte de données: ex: Supervision limitée en Gambie
1. Est ce que les questionnaires sont bien remplie? La sélection aléatoire des élèves est elle bien
faite au niveau de l’école?
2. Rester en contact avec les enumérateurs et partager leur expérience avec les autres.
Cela évite que les autres commettent les mêmes erreurs.
![Page 27: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/27.jpg)
Apurement
1. Détecter les erreurs dans les données: Incohérences des réponses (Ex. Élève de 5 ans
au CM2) et d’autres erreurs. Gambie: Beaucoup d’erreurs dans les données
soumises. Près de 3 mois pour tout corriger.
2. Correction des erreurs: Revoir les questionnaires, contacter les
enumérateurs, éventuellement retourner sur le terrain.
Solution non désirable: Exclure de l’analyse les observations erronées, mais il faut documenter toute suppression.
![Page 28: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/28.jpg)
Technologie d’apurement
Questionnaires remplis directement sur des ordinateurs portables.
Détecter immédiatement les incohérences Economie au niveau de la saisie Utile surtout pour les enquêtes de grande
envergure. Utilise en ce moment en Tanzanie
![Page 29: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/29.jpg)
Une fois données de base collectées
La collecte des données de suivi est beaucoup plus simple, moins couteuse, rapide.
Utilisation des mêmes instruments Non nécessité de faire un nouveau pré-
test Moins besoin de formation (si on utilise
les même)
![Page 30: Second Cross-country Workshop of the Africa Program for Education Impact Evaluation Dakar, December 2008 Data Collection Top to Bottom (Skipping some stuff](https://reader036.vdocuments.us/reader036/viewer/2022070309/551d9d80497959293b8b9c6e/html5/thumbnails/30.jpg)
Conclusions
La collecte des données nécessite beaucoup d’argent et d’efforts
Mais cela est essentiel pour l’évaluation d’impact.
ET peut répondre a plusieurs attentesConcevoir en tenant compte de tous
les aspectsSuivi attentifApurement (data cleaning) minutieux
Meilleurs connaissances des écoles!