how an impact evaluation will look likepubdocs.worldbank.org/pubdocs/publicdoc/2015/8/... · 8....

Robin AudyWorld Bank

May 5 – 8, 2015Nairobi, Kenya

IMPACT EVALUATION RESEARCH WORKSHOP

How an impact evaluation will look like

Key Messages

• Impact evaluations requires careful planning from the start of a project It’s all about details

• Collecting data electronically is now accessible to all

• Use technology like GPS to create new indicators,

1. The steps of an impact evaluation

1. Identify the problem

3. Design an

evaluation

4. Do Initial Survey

5. Randomize the program

6. Conduct follow-up

survey

7. Crunch the numbers

8. Announce the result

2. Create a new or target an ongoing

program

1. The steps of an impact evaluation

4. Do Initial Survey

5. Randomize the program

6. Conduct follow-up

survey

2. Who do we interview?

Who are the final beneficiaries

Who should be interviewed, and when

How to identify them?

2. Who do we interview?

2. Who do we interview, and when?Imagine a simple conditional cash transfer for the poor

2. Who do we interview, and when?You will target among your population. A census can help determine who is eligible.

2. Who do we interview, and when?From the list of targeted eligible individuals, do your baseline survey

Do the baseline survey before assigning treatment status!

2. Who do we interview, and when?Only then should you assign treatment and control statuses

2. Who do we interview, and when?Perhaps take-up will not be complete among all those in the treatment group.

2. Who do we interview, and when?Perhaps take-up will not be complete among all those in the treatment group. You could also have individuals from the control who manage to get treatment.

2. Who do we interview, and when?Perhaps take-up will not be complete among all those in the treatment group. You could also have individuals from the control who manage to get treatment.

At endline, follow all the individuals from the baseline

2. Who do we interview, and when?Another type: clustered randomization by village

2. Who do we interview, and when?

2. Baseline survey


• Other examples to share?

3. What does a baseline interview look like?

Video about electronic data collection!

19

3. What about electronic data collection?Preparation is key!

• Design questions and prepare deployment

• Test, pilot and refine design

• Train and run interviews

4. Costs and benefits of electronic data collection

• Faster interviews

• Fewer errors

• Provide on-time feedback on quality of the

data collected by each enumerator

• More types of indicators (e.g. sound,

pictures, gps)

4. Costs and benefits of electronic data collection

• More planning beforehand

• Time to learn to make questionnaires

• Investment to deploy the system

• Additional training for enumerators

• Procurement of devices

• Maintenance of devices

• Device theft

• Battery life

• Availability of electricity and 3g network

5. Resources for electronic data collection

• solutions.worldbank.org

5. Resources for electronic data collection

• SurveyCTO (ODK) www.surveycto.com

• Kobo Collect (www.kobotoolbox.org)

6. What does a questionnaire look like?

• Contextual setting variables• Age, education, poverty, community characteristics

• Intermediate outcomes• Enrolment in school, frequency of health care visits, sanitation behavior

• Final outcomes • Measures of learning, health outcomes, measures of deforestation

• Contact information for follow-up• Map, name, phone number, etc…

6. Types of indicators to consider

7. Technology allows for new types of indicators

27


28


29

8. Randomly assign treatment and comparison


• The lottery assigns who gets what

• Perception of fairness

• Transparency: -> Big public event with dignitaries and all stakeholders

• Communications plan is important

• Explain equal chances to all


• An early childhood development program has 3 components:• Free access• Free access + incentive• Control group

• We have 120 participating villages

• We want to divide the villages into 3 equal groups:

Example:

Free access Free access + incentive

Control Totalvillages

40 40 40 120


• A program tests the attention level of workshop attendees:• Stretch• Caffeine• Control group

• We have 12 participating tables

• We want to divide the tables into 3 equal groups:

Example:

Stretch Caffeine Control Total

4 4 4 12


All 120 villages are assigned to a ticket

A, B or C

These tickets can be shown to

stakeholders in advance of

randomization

In the lottery, we randomly assign a

treatment to each letter

Example:Village Ticket

Bansko A

Belitza B

Blagoevgrad C

Gotse Delchev A

Garmen C

Krupnik B

Mikrevo A

Petrich B

Sandanski C

Satovcha B

Simitly A

Yakorouda C

Ajtos A

Burgas B

Karnobat B

Pomorie A

Vresovo B

Troianovo C

Ahtopol A

...


All 12 tables are assigned to a ticket

A, B or C.

It is possible to stratify. For example, 2

from the middle and 2 from the front

on each ticket

These tickets can be shown to

stakeholders in advance of

randomization

During the lottery, we randomly assign a

treatment to each letter…

Example:Village Ticket

5 A

8 A

10 A

7 A

2 B

6 B

9 B

1 B

12 C

3 C

11 C

4 C

8. Randomly assign treatment and comparisonExample:

Table Ticket

5 A

8 A

10 A

7 A

2 B

6 B

9 B

1 B

12 C

3 C

11 C

4 C

Treatment Ticket

Stretch ?

Coffee ?

Control group ?

8. Randomly assign treatment and comparisonLottery results

9. Follow-up surveys

• Initiate intervention

• Option: collect additional intermediate outcomes

• Option: collect program implementation information

• Option: collect costing information for CBA / CEA analysis (opportunity costs

need to be collected

• Follow up beneficiary survey: back in the fields!

Key Messages

• Impact evaluations requires careful planning from the start of a project Know you who, when and how

• Collecting data electronically is now accessible to all

• Use technology to collect new indicators like GPS

Questions?

how an impact evaluation will look likepubdocs.worldbank.org/pubdocs/publicdoc/2015/8/... · 8....

Documents