impact evaluation design kenya - world...

Impact Evaluation Design Kenya

Country Team 1. Anne Amadi 2. Justice Daniel Musinga 3. Dr. Paul Kimalu 4. Kennedy Mugonyi

ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015

: #ieGovern

What is the problem?

What is the problem? - Case backlog & case delays Intervention(s) - Regular feedback on performance (case backlog and

clearance rates to the courts). - Emails, phone calls, heads-of-court stations quarterly meetings, visits

by Judiciary officials. - *Considering tailoring printed feedback with influential images or

message (such as sending report with picture of the Chief Justice).

- Enhanced Court User Committees (CUCs) engagement - Greater stakeholders engagement – (Representatives from police,

prisons, advocates, CSOs, probation dpt etc)

1

Research questions • What is the most effective way of reducing case backlog and

improving case clearance rates? • Discuss mechanisms (e.g. multiple arms) - Monitoring and feedback without rewards or sanctions - Court stations stakeholders engagement. • Briefly mention the Theory of Change: Program -> A -> B -> C -

> X -Feedback Appreciation, recognition, managerial assistance, career progression/promotion Reduction in backlog -Stakeholders engagement Improved coordination, peer pressure Reduction in backlog.

Research Questions

2

Method Randomized Control Trial (RCT)

Clusters / sample size How many clusters? 160 courts How big is the sample size? 160 courts X 12 months = 1920 court months.

Design Cross-cutting design.

Methods & Design

3

Feedback

Stakeholders consultations

(CUC) 40 40

40 40

Outcomes:

- Speed -Case clearance rate.

-Disposition time.

-Case turnover ratio.

- Quality -Court users and employee satisfaction survey.

Unit(s) of observation: - Courts

Outcomes & Measurement

4

#ieGovern Impact Evaluation Workshop

Istanbul, Turkey January 27-30, 2015

Thank You!

facebook.com/ieKnow

#impacteval

blogs.worldbank.org/impactevaluations

microdata.worldbank.org/index.php/catalog/impact_evaluation

http://dime.worldbank.org WEB

Post Crisis Needs Assessment Impact Evaluation FATA & KPK

ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015

: #ieGovern

Background • Post Crisis Needs Assessment Strategic Objectives, mainly

Citizen – State Trust

• Impact Evaluation & capacity building of country systems

• Rule of law (Judicial)

• Governance

• Infrastructure

• Health

• CDD

Evaluation Objectives

• To measure impact of key PCNA interventions with credible data and correctly specified indicators

• To establish credible evidence and inputs for governance and service delivery reforms for further programming decisions

• To institutionalize evaluation in country systems for evidence based decision making and development planning

Projects Evaluation

Type Evaluation Method Sample Size (with Over

Sample)

Off-Season Vegetable - Agriculture Ex-Ante Randomize Control Trials RCT

430 Farmers (Adopters & Non Adopters) were interviewed in survey against 1381 Farmers in Khyber & Mohmand Agency which is 30% of the total farmers.

Anti-corruption Hotline - Governance

Ex-Ante Randomize Control Trials RCT

16,000 Individuals

TB Control Program FATA - Health Ex-Ante Randomize Control Trials RCT

124 Villages with 871 Respondents

MNCHP - Health Ex-Post Randomize Control Trials RCT

22 CMWs , 22 LHWs

Right to Information - Governance Ex-Ante Randomize Control Trials RCT

240,000 Individuals

e-Citizen Grievance - Justice Ex-Ante Randomize Control Trials RCT

220,000 Individuals

Right to Access to Public Services - Governance

Ex-Ante Randomize Control Trials RCT

215,000 Individuals

Review Request

• Define Citizen Trust Restoration

• Quality of service satisfaction (17 measures)

• Confidence in institutions (9 measures)

• Social satisfaction – self/family/other (17 measures)

• Use of technology mobile/internet for transparency and increase in trust in the government (43 measures)

What is the problem? • The effectiveness and utilization of the formal

judicial system • Back log of cases • Parallel judicial systems

• Build capacity of judicial system through trainings • Exploit variations in the timing of training (both experimental and

non-experimental)

• Exploit variations in where trained judicial officials are located

• Exploit variation in the type of training received

• Information about the availability of improved alternatives to formal court cases

Research Questions

• Who are the key actors underlying an effective Judicial System?

• What are barriers to citizen utilization of state supported Judicial System?

• Poorly trained Judicial staff leads to bad decisions and that leads to citizens distrust and under utilization

• On the supply side the judicial staff is trained

• On the demand side use awareness campaigns • Improved efficiency • Low cost mediation

Methods & Design • Triple Difference for the impact of judicial trainings

• Across judges

• Across time

• Across judicial staff

• RCT on the awareness campaign

• 350 judges across 27 districts

• Universe of cases that are brought forward and cases that are brought forward for mediation

Outcomes &

Measurement • Case specific outcomes

• Mediation vs formal court litigation • Time to completion • Number of hearings • Third party evaluation of decisions (random order)

• Litigant-specific outcomes • Satisfaction with decision • Cost incurred • Willingness to bring cases forward in the future

• Citizen outcomes • Willingness to access state institutions • Most recent form of dispute resolution • Trust in the state

• Court level outcomes • Backlog of cases • Case load per Judge

#ieGovern Impact Evaluation Workshop

Istanbul, Turkey January 27-30, 2015

Thank You!

facebook.com/ieKnow

#impacteval

blogs.worldbank.org/impactevaluations

microdata.worldbank.org/index.php/catalog/impact_evaluation

http://dime.worldbank.org WEB

Judicial Development Project KYRGYZ REPUBLIC

ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015

: #ieGovern

What is the problem?

What is the problem? - Problems of access and efficiency deter (M)SMEs from using

courts to resolve their disputes and enforce their rights. Interventions - Arm 1: SMEs receive training on laws/procedures that commonly

affect SMEs (staggered) - Arm 2: SMEs receive court infrastructure (staggered) - Arm 3: SMEs receive infrastructure & training

1

Research questions • How do improvements in court infrastructure impact access &

perceived efficiency of justice for (M)SMEs? • How does targeted training impact (M)SMEs? • Which of the interventions provides the greater impact on (M)SMEs? Theory of change: • Demand-side interventions that raise awareness of law/procedure

among potential court users are more likely to increase capacity to resolve legal disputes than supply-side courthouse infrastructure.

Research Questions

2

Method - RCT built into quasi-natural experiment (courthouse)

Sample size - Baseline survey of 500 (M)SMEs + repeat surveys - Control for revenue, no. employees, distance to courthouse, gender

and ethnicity of manager/owner - Caseload data on timeliness, project data, case tracking

Spillovers: - Information spillovers - Understanding of preferences/behaviors of (M)SMEs on all topics - Increase in formality of (M)SMEs

Methods & Design

3

Outcomes:

- Project level: inform Kyrgyz Republic Phase 2 project

- World Bank level: inform design of future projects

- National/international level: inform decision-making by Kyrgyz (and other governments) on how to improve justice / business climate

Units of observation:

- Firm level

Outcomes & Measurement

4

#ieGovern Impact Evaluation Workshop

Istanbul, Turkey January 27-30, 2015

Thank You!

facebook.com/ieKnow

#impacteval

blogs.worldbank.org/impactevaluations

microdata.worldbank.org/index.php/catalog/impact_evaluation

http://dime.worldbank.org WEB