issues and challenges in m&e of rural livelihoods programs: designing feedback loops rohini...
TRANSCRIPT
Issues and Challenges in M&E of Rural Livelihoods Programs: Designing Feedback Loops
Rohini PandeMohammad Kamal Professor of
Public Policy12 August, 2013
1
Evidence for Policy Design Initiative@CID, Harvard University
2
The NRLM FrameworkProposal:Increase household income of rural poor via sustainable livelihood enhancement and improved access to financial and non-financial services
Mission components: Social, financial, and economic inclusion
Mission features:• Decentralized implementation• Prospective design of M&E framework
3
NRLM M&E: What we want to know?
1. Why do we expect NRLM to achieve its goals?
2. How do we know that, in practice, it is achieving these goals?
3. If it isn’t, how can we modify it so that it does achieve the goals?
4
Talk Outline
A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops
Working with smart policy design and feedback loops• Testing design features for microfinance• MIS systems as feedback mechanisms: Data
visualization in MNREGA
• Final summing up
5
Talk Outline
A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops
Working with smart policy design and feedback loops• Testing design features for microfinance• MIS systems as feedback mechanisms: Data
visualization in MNREGA
• Final summing up
6
Smart Policy Design: Policy Development
What is the policy goal
Is existing evidence consistent with the
theory?
What is your theory for how to achieve
the goal?
Refine objectives, theory Policy design
7
Smart Policy Design: Monitoring and Feedback Loops
Policy design and pilot rollout
Is policy achieving its goals
Refine objectives, theory
Large scale rollout but
keep monitoring
monitor
8
Talk Outline A Holistic view of M&E: Smart Policy Design
Policy development Monitoring and Feedback loops
Working with smart policy design and feedback loops• Testing design features for microfinance• MIS systems as feedback mechanisms: Data
visualization in MNREGA
• Final summing up
9
Objective:Promotion and growth of micro-enterprises
Smart Policy Design: Testing Design Features in Microfinance
10
From Objective to Policy
Objective:Support growth of micro-enterprises.
Hypothesis:Micro-enterprises have high growth potential but are limited by lack of financing. Policy: Access to Microfinance
11
Smart Policy Design: Policy Development
What is the policy goal
Is existing evidence consistent with the
theory?
What is your theory for how to achieve
the goal?
Refine objectives, theory Policy design
12
Is existing evidence consistent with theory?The typical evidence provided to examine microfinance impacts:1. High returns to providing enterprises cash• 4-5% monthly returns
2. High rate of growth of microfinance use through India3. High repayment rates• With most MFIs posting over 90% repayment rates
Concludes: Microloans are a sustainable way of providing the poor finance
13
Is existing evidence consistent with hypothesis?But is that the correct evidence support the hypothesis?
– Directly test impact of microfinance on enterprize growth– Rigorous impact evaluations show small average impact
Triangulate evidence to reformulate design1. Why could this be given that micro-entrepreneurs have
high growth potential and site credit constraints as a predominant obstacle to the growth of their businesses?
2. How can we design an evaluation to test where the answer lies?
For redesign: Pilot new policies
14
How should policy be redesigned?or what feedback is needed? One possibility: Replace private sector providers by community organizationsBut why will this solve the problem?
Another possibility (and one we test)Maybe the microfinance contract is too rigid and this implies entrepreneurs don’t undertake high risk high return activities.
15
Refine policy: What limits returns to microfinance?
Policy design
Theory: Entrepreneurs
limited by lack of credit
Empirics: High returns to capital, but
impact of microfinance
limited
Conc
eptu
alfr
amew
ork
Policy goal : Encourage entrepreneurship
From Objective to Policy Goal
16
Smart Policy Design: Monitoring and Feedback Loops
Does the evidence enable better policy design?
How will we monitor?
What is the pilot policy
17
Identify pilot policyWhat do we know about microenterprises?•Subject to frequent household and demand shocks•High rate of business closings•Lack of insurance to overcome shocks•Irregular income streams with businesses highly subject to seasonality
Conclude: •Microentrepreneurs unable to make profitable investments that require a longer return horizon•Need flexibility in loan contract to make such investments
18
What do we know about microfinance contracts?•Standard Grameen microcredit loan contract– Immediate repayment obligation: One week between
loan disbursement and first repayment– Frequent loan repayment: Weekly repayment
•Rigid contract
Conclude: •Test impact of increasing repayment flexibility to microentrepreneurs.– Provide a grace period between loan disbursal and
repayment
19
Competing theories!1. Our theory: Presented above2. Competing theory: Contractual flexibility can have adverse effects
Recall need for evaluation
20
Smart Policy Design Method
What is the pilot policy?
How to monitor
21
Isevidence consistent with the theory?
The question: Does a grace period before loan repayment starts enable enterprise growth?
Method to test the question1: •Work with large MFI provider in Kolkata•By lottery select some MFI groups that receivethe grace period contract•Other groups remain on standard contract
1 Field, Pande, Papp, and Rigol (2013)
22
Pilot design
• Half of study sample receives standard contract‒ Immediate repayment obligation:
Two weeks between loan disbursement and first repayment
‒ Loan repayment:Fortnightly repayment
• Half of study sample receives contract with a grace period
‒ Immediate repayment obligation:Two months between loan disbursement and first repayment
‒ Loan repayment: Fortnightly repayment
23
Results
24
Smart Policy Design: Monitoring and Feedback Loops
Policy design and pilot rollout
Is policy achieving its goals
Refine objectives, theory
Large scale rollout but
keep monitoring
monitor
25
Using this evidence to enable smart policy design for NRLM
1. Under NRLMs SHGschoose loan repayment terms for group members.
2. IS Trade-offs faced by SHG similar to MFI providers? • If SHG has incentives to ensure high repayment but the
group doesn’t benefit from single member’s success then they may go for conservative rigid repayment schedule
• Policy question: How can SHG incentives be aligned with those of individual member? • What is the hypothesis? • What evaluation mechanism exists for feedback?
26
Talk Outline
A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops
Working with smart policy design and feedback loops• Testing design features for microfinance• MIS systems as feedback mechanisms: Data
visualization in MNREGA
• Final summing up
27
MIS and Smart Policy Design: Example of NREGA
Smart policy design is about building feedback loops into the policy process.
The NREGA data visualization project seeks to do this by:• Identifying key program performance indicators• Presenting descriptive statistics with an eye toward
digestibility and aesthetics• Facilitating basic hypothesis testing – investigate
correlations between key outcomes and administrative inputs
• Making program data easier to access for the public and academics (encourage external feedback)
28
Smart Policy Design Framework:First causal hypotheses
•Use simple descriptive statistics to identify problem areas and confirm/reject initial hypotheses
•Develop and investigate explanations for observed phenomena
• Start with simple correlations between policy inputs and outcomes; move on to more complex regression analysis, field evaluations; update hypotheses at each iteration
• Idea to build increasingly defensible causal explanations
29
NREGA implementation: Need for focus
• The NREGA MIS collects dozens of indicators at multiple levels of granularity
– Even highly motivated implementers may not have capacity to sift through information and get a clear picture of program performance
• These need to be distilled into a few key measures of program performance such as female and SC/ST participation or average time to payment.
30
Example: female participation in NREGA
Getting data on female participation currently takes 4+ clicks (if you know where to look).
District-level data only available state-wise, block-level data only available district wise – hard to get a broad view of performance without resorting to data scraping
Increasing data accessibility is low-hanging fruit
31
Reusable charts can increase data accessibility
Web technology exists to pull data from MIS and display it as interactive graphics
Charts are reusable – we can use the same chart format to display many user-selected indicators from the MIS
Technology (JavaScript and the d3.js library) is open source with a large support community
32
Reusable charts can increase data accessibility
33
Reusable charts can increase data accessibility
34
Ongoing work: Data visualization as a feedback loopHow should administrators use MIS/data visualization to identify potential policy levers?
Possible hypotheses:1. Demand Side: Absolute supply of work affects female
participation through male-first work allocation2. Supply side: Higher female empowerment implies greater
female participation
First step (work in progress!): Develop software that combines different data to generate user-defined scatterSuch initial diagnostics can help kickstart smart policy design.
35
Demand side: Examining correlations
36
Supply side: Examining correlations
37
Talk Outline
A Holistic view of M&E: Smart Policy Design Policy development Monitoring and Feedback loops
Working with smart policy design and feedback loops• Testing design features for microfinance• MIS systems as feedback mechanisms: Data
visualization in MNREGA
• Final summing up
Smart Policy Design: The process
38
• Identify your objective• Develop a theory that explains why a specific policy
will achieve the objective– What is the mechanism by which the policy
causesthe objective to be achieved?– What is evidence that these causal links will work?• How similar is problem to other contexts from
which we have evidence on similar policies?• What more do you need to know?
• Use monitoring and feedback loops to generate this evidence
Takeaways for Designing NRLM M&E
39
M&E should not stand apart from policy design and implementation process.Rather it’s design should ensure feedback loops with policy process.•Shouldhavelittle to no distinction between how to think about monitoring versus evaluation.•Monitoring mechanisms should enable concurrent evaluation.•In all cases, iterate between theory, existing evidence and pilots.