Randomized Experimentationfor the Program Manager:

A Quick How-To Guide

Jonathan ZinmanAssistant Professor of Economics, Dartmouth CollegeResearch Associate, Innovations for Poverty Action

May 1, 2007Presentation for IFC M&E Conference

What’s Your Objective?

• If it’s “beat or meet the market”….

• Can’t afford to focus on evaluation per se

What’s Your Strategy?

• Focus must be….

Using evaluation to feed innovation

• (Abhijit’s “dynamic learning”)

Experimentation &the Learning Organization:

A Virtuous Cycle

Evaluate

Innovate

Experiment

Randomized Evaluation:A Quick How-to Guide

1. What should I evaluate?2. How do I design and implement the

evaluation?3. How do I interpret and apply the results?

At each step will highlight how randomized-control trials (RCTs) can be part of an innovation strategy producing comparative advantage for:

• Client financial institutions (FIs)• Wholesale investors like IFC

What Should I Evaluate?

• Which interventions (“treatments”)?• Which outcomes?

Which Interventions?

• Use RCTs to address critical questions:– Business/programmatic/policy strategy– Stakes should be high, given fixed costs of

experimentation and related data collection• Existing programs you’re funding (of

course!)• Also….

Which Interventions?• But also: “Microfinancial Engineering”

– Analogy to “high” finance– “Design-build” partnerships between academics &

financial institutions– Input from funders, program managers, policymakers

• Don’t just “take opportunities” to do systematic experimentation

• Create opportunities. What’s a “program”?– Innovation challenges– Brokering– Funding

Which Interventions?

• RCT interventions can be built into any critical business function– Product development– Pricing– Marketing and targeting– Monitoring and enforcement– Risk assessment

Which Interventions?Product Development examples• Savings products with new features:

– Commitment– Goals– Method and nature of communication with client

• Smoking cessation performance bond– Smokers “bet themselves” they can quit (as measured by urine test in 6

months)– Financial institution fills missing market: enforcement

• Micro-insurance

Evaluate by randomizing• Whether product offered• Product features

Which Interventions?

Pricing examples• Loans: commercial, consumer• Savings: various products• Micro-insurance

Evaluate by randomizing• Initial offer• Dynamic pricing (future offers)

Which Interventions?Marketing and targeting examples• Marketing: what speaks to the client and why?

– Direct mail: randomize content– Text messaging: randomize content

• Targeting: finding your market or intended beneficiaries– Health insurance for the poor in the Philippines– Working with government, its contractors to experiment with:

• Measurement techniques• Incentives for using them

– Compensation– Punishment (monitoring, auditing)

Which Interventions?Loan monitoring/enforcement examples:• Require peer monitoring or liability?• Incentivize peer monitoring (referrals)?• How monitor and communicating with

(delinquent) borrowers?• What are the most (cost-) effective threats and

penalties?

Evaluate by randomizing:• Mechanisms, incentives, protocols

Which Interventions?Risk assessment• Possible to lend profitably to some rejected applicants?• Innovations to test:

– Improve risk assessment model (credit scoring; other projects)– Provide incentives to overcome loan officer conservatism

(details to follow)

Evaluate by randomizing:• The approve/reject decision (within reasonable bounds)

How-to Example From One Project

• Experiment with risk assessment• Objective: measure impacts of expanding

access to consumer loans– Popular product in South Africa– Worked with a leading for-profit “microlender”– “Traditional” micro(enterprise)credit largely

absent in South Africa– 4-month installment loans; 200% APR

Example: Expanding AccessWhat we did. Overview of the experiment:• Intervention: randomly approve loans for some close-to-creditworthy

(“marginal”) applicants who would normally be rejected– This is the “treatment group”

• Other marginal applicants remain rejected– This is the “control group”

• Measure impacts– Difference between treatment and control

• For the many outcomes of interest– Lender: use its data to calculate profits– Applicants: survey data on economic and well-being outcomes

• See “Expanding Access: Using Randomized Supply Decisions to Estimate the Impacts” (with Dean Karlan): http://www.dartmouth.edu/~jzinman/Papers/Derationing.Karlan-Zinman.pdf

How Did ThisEvaluation Come About?

• Remember Step 1: What do we evaluate?– Which intervention(s)?– Which outcomes?

• This project very much design-build• Lender motivation

– Prior experiments with Lender• Identified specific market failures (asymmetric information problems)• Identified binding liquidity constraints for borrowers

– These reinforced Lender’s priors that loan officers being too conservative

• Profitable deals being left on the table• Open to RCT as systematic way to control and evaluate risk of

liberalizing criteria

Motivation for this Evaluation:What do we Evaluate?

Researcher/policy/funder angles:• Consumer credit controversial

– Policy tends to restrict rather than encourage access– But why? Economic arguments for restricting tenuous

• But consumer (micro)credit markets growing• Our methodology applicable to microenterprise

credit as well

Step 2. How do we do it?Design and Implementation

3 key issues in this case:A. Scope of studyB. Implementing the intervention: how to

randomize loan approvals decisionsC. Tracking and measuring outcomes

Design & Implementation: Scope

A. Scope: How big? Where?• Required deal flow for a conclusive evaluation?

– How big a sample do we need to answer the questions of interest (statistical power)

– Researchers identify• Best way to obtain the required deal flow? Researchers

and Lender worked together to identify:– A practical definition of “marginal” applicant– Timeframe for the intervention (2 months)– Participating branches. Chose 8 branches that would

• Produce required deal flow in the 2 month timeframe• Be relatively easy to monitor and train• Be representative enough to draw conclusions re: whether or not to

scale up the intervention to other branches

Design & Implementation:The Intervention

B. How actually randomize loan approvals?• Insert research protocols into loan assessment

process. In this case 2 additional steps:1. Loan officers rejected applicants into “marginal” and

“egregious”2. New software randomizes “marginal” into “keep

rejected” or “approve” (second look)– New implementations streamline this with introduction of

pure credit scoring model• Train branch personnel• Monitor (& incentivize) branch personnel to

comply with protocols

Design & Implementation: Measurement

C. Tracking & measuring outcomes• Lender data on sales, costs, & hence profitability• Follow-up survey data on applicant outcomes:

– Economic (employment status, income, consumption)– Subjective well-being (decision power, optimism,

mental health)• Researchers designed household survey

instrument– Contract survey administration to survey firm– Close monitoring from pilot to final survey

Results: Lender Outcomes• Lender made money: marginal loans were

profitable– Less profitable than loans above the bar– But profitable nonetheless

• Even on initial loan• Profits from acquiring new clients even bigger

• Did Lender scale up? That was the plan, but….– Then Lender was merged into a larger bank– New senior mgmt hostile to “consultants”– Old senior mgmt (our partners) banked knowledge

and took to new firms

Results: Applicant OutcomesLarge, positive, statistically significant impacts on:• Economic self-sufficiency (employment, income, above

poverty line)• Consumption (avoiding hunger, food quality)• Outlook and control (decision power, optimism)No significant impacts on:• Investment (education, housing, self-employment)• Physical healthNegative impact (90% significant) on mental health

(depression, stress)Overall impact significant and positive• If weight all outcomes equally

Step 3. How Apply The Results?

The intervention itself• Do (social) costs exceed benefits?

– In this case interpreting results simple: win-win– Often there are tradeoffs: weighing costs and benefits

requires some insight into the “whys” of impacts• Here evidence of market failures from earlier experiments• Prior and project evidence of binding liquidity constraints

– Opportunity cost of intervention(s)?• In consumer credit key is ruling out negative effects: default

policy/programmatic approach is to restrict access• Unlike microenterprise credit, where default approach is to

expand/subsidize access, and hence opportunity cost of subsidy matters

How Apply The Results?Applying the results (external validity)• Scalability• Replicability

Three complementary approaches:1. Design so that get answers re: why

interventions do or don’t work2. Choose sites/markets/partners carefully3. Do lots of RCT experimentation

Take-AwaysRCTs deliver:• Gold-standard measures of impacts• Insights into the “why” questions that:

– Affect scalability– Feed back into innovation

RCTs are doable:• Design-build partnerships with researchers for:

– Microfinancial Engineering– Innovation that is scalable and replicable

Experimentation &the Learning Organization:

A Virtuous Cycle

Evaluate

Innovate

Experiment