project presentations - world bankpubdocs.worldbank.org/.../day-4-project-presentations.pdf ·...

Project Presentations

Target for Growth: Impact Evaluation Workshop

Mexico City

February 27th- March 3rd, 2017

How we’re doing this today

• In pairs: two presentations of 5 minutes, followed by 15 minute Q&A

• There is a lot you would want to talk about


• but resist the temptation and deliver


Zambia – Tunisia

Mexico – Vietnam

Bangladesh – Greece

Kenya – Senegal

Argentina – Colombia

Brazil – Peru

Ethiopia - Kosovo

Intermediation and Matching Grants for Agribusinesses in Zambia


Mexico City


Intervention(s) to be Evaluated

Executed by the Ministry of Commerce, Trade and Industry

Overall goals: Increased market linkages and firm growth in agribusiness.

Components: Market Linkages in Agribusiness (US$28.4 million)

• Subcomponent 1a. Building Productive Alliances in Zambia (US$16.7 million)

(a) matching grant investments in Productive Organizations (POs) (approximately US$9.7 million); (b) facilitation of and capacity building in POs and PAs (approximately US$3 million);

• Subcomponent 1b. SME Supplier Development Program (US$11.7 million)Invest in MarketConnect (US$6 million) (training and BDS program) and Business Linkage Fund (US$5.7 million – matching grants focusing on product upgrade/standard development equipment/ alternative energy) (

Scope

• Productive Alliances in Zambia – FarmersConnect: - 180 PAs (by end year 5), 4500 emerging farmers, PAs need to include minimum of 10 farmers, each farmer MG capped at 3000 USD, 30 % counterpart financing required

• MarketConnect- expected to reach a total of 300 SMEs, of which 140 SMEs expected to participate in the Business Linkage Fund, 10,000 – 100,000 royalty bearing matching grants

• Minimum eligibility criteria and definitions for emerging farmers and growth-oriented SMEs

• Reach – covers five provinces which include the highest density of poverty and agribusiness

Theory of Change (Productive Alliances in Agribusiness/Market Connect for SMEs)

LONG-TERM OUTCOMESSHORT-TERM OUTCOMESOUTPUTS

Aggregation of farmers

into POs / activation of

existing coops

ACTIVITIES

Information campaign

about structure of

program and

selection processHigher revenue,

profitability,

productivityIntermediary Service

Provider providing

coordination and

planning T.A.

Handholding PA/SME

to allow absorption

and assistance in

implementation

Business plan formed for

PAs/SMEs

Invest in PO’s to form

productive PA’s (High

potential POs matched

with sellers, )

business plans developed

and submitted for reviewImproved market

access, reduced

transaction challenges

MEDIUM-TERM OUTCOMES

Contract Fulfilled,

sustained over medium-

run

Technical Review of

proposals for PA’s

and SME’s

Formal agreement

between buyers and

PO’s/SMEs (Contracts)

Behavioral change:

investment, managerial

practices

Reduction in

costs/Increase in

markup

Improved reputation

amongst buyers and

retailersPA’s receive MG based

on achievement of first

sets of milestones

SMEs put in place

Marketing strategy*

+

Subset apply for and

receive MG

Changes in product mix*

Increased Sales,

employment*

Contract specific

investment aimed at

improving productivity

Matching Grant to

elegible PA’s & SMEs

Eligible PA’s & SMEs

selected to receive TA &

matching grant *outcomes formarket connect

Impact Evaluation Questions• Overall Question: What is the impact of intermediation and providing matching grants on income,

productivity, firm size and transaction challenges? • Conditional on being matched to a buyer, what is the effect of receiving support in developing business plans?

(T1)• Conditional on receiving business plan support, what is the effect of receiving a matching grant? (T2)

(Two distinct interventions: 1) Matching farmers in POs to buyers (Productive Alliances Program) and 2) Matching SMEs to buyers (Market Connect Program)

(Intermediaries do 3 functions: matchmaking between PO and buyer, support to development of business plan, and support to execution of business p (financed by the MG)

• Main indicators and data• Key indicators: productivity, sales, income, employment, reputation (e.g. scores/reviews), transaction challenges

• Data availability: admin data, project-specific surveys, Zambia Enterprise Survey, Zambia Business Survey

• Upstream Challenges and Delivery Mechanisms:

• Does intensive handholding lead to a higher probability of bringing farmers together to form a Productive Organization? (Randomize villages for intensive PO-outreach)

• Conditional on forming a productive organization, what is the effect of being matched with a buyer? (Randomize buyer to match to PO)

• Conditional on being selected to receive a Matching Grant, what is the optimal subsidy level to induce take-up? (Needs further work)

Methodology

Total sample size

540 eligible PAs

Control Group

[180 PAs]

T2

[180 PAs]

[Business Plan +

Matching Grant]

T1

[180 PAs]

[Business Plan ONLY]

Treatment Group

[360 PAs]

Challenges

• Take Up• Farmers/Producer Organizations (testing handholding intensity, information

channels)• Buyers/Producer Alliances (intermediary intensity and quality)

• Attrition• POs not able to provide quantity/quality as promised, buyer pulls out, PA

collapses (intermediary intensity and quality)

• Contamination (examining spillover measures)

• Implementation• Producers buying into counterpart financing• Sufficient Governance structures of POs• High intensity handholding during implementation/disbursement

Impact Evaluation Timeline

Project Effectiveness

Mar 2017

Implementation

June 2017

Baseline

Sep 2017

Follow up 1

May 2018

Follow up 2 -4

May 2019-21

Final dissemination

MMM YYYY

• Operations Approval Process

• Information Campaign - Apr

• Intermediaries Hired – June

• First Sets of Proposals – July

• Proposal Deadline – August

• Business Plan Support –September/October

• 1st milestone disbursed -November

• Cleaned dataset

• Data analysis• Client

workshop• Technical

paper• Policy note

• Cleaned dataset

• Data analysis

• Technical paper

• Policy note• Client

workshop

Thank you!

How Effective is Productive Alliance to help grow the Small Scale Producers in Tunisia


Mexico City


What is a Productive Alliance


• Context• Agencies involved: Ministry of Agriculture, Ministry of Industry, Ministry of Commerce

and Trade• Key overall goals: Improve small-scale producers’ income and job creation• Components: create productive alliance for small-scale producers and buyers

• Scope: • 20 productive alliance will be formed in 10 villages. Each alliance has about 50 farmers.• On average, 200,000 USD matching grants will be offered to each alliance. • Buyers offer suppliers’ contracts and product specific training to ensure product

quality/variety

• Only small producers with irrigated land less than 2 hectares can participate

• Reach: the first phase of the program will be carried out in 10 villages, the program will be expanded to more villages in the future.

Theory of Change


Application for joining

productive alliance

Number and value of

contracts, price of

inputs, product quality,

product variety,

investment, PA-jobs

ACTIVITIES

Information session

on benefits of

participating in

productive alliance

Employment

Randomly select

applicants to join the

productive alliance

Form partial

productive alliance

(productive alliance

without matching

grant) in selected

control villages

Training/Capacity

Development -

knowledge on product

quality, market

information, marketing

strategies, management

practices etc.

Form productive

alliance

Form partial productive

alliance


Productivity, profit

Impact Evaluation Questions

• Overall What is the impact of Productive Alliance on profitability, growth, and productivity of Small-scale producers• Specific question (arm 1): what’s the impact of joining productive alliance on

farmers’ market access, income, productivity, and employment?

• Specific question (arm 2): what’s the impact of joining partial productive alliance (networking with buyers but infrastructure investment and grants) on farmers’ market access, income, productivity, and employment ?

• Main indicators and data• Main indicators: market knowledge, sales, profit, productivity, jobs

• Data: surveys of small-scale producers, administrative farm census data from the Ministry of Agriculture

Methodology

• Small-scale producers will be randomly selected to participate in the Productive Alliance in 10 villages

• Sample villages will be randomly selected (out of 100 eligible villages) for “partial” productive alliance

• Small scale producers in treated villages for “partial productive alliance will be randomly selected to participate in partial productive alliance

• Surveys and admin data will be used for evaluation purpose

Total sample size

3000 small-scale

producers in 30 villages

Control Group

[1000 farmers in 10

villages]

[None]

Arm 2

[1000 farmers - 20

productive alliance in 10

villages and 20 buyers]

[Partial productive

alliance]

Arm 1

[1000 farmers - 20

productive alliance in 10

villages and 20 buyers]

[Productive alliance]

Challenges

• Challenge 1: Take-up of producers and buyers• Lack of trust, lack of knowledge of the benefits of productive alliance • Mitigation: face-to-face meetings, information session.

• Challenge 2: Measuring spillovers• Non-participating farmers live in the same village with participating farmers• Mitigation: compare non-participating farmers in treated villages with farmers in control

villages.

• Challenge 3: Attrition• Farmers/buyers exit the productive alliance over time• Mitigation: state clear in the contract about the punishment of exiting.

• Challenge 4: Decrease in market demand• Economic crisis • Mitigation: assist in improving product quality/variety; find new markets.


Concept

Aug 2017

Baseline

Oct-Dec 2017

Implementation

Jan 2018

Follow up 1

Jan 2020

Follow up 2

Jan 2021

Final dissemination

Sep 2021

• Funding application

• Approval process

• Cleaned dataset


workshop

• Monitoring report

• Lessons note

• Cleaned dataset




• Cleaned dataset

• Data analysis

• Technical paper


workshop

Thank you!

Discussion

Connecting Lead Firms to Local Suppliers in Mexico

Target for Growth: Impact Evaluation WorkshopMexico City


Background

•Context: Four new SEZs in Mexico (Chiapas, Veracruz, Michoacan, Oaxaca)•Motivation: A major objective is to generate spillovers to local suppliers in the “zona de influencia” •Proof of Concept: Design an online marketplace to foster linkages between lead firms and potential domestic suppliers, determine the value of these relationships •Priors:

•Finding local suppliers is costly for lead firms•The quality and specifications of locally produced goods may not initially satisfy lead firms’ sourcing needs

Rubén

Theory of ChangeLONG-TERM OUTCOMESSHORT-TERM OUTCOMESOUTPUTS

General training

for all firms,

intense

mentorship from

lead firms

ACTIVITIES AND INPUTS

Capacity

building with

local firms:

training,

diagnostics,

mentoring

Gather

information:

surveys, photos,

reliability

rankings

Use platform to

determine

matching,

prices

More eligible firms,

more initial

contracts created

Matching system

for local firms and

lead firms,

arrange

relationships


Transfer of

expertise to local

firms Greater overall use

of local firms:

product line

diversification

Arrange contracts

and financial

support for local

firms

Repeated

contracting and

relational

investment

Local firms

increase future

productivity,

investment

Increase lead firms'

interest and lower

their risk aversion

in working with new

suppliers

Lead firms switch

from importing to

local sourcing

Rafael/

Priam

Impact Evaluation Questions Rafael/

Priam

•What is the impact of training/support on lead firms' demand for contracts/relationships/links with local firms?•How can intensive training/mentorship increase the likelihood of link formation?•How does differential information about potential partners impact link formation?•What is the impact of price discounts/assistance on continued lead firm-supplier linkages?

Methodology

•Local suppliers randomly sampled from SEZs, lead firms are in negotiations

Treatment:

High-intensity training,

1000 firms

Control:

Low-intensity training,

1000 firms

Partial

information:

500 firms

Full information,

linkage support:

500 firms

Full information:

500 firms

Training Phase:

Matching Phase:

Full information:

500 firms

Challenges

•Challenge 1: Lead firms might not want to work with local firmslA feature, not a flaw?lStickiness of existing relationships

•Challenge 2: Too few lead firms or local suppliers•Challenge 3: Firms contract off the platform around us

lThey may renegotiate the terms we set up or leave the platform•Challenge 4: Platform implementation

lPartnering with an existing platform vs competing with them•Challenge 5: Financial incentives

lSubsidies are controversial; insurance may be hard to implement•Challenge 6: Underbidding and default

lFirms might put in unrealistic bids to win, then fail to deliver


ConceptAug 2017

BaselineJanuary 2018

TrainingApril 2018

MatchingJune 2018

EndlineDec2019

DisseminationDec 2020

•Funding application•Approval process

•Survey, photograph, and screen local firms•Survey lead firms •Develop platform

•Conclude low-cost and high-cost training•Solicit needs from lead firms, proposals from suppliers

•Begin arranging matches on platform•Evaluate impact of training on link formation

•Survey lead and local firms on key outcomes•Evaluate impact on firm and relational outcomes

•Academic papers•Policy note•Demand models for SEZ prediction•Transfer platform to government

Thank you!

Vietnam: facilitating FDI-SME linkages


Mexico City



• Project objectives:• To expand the integration of local SMEs into GVCs and increase overall domestic

value added

• Participants: • Buyers: FDI firms in targeted value chains (automotive & electronics)

• 15-20 lead firms and potentially 100+ tier-1 or tier-2 suppliers

• Sellers: local SMEs in relevant upstream sectors (plastic, rubber, metal processing)• Around 400 firms with potential to participate

• Implementing agency: Ministry of Industry & Trade, Supporting Industry Enterprise Development Center


• Interventions:• Online supplier database

• Scoring firms on potential to be successful suppliers

• Coaching program, in conjunction with demand side, for improving success as suppliers

Theory of Change

• Develop the chain of effects from interventions to be evaluated to outcomes

• Discuss potential unintended effects and assumptions


Improved quality of

informationTransaction cost (time

spent searching for

potential suppliers)

Usage, WTP

ACTIVITIES

Online database

Scoring of firms

Growth (sales,

employment)

Productivity

Investment

Spillovers

Supplier development

programImprove performance

score of SMEs

Number of firms trained

Adoption of improved

production management

by SMEs


Increased number of

contracts and sales to

FDI

Impact Evaluation Questions• Does the program increase linkages between SMEs and FDI firms?

• What is the effect of providing scoring information in the online database?

• What is the effect of the supplier development program?

• Main indicators and data• Number & value of contracts

• Firm outcomes: sales, employment, profits, productivity, investment, spillovers

• Data:• SIDEC & Ho Chi Minh supplier databases: 300-400 suppliers with potential to join

• Administrative data from the SDP + survey data

• Existing and potential FDI buyers (lead firms & tier-1 suppliers of lead firms): Foreign Investment Agencies + provincial Planning & Investment departments + IZ Authorities

Methodology

Total sample size:

400 local firms with

potential to become

suppliers

Control Group

[175]

Arm 2: Providing

“readiness score” in

online database

[200]

Arm 1: Coaching

program to meet buyer’s

standards

[50]

Challenges

• Discuss potential challenges/risks and how to tackle/mitigate them• Power (sample size, treatment group size, non-compliance, general

participation)• High-frequency data collection

• Spillovers (+/-)• Collect network data to measure effects (market competition, firm interactions)

• Implementation failures:• In baseline: build on rationale for theory of change & identify potential obstacles


Concept

Aug 2017

Baseline rounds

Fall 2017

Implementation

Early 2018- June 2021

Follow up rounds

Throughout & after

implementation

Final dissemination

Dec 2021

Thank you!

Discussion

One-Stop Shop services for private investors in Bangladesh


Mexico City



• We will evaluate the implementation of a One-Stop Shop (OSS) for investor services and process simplification in Bangladesh.• Agencies involved: Bangladesh Investment Development Authority (BIDA), in

collaboration with line agencies.• OSS will simplify the process for both foreign and domestic investors to obtain entry

and operational licenses for investment projects in Bangladesh.

• Scope: BIDA receives approximately 10,000 license applications per year. The list of recent applicants for investment projects will form the basis of our sampling frame.

• Eligibility: The current bill focuses on all foreign and domestic investors.

• Reach: The OSS will serve all sectors and regions of the country through a combination of main and regional offices, and an online platform.


• Overall: What is the impact of the process simplification and lowered transaction costs on the growth of new private investment?• Does the OSS reduce time and cost of private investment entry?• Does the OSS improve satisfaction with the process of private investment entry?

• Main indicators and data• Indicators: Number of applications submitted to OSS vs. “status quo” process,

satisfaction with process, time to approval, share of applications completed/approved, total investment proposed, total investment realized, incidence of corruption

• Data availability: BIDA investor tracking systems/database, companies register, Central Bank survey of foreign investors, feedback surveys, internal databases held by line agencies and OSS, baseline/endline surveys by IE team

Methodology

• Once the OSS is officially established and launched, we will implement a randomized encouragement design to induce exogenous variation in the use of OSS.

• (Institutional constraints will not allow restricted-access pilots or geographic staggering.)

• We will base the sampling frame on the former business investment license applicants to BIDA (~10,000 annually, use most recent year as initial frame; additional years as available).

• We will use existing administrative data (applicant/process files) + baseline and endline survey.

Total sample size

Estimated maximum

~10,000

Control Group

33%

No incentive or

matching service

Arm 2

33%

Encouragement/incentiv

e to use OSS

+

BIDA buyer/supplier

matching service

Arm 1

33%

Encouragement/incentiv

e to use OSS

Challenges• Four common examples:

• Challenge 1: Take up• Investors and/or intermediaries may prefer the well-established process with which they are familiar (or, lack

confidence in new OSS process; potential perception by investors of likelihood of increased effectiveness of OSS). • Possible encouragement incentives may be too small for relatively large investors.• We will have further consultation with government and investors/firm owners to determine appropriate level of

incentive.• We will allow for an expanded study horizon within which to measure take-up.

• Challenge 2: Contamination of control group.• We believe this to be unlikely based on sampling frame.• Will frame the intervention as a randomly-assigned inaugural promotion which should reduce likelihood feelings

of unfairness.

• Challenge 3: Implementation failures. Possible causes:• Resistance of line agencies to participate in OSS or no line agency process simplification. • Intermediaries may block take-up of OSS. • Strong commitment of client to ensure establishment of OSS and IE.

• Challenge 4: Confounding effect of incentive• Instead of a cash incentive, we will use credit for consumer goods so as to not affect investor’s business/financial

situation directly.

• Challenge 5: Proper measurement of corruption• Will be based on self-reports in baseline/endline surveys.


Concept

Aug 2017

Pilot

by September 2017

Implementation: OSS online

by June 2018

Follow up 1

by June 2019

Follow up 2

by June 2020

Final dissemination

by June 2021



• Sampling frame available

• Client workshop

• Pilot

• Baseline survey

• Assignment/incentive offer

• Intervention monitoring

• Obtain post-intervention admin. data

• Midpoint survey + report

• Client workshop

• Endline survey• Obtain post-

intervention admin. data

• Data analysis• Draft technical

paper

• Technical paper

• Client workshop

• Policy note

Theory of Change

• Develop the chain of effects from interventions to be evaluated to outcomes

• Discuss potential unintended effects and assumptions


1) Establishment of One-

Stop Shop (OSS) services

• Minimum five (5) line

agencies linked to OSS

• Necessary institutional

structure to support

OSS

2) Information widely

available on online portal

3) New investment

process simplified

4) Reduce human

interaction

• Use of OSS for new

investment projects

• reduced time and cost

for regulatory approval

for private investment

• Increased investor

satisfaction with

approval process

• Increase in applications

• Increase in share of

project applications

completed/approved

ACTIVITIES

• Formulation and

enactment of OSS

law

• Development of

rules and

regulations

• Integration of

services and

process

simplification with

line agencies

• Establishment of

online OSS portal

• Training of

involved staff

• Strategic

communications to

private investors

• Increased job creation


• Increased private

investment

• Increased business

entry

• Reduced incidence of

corruption related to

processes

• Increase in economic

diversification

Thank you!

Investment Licensing Reform in Greece


Mexico City


• Δ/νση Αδειοδότησης Επιχειρήσεων & Επιχειρηματικών Πάρκων, ΓΓΒ/ΥΠΟΑΝ (Ιαν' 2017)


• Describe the interventions to be evaluated

Optimize the inspection system to increase efficiency and maintain (or increase) safety

Reduce the time (and cost) to notification for businesses

What are key overall goals? What are the components?

• Goals: Improve the business environment, improve efficiency of the inspection system (optimize resources), preserve or improve public safety

• Components: Assist businesses during the initial notification process, vary the inspection schedule for checking for compliance

• Scope

Target sectors: Food and Beverage production, Shops with hygiene requirements (restaurants, hairdresser, night clubs, etc)

• Eligibility: Both new and old firms (excluding very high risk firms)

• Reach: Pilot in Attica Region (1 of 13 in Greece), main project cross country

Activities Intermediate outcomes

Improve knowledge of firmsImprove time to notificationReduce cost to notificationImprove quality of information provided

Improved economic

activity (job

creation)

Increased number of

firms created

Components/inputs

1

Simplified regulatory framework

for business

2

Improved

inspection

system to

check for

compliance

with rules,

and enforce

sanctions for

non-

compliance

Long-term

outcomesOutputs

Inspection

Improved Inspection

quality and efficiency

• Change license to

notification

• Reduce approvals

• Provide information on

requirements

• Develop inspection

probability system by risk

and time

• Inform some businesses of

probability risk

• Develop inspection quality

check/system

• Develop information,

guidance to support

compliance

• Notification system

including information on

key steps

• No. of inspections

• No. of quality checks

conducted

• Information/guidance

support delivered

• Sanctions enforced (%)

Improved firms’

compliance with safety

standards, reduced

safety violations

Improve safety /

mitigate safety risks

3

Improved information

to/from consumers

a) Scorecard system

b) Digital feedback

Theory of Change

Unify penalty system

Develop scoring system from checklistDevelop platform for customer feedback

• Scoring system displayed

in businesses

Notification System

Framework Law


• How can we optimize inspection system to increase efficiency and maintain (or increase) safety?• How does safety and efficiency vary with inspection probability?

• How does safety and efficiency vary depending on disclosing inspection probabilities?

• How does safety and efficiency vary depending on timing of inspection?

• How can we reduce the cost of and time to notification for businesses?• How does information provided to businesses at registration affect

notification compliance and cost?

Main indicators and data

• Outcome Data for Inspections• Compliance with safety regulation as determined at inspection (0 or 1)

• Safety score from inspection checklist ( continuous)

• Quality of inspections by shadow inspectors (effort and corruption)

• Consumer reporting of suspected violations

• Cost of inspections

• Firm reported profits

• Outcome Data for Notifications• Time to notification

• Quality of information provided

• Cost to businesses

Other Impact Evaluation Questions we were interested in (and will look at later)• Does advertising the simplified notification system lead to an increase

in the number of new businesses?

• Does displaying a public rating system (based on compliance) improve safety and affect profits for firms?

Methodology - Inspections

Total sample size

11,000

Control Group

Inspect based on current

risk probability

calculations (status quo)

Arm 2

Unknown probability of

inspection

(Vary probability of

inspection)

Arm 1

Known probability of

inspection

(Vary probability of

inspection)

Information treatment for notification (for half of all

above firms)

Stratify by high risk/low

risk

Challenges

• Challenge 1: Compliance with the inspection schedule• Challenge 2: Measuring spillovers


Concept

Aug 2017

Baseline

June-Nov 2017

Implementation

Nov 2017 – May 2018

Follow up 1

June 2018-Nov 2018

Final dissemination

Dec 2018



• Cleaned dataset

• Data analysis


• Lessons note

• Cleaned dataset




• Technical paper


workshop

Thank you!

Discussion

Business Plan Competition in KenyaTarget for Growth: Impact Evaluation Workshop

Mexico City



• Program: Kenya MSEA’s Business Plan Competition with the creating jobs for targeted youth*

• Scope: 7,500-10,000 applicants; 3,000 screened for preparing business plans; 1,500 support in Business Plan preparation; 750 receive grants (500 through screening process)

• Eligibility: Young entrepreneur up to 35 years old with a registered youth-owned business** or intending to register one; Kenyan

• Reach: National level

*youth between 18 and 29 years of age, who are without jobs and have experienced extended spells of unemployment or who are currently working in vulnerable jobs (working on their own, a contributing family worker, or working for wages in a household enterprise with fewer than 10 workers. These are mostly informal workers). The level of education of targeted beneficiaries will be up to Form 4.

**75 percent owned by individuals up to 35 years old

Theory of Change

LONG-TERM OUTCOMES

(12-60 months)

SHORT-TERM OUTCOMES

(1~3 months)OUTPUTS

• 750 business ideas

with potential for

growth identified

• Businesses created/in

operation

ACTIVITIES

• Application &

Selection

• Awareness

& Call

• Application

• Initial

Screen

• Review of

submitted

business

plans

• Sustained targeted

youth employment

• Targeted youth with

access to continued

skills training.

• Support for

Business Plan

Preparation

• Grants of $9,000

• Grants of $36,000

- Business grants to new

and existing

businesses.

- 500 businesses with

$9k

250 businesses with

$36k


profits


investment


(3 -12 months)

• Greater number

employees hired

• Higher profits

• Businesses’ with

increased access to

finance


increased access to

networks


increased access to

additional skills

• 1,500 youth

entrepreneurs trained

to create a business

plan

• Refined business ideas

with high potential.

• High-impact

entrepreneurs with

business plans

• Greater number of

target youth employed

• Higher earnings for

targeted youth

• Increased quality of

employment.


• What is the impact of a grant to high growth entrepreneurs on job creation for targeted youth in the context of a business plan competition?• What is the differential impact of providing a large grant ($36,000) versus a small

grant ($9,000)?

• Does support for business plan preparation increase the likelihood of winning the business plan competition?• And the ultimate success of the businesses? [conditional on power]

• How effective is the business plan competition screening process at selecting firms that will provide job creation for target youth?

• What is the effect of the business plan competition screening process on firm impact?• For those who do not participate in the business plan competition screening process,

what is the effect of a small grant ($9,000) on job creation for target youth?

Impact Evaluation QuestionsOutcome Indicator

Business in operation Self Report (SR), Online check, Physical check

Access to skills SR, Expert Evaluation

Access to finance SR, KRA records, financial due diligence

Network characteristics SR

# employees / employees in target youth SR, Observation, KRA

Proportion full time SR, Audit

Income of workers SR, KRA

Type of employment SR, Audit

Business profits SR, KRA

Business investments SR, Audit

Methodology Wednesday 1

Total

sample size

10,000

5. Selection + $0+ Support

6. Selection + $0

250

3. Selection + $9k + Support

4. Selection + $9k

250

1. Selection + $36k + Support

2. Selection + $36k

250

7. $9k

2508. $0k

250

Eligible

9,000

11. Support

1,500

10. No

Support

2,500

Plan

Submitted

2,000

Screen

Passed

4,000

Winners

750

(225 Support + 225 No Support + 300)

Ineligible

1,000

Screen

Failed

4,500

Losers

1,250

Not

Submitted

2,000

9.

Screening

Sample

8,500

(500)

Groups

Group Selection Support Winner Grant Size

n

1 X X X $36k >75

2 X X $36k >75

3 X X X $9k >75

4 X X $9k >75

5 X X X $0 >75

6 X X $0 >75

7 $9k 250

8 $0 250

9 X ? ? ? 500

10 X ? ? -

11 X X ? ? -

Effect Comparison

Any grant (1+2+3+4) - (5+6)

Grant Size (Large vs. Small) (1+2) - (3+4)

Support (on winning) (10) – (11)

Support (on winner outcomes) (1+3+5) – (2+4+6)

Selection on picking winners (4+6+10+11) – (7+8)

Selection on firm impact (9) – (8)

Grant (cond. no selection) (7) – (8)

Challenges

• Challenge 1: Not enough applicants to the Business Plan competition

• Applicants do not have access to the internet

• Online platform is not operational (downtime, slow, confusing)

• Phishers impersonate the website to collect user information

• Multiple applications by an individual or firm artificially increases the sample size.

• Too difficult to submit the business plan

• Business plans are copies or fake

• Challenge 2: Difficulty in determining eligibility

• Criteria is difficult to measure or verify

• Criteria is too strict

• Verified eligibility reveals that winners were not eligible ex-ante

• Challenge 3: Attrition

• Applicants do not attend the support program

• Non-response from people who do not win the competition

• Challenge 4: Non-compliance of grant recipients

• Grant recipients do not adhere to tranche rules

• Challenge 5: Mistakes in carrying out research implementation

• Process fails to adhere to randomization policy

• Political landscape forces changes to implementation

• Implementing firms fail to carry out project as intended

• Challenge 6: Spillovers

• Collusion among potential candidates


Concept

Aug 2017

Implementation

Jun 2018

Baseline

Jan 2019

Follow up 1

Sep 2019

Follow up 2

Jan 2021

Final dissemination

Dec 2021




• Lessons note

• Cleaned dataset


workshop

• Cleaned dataset




• Cleaned dataset

• Data analysis

• Technical paper


workshop

Impact Evaluation TimelineWednesday 2

Thank you!

Senegal Business Plan competitionTarget for Growth: Impact Evaluation Workshop

Mexico City



• The intervention: Business Plan Competition

➡Which agencies are involved? ADEPME

➡What are key overall goals? To encourage innovative firm creation and/or growth

➡What are the components? Outreach, selection, training, grants, aftercare services

• Scope: 4 rounds; up to 150 awardees per round (500 total); 4-5 days Business Plan Training 1,500 trainees per round (6000 total). Grants range: 5,000-50,000 USD

• Reach: The program will be implemented on a national level

Theory of Change


Outreach campaign

completed

Sufficient number of

entrepreneurs applied

Public awareness on

entrepreneurship

increased; motivation of

aspiring entrepreneurs

increased

ACTIVITIES

Outreach &

communication

campaign and selection

process

Job creation

Business plan training

and submission of

business plans

Selection of finalists,

grant funding and

aftercare services and

support

Entrepreneurs utilizing

knowledge, services and

money of the program to

scale their businesses

Selected entrepreneurs

trained

Business plans developed

and submitted

Finalists submissions

evaluated

After-care Services

delivered

Funding disbursed

Revenue increase


Spillover effect

Increased innovation

Survival rate


• Overall: What is the impact of BPC on the innovative firm creation and growth in Senegal?

• Arm 1: Does BPC as currently implemented affect innovative firms creation?

• Arm 2: What is the additional impact of the follow up customized support (aftercare) to the winners on the firms survival rate and growth?

• Arm 3: What share of the program impact on the firms survival and growth comes from public recognition and celebration as “winners”?

• Main indicators and data• Key indicators: firms created, firm survival rate, employment, revenues, innovation• Data available: project data (ERP), administrative data (RCCM, RNEA); impact

evaluation data (new survey, including baseline, midline and end-line surveys)

Outcomes Definition

• Originality, relevance, feasibility (Innovation index to be refined)

• Firms still existing after the 1st and the 3rd year

• Turnover

• Number of employees present by the day of the survey

• Increase of number of applicants among respondents baseline network• Negative effects on competitors

• Index of self-confidence• Impact on marketing expenses

Methodology

800 selected applicants

BPC : 350 WINNERS

BIG CELEBRATION

OF THE WINNERS

NO CELEBRATION

ADDITIONAL AFTERCARE SUPPORT

A B 175

NO AFTERCARE SUPPORT

C D 175

175 175

Control Group

450

• Baseline: at the selection process,

• 2 follow-up surveys

Challenges

• Challenge 1 : Is randomization acceptable by project stakeholders? ➡Mitigation: Additional consultation to explain the benefits and to discuss the

best way to communicate the transparency of the method

• Challenge 2: Take up➡Based on the pilot implementation, the demand is high

• Challenge 3: Contamination➡Phased in implementation

Thank you!

Discussion

Buenas Prácticas exportadoras(Good Exporting Practices)


Mexico City



• Describe the interventions to be evaluated• Agencies are involved: AAICI, INTI, STP, Secretary of commerce• Overall goal: improve export performance• Components: Diagnostic of best practices in 7 specific areas (strategy, market

identification and segmentation, design and development, production, communication,distribution and administration) and technical assistance

• The diagnostic and the technical assistance will be performed by specialists in exportpromotion, commercialization, design, quality, and strategy

• Scope: 300 SMEs, budget: USD1.5 million

• Eligibility: SMEs in the food industry, 5 to 250 employees, export experienceor willingness to export

• Reach: national level

Theory of Change


Report of results in the

7 areas mentioned

previously

Change in practices

ACTIVITIES

Diagnostic

Export performance

(value, volume)

Employment

Diversification of

exports (number of

products , number of

markets, sophistication

of export markets)Technical assistanceImplementation of a

plan to change practices


Change in practices

Export performance

(value, volume)

Employment

Diversification of

exports (number of

products, number of

markets, sophistication

of export markets)


• Lack of knowledge of good business practices constrains export potential

• Does adoption of good export practices lead to better export performance?

• How to facilitate information of good export practices to SMEs? • providing a diagnostic incentivizes adopting good exports business practices? Or• Do you need to combine both diagnostic + technical assistance

• Main indicators and data• Index of export practices adoption • Exports (value, volume, quantity of export destination, sophistication of market)• Employment

Methodology

• The budget is only enough to perform the diagnostic on 200 firms and the technical assistance on 100 firms

• If the number of firms that express interest in participating exceeds 200, we will randomly assigned 200 firms to receive the diagnostic, and100 firms to receive the intervention, so the remaining for will be assign to a control. Otherwise, we will assign half of the firms to receivethe diagnostic, and half of the firms to receive the diagnostic and the technical assistance

• Two sources of data: administrative records and surveys conducted by the impact evaluation team (3 surveys, a baseline and two follow upstaken by firms in Arm 1 and Arm 2)

Total sample size

300 (or n, if different

from 300)

Control Group

[100, or n-200]

Arm 1

[100]

[Diagnostic]

Arm 2

[100]

[Diagnostic + technical

assistance]

Challenges

• Ensure adequate quality in both implementing the diagnostic and the diagnostic + technical assistance• Training of consultants• The diagnostic has been piloted• Monitoring of the consultants

• Four common examples:• Challenge 1: Take up

• There are multiple strategies (mailing to firms in the export database, information posted on Agency’s website and social media, mailingto members of 50 chambers of commerce, mailing to provincial export promotional agencies database, etc.) to achieve the target takeup of 300 firms

• Challenge 2: Contamination of control group• All other interventions by the government agencies are orthogonal to the treatment (even if they are other interventions, all firms are

targeted independently of treatment status)• Blind process: Consultants surveying firms are not informed on whether the firm they are surveying will be treated or not

• Challenge 3: Implementation failures• Potential non compliance. This will be addressed using an Instrumental Variables approach• Attrition. No problem for administrative outcomes

• Challenge 4: Measuring spillovers• Positive spillovers would arise if some firms can obtain information on exporting practices even if they are not treated. However, this is

unlikely to be the case since the nature of the program is to be “taylor-made”


Concept

Jan-Feb 2017

Baseline

May – Jul 2017

Implementation

Aug - Jan 2018

Follow up 1

May – Jul 2018

Follow up 2

May – Jul 2019

Final disseminatio

n

Aug-Dic2019

• Cleaned dataset


workshop


• Lessons note

• Cleaned dataset




• Cleaned dataset

• Data analysis

• Technical paper


workshop

Team

Impact Evaluation Team• Impact evaluation TTL - Leonardo Iacovone

• Principal investigator(s) - Martin Rossi

• Research analyst(s) - Maximiliano Lauletta

• Field coordinator - (graduate student)

Project Team• Project TTL - Pablo Abriani and Andrea González

• Specialist(s) - Juan Carlos Hallak

• Government counterpart 1 - Agencia Argentina de Inversiones y Comercio Internacional

• Government counterpart 2 - INTI

• Government counterpart 3 - Secretaría de Comercio

• Government counterpart 4 - Secretaría de Transformación Productiva

Budget

Item Year 1 Source Year 2 Source Year 3 Source Total

IE Team: concept note preparation 8,000 8,000

Baseline survey cost 210,000 210,000

IE Researchers: baseline survey & analysis 8,000 8,000

IE follow up surveys 125,000 125,000 250,000

IE Researchers: follow up survey & analysis 8,000 8,000

IE Coordinator 4,500 4,500

IE Dissemination workshops - 10,000 10,000

IE travel -

Total cost of impact evaluation

Cost of intervention(s) (diagnostic+TA) 690,000 690,000

Platform 200,000 200,000

Miscellanous 50,000 50,000 100,000

Thank you!

Colombia productivaTarget for Growth: Impact Evaluation Workshop

Mexico City


Colombia Productiva

• The program consists of three stages: (1) a comprehensive diagnostic of the firm in 7 areas of performance, (2) 208 hours of technical assistance by specialized consultants over a 6 month period, for the areas in which the firm is found to be lacking, for a subset of the firms participating in the first stage, and (3) participation in business match-making forum, for a subset of the firms participating in the stages 1 and 2 (stage 3 subject to sample size and funding availability).

• Firms will be evaluated on the following areas of performance:• Human resource management• Energy efficiency• Adoption of Information Technologies (IT)• Processes’ optimization• Quality standards• Products’ readiness for international markets• Management practices for exporting

• The agency in charge of implementation will be Programa de Transformación Productiva (PTP), a unit of the Ministry of Trade, Industry and Tourism of Colombia. PTP will work with the agency for export promotion, ProColombia, as an ally. ProColombia will be directly in charge of implementing stage 3.

• The program aims to (1) increase firms’ productivity; and (2) increase firms’ exporting capabilities measured by exports growth and continuity.

Colombia Productiva

• The expected cost of the program is of USD 1,302,000 (an estimated cost of USD 460 per diagnostic for 200 firms plus USD 13,000 per 208 hours of consulting support for 100 firms) + USD 703,000 (an estimated cost of USD 460 per diagnostic plus USD 13,000 per 208 hours of consulting support for 50 additional firms + USD 30,000 for business match-making forum). Stage 3 subject to sample size and funding availability.

• PTP will reach 200 firms in the first wave of the program in the diagnostic stage and 100 of them with the technical assistance package. (250 in the first wave and 150 with technical assistance + 50 with participation in match-making forum, if funding and sample size permit including Stage 3).

• Participating firms will be required to pay for part of the technical assistance they will receive [a flat rate per firm].

• Eligible firms must (1) be at least 3 years old (formally registered at the chamber of commerce) (2) be able to show sales of at least 500 minimum monthly wages on the last year, (3) not have exported more than USD TBD on any of last three years; (4) be a producer and not only a trader; and (5) belong to a set of specific sectors [chosen to ensure the necessary sample size for evaluation]

• The program will be aimed to all eligible firms, regardless of their location.

Theory of Change

LONG-TERM OUTCOMES

More than 24 months

SHORT-TERM OUTCOMES

(3 to 12 months)OUTPUTS

Diagnostics of firms in 7

areas of performanceStandardized processes

ACTIVITIES

Design of diagnostic

toolProductivity

Production of

contents for each

module of technical

assistance

Energy consumption per

unit producedHours of technical

assistance

Comtinuous exportimg

activity


(12 to 24 months)

Quality standards

certification

ProfitabilityExports growth

Adoption of Information

Technologies (IT)

Training of personnel

who will apply

diagnostic tool

Definition of final

eligibility criteria for

participation

Training and hiring of

consultants

Invitation to

participate

Product readiness for

international markets

Exports growthBusiness match-making

forum (subject to

sample size and funding

availability for Stage 3)

Organization of business

match-making forum (subject

to sample size and funding

availability for Stage 3)


• What is the impact of offering firms careful diagnostics of their performance in different areas, on their productivity, profitability, exports growth, and continuity as exporters?

• What is the impact of offering firms not only careful diagnostics of their performance in different areas but also technical assistance during a 6 month period, on their productivity, profitability, exports growth, and and continuity as exporters?

• What is the impact of offering firms participation in a business match-making forum after offering both a careful diagnostics of their performance in different areas plus technical assistance during a 6 month period, on their productivity, profitability, exports growth, and and continuity as exporters?

Main results indicators and data

LONG-TERM OUTCOMES

More than 24 months

SHORT-TERM OUTCOMES

(3 to 12 months)

Number of standardized

processes

Sales per worker

Change in KWh per

dollar of sales

Number of consecutive

semesters with exports

of at least USD 10.000


(12 to 24 months)

Number of certifications

obtained

Return on Assets (ROA)Number of new software

solutions adopted

Number of products that

comply with market

requirements

Exports growth in USD

Number of days from

order to shipment

Number of new products

exported

Number of new export

destinations

Exports growth in USD


exported


destinations

Change in managerial

and organizational

practices

Data availability

LONG-TERM OUTCOMES

More than 24 months

SHORT-TERM OUTCOMES

(3 to 12 months)

Number of standardized

processes:

Survey

Sales per worker:

Financial statements /

Survey + Official records

of formal employment

(PILA)Change in KWh per

dollar of sales:


Survey

Number of consecutive

semesters with exports

of at least X value:

Official trade records

(DIAN- DANE)


(12 to 24 months)

Number of certifications

obtained - Survey

Return on Assets (ROA):


Survey

Number of new software

solutions:

Survey

Number of products that

comply with market

requirements - Survey

Exports growth in USD:


(DIAN - DANE)

Number of days from

order to shipment:

Survey


exported:


(DIAN - DANE)


destinations:


(DIAN - DANE)

Exports growth in USD:


(DIAN- DANE)


exported:


(DIAN- DANE)


destinations:


(DIAN- DANE)

Change in managerial

and organizational

practices:

Survey

Methodology

• Participants will be invited to apply to be part of the program. They will then be randomly selected into one of three groups, a control group and two treatment groups

• Modules will be added to an official survey from the National Office of Statistics to collect information that is not available from administrative records.

300 firms

Control Group

100

Application to program

Arm 2

100

Application to program +

Diagnostic of 7 areas of

performance + Technical

Assistance

Arm 1

100



performance

Methodology (alternative with Stage 3)

• Participants will be invited to apply to be part of the program. They will then be randomly selected into one of four groups, a control group and three treatment groups

• Modules will be added to an official survey from the National Office of Statistics to collect information that is not available from administrative records.

350 firms

Control Group

100

Application to program

Arm 2

100




Assistance

Arm 1

100



performance

Arm 3

50




Assistance + participation in

business match-making forum

Challenges

• Challenge 1: Take up• Possible reasons: poor marketing, lack of knowledge (don’t know what they don’t know). Will

require using all channels of communication to attract eligible firms.

• Challenge 2: Obtaining information from control group• Possible reasons: Lack of contact with government agency after application and rejection.

Will be addressed by including module for all firms in the evaluation in official survey. Alternatively, providing information should be made easy and awarded with something of value to the firm (i.e. comparison to firms in the US after answering?)

• Challenge 3: Contamination of control group• Possible reasons: participation in other government programs or other programs. Will be

addressed by collecting information about program participation.

• Challenge 4: Implementation failures• Possible reasons: quality of consultants and contents of technical assistance as well as

standardized delivery. Will require careful planning and training as well as standardized guidelines for assistance.


Concept

Aug 2017

Baseline

June 2017

Implementation

June 2017-Feb 2018

Follow up 1

June 2018

Follow up 2 and 3

March and Sept 2019



• Cleaned dataset


workshop


• Lessons note

• Cleaned dataset




• Cleaned dataset

• Data analysis

Thank you!

Discussion

Sebrae na sua EmpresaTarget for Growth: Impact Evaluation Workshop

Mexico City



• Describe the interventions to be evaluated• Which agencies are involved? Sebrae-RJ

• What are key overall goals? Increase adoption of ‘best’ business practices (ideally via Sebrae’s material)

• What are the components? 1 hour in loco visit where consultants run a diagnostic on business practices

• Eligibility: micro (80%) and small (20%) formal firms

• Reach: Metropolitan region of Rio de Janeiro

Monday

Theory of Change

LONG-TERM OUTCOMESInterventionsOUTPUTSACTIVITIES Short-TERM OUTCOMES

Consultants visit firms to collect information on business practices and entrepreneur’s profile

# of firms visited

# of firms that accessed the materials

Emails and SMS sent to firms with links to materials (free)

# of ‘best’ business practices (BBP) adopted

# of firms that contacted Sebrae-RJ for further support

Increased sales Higher profitsHigher productivityMore employees

Diagnostic on business practices

Adoption of BBP according to entrepreneurs profile

Diagnostic on entrepreneur’s profile

Feedback with benchmark on best practices

‘Entrepreneur’s spirit’Feedback with benchmark on best practices + info on expected returns + videos with testimony of successful cases

Feedback with benchmark on best practices + info on expected returns

Standard feedback (current approach)

Adoption of BBP via Sebrae’smaterials/services


• Overall: What is the impact of best business practices on firm’s performance?

• Phase 1: 2017 (refine diagnostic + pilot)

• Diagnostic with the intention to pre-identify high growth potentials

• Pilot:• Specific question 1: does feedback (with benchmark) on business practices affect

adoption of best practices

• Specific 2: Is the adoption of best practices via Sebrae’s materials?

• Specific question 3: is adoption different according to entrepreneur’s profile/expectations?

• Specific question 4: Do reminders via SMS affect access to and use of Sebrae’s materials?

Impact Evaluation Questions• Overall: What is the impact of best business practices on firm’s performance

• Phase 2: 2018

• Specific question 1 (lack of info): does feedback (with benchmark) on business practices affect adoption of best

practices

• Specific question 2 (lack of info + misperception): does benchmark feedback + info on expected returns associated

to adoption of business practices affect adoption of best practices?

• Specific question 3 (lack of info + misperception + lack of motivation): does benchmark feedback + info on returns +

demonstration of successful cases affect adoption of best practices?

• Specific question 4: Is the adoption via Sebrae’s materials?

• Specific question 5: Do (SMS) reminders affect use of Sebrae’s materials?

• Specific question 6: Is adoption different according to entrepreneur’s profile?

• Main indicators and data

• Indicators: business practices, entrepreneur’s spirit, subjective expectations, and number of employees, wages, revenue, and profits

• Survey data collected in loco during consultants’ visits and administrative data

Methodology Total sample size

5000

Arm 2: Current Approach

+ Benchmarking

[n = 1000]

Arm 1: Current Approach

[Diagnosis + Feedback]

[n = 1000]

Arm 4: Arm 3 +

Short videos with testimony

of successful entrepreneurs

[n = 1000]

Arm 3: Arm 2 +

Information on Returns

[n = 1000]

Control Group

[1000]

[No intervention]

Arm 1:

[333]

[1 SMS]

Arm 2:

[333]

[2 SMS]

Arm 3:

[333]

[3 SMS]

Arm 1:

[333]

[1 SMS]

Arm 2:

[333]

[2 SMS]

Arm 3:

[333]

[3 SMS]

Arm 1:

[333]

[1 SMS]

Arm 2:

[333]

[2 SMS]

Arm 3:

[333]

[3 SMS]

Arm 1:

[333]

[1 SMS]

Arm 2:

[333]

[2 SMS]

Arm 3:

[333]

[3 SMS]

We may stratify the sample according to entrepreneur’s profile before randomizing to be able to test for heterogeneous effects

Challenges

• Discuss potential challenges/risks and how to tackle/mitigate them

• Take up: the risk of having firms denying to answer the survey exist but it shouldn’t affect take-up because there is a long backup list where new firms can be drawn from

• Contamination of control group: this is a real risk and we are considering a cluster randomization at neighborhood level to mitigate the problem

• Implementation failures:

• Measuring spillovers: if cluster randomized trial, we will explore the idea of conducting a saturation experiment where the proportion of treated firms varies in each neighborhood


Concept

Aug 2017

Baseline

July-Sept 2018

Implementation

July-Sept 2018

Follow up 1

July-Sept 2019

Follow up 2

July-Sept 2020

Final dissemination

March 2021

Funding applicationApproval process

Cleaned datasetData analysisClient workshop

Monitoring reportLessons note

Cleaned datasetData analysisClient workshopTechnical paperPolicy note

Cleaned datasetData analysis

Technical paperPolicy noteClient workshop

For 2017: • Non-experimental ex-post IE to measure effects on adoption of business

practices (assessment of the current version of the program)• Pilot to measure effects of providing benchmarking based on the practices

adopted by the best peers in a given sector • Experiment the number of reminders via SMS seeking to increase adoption

of good practices

Thank you!

March 2017, Mexico

Entrepreneur versus Ecosystem?Examining the Impact of Radical Innovation

Programs on Firm Growth

Motivation

Source: BCRP, RICYT, Concytec.

Total Factor Productivity slowdown in Peru

3,0

6,3

5,2

3,7

-1

1

3

5

7

1997-2003 2004-2010 2011-2013 2014-2015

Empleo ajustado por Capital HumanoCapital FísicoProductividad Total de FactoresPBI Potencial

Growth Potential(annual change as % of GDP)

117Source: RICYT

Innovation(R&D as % of GDP)

0,15% 0,20%

0,50% 0,50%

0,80%

1,20%

2,40%

2,73%

0,00%

0,50%

1,00%

1,50%

2,00%

2,50%

3,00%

Perú Colombia Chile México Prom.

LatAm y

Caribe

Brasil OCDE EEUU

• Innovation (by SMEs) a key determinant of TFP… but also low!

Motivation

2 novel initiatives

118

Entrepreneur exposure: since 2015, 2 entrepreneurs per year

119

Gastronomic tourism

2 novel initiatives

120

Gastronomic tourism

2 novel initiatives

“Innovation Champion”

Behavior change: Future vision Collective message Revaloración de

productos nativos Sinergias entre todos

los actores del ecosistema

A success story

Scaling up 2 novel initiatives

121

2 Interventions

T2: “Ecosystem” (firm-level)

Change the behavior

Innovation Champion

Helps to connect the ecosystem

T1: “Entrepreneur-exposure”

Change the mind-set of the open to change

- Risk-Taking

- Ambition for growth

- Appetite for experimentation

- De-stigmatizing failure

Research Questions

122

1. What is the impact of “entrepreneur versusecosystem” innovation interventions on firmperformance?

Main outcomes Productivit Sales Profits

2. Which innovation focus (or both) drives more radical innovations?

3. What are the internal and external innovation-related outcomes that result from more radical innovations?

Internal innovation-related outcomes (4Ps) Processes Products People Pricing & mark-ups

External innovation-related outcomes (4Cs): Competition Customer Chain Collaborators

Intervention-directly-related outcomes Continuum of incremental to radical innovations (TBC)

123

16 sectorsidentified by

Government as

strategicallyimportant for

economic growth

5 sectors chosen by the

Government to focus for the

lending operation

2 sectors for the

RCT

Research Design: Sample

124

Field work to be used to select the sectorsSectors with potential big impact on innovation, productivity, and jobs.To be defined: we will ignore the subsistence level and the big firms. Largesample of micro (5+; rule out 0-5), small, and potentially medium-sized firms;large firms will be excluded.

Screening steps to identify the study sampleGovernment register plus door-to-door marking of the programPitch the program and only bring in interested candidates/firms (Self-reported interest)

Screening process to identify firms with higher potential to innovate (Data)Proxies: GPI score cardPractices-BPI (or a mixed of both)Other restrictions? (i.e., language for training, industries/products)

Target sample size 1500 firms only for the study

Research Design: screeninng process to build the sample

125

Unit of analysis: individual firm owner

Randomization: stratified at the sector level and randomization within sectors

Research Design: Experiment

Sample1500 firms (screened)

Sector A(N=750 screened into sample)

T1 (N=250 firms)

T2 (N=250 firms)

C (N=250 firms)

Sector B(N=750 screened into sample)

T1 (N=250 firms)

T2 (N=250 firms)

C (N=250 firms)

126

Baseline

Timeline

T1= 12 months

Follow up 1

T2= 24 months

Follow up 2

Discussion

Ethiopia NQITarget for Growth: Impact Evaluation Workshop

Mexico City


Ethiopia

THE NATIONAL QUALITY INFRASTRUCTURE (NQI)

TECHNICAL INFRASTRUCTURE GOVERNANCE

NQI Technical Committee (NQITC)

MOST National Science, Technology and Innovation Council (NSTIC)

National Quality Forum (NQF)

REGULATORY GOVERNANCE

MOT Inspection and Regulatory Directorate

Ethiopian Consumer Affairs

Ministry 1

Ministry 2

Ministry 3

Ministry 4

Ministry n

MOT

WTO Notification Authority

Import and Export Goods Co-ordination Control

Office for Co-ordination of Technical Regulations (TRO)

Legal Metrology

WTO National Enquiry Point

Technical Regulations Forum

MOST

NQI Directorate

TECHNICAL INFRASTRUCTURE

1

2

3

45

6

7

8

9

1011

13

12

TECHNICAL INFRASTRUCTURETechnical Infrastructure Governance

MOSTNQI Directorate (General)

SPS TBT

Private Conformity Assessment service providers

Proposed Development Objective

to improve the delivery ofquality assurance services toenterprises in the targetedsectors.

Interventions to be Evaluated

• Background: Ethiopian firms struggle to export due to trouble meeting export requirements (high rejection rates from advanced markets)

• Four institutes (standards, metrology, accreditation, conformity assessment)

• Target firms: • Potential exporters in leather, textile and garment, agro processing industries

• ~500 firms


• Does the preferred access to QI services increase firms’ usage of QI services? [improving supply]

• Does the provision of tailored diagnostic support in addition to preferred access increase firms’ usage of QI services? [improving demand]

• Does the increased usage of QI services increase firm growth and productivity (profits, volume of exports, number of new export contracts, employment)? [impact]

• Does the increased demand and usage of QI services increase the quality of input from suppliers to QI service users? [spillover]

Methodology

Total Sample Size

Firms screened for

interest

300-400

Treatment 2

Preferred Access

And

Detailed Diagnostic

100-133

Treatment 1

Preferred Access

100-133

Control Group

100-133

Potential Exporters

500

Methodology / Data Collection

• Firm survey of potential exporters

• Mapping exercise of supply chains to understand linkages and potential spillovers

• Surveys of actors down the supply chain• Ex. Leather goods sector: Traders, farmers, intermediaries

• Potential export market analysis

Theory of Change


Provision of

internationally

recognized QI services

Increased firm

compliance to

quality requirements

ACTIVITIES

Provision of QI

ServicesIncreased export

Increased

productivity

Firm growth

Provision of tailor-

made diagnostic

services

Preferred access to

QI Services

Reduced compliance

cost of businesses

Firms have a better

understanding of

quality improvements

Reduced QI service

timeIncreased quality

of suppliers down

the supply chain*

Its own theory of

change


Access to new

buyers

(domestic and

abroad)

Ability to meet

export

requirements

Concluding Remarks

Without Effective and Trusted NQI System,

• No Strong Enterprises፣

• Neither Development nor Competitiveness are possible.

Increasing competitiveness and jobs through matching grants: evidence from Kosovo


Mexico City


Intervention to be Evaluated

• Implementing agency: Kosovo Investment and Enterprise Support Agency, KIESA (good buy-in, limited capacity)

• Overall goals: Increase SME exports and improve SME performance

• Components? Matching grant scheme (co-finance 50%) to provide (i) quality certification and standards, and (ii) access to business development services

• Scope: Plan is to support 150 SMEs with average size of grant being $16-17k implemented over 1-1.5 years (certification may take 1.5 years if firm is far from ready for certification)

• Eligibility: SME registered in Kosovo, 10-50 employees, 2 years operating, not in natural resources and not harming environment

• Reach: the program will cover SME at national level

Theory of Change


Number of firms

applying to matching

grant scheme

ACTIVITIES

Encouragement to

help a subgroup of

firms apply to the

matching grant

scheme

Volume of exports per

firm in USD

Numbers of new

products exported and

new markets reached

Delivery of matching

grant – for product

quality certification

and standards

Delivery of matching

grant – for access to

BDS

Number of quality

certifications provided

Number of standards

implemented

Number of BDS

delivered (in total and

by categories)

Firm performance

(revenue, profit,

employment,

investment,

productivity)


Increased capacity and

know-how of firms and

training of employees


• What is the impact of providing business development services on firm exports, employment, and productivity?

• What is the impact of providing quality certification and standards along with business development services on firm exports, employment, and productivity?

• Main indicators and data• Using the results chain, identify and define key indicators: volume of exports per

firm, numbers of new products exported and new markets reached per firm, measures of firm performance (revenue, profit, employment, investment, productivity)

• Describe data availability: exporter customs data, Kosovo Statistics Agency firm annual survey, Tax administration data, Business registry data, application form data, we will need to conduct follow-up surveys (ideally 2 or 3)

Methodology

• Sampling: call for applications, among applicants eligibility criteria will define a cutoff point, among eligible applicants above cutoff point we will randomly select groups for treatment and control

• Expectation would be 1000 applicants out of which 300 would be above cutoff

• Matching grant scheme has funding to provide grants to 150 beneficiaries

• Depending on actual applications we will know how many firms will get treatment in each of bullets in Treatment 1

• Data will be collected via surveys and existing administrative data

Total sample size

300

Control Group

150

Treatment 2

May be added during

implementation

Treatment 1

150

• Matching grant used to access to

Business Development Services

• Matching grant for product conformity

• Matching grant used for both above

activities

Challenges• Challenge 1: Take up at application and eligibility phases

• At application phase • Very good communication campaign, an easy application process, telephone and online help• Encouragement design (drawing lessons on failed matching grants in Africa & fear that not enough firms will apply for 50-50 grant when

previous schemes in Kosovo were 75-25)

• At eligibility phase • Spell out clear and objective criteria (e.g., business plan must include X, Y, Z sections)• Ask applicants for proof of availability of co-financing to reduce drop out at implementation stage

• Challenge 2: Implementation failures – at disbursement phase• To easily disburse: make payment directly to vendor rather than firm paying upfront and getting reimbursed• Require progress reports from beneficiaries

• Challenge 3: Availability and quality of consultants in Kosovo• Beneficiaries will be provided with information on a network of consultants in the Balkan region

• Challenge 4: Political buy-in of randomized treatment and private sector reluctance to accept• Complete buy-in from KIESA CEO and good communication with public and private sector explaining transparency of

lottery

• Challenge 5: Contamination of control group• Keep good data on control firms and may enable us to measure additionality of matching grant scheme

• Challenge 6: Time frame may be too short to see export increases


Concept

Aug 2017

Baseline

1st round

Q1 18

Implementation

Q2 18-Q2 19

Follow up 1

Q2 19

Follow up 2

Q2 20

Final dissemination

2021 (?)



• Cleaned dataset


workshop


• Lessons note

• Cleaned dataset




• Cleaned dataset

• Data analysis

• Technical paper


workshop

Thank you!

Discussion

THE END

project presentations - world bankpubdocs.worldbank.org/.../day-4-project-presentations.pdf ·...

Documents