innovative techniques to evaluate the impact of private ......the government of rwanda recently...

Innovative techniques to evaluate the Impact of private sector development reforms:

An application to Rwanda and 11 other countries

Sachin Gathani Massimiliano Santini Dimitri Stoelinga1

This version: February 06, 2013

Accepted by the MPSA Annual Conference, 11-14 April 2013

Abstract The objectives of this paper are twofold: (1) to show how the synthetic control methodology can be used to measure the impact of private sector development reforms, and (2) to introduce a new technique, the proximity control methodology, that offers similar advantages but greater flexibility than the synthetic control methodology to test the validity of results. While maintaining the technical rigor of other econometric techniques used to conduct impact evaluations, these two methods are quicker cost-effective alternatives that can be applied to measure ex-post the impact of policy reforms in a given country or region. They can also easily be replicated to similar reforms in other countries. We illustrate this by using both methodologies to estimate the impact of the introduction of a one-stop shop for business registration on new firm creation in Rwanda and 11 other countries. Both approaches yield similar and comparable results and show that one-stop-shops can have a very large impact: in Rwanda for example, we observe a 186% average increase in new firm creation after the reform was introduced, in Tajikistan a 132% increase, and in Belarus a 103% increase. In this paper we also propose a new way of looking at the Doing Business dataset by introducing a measure of the similarity of the business environment of a pair of countries. This metric offers a more accurate comparative representation of a country’s business environment than more traditional metrics.

JEL Classification: C21, C23, G18, L51, M13

Key Words: Entrepreneurship, Business Environment, Investment Climate, Impact Evaluation, Synthetic Control, Proximity Control, Doing Business, Rwanda

1 Massimiliano Santini (email: [email protected]) is an Economist at the World Bank Group. Sachin Gathani (email: sgathani@laterite-‐africa.com) and Dimitri Stoelinga (email: dstoelinga@laterite-‐africa.com) are Partners at Laterite Ltd. (www.laterite-‐africa.com), a research firm based in Rwanda and Malawi. This paper was prepared with assistance from Gabriela Armenta and Maria Paula Gomez. Thanks to Miriam Bruhn, Alexis Diamond, Jeffrey Grogger, Robert Lalonde, David McKenzie, and Ricardo Sabates for very helpful comments. We also thank the Rwanda Development Board for sharing updated monthly firm registration data. All mistakes in this paper are our own.

Innovative techniques to evaluate the Impact of private sector development reforms.

2

Table of Contents

1. Introduction ........................................................................................................................................... 3

2. Why business entry reforms matter: a brief summary of the literature ........................................ 4

3. Using the synthetic control methodology to measure the impact of private sector development reforms ................................................................................................................................ 6

3.1 Data ................................................................................................................................................. 7

3.2 Rwanda: measuring the impact of a one-stop shop with the synthetic control methodology ................................................................................................................................................................. 8

3.3 Testing the robustness of the results obtained using the synthetic control methodology . 13

3.4 Rwanda’s one-stop shop and the link to new business creation .......................................... 15

4. Introducing the proximity controls methodology ............................................................................ 19

4.1 Measuring and visualizing the Doing Business similarity space .......................................... 20

4.2 The properties of the Doing Business similarity space .......................................................... 24

4.3 Rwanda: measuring the impact of a one-stop shop with the proximity controls methodology ........................................................................................................................................ 26

4.4 Testing the robustness of the results obtained using the proximity controls methodology ............................................................................................................................................................... 30

5. Replicating and comparing both methodologies to 11 other countries ...................................... 39

6. Conclusion .......................................................................................................................................... 44

Bibliography ............................................................................................................................................. 47

Annexes .................................................................................................................................................... 49


3

1. Introduction

In recent years, the development community has been facing increasing pressure from

donors and client countries to show the impact of its interventions. Delivering technical

assistance and lending money are not enough anymore. By isolating the intended result of the

project from everything else that happened at the same time, rigorous impact evaluations tell us

what works and why, making a case for successful policy interventions to be scaled up and

replicated.

Unfortunately, impact evaluations are usually costly and time consuming, and in the real

world are implemented only in a handful of projects. As a result, all sort of assumptions need to

be established to extrapolate the results obtained by one country-specific project to another

country’s similar intervention (for example, to which degree can we assume that conditional

cash transfers that increase returns to education in Mexico will have the same effect in the

Kyrgyz Republic?).

While maintaining the technical rigor of other econometric techniques used to conduct

impact evaluations, the synthetic control methodology is a low-cost alternative that can be

applied to policy changes with aggregate country-level effects and easily replicated to similar

policy changes in other countries. This methodology was first introduced by Abadie and

Gardeazabal in 2003.

he objectives of this paper are (1) to show a step-by-step application of the synthetic

control methodology to measure the impact of the introduction of a one-stop shop for business

registration on new firms created in Rwanda, and replicate the methodology to eleven other

countries; and (2) to introduce a new technique, the proximity control methodology, that offers

the same advantages as the synthetic control methodology but may be easier to apply, and to

show its application to the same business entry reforms in Rwanda and the eleven other

countries.

The principle behind both approaches is relatively straightforward: both techniques use a

linear combination of control countries – i.e. countries without one-stop shops – to create a

“synthetic control region” that accurately fits the reference country on certain variables of

interest before the introduction of the one-stop shops. If the synthetic region closely resembles

the reference country, and accurately predicts new firms creation in the reference country

before the introduction of the one-stop shop, then the synthetic control region is likely to be a

relatively accurate predictor of what would have happened in the reference country had the one-

stop shop not been introduced.

T


4

In addition, we will also show how to compare the business environments of countries by

using a metric of business environment similarity developed for the proximity control

methodology, which we argue presents many advantages over the Doing Business aggregate

index of the ease of doing business.

The paper proceeds as follows. After the introduction and a brief overview of the

literature on the impact of business entry reforms, chapter three will make the case for using the

synthetic control methodology to estimate the impact of private sector development reforms,

and apply the methodology step-by-step to evaluate the impact of the introduction of a one-stop

shop for business registration in Rwanda. Chapter four will introduce the proximity control

methodology and apply it to the case of Rwanda too. Chapter five will summarize the results

obtained using both methodologies estimating the impact of one-stop shop reforms in eleven

other countries, before we conclude assessing the pros and cons of each methodology, and

how they may be used to conduct cost-effective and rigorous impact evaluations of private

sector development reforms.

2. Why business entry reforms matter: a brief summary of the literature

In recent years, many countries have reformed the business entry process by reducing

the time, cost, number of procedures and minimum capital necessary to start a business, with

the declared intent of stimulating private sector activity. Typical reforms included the reduction

of unnecessary license requirements, the streamlining of business entry processes, the

reduction of overhead costs, and the improved coordination between regulatory agencies.

Business entry reforms are relatively easier to implement than other private sector development

reforms, and policy makers have found it straightforward to build consensus for their

implementation. Approximately 80% of the 183 economies measured by the Doing Business

report have made it easier to start a business since 2003.2 Specifically, 348 business

registration reforms were introduced in 146 countries in June 2003-May 2011, about 20% of all

investment climate reforms reported in the same period by the Doing Business report.3

One of the most popular business entry reforms is the introduction of one-stop shops for

business registration, which provide entrepreneurs with a single place where they can fulfill all

the requirements necessary to start their businesses. A literature review on business entry

reforms found that (1) the introduction of significant business entry reforms, like a one-stop shop

2 Doing Business database, 2005-‐2012. 3 Ibid.


5

for business registration, is directly associated with an increase in the number of firms, and that

(2) a significant reduction in business registration costs affects new firm creation more in

industries with low barriers to entry than in those with high barriers.4

Klapper and Love (2010) show that business registration reforms that cut cost and/or

time by more than 40% in 92 countries during the 2003-2008 period, like the introduction of a

one-stop shop, had a statistically significant impact on new business creation. Conversely,

smaller reforms – such as the city of Lima’s (Peru) simplification of just the process to obtain a

license to start a business – seem to have had no significant effect on firm performance.5 In

Portugal, the introduction of a one-stop shop in 308 counties, which decreased the number of

days to register a business by 91%, led to an increase of new firms created by over 17%.6 On

the other hand, a study of SMEs in Vietnam found that the decision of firms to formalize during

the period 2007-2009 led to an increase in firms’ gross profits and investments.7

Preliminary evidence shows that significant business entry reforms can encourage job

creation too. In Mexico, the introduction of a one-stop shop for business registration was

associated with an increase of 2.2-8% in employment.8 In Portugal, the introduction of the one-

stop shop led to an increase in employment by 21%. A cross-country study showed that a

decrease of 61% in the number of days to register a business is associated with an increase of

0.4% in (manufacturing) employment.9

In the next chapter, we will explain step-by-step how we can use the synthetic control

methodology to estimate the impact of the introduction of the one-stop shop on new firm

registration in Rwanda in 2009. The Government of Rwanda recently implemented two major

waves of business environment reforms: the first in April/May 2009, when the one-stop shop for

business registration was created; the second in April/May 2010. Due to limited data availability,

we focus only on the impact of the first wave of reforms and in particular the impact of Rwanda’s

one-stop shop for business registration on new firm creation in Rwanda in 2009. We will also

discuss various approaches to test the validity of the results obtained.

4 Motta et al. (2010). 5 Lorena Alcázar, Miguel Jaramillo, “Panel/tracer Study on the Impact of Business Facilitation Processes on Microenterprises and Identification of Priorities for Future Business Enabling Environment Projects in Lima, Peru”, Mimeo, Grade, June 2011. 6 Branstetter, Lima, Taylor, Venancio, “Do Entry Regulations Deter Entrepreneurship and Job Creation? Evidence from Recent Reforms in Portugal” NBER Working Paper 16473, October 2010. 7 John Rand, Nina Torm, “The Benefits of Formalization: Evidence from Vietnamese Manufacturing SMEs,” World Development Vol. 40, No. 5, pp. 983–998, 2012. 8 Bruhn, “License to Sell: The Effect of Business Registration Reform on Entrepreneurial Activity in Mexico,” World Bank Policy Research Working Paper No. 4538, January 2008. Kaplan, Piedra, Seira, “Entry Regulation and Business Start-‐Ups: Evidence from Mexico,” World Bank Policy Research Working Paper No. 4322, June 2007. 9 Ciccone and Papaioannuou, “Red Tape and Delayed Entry”, Journal of the European Economic Association, vol.5, no.2-‐3, pp.444-‐458, 2007.


6

3. Using the synthetic control methodology to measure the impact of private sector

development reforms

The synthetic control methodology developed by Abadie and Gardeazabal enables

researchers to conduct aggregate-level impact evaluations at national, regional and sectoral

levels. In the next chapter, we will show how it can be applied to estimate the impact of the

introduction of a one-stop shop on new firms created in Rwanda.10 Despite significant

advantages over alternative techniques (both from a theoretical and cost-effective

perspective)11, this approach to impact evaluations has been underutilized in both policy and

academic spheres. By illustrating how it can be applied in a concrete, step-by-step way, we

hope to encourage its wider use by policy makers and researchers, and unlock its potential use

for measuring the impact of private sector development reforms.

As its name suggests, the main objective of the synthetic control methodology is to

create a control region for a geographic area where a policy change (the “intervention”) has

taken place (the “treatment region”). The control region is called “synthetic” because it is

constructed using a linear combination of alternative regions where the intervention has not

taken place (the “control region”). By definition, a good control region is a region that perfectly

matches the treatment region on a number of key characteristics, before the intervention takes

place.12 If two regions – the treatment and the synthetic control regions – are relatively similar

on key characteristics and have a similar performance on the variable of interest over a certain

period of time, then the synthetic control region is likely to be a good predictor of what would

have happened in the treatment region had the event or intervention not taken place. The

observed difference between the performance of the treatment region and the control region on

the variable of interest after the intervention is our estimate of impact.

For example, in their paper on the impact of terrorism on economic growth in the Basque

region, Abadie and Gardeazabal (2003) construct a “synthetic Basque region,” using a linear

combination of other Spanish regions that minimizes the difference between the synthetic

Basque region and the actual Basque region on the following indicators: real GDP per capita,

investment ratio, population density, sector shares as a percentage of GDP, and human capital

indicators (illiteracy rate, primary and secondary education enrollment rates). The resulting

synthetic Basque region closely resembles the Basque region on these economic determinants

before the beginning of terrorist activity, and it also perfectly matches economic growth in the 10 For the purposes of this paper, we use the term “firm” as a synonym of business, company, partnership and corporation. 11 See Abadie et al. (2003). 12 Ibidem.


7

Basque country for a period of 20 years before the beginning of terrorist activity. The observed

deviation between the Basque country and the synthetic Basque region after the start of terrorist

activity is the author’s estimated impact of terrorist activity on GDP growth in the Basque

country.

3.1 Data

In order to measure the impact of improving business entry regulation on new firms

creation, we build a dataset that codes the 183 countries measured by the Doing Business 2012

report based on whether or not they have introduced a one-stop shop for business registration.

The one-stop shop for business registration is defined as “an organization that (i) receives

documents for business registration and (ii) carries out at least one other function related to

business start-up. (e.g. tax registration, social security registration, statistical agency

registration, etc.).”13 In the same dataset, we indicate the year of introduction of the one-stop

shop and source this information for each country.

Our outcome of interest is “new firms registered.” We use the dataset from the 2010

World Bank Group Entrepreneurship Snapshots (WBGES),14 which contains country-level data

on “new firms” and “entry density” from 2004 to 2009. New firms are defined as private

companies with limited liability, which is the same definition used by the Doing Business reports.

Entry density is defined as the number of new firms per 1,000 working-age people (15-64 years

old). Throughout the paper, we use the variable “entry density” to do all the calculations, but we

often interpret the results in terms of the variable “new firms” because of its more common use.

While the WBGES is the most comprehensive cross-country dataset currently available

on firms’ registration, it excludes the registration of sole-proprietors. The Doing Business

dataset on the ease of the business environment only includes data affecting limited liability

companies, and it excludes data on sole proprietors as well. By limiting our analysis on the

impact of the creation of limited liability companies, we may miss some implications on easing

business regulations on micro and small firms, in particular in the context of the determinants of

informality. On the other hand, we think that the results could serve as an accurate proxy for the

dynamics of the formal sector as a whole.

13 Investment Climate Advisory Services, 2010. “How many stops in a one-‐stop shop? A review of Recent Development in Business registration,” Flagship report, The World Bank. 14 Publicly available from: http://go.worldbank.org/C8Q8EGTTH0.


8

3.2 Rwanda: measuring the impact of a one-stop shop with the synthetic control methodology

In order to measure the impact of a one-stop shop on new firm creation in Rwanda, we

create a “synthetic” counter-factual of Rwanda following a 5-step process:

Step 1.

The first step in constructing a synthetic control for Rwanda is clearly defining the

variable of interest. We are interested in measuring the impact of the introduction of the one-

stop shop in Rwanda on new firms created. Based on available data, there are two ways of

measuring new firm creation: (i) by number of new firms registered; (ii) or by new business

density (NBDEN), measured as the number of new firms registered per 1,000 inhabitants.

In order to compare countries using the synthetic control methodology, the latter is

preferable, as we can compare countries with similar characteristics keeping population

constant. This is analogous to comparing two countries based on their GDP per capita rather

than their total GDP. Once results have been obtained in terms of new business density, we

translate them back into number of new firms registered, a more tangible metric for policy

makers.

Step 2.

Which predictors do we select to match Rwanda to its synthetic control region, given that

we are interested in new business density? Our objective is to create a region that is similar

enough to Rwanda prior to the introduction of the one-stop shop, on both (i) new business

density; and (ii) key characteristics that play an important role in determining the level of new

firm creation. The selection of predictors should reflect our knowledge on the variables that are

good predictors of new business density.

Given the nature of new business creation, we choose to focus on selected macro-

economic variables that capture information on the structure and level of economic development

of the economy: GDP per capita, agriculture (% GDP), industry (% GDP), services (% GDP),

gross fixed capital formation (% GDP), trade balance (% GDP), and urban population (% total

population). As we can see in graph 1 below, these variables are good predictors on average of


9

the level of new business density (R²=0.49, using average 2000-2009 data from 105 different

countries)15.

Step 3.

The third step in constructing a synthetic control for Rwanda involves selecting the time

period during which the difference between Rwanda and the synthetic Rwanda is minimized.

Given that Rwanda introduced the first package of business registration reforms connected with

the one-stop shop in 2009 – with immediate effect – we use data prior to 2009 to match Rwanda

to the optimal linear combination of control countries. Data on new business density in Rwanda

is only available starting in 2003, so we match Rwanda to its synthetic control using 2003-2008

data; we call this the input period. The output of the synthetic control method is an estimate of

new business creation in Rwanda’s Synthetic Control both before and after the introduction of

the one stop shop. We call the 2003-2009 period the output period. We exclude the year 2010

and the impact of Rwanda’s second major reform package from the analysis because business

registration data is not available for some of Rwanda’s comparators.

15 We predict new business registration data with a simple regression of selected explanatory variables on average values for the 2000-‐2009 period.

R² = 0.49062

-‐4

-‐3

-‐2

-‐1

0

1

2

3

-‐4 -‐3 -‐2 -‐1 0 1 2 3 4 Log New

Business R

egistrako

n (Predicted

)

Log New Business Registrakon (Actual)

Graph 1. Log New Business Density: Predicted vs Actual (2000-‐2009)


10

Step 4.

Then, we identify a pool of potential control countries from which the synthetic Rwanda is

constructed – borrowing from the statistical literature on matching, Abadie et al. (2007) call this

the donor pool. We establish the donor pool using three criteria:

1. Given that the “treatment” is the introduction of one-stop shop, we eliminate from the donor

pool all countries that already had or introduced a one-stop shop during the output period –

i.e. before and after Rwanda introduced its one-stop-shop. This leaves us with a pool of

countries where the “treatment” did not take place. This means that any Synthetic Country –

constructed as a linear combination of any one of these control countries – did not

experience the introduction of a one-stop shop at any point between 2003 and 2009.

2. We eliminate from the donor pool all countries for which we do not have the required data

during the period 2003-2009. This includes: (i) all countries for which new business density

data is missing during the 2003-2009 period; (ii) all countries for which we do not have at

least one data point during the input period for each of the predictors.

3. In order to avoid biases caused by interpolating across regions with very different

characteristics,16 we eliminate from the donor pool all countries that on average during the

input period had a GDP per capital level greater than USD 1,000 (constant USD), compared

to USD 275 for Rwanda. The objective is to strike a balance between the size of the donor

pool on one hand, and how similar the characteristics of countries within that donor pool are

on the other.

This leaves us with a donor-pool of seven countries for Rwanda: Cambodia, Ethiopia,

Indonesia, Moldova, Malawi, Pakistan, and Uganda.

Step 5.

With the variable of interest, the predictors, the time-period and the “donor pool” now in

place, we can construct a synthetic Rwanda following the methodology outlined by Abadie et al.

(2003 and 2007). Synthetic Rwanda is constructed as the linear combination of countries in the

“donor pool” that most closely resemble Rwanda in terms of the variable of interest and the

predictors prior to the introduction of the one-stop shop (i.e. during the 2003-2008 period). 16 See Abadie et al. (2007).


11

Table 1: Predictors (averages 2003-2008)

The resulting Synthetic Rwanda consists of a linear combination of Cambodia (40.5%),

Malawi (32.5%) and Ethiopia (27%). As we see on table 1, this Synthetic region closely matches

Rwanda on average on most of the selected predictors during the 2003-2008 period (pre-OSS):

GDP per capita, agriculture (% GDP), the trade balance, and the level of urbanization are

almost identical in both regions. Synthetic Rwanda is slightly more industrialized however

(19.9% vs. 13.8%), less service intensive (44.2% vs. 49.5%), and has a slightly higher

investment rate (20.3% vs. 17.6%). These difference are however relatively small and remain

constant during the 2003-2009 period; i.e. they do not explain the observed jump in new

business registration between 2008 and 2009.

In addition to fitting Rwanda on the selected explanatory variables, synthetic Rwanda

predicts new business density in Rwanda during the 2003-2008 period very well, before the

introduction of the one-stop shop (see graph 2).

Predictors Rwanda Synthetic

Rwanda

GDP per capita (constant 2000 USD) 279 263

Agriculture (% GDP) 36.7 35.9

Industry (% of GDP) 13.8 19.9

Services (% of GDP per capita) 49.5 44.2

Trade balance on goods and services (% GDP) -14.9 -14.0

Gross Fixed Capital Formation (% of GDP per capita) 17.6 20.3

Urban population (% total population) 17.5 18.2


12

The graph shows that while new business registration in Rwanda and Synthetic Rwanda

followed a relatively similar growth path between 2003-2008, new business registration in

Rwanda greatly increased in 2009 after the introduction of the one-stop shop. Following Abadie

et al. (2003), a simple difference-in-difference calculation enables us to estimate the impact of

Rwanda’s one-stop shop on new business density. We estimate that that the introduction of the

one-stop shop and related reforms led to the registration of 2,041 new firms in 2009 alone,

which is equivalent to an increase of 188% in new firms created after the one-stop shop was

introduced (note that the one-stop shop was created in May 2009). In other words, after the

introduction of the one-stop shop, 188% more firms registered than they would have registered

had the one-stop shop not been introduced.

In the next section we test the robustness of these results, before focusing in chapter 3.4

on why this increase may be attributed to the introduction of the one-stop shop and not to other

factors happening in the country at the same time.

0

500

1000

1500

2000

2500

3000

3500

2003 2004 2005 2006 2007 2008 2009

New

Firm

Registrako

n

Graph 2. New Firm RegistraGon in Rwanda and SyntheGc Rwanda

Rwanda

Synthekc Rwanda


13

3.3 Testing the robustness of the results obtained using the synthetic control methodology

The final step involves inferential analysis and testing the validity of the results.

Following Abadie et al. (2003, 2007) we propose two different techniques, the falsification test

and the Mean Squared Prediction Error test.17 The objective of the falsification test is to ensure

that Synthetic Rwanda did not experience a shock itself in 2008-2009 (a “treatment”) as this

would entail that we are either under-estimating or over-estimating the impact of Rwanda’s one-

stop shop on new business registration. To test whether Synthetic Rwanda experienced a

positive or negative shock in 2008-2009, we apply the synthetic control methodology described

above (steps 1-4) to Synthetic Rwanda (which consists of a linear combination of Ethiopia,

Malawi and Cambodia), i.e. create a synthetic region for Synthetic Rwanda. We call the newly

created synthetic region, the Placebo region.

As can be seen on graph 3, Synthetic Rwanda and its Placebo region do not differ

significantly during the 2003-2009 period. The match between the two regions is not perfect –

Synthetic Rwanda seems to experience a small bump in 2006 - but the results strongly suggest

that Synthetic Rwanda did not experience an impact (or shock) in 2008-2009 that would explain

the observed difference between new business registration in Rwanda and Synthetic Rwanda.

17 We define the Mean Square Prediction Error as the mean of the squared differences between new business density in one region and another over a certain period of time.

-‐0.2

-‐0.1

0

0.1

0.2

0.3

0.4

0.5

2003 2005 2007 2009

Diffe

rence in NBD

EN between reference

coun

try and Syntek

c Co

ntrol

Graph 4. EsGmated New Business Density Increase in Rwanda and other

controls

Rwanda

0

500

1000

1500

2000

2500

3000

2003 2005 2007 2009

New

Firm

Registrako

n Graph 3. New Firm RegistraGon in SyntheGc Rwanda and Placebo

Rwanda

Synthekc Rwanda

Placebo


14

The difference between Synthetic Rwanda and the Placebo was 180 businesses in 2008,

compared to 172 in 2009, which is almost identical. The observed bump in Synthetic Rwanda is

due to Cambodia; in order to ensure that Cambodia (which accounts for 40.5% of Synthetic

Rwanda) is not significantly skewing results, we repeat the exercise without Cambodia and find

an impact of 180%, slightly lower than the estimated 188%. While this suggests we could be

overestimating the impact of the one-stop shop by 8 percentage points, it does not significantly

alter our overall conclusion that the one-stop shop has had an impact on new business

registration in Rwanda.

The second test asks the question: how unusual is the impact estimate obtained? Is it

due to chance? To answer this question we conduct synthetic control tests on all the countries

in Rwanda’s “donor pool” (testing for an impact in 2009) and compare the results to the

estimated impact of the one-stop shop on new business density in Rwanda. If the estimated

impact for Rwanda is unusual – higher than in other “donor pool” countries – then this is

additional evidence that the one-stop shop had an impact on new business density in Rwanda.

If this is not the case, then the observed difference between new business density in Rwanda

and Synthetic Rwanda could be due to chance rather than the one-stop shop. Abadie et al.

(2007) demonstrate that this iterative effort leads to exact inference.

Graph 4 clearly indicates that the estimated impact on new business density in Rwanda

is unusual compared to the seven other countries in Rwanda’s “donor pool”, thereby providing

additional evidence that the one-stop shop had a statistically significant impact on new business

density in Rwanda.

Another way of looking at these results is to compare the ratio of the Mean-Square

Prediction Error (MSPE) before and after the introduction of the one-stop shop in Rwanda to

that of the other countries/controls in Rwanda’s “donor group”. We formally calculate the MSPE

for country i using the following formula:

𝑀𝑆𝑃𝐸! =(𝐴𝑐𝑡𝑢𝑎𝑙!,!""# − 𝑆𝑦𝑛𝑡ℎ𝑒𝑡𝑖𝑐 𝐶𝑜𝑛𝑡𝑟𝑜𝑙!,!""#)!

(𝐴𝑐𝑡𝑢𝑎𝑙!,!""#!!""# − 𝑆𝑦𝑛𝑡ℎ𝑒𝑡𝑖𝑐 𝐶𝑜𝑛𝑡𝑟𝑜𝑙!,!""#!!""#)!!!

6

An MSPE smaller than 1 indicates that the observed impact in 2009 is not unusual i.e. it

is smaller than in other years before the intervention, while an MSPE of more than 1 indicates

that the observed impact is larger than in other years. The size of the ratio enables us to

compare how unusual the observed impact is across countries. Not surprisingly, we find that


15

this ratio is much higher in Rwanda than in other countries (see table 2) – indicating that the

impact we observe in Rwanda is indeed unique.

We conclude that the introduction of the one-stop shop in Rwanda had a significant

impact on new business density and led to an approximate increase of 188% in new firms

created.

Table 2: MSPE test

Country MSPE Ratio

Rwanda 388.21

Ethiopia 1.00

Indonesia 0.71

Cambodia 0.02

Moldova 0.35

Malawi 0.60

Pakistan 1.80

Uganda 1.98

3.4 Rwanda’s one-stop shop and the link to new business creation

Questions remain on whether the increase in new firms registered can be attributed

exclusively to the introduction of a one-stop shop or instead to other investment climate reforms

occurring concurrently or other changes in the economy. We argue that new business

registration can be linked to the introduction of the one stop-shop but is unlikely to be linked to

the introduction of concurrent reforms happening at the same time. Moreover, as we show in the

Annex, we find the same substantive and significant impact on new firms created in eleven

other countries after the introduction of a one-stop shop.

Even before introducing major reforms in 2009 and 2010, Rwanda had been an active

reformer of its business environment: in 2001, the Government introduced a new labor law; in

2002 a property titling reform; in 2004, it simplified customs procedures, improved the credit

registry and undertook court reforms; in 2007, Rwanda reformed property registration and

further improved customs procedures; and in 2008 certain judicial reforms came to completion,

leading to the introduction of new commercial courts.18 Despite these reforms, operating a

18 Doing Business 2009, World Bank Group.


16

business in Rwanda at the beginning of 2009 was not easy. Rwanda ranked 139th in the Doing

Business 2009 report (published in September 2008) and at about 0.19 firms per 1,000 people

(of working age), new business registration was one of the lowest in the world.

The year 2009 marked a major acceleration in the improvement of Rwanda’s investment

climate: on April 27th 2009 Rwanda enacted a new Companies Act, followed by the Mortgage

Law, the Secured Transactions Law and the Insolvency Law in May 2009.19 The Companies Act

strengthened investor protection and created the one-stop shop for business registration, which

opened its doors in May 2009. The creation of the one-stop shop led to an impressive reduction

in the time, cost and number of procedures required to start a business: the number of

procedures was reduced from 8 to 2, the time from 14 days to 3, and the cost from 108.9% of

GDP per capita to 10.1%. The Secured Transactions Law improved access to credit by

increasing the range of assets that can be used as collateral; the Insolvency Law eased the

process of filing for bankruptcy and closing a business; while the Mortgage Law shortened the

process of property registration. This mix of reforms resulted in Rwanda greatly improving its

business environment.20 This first reform package was followed by a second package in April

and May 2010, when the business environment was further facilitated by the introduction of free

online registration, a reduction in registration fees, new regulations regarding construction

permits, a further reduction in the documents required for exports and reforms in the access to

credit space.

May 2009, when the first package of business reforms were implemented, also marked a

turning point in new business registration (see graph 5). New business registration increased

from an average of about 100 firms per month from January 2008 through to April 2009, to an

average of about 300 firms between May 2009 and December 2009. In the sixteen months from

January 2008 to April 2009, 1,552 firms were registered in Rwanda; an equivalent number of

firms were created in just five months after these business reforms were enacted. After the

second package of reforms were enacted, new business registration further increased to reach

400-500 firms per month.

19 See Official Gazette n° special of 14/05/2010. 20 Rwanda improved its ranking in the Doing Business 2010 report from 139 to 67 in the space of a year, becoming the world top reformer.


17

Source: the Rwanda Development Board (2012)

No macro-economic changes in April and May 2009 can justify the observed increase in

new business registration. On the contrary, Rwanda, like many other developing countries,

suffered a slowing down in economic growth in 2009 as a direct result of the global financial

crisis. Rwanda’s real growth rate in 2009 was 6.2%, compared to 11.2% in 2008. In most

countries, the crisis translated into a drop in new business registration in 2009, which is what we

would have expected to see in Rwanda given the 5 percentage point drop in GDP growth.21

Instead, the timing of the increases in new business registration suggests that the

implementation of investment climate reforms, and in particular the one-stop shop for business

registration, are responsible for the increase in new business registration. Both accelerations in

new business registration in 2009 and 2010 occurred in the months of May and June, just after

the reform packages were implemented (see graph 5).

While it is impossible to disentangle the respective impact of each investment climate

reform – given that they were all passed approximately at the same time and that we have

limited data on the implementation timelines of these reforms – the most likely explanation as to

why new business registration accelerated immediately after the reforms were passed is the

creation of the one-stop shop and the associated decrease in the procedures, cost and time of

starting a new business. The one-stop shop had an immediate impact: it was widely publicized,

it provided a direct window for new business to register, and offered a very efficient service to

entrepreneurs.

21 Klapper, Leora and Love, Inessa, 2011. “The impact of the financial crisis on new firm registration,” Economics Letters, vol. 113(1), pages 1-‐4, October.

0 50 100 150 200 250 300 350 400 450 500

Jan-‐08

Mar-‐08

May-‐08

Jul-‐0

8 Sep-‐08

Nov-‐08

Jan-‐09

Mar-‐09

May-‐09

Jul-‐0

9 Sep-‐09

Nov-‐09

Jan-‐10

Mar-‐10

May-‐10

Jul-‐1

0 Sep-‐10

Nov-‐10

No firms p

er m

onth

Graph 5: New business registraGon per month in Rwanda

Reform package 1

Reform package 2 1


18

Other reforms certainly increased the attractiveness of starting a business, but the pass-

through effect is likely to have been much more gradual:

• Access to credit was improved, in theory, by the passing of new laws regarding collateral,

but credit to the private sector actually decreased in Rwanda between the months of May

and November 2009, making it unlikely that increased access to credit was behind the

rapid increase in new business registration in May and June 2009 (see graph 6).

• The new insolvency law may have improved insolvency procedures, but even today

Rwanda remains one of the least business friendly places in the world to close a

business (Rwanda ranked 165th on closing a business in DB2012).

• The 2009 reform to construction permits streamlined processes by combining the

applications for location clearance and the building permit in a single form and

introducing a single application form for water, sewerage, and electricity connections.

While this resulted in a reduction in the number of procedures and the time required for

dealing with construction permits, it nevertheless still required more than 200 days and

600% of income per capita to obtain a construction permit in 2009.22 Kigali City’s One

Center for construction permits, which resulted in a further decline in the time required to

get construction permits only started in operations in April 2010.23 Moreover, most new

firms created in Rwanda since 2009 are in the retail/wholesale sector and do not

necessarily involve construction.24 Construction permit reforms in 2009 fail to explain the

immediate post-reform increased in new business registration in the months of May and

June 2009.

22 IFC and World Bank, 2012. “Doing Business in a more transparent world. Economy Profile: Rwanda” 23 Official Gazette n°22 bis of 31/05/2010 24 Rwanda National Institute of Statistics, 2012. “Establishment Census 2012”, www.nisr.gov.rw


19

The combination of these factors strongly suggests that while concurrent reforms to the

one-stop shop, enacted in May 2009, contributed to improving Rwanda’s general business

environment, they are unlikely to have had an immediate impact on new firm creation.

4. Introducing the proximity controls methodology

The proximity control methodology is largely inspired by the synthetic control

methodology, but it relies on a different technique - and application - to construct the “synthetic”

region. By using the Doing Business indicators, we show that it is possible to construct a

relatively accurate control region for a reference country using linear combinations of countries

with the most similar business environment to the reference region. We also highlight a different

way of looking at the Doing Business indicators – comparing countries not by ranking but by

how similar their Doing Business indicators are. This can lead to many other interesting

applications and presents a new way of representing and communicating the findings of the

Doing Business reports.

The logic of the approach we propose is straightforward: if we assume that a country’s

business environment as measured by Doing Business is one of the most important

determinants of new firms creation - and that changes in a country’s business environment as

measured by Doing Business can significantly impact new firms creation - then it is likely that

countries with very similar business environment as measured by Doing Business will have

similar new business density or new business density growth. If this assumption holds, then it

should be possible to estimate new business density or new business density growth in a

250 270 290 310 330 350 370 390 410

Jan-‐08

Mar-‐08

May-‐08

Jul-‐0

8

Sep-‐08

Nov-‐08

Jan-‐09

Mar-‐09

May-‐09

Jul-‐0

9

Sep-‐09

Nov-‐09

Jan-‐10

Mar-‐10

May-‐10

Jul-‐1

0

Sep-‐10

Nov-‐10

Rwf (bn

)

Graph 6. DomesGc credit to the private sector (by month) Source: Na0onal Bank of Rwand


20

reference country by looking at new business density in the countries with the most similar

business environments. Should these comparisons result in accurate estimates of new business

density in the reference country before a policy change (e.g. the introduction of a one-stop

shop), then we could use these comparator countries as control regions for the country of

interest and conduct a counter-factual analysis.

In the next chapters we develop and explain this approach, which is inspired by an

analogous method called proximity control, where export similarity networks are used to conduct

counter-factual analysis.25

4.1 Measuring and visualizing the Doing Business similarity space

Cross-country comparisons show that countries with similar overall rankings in the Doing

Business report can have different business environments. To illustrate this, imagine the

extreme case where country Alpha is the world’s best performer on half of the Doing Business

indicators, and the world’s worst performer on the other half, and where country Beta is the

exact opposite (i.e. the best where country Alpha is the worst, and the worst where country

Alpha is the best). Then country Alpha and Beta would have the same overall ranking in the

doing Business report, but in reality they would have radically different business environments.

In order to avoid such scenarios and get a more accurate estimate of how similar is the

business environment of a pair of countries, as measured by the Doing Business report, we

introduce the new metric of Doing Business similarity (DBSim). The metric we propose is very

simple, and it assumes that all Doing Business indicators (we use 32 indicators in total) are

equally important in determining how similar or dissimilar are the business environments of two

countries, as measured by the Doing Business report.

To measure DBSim between any pair of countries a and b, we first calculate a measure

of the distance (𝑑!,!) between their business environment as measured by Doing Business

(basically a measure of how dissimilar their business environments are) by: (i) standardizing

each country’s score on all the Doing Business indicators; (ii) summing the squared difference

between the standardized scores of both countries across all the indicators; and (iii) dividing by

the number of indicators for which data is available for both countries, so that we get the

average distance. Finally, to obtain a measure of similarity, rather than dissimilarity, we use the

exponent of minus the distance. Formally, this can be written as:

25 See Gathani et Stoelinga (2013).


21

𝑑!,! =1𝑁

𝑆!,! − 𝑆!,!!

!

!!!

and,

𝐷𝐵𝑆𝑖𝑚!,! = 𝑒!!!,!

where 𝑆!,! is country a’s standardized score on indicator i.

With this measure of similarity, we can now answer questions such as: which ten

countries have the most similar business environment as measured by Doing Business to

country Alpha? Let us illustrate this with an example, by identifying the countries with the most

similar business environment to Rwanda and Georgia based on data from the Doing Business

2010 report (see table 3).

Table 3: Countries with most similar business environment to Rwanda and Georgia

While Georgia was ranked 11th in the Doing Business 2010 report, its 10 closest

comparators were ranked between 5th and 51st. As for Rwanda the spread is even larger:

Rwanda was ranked 67th in the Doing Business 2010 report, but its 10 closest comparators

were ranked 32nd to 147th. Countries can have very different Doing Business rankings but have

quite similar business environment as measured by Doing Business and vice-versa.

Similarity to Georgia in Doing Business 2010 (ranked 11th)

Similarity to Rwanda in Doing Business 2010 (ranked 67th)

Similarity rank Country d DBSim DB2010

rank Similarity rank Country d DBSim DB2010

rank 1 Estonia 0.26 0.77 24 1 Mongolia 0.89 0.41 60

2 Saudi Arabia 0.42 0.66 13 2 Botswana 1.07 0.34 45

3 Lithuania 0.44 0.64 26 3 Kyrgyz Republic 1.26 0.28 41

4 Mexico 0.49 0.62 51 4 South Africa 1.29 0.28 34

5 Sweden 0.49 0.61 18 5 Paraguay 1.32 0.27 124 6 Germany 0.50 0.60 25 6 Azerbaijan 1.36 0.26 38

7 Chile 0.51 0.60 49 7 Burkina Faso 1.37 0.25 147

8 FYROM 0.51 0.60 32 8 FYROM 1.41 0.24 32 9 Thailand 0.53 0.59 12 9 Vanuatu 1.42 0.24 59 10 UK 0.53 0.59 5 10 Malawi 1.43 0.24 132


22

This way of looking at the Doing Business indicators also provides a different

perspective to policy makers. Rather than aiming at increasing their Doing Business rankings,

Governments should aim at becoming more similar to a certain “role model” or “compass

country”. Georgia, for example, should become more similar to the UK, and Rwanda to South

Africa. This approach also highlights the country’s weaknesses and strengths better; while

Rwanda’s business environment as measured in Doing Business 2010 report was good enough

to be compared to a country like FYROM (ranked 32), it was also weak enough to be compared

to Burkina Faso (ranked 147).

One of the side benefits of measuring the Doing Business similarity between all pairs of

countries for which data exists is that we can place countries within a Doing Business similarity

network. The closer countries are to each other in this network, the more similar their business

environment as measured by Doing Business are and vice-versa. This provides a powerful

visual tool that we can use to represent Doing Business results. We illustrate the use of this

tools in graph 7-A and 7-B.

Graph 7-A: Countries with similar business environments and their GDP level


23

Graph 7-B: with similar business environments and their depth of reforms

Graph 7-A is a network representation of all countries for which data is available with

links to their three closest comparators in the Doing Business similarity space (using data from

the Doing Business 2010 report). Countries (the nodes in the network) are colored by their GDP

per capita levels, ranging from green - for countries with the lowest GDP per capita levels - to

red - for countries with the highest levels of GDP per capita. The graph reveals that countries

with similar levels of GDP per capita tend to have similar business environments as measured

by Doing Business – all the reds are clustered together, as are the countries in orange, yellow

and green. Yet there are a number of clear outliers: Georgia, for example, performs much better

in the Doing Business indicators than we would expect given its level of GDP per capita

(Georgia, in yellow, is surrounded by countries in red and orange); while Brazil, on the other

hand, performs much worse (Brazil is in dark orange, but it is surrounded by countries in light

orange, yellow and green).

Graph 7-B represent the same network, only this time countries are colored not by their

GDP per capita levels, but by how large their reforms to the business registration process were

during 2008-2009 (as captured by the Doing Business reports 2009 and 2010). The greater the

red shading of the country (or node), the deeper the reforms. Countries in white did not


24

introduce any reforms during this period. This network reveals that in 2008-2009 the countries

that made the biggest reforms to business registration regulations were Rwanda and Belarus.

Also, it shows that about half the countries in the world carried out some form of reform to

business registration regulations during that period.

4.2 The properties of the Doing Business similarity space

After having established a way to measure Doing Business similarity, we need to better

understand the properties of the similarity space and how we can use them to conduct

inferential analysis. In this section, we show that countries with similar business environment as

measured by Doing Business are similar on a whole range of other economic indicators.

We choose to focus on two properties of the Doing Business similarity space, which are

relevant to our analysis. These properties highlight a clear link between the business

environments of countries and their levels of GDP per capita, GDP growth, new business

density, and new business density growth. These results are new to the Doing Business

literature and are very strongly statistically significant. They are very much in line with the

findings on export similarity space.26

Property 1: The more similar the business environment of a pair of countries, the more

similar on average are their levels of GDP per capita and their levels of GDP per capita

growth

As graphs 8 and 9 show, there is a very strong and statistically significant association

between DBSim and the average difference in GDP per capita and GDP per capita growth

between pairs of countries. What this means is that countries that have similar business

environment as measured by Doing Business are likely to grow at relatively similar GDP per

capita growth rates (+/- 1% on average) and are likely to have relatively similar levels of GDP

per capita. While these results do not imply a causal relationship in any way, they do reveal that

Doing Business similarity well captures how similar countries are on two fundamental economic

variables: GDP per capita and GDP per capita growth.

26 See Gathani et Stoelinga (2013).


25

Property 2: The more similar the business environment of a pair of countries, the more

similar on average their levels of new business density and their levels of new business

density growth (see graph 10-11).

The results we obtain for new business density (NBDEN) and new business density

growth are similar to the results we obtain for GDP per capita and GDP per capita growth.

These results show a clear link between country’s performance on Doing Business and new

business registration.

R² = 0.96426

0

0.5

1

1.5

2

2.5

3

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Average diffe

rence in log GD

Ppc be

tween pairs

of cou

ntrie

s (2009)

DBSim of pairs of countries (DB2010)

Graph 8. Average difference in GDP per capita between pairs of countries based on

their DB similarity score (DB2010)

R² = 0.84234

0%

1%

2%

3%

4%

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Average diffe

rence in GDP

pc growth

betw

een pairs of cou

ntrie

s (2004-‐2009)


Graph 9. Average difference in GDP per capita growth between pairs of countries based on their DB similarity score (DB2010)

R² = 0.87095

0

0.5

1

1.5

2

2.5

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Average diffe

rence in log NBD

EN between

pairs of cou

ntrie

s (2009)


Graph 10. Average difference in NBDEN between pairs of countries based on their

DB similarity score (DB2010)

R² = 0.67694

0%

10%

20%

30%

40%

50%

60%

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Average diffe

rence in NBD

EN growth

betw

een pairs of cou

ntrie

s (2009)


Graph 11. Average difference NBDEN growth between pairs of countries based on their DB similarity score (DB2010)


26

These two properties suggest that countries with similar business environment as

measured by Doing Business are quite similar on the main explanatory variables that we use in

the synthetic control tests. A linear combination of a country’s closest neighbors in the Doing

Business similarity space should therefore result in a control region with similar levels of new

business density, GDP per capita, GDP per capita growth, and other relevant explanatory

variables to the country of interest. In the next chapter, we show that these synthetic countries

can be used as control regions to test the impact of policy changes, and we choose to test this

methodology to the introduction of one-stop shops on new business registration in order to

compare it to the synthetic control methodology.

4.3 Rwanda: measuring the impact of a one-stop shop with the proximity controls methodology

We apply the concepts of proximity control and randomized permutations27 to Rwanda,

which introduced a one-stop shop for business registration in 2009. Our objective is to measure

the impact of Rwanda’s one-stop shop and related reforms on new firms creation by using a

control region – or proximity control - for Rwanda which fulfills the following three criteria: (i) the

proximity control should accurately estimate new business density in Rwanda before the

introduction of the one-stop shop; (ii) the proximity control should have similar levels of GDP per

capita and other selected explanatory variables to Rwanda; and (iii) the proximity control should

be composed of countries which did not have a one-stop shop during the period under

consideration.

The following three steps are required to construct a proximity control for the Rwanda

during the 2003-2009 period:

Step 1.

As in the case of the synthetic control approach, it is necessary to select a time period

during which the difference between Rwanda and the proximity control Rwanda is minimized.

Given that Rwanda introduced its one-stop shop in 2009, we use data prior to 2009 to construct

the proximity control Rwanda. Given that new business density in Rwanda is available for the

2003-2009 period, we match Rwanda to its synthetic control using 2003-2008 data.

27 Ibidem.


27

Step 2.

Next we identify a pool of potential control countries from which proximity control

Rwanda can be constructed – we call this the “donor pool”. The process we follow is different to

the synthetic control approach and imposes fewer restrictions on the quality of the data:

1. We eliminate from the donor pool all countries that already had or introduced a one-

stop shop during the 2003-2009 period. This leaves us with a pool of countries where

the “treatment” did not take place.

2. From the remaining countries in Rwanda’s Doing Business similarity space (using

data from the Doing Business 2009 report),28 we select Rwanda’s five closest

neighbors (i.e. the countries with the highest Doing Business similarity to Rwanda).

According to the properties of the Doing Business similarity space, these are the five

non-OSS countries which in 2008 were the most likely to have similar levels of new

business density, GDP per capita, and GDP per capita growth, as well as similar

Doing Business indicators. Depending on the quality of the data and the quality of

the resulting fit, we can choose to include more or less countries in the pool – we

leave this to the discretion of the researcher. The more countries are in the pool, the

more likely we are to obtain a good fit for the variable of interest (in this case new

business density); however, the difference between the business environments of

the country of interest and the proximity control will be larger as each extra country

added to the pool is farther away from the country of interest in the Doing Business

similarity space.

3. Finally, to ensure the quality of the resulting fit, we eliminate from this list countries

for which we have less than a minimum number of observations in time. Here we are

interested in the 2003-2009 period, so we eliminate from the sample countries for

which new business density data is not available during this period. In the case of

Rwanda, we remain with a donor pool of 5 countries: Malawi, Ethiopia, Niger,

Uganda, and Croatia.

28 Data from the Doing Business 2009 report were collected in July 2008–June 2009, before the introduction of the one-‐stop shop in Rwanda.


28

Step 3.

From this pool of five countries, we select combinations of countries to construct a

proximity control for Rwanda and conduct inferential analysis. There are infinite combinations of

these five countries when both weights and the different country combinations are taken into

consideration. Moreover, by construction, not all combinations are good predictors of new

business density in Rwanda before the introduction of the one-stop shop. To overcome these

two challenges, we generate 15,000 random permutations of these five countries (in groups of

4) and select a limited number of permutations (in this case 12) that best match NBDEN in

Rwanda during the 2003-2007 period. We take the mean of these best permutations as our

proximity control for Rwanda. We explain in more detail below how to select the optimal number

permutations to construct the proximity control, and why in the case of Rwanda this number is

12.

The resulting proximity control for Rwanda is composed of the linear combination of

countries as shown in Table 4.

Table 4: Composition of proximity control

Composition of proximity control Weight

Malawi 72.37%

Niger 14.45%

Ethiopia 11.11%

Croatia 2.06%

This linear combination of countries performs slightly better than Rwanda’s synthetic

control in fitting Rwanda’s predictors during the 2003-2008, except on balance of trade which is

more negative in the proximity control (see table 5). This suggests that Rwanda and its

proximity control are very similar on the most relevant variables of interest.


29

Table 5: Predictors (averages 2003-2008)

Also, as can be seen in the graph 12, Rwanda’s proximity control is a relatively good

predictor of new business density in Rwanda during the 2003-2008 period, before the

introduction of the one-stop shop. The average difference in new business density between

Rwanda and its proximity control is about 125 businesses on average per year, compared to 87

for the synthetic control, making it a slightly less accurate predictor than Synthetic Rwanda. A

difference of 125 new firms however remains small compared to the 2,900 new firms that were

registered in Rwanda in 2009.

0

500

1000

1500

2000

2500

3000

3500

2003 2004 2005 2006 2007 2008 2009

New

firm

reigstrako

n

Graph 12. New Firm RegistraGon in SyntheGc Rwanda and Placebo

Rwanda

Proximity Control

Synthekc Rwanda

Predictors Rwanda Synthetic Rwanda

proximity control

GDP per capita (constant 2000 USD) 279.4 263.2 282.6

Agriculture (% GDP) 36.7 35.9 33.5

Industry (% of GDP) 13.8 19.9 16.9

Services (% of GDP per capita) 49.5 44.2 49.6

Trade balance on goods and services (% GDP) -14.9 -14.0 -19.0

Gross Fixed Capital Formation (% of GDP per capita) 17.6 20.3 20.5

Urban population (% total population) 17.5 18.2 17.8


30

We can now estimate the impact of Rwanda’s one-stop shop on new business density.

While Rwanda and its proximity control had a similar performance on new business registration

during the 2003-2008 period, graph 12 shows that new business registration increased

exponentially in Rwanda in 2009 following the introduction of the one-stop shop. In 2009, the

difference in new business registration between Rwanda and its synthetic control was 2,260

firms compared to only 125 on average during 2003-2008. A simple difference-in-difference

calculation using 2008 as the base year enables us to estimate that the introduction of a one-

stop shop in Rwanda led to the creation of 1,994 firms, which is equivalent to an increase of

184% in new firm registration; not too different from our previous estimate of 188% using the

synthetic control methodology.

4.4 Testing the robustness of the results obtained using the proximity controls methodology

The most important questions to answer when attempting to measure the impact of an

intervention is whether it is possible to produce valid statistical inference. We do it here using six

different approaches.

Test 1: Random permutations test

The first approach consists in testing the sensitivity of the proximity control to changes in

its composition. As discussed above, to construct a proximity control for Rwanda we first

created 15,000 randomly generated different combinations of the 5 countries in Rwanda’s

“donor pool”, namely Malawi, Niger, Ethiopia, Croatia and Uganda, which were the 5 countries

with the most similar business environment to Rwanda in 2008. Each of these 15,000

combinations is a potential proximity control with different weights assigned to each country in

the pool. If Rwanda’s one-stop shop, introduced in 2009, had led to no impact or little impact in

that same year, then we would expect the difference-in-difference of new business density in

Rwanda in 2009 compared to these 15,000 different combinations to be close to 0 on average

and/or not statistically significant. Yet, as can be seen in table 6 below where we take 2008 as

the base year to calculate difference-in-difference impact estimates, we estimate that the

minimum impact of the OSS reform was at least 1,304 registered newly registered firms. In all

15,000 cases, the estimated impact of Rwanda’s one-stop shop in 2009 was highly positive and


31

we can therefore reject the null-hypothesis that the difference-in-difference in 2009 was nil (t-

statistic: 305).

Table 6: Difference-in-difference estimates of new business density in Rwanda compared

to 15,000 random combinations of control countries

Difference-in-differences year on year

Year 2004 2005 2006 2007 2008 2009 Minimum number of new firms registered

-1064 -2314 -2842 -953 74 1304

Maximum number of new firms registered

949 -6 329 217 1103 6225

However, as discussed, not all combinations are good predictors of new business

density in Rwanda before the interventions and are hence unlikely to be good controls

thereafter. In order to test whether the selected proximity control – which is the average of the

12 best matching combinations – is a good estimate of impact or not, we repeat this exercise

with the 12 combinations that best match new business density in Rwanda between 2003 and

2008. As can be seen in graph 13, which shows the distribution of estimated impact using each

of these 2 combinations as a control region for Rwanda, every single combination puts the

impact in 2009 between 1,940 and 2,025 new firms, which corresponds to a minimum impact of

179% and a maximum impact of 186%. The maximum of the distribution curve comes at about

2000 firms, which is close to our estimated impact of 1,994 new firms registered.

0% 5%

10% 15% 20% 25% 30% 35% 40% 45%

1940 1950 1960 1970 1980 1990 2000 2010 2020

Share of alte

rnak

ve Proximity

Con

trols

Eskmated Impact (number of firms)

Graph 13. DistribuGon of esGmated impact using 12 closest combinaGons


32

While the distribution above confirms the impact estimate, these differences could be

due to marginal changes in the 12 combinations. This would be the case for example if all 12

combinations had Malawi contributing between 46% and 50% to the linear combination, Niger,

between 23% and 25%, and so forth. It would imply that the 12 different combinations of

controls are actually not “alternative scenarios,” but slightly different versions of the same

scenario. It would also mean that the 7 percentage point spread we observe between the

minimum and maximum impact estimates is due to very small changes in the composition of the

control region – therefore, the proximity control is actually quite sensitive to small changes in its

composition. If this were the case, it would be necessary to increase the sample size, and focus

not on the best 12 matches, but a larger sample.

To check whether this is indeed an issue, we propose a measure of the composition

diversity of the selected control regions, which also enables us to identify the optimal number of

different combinations from which to construct the proximity control. We measure composition

diversity using the following formula:

𝐷𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦! =(𝑀𝑎𝑥𝑆ℎ𝑎𝑟𝑒! −𝑀𝑖𝑛𝑆ℎ𝑎𝑟𝑒!)!!!

5×(0.25)!

where 𝑀𝑎𝑥𝑆ℎ𝑎𝑟𝑒! is the maximum share of control region i in either of the N different

linear combinations, 𝑀𝑖𝑛𝑆ℎ𝑎𝑟𝑒! is the minimum share of control region i, and where i ∈ ( 0,5 ). If

𝐷𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦! is greater than 1 then there is more variation in the composition of the N linear

combinations than if the contribution of each control region varied by 25%; if on the contrary

𝐷𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦! is smaller than 1, then there would be less variation in the composition of the N

linear combinations than if the contribution of each control region varied by 25%. 𝐷𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦! is

an increasing function of N: the higher the number of linear combinations selected (N), the

larger the diversity of the combinations selected. With 5 control regions in the donor pool,

maximum diversity is achieved when 𝐷𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦!=16, which would imply that each of the control

countries is represented in the N different linear combinations with shares ranging from 0% to

100%. A diversity level of 16 however is not desirable as mathematically it is impossible for all

linear combinations of control countries to be good predictors of new business density in

Rwanda before the one-stop shop was introduced.

In order to ensure a minimum level of variation in the selected linear combinations that

make up the proximity control, we propose that the researcher select the smallest possible N

such that 𝐷𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦! is greater or equal to 2. This would mean that there is at least 2 times


33

more variation in the linear combinations that contribute to the proximity control, than if the

contribution of each control region in the donor pool varied by 25%. Moreover, by selecting the

smallest possible N, we are selecting the best possible match between the proximity control and

the reference country on the variables of interest.

Table 7: Calculating 𝑫𝒊𝒗𝒆𝒓𝒔𝒊𝒕𝒚𝟏𝟐 to test Diversity in Rwanda’s proximity control

Countries in donor Pool

Minimum share in selected linear combinations

Maximum share in selected linear combinations

Squared difference between minimum

and maximum

Malawi 35.0% 98.7% 0.406

Niger 0.0% 36.9% 0.136

Ethiopia 0.0% 62.3% 0.389

Croatia 1.3% 2.6% 0.000

Uganda 0% 0% 0.000

Sum of squared differences 0.931

Diversity 2.980

Based on this metric we find that the optimal number of linear combinations needed to

construct Rwanda’s proximity control is 12. The diversity of these 12 best linear combinations is

2.98, well above the 2-point threshold. Within these 12 linear combinations, the contribution of

Malawi varies from 35% to 98.7%, the contribution of Niger from 0 to 36.9%, and the

contribution of Ethiopia from 0 to 62.3% (see table 7), thereby ensuring that the 7 point spread

we observe in the impact estimates is not due to small variations in the composition of the 12

linear combinations, but significant changes in their composition.

Test 2: Sensitivity to changes in donor pool

Another way of testing the sensitivity of the proximity control to changes in its

composition is to change the countries in the donor pool. We test 21 different scenarios: (i) first

replacing each of the countries in the pool with the 6th closest country to Rwanda in the Doing

Business space; (ii) second replacing each of the countries with the 7th closest country to

Rwanda; (iii) third testing the 10 possible combinations of replacing two countries out of the 5 in

the donor pool simultaneously with the 6th and 7th closest countries; and (iv) lastly replacing all 5

countries with in the pool with the subsequent 5 countries closest to Rwanda in the Doing

Business space. As the results in graph 14 indicate, the proximity control is not very sensitive to


34

changes in its composition. While impact estimates using the 21 alternative controls range from

1,794 to 2,001 new firms created - which is equivalent to an impact of respectively 165% and

184% - the majority of estimates are clustered around the 1,960-2,000 new firms mark.

Test 3: Sensitivity to changes in time-frames

One of the main risks with forcing a match between two regions, in this case Rwanda

and its proximity control, is over-fitting. Over-fitting would mean that Rwanda’s proximity control

does not describe Rwanda, but instead ‘fits’ the random noise or random errors generated by

changes in Rwanda’s new business registration figures between 2003 and 2008. If there were

over-fitting, we would expect changes in the time frames during which we force the fit between

Rwanda and its proximity control to lead to significant changes in our impact estimates. We

therefore test for the possibility of over-fitting by varying the time-frames during which Rwanda

is matched to its proximity control and compare the outcomes to our original impact estimate of

1,994 additional firms registered in 2009 or an impact of 184%.

The results do not vary substantially with impact estimates ranging from 1,978 additional

firms to 2,067 additional firms in 2009, which corresponds to +/- 3.5 percentage points from the

original estimate of 1,994 firms (see table 8). Rwanda’s proximity control is therefore robust to

changes in the periods of fit, implying that it is unlikely that the proximity control suffers from

over-fitting.

0%

10%

20%

30%

40%

50%

60%

1750 1800 1850 1900 1950 2000

Share of observako

ns

Graph 14. DistribuGon of Impact EsGmates based on 21 alternaGve Proximity Controls


35

Table 8: Comparing impact estimates using 8 alternative periods

Period of fit # firms Impact estimate

2003-2007 1978 182.0%

2003-2006 2065 190.0%

2003-2005 2067 190.2%

2004-2008 2018 185.7%

2004-2007 2042 188.0%

2004-2006 2044 188.1%

2005-2008 2019 185.8%

2005-2007 2047 188.4%

Test 4: Falsification test

To test the validity of the results obtained, it is also possible to replicate the two main

tests carried out to check the robustness of the synthetic control methodology, namely the

falsification test and the MSPE test.

A simple way to construct a falsification test is to run the proximity control method on

each of the countries that make up Rwanda’s proximity control (namely Malawi, Niger, Ethiopia,

Uganda, and Croatia) and compare the weighted average impact estimate on the proximity

control to the estimated impact for Rwanda. Each country is weighted according to its

contribution to the proximity control and the proximity control method is run using the same

parameters as in the case of Rwanda to ensure comparability. If the proximity control region is a

valid control region for Rwanda, then it should not have experienced any large impact during the

period of interest, which is 2003-2009.

In graph 15, we compare impact estimates for Rwanda to that of its proximity control.

Impact estimates for the proximity control during the 2006-2009 period range between -36 firms

to +2 firms, which is very small compared to the impact estimate for Rwanda. We conclude that

the proximity control region did not itself experience any unusual changes in new business

registration during the period of interest.


36

Test 5: Mean Square Prediction Error test

In addition to the falsification test, we conduct a Mean Square Prediction Error (MSPE)

test and compare the ratio of MSPE before and after the introduction of the one-stop shop in

Rwanda to that of the other countries/controls in Rwanda’s “donor group”. What we are testing

for is an hypothetical impact in 2009. As in the case of the synthetic controls methodology an

MSPE ratio smaller than 1 indicates that the observed impact in 2009 is not unusual, i.e. it is

smaller than in other years before the intervention, while a ratio of more than 1 indicates that the

observed impact is larger than in other years. If the ratio significantly larger than 1 then the

control region might have experienced an impact during the period of interest. Not surprisingly,

this ratio is much higher in Rwanda than in other countries (see table 9) – indicating that the

impact we observe in Rwanda is indeed unique. Moreover, for all countries except for Uganda

this ratio is around 1, indicating that countries in Rwanda’s donor pool did not experience a

large impact in 2009.

-‐1000

-‐500

0

500

1000

1500

2000

[2003-‐2004]

[2004-‐2005]

[2005-‐2006]

[2006-‐2007]

[2007-‐2008]

[2008-‐2009] Num

ber o

f firm

s

Graph 15. Comparing Impact EsGmates for Rwanda and its Proximity Control

Rwanda

Proximity Control (Placebo)


37

Table 9: MSPE test

Country MSPE Ratio

Rwanda 163.2

Malawi 0.1

Niger 1.5

Ethiopia 1.1

Croatia 1.8

Uganda 8.1

Test 6: Excluding Doing Business reforms in the proximity control

Lastly, the results obtained could be over-estimating or under-estimating the actual

impact of the introduction of the one-stop shop because of other reforms impacting time, cost,

procedures and minimum capital required to start a new business. To test for this eventuality we

compare average year-on-year changes in the “starting a business” indicators of Rwanda and

its proximity control (see graph 16). We find that on average the countries that comprise

Rwanda’s proximity control did not implement major “starting a business” reforms during the

2003-2009 period (covered by the Doing Business reports from 2004 to 2010). We define major

reforms using the threshold defined by Klapper and Love (2010), who show that only business

registration reforms that cut cost and/or time by more than 40% in 92 countries during the 2003-

2008 period had a statistically significant impact on new business creation. In Rwanda’s

proximity control the maximum average change during the 2003-2009 period was a 10% cut in

the cost and time required to start a business, achieved in 2008. Moreover, we know already

from graph 15 that these reforms do not seem to have affected new business registration in any

way.


38

*Minimum capital requirements is excluded from the calculation of “Starting a Business”

improvements as this has been 0 for Rwanda from 2003 through to 2009

From 2003 to 2007 (Doing Business reports from 2004 to 2008), Rwanda and its

proximity control reformed at about the same pace, with gradual reductions in the cost of

starting a business as a share of GDP. A large part of this relative reduction in costs was the

result of high GDP growth rates (i.e. changes in the denominator), rather than actual reductions

in the costs of starting a business. Business reforms accelerated in Rwanda in 2008, with a

small reduction in the number of procedures and the time required to start a business

(procedures were reduced from 9 to 8 and the time required from 16 days to 14 days). But the

major breakthrough came in 2009 (Doing Business 2010). While the average time, cost, and

procedures required to start a business in Rwanda was reduced by 81% in 2009 (Doing

Business 2010), the corresponding figure for Rwanda’s proximity control was just 9%,

unchanged from the average of the 2003-2008 period.

Our calculations of impact estimates seem robust: they are not sensitive to (i) changes in

the composition of the proximity control, (ii) changes in the donor pool, and (iii) changes in time

frames used to construct the proximity control; (iv) they are not due to any large deviations nor

events in the proximity control region; (v) they are unique compared to other countries in

Rwanda’s donor pool; and finally (v) they are not due to alternative reforms to “starting a

business” indicators during the period of interest.

-‐90% -‐80% -‐70% -‐60% -‐50% -‐40% -‐30% -‐20% -‐10% 0%

10% DB2005 DB2006 DB2007 DB2008 DB2009 DB2010

Change in starkn

g a bu

siness ind

icators

Graph 16. Changes in StarGng a Business Indicators (Gme, cost, and procedures) *

Rwanda

Proximity Control


39

5. Replicating and comparing both methodologies to 11 other countries

Table 10: Summary of impact results obtained in 12 countries29

Country Year of Reform

First year of impact estimate

Estimate of # firms increase

using synthetic control

Estimate of # firms increase

using proximity control

Average estimate of

# firms increase

among the two

methods

Estimate of % firms increase

using synthetic control

Estimate of % firms increase

using proximity control

Average estimate of

% firms increase

among the two

methods

Rwanda 2009 2009 2,041 1,994 2,018 187.9% 183.5% 186%

Tajikistan 2008 2009 1,227 1,200 1,214 133.0% 130.1% 132%

Belarus 2007 2007 1,800 1,749 1,774 104.8% 101.8% 103%

Albania 2007 2007 709 640 676 57.2% 51.8% 55%

Oman 2006 2006 672 702 688 50.9% 53.2% 52%

Senegal 2008 2008 591 531 561 47.0% 42.2% 45%

Kyrgyz Rep. 2008 2009 1,625 1,595 1,610 44.0% 43.2% 44%

Georgia 2006 2006 1,408 1,287 1,348 37.1% 33.9% 36%

Tunisia 2006 2007 1,682 1,466 1,574 35.0% 30.5% 33%

Denmark 2006 2007 3,681 2,988 3,335 17.5% 14.2% 16%

Canada 2005 2006 13,691 14,663 14,177 8.5% 9.1% 9%

Netherlands 2007 2008 2,804 932 1,868 8.2% 2.7% 6%

We replicate the synthetic control and proximity control methodologies to estimate the

impact of the introduction of a one-stop shop on new business registration in 11 other countries,

namely Albania (2007), Belarus (2007), Canada (2005), Denmark (2006), Georgia (2006),

Kyrgyzstan (2008), Netherlands (2007), Oman (2006), Senegal (2008), Tajikistan (2008), and

Tunisia (2006).30 In some cases the synthetic and proximity control methodologies work better

than in others, but overall the results are robust and comparable across countries. The

magnitude of the observed impact (see table 10 or the Annexes for disaggregated results) is

impressive: in Rwanda, we estimate that the introduction of the one-stop shop led to a 184-

188% increase in new firms created the same year it was introduced (depending on the

methodology utilized), in Tajikistan the increase was almost 132%, in Belarus it was 103%.

Countries as diverse as the Kyrgyz Republic, Oman, Albania and Senegal showed an increase

of 40-55% in terms of new firms created, while in Tunisia and Georgia the impact was close to 29 The results obtained for Canada and the Netherlands are not statistically significant. 30 In parenthesis, the year when the one-‐stop shop was introduced.


40

35%. These results confirm that burdensome regulations for business registration can be a

major obstacle to new firm registration. Table 10 summarizes the impact estimates obtained for

each of the 12 countries (Rwanda + 11 other countries) using both the synthetic control and

proximity control methodologies.

While impact estimates are strongly positive, they vary from 5.5% in the case of the

Netherlands to 184%-188% in the case of Rwanda. Why? One possible explanation is that in

some countries the introduction of the one-stop shop led to a larger improvement in the “ease of

starting a business” than in others. One would expect the marginal improvement in the “ease of

starting a business” in countries such as Netherlands, Canada and Denmark, which had a good

initial business environments, to be smaller than in countries such as Rwanda, Tajikistan or

Belarus, where the initial regulatory burdens of starting a business were high. To test this

hypothesis we compare the estimated impact of the one-stop shop with the “depth of the reform”

induced by the one-stop shop. We proxy for the depth of the reform using the average annual

percentage change in the four “Starting a Business” sub-indicators: the cost, time, number of

procedures and minimum capital requirements to start a business. In graph 17, we plot the first

year of impact estimates for all 12 countries with the corresponding improvement of the

business registration process in the same year the one-stop shop was introduced.

Graph 17: Depth of business entry reforms.

RWA

KGZALBOMN

DNKCANNLD

TJK

SEN

BLR

GEOTUN

020

4060

8010

012

014

016

018

0

Est

imat

ed Im

pact

Yea

r 1 (%

incr

ease

in fi

rm re

gist

rata

tion)

0 10 20 30 40 50 60 70 80 90Depth of reform (% change)

Estimates Impact (Year 1) and Depth of Reform


41

We find a statistically significant linear association between the depth of the reforms

induced by the one-stop shop and the increase in new firm registration, suggesting that the

impact of a one-stop shop is proportional to the scale of the resulting regulatory change. On

average, a 1 percentage point improvement in the ease of starting a business (measured as the

average percentage change on the four starting a business indicators) is associated to a 1.87

percentage point increase in new business registration (𝑅! = 0.65). This proportional

relationship between impact and reform is further evidence that what was holding back new firm

registration in the 12 countries in our sample were regulatory barriers to starting a business. The

more these barriers were alleviated in relative terms, the greater the impact of the one-stop

shop.

Increases in new firm registration can be the result of: (i) companies that were previously

in the informal sector shifting towards the formal sector; (ii) new entrepreneurs deciding to start

a business, as the barriers are lower; (iii) foreign investors deciding to register local businesses;

or (iv) simply re-registration requirements. While additional data and research is needed to

understand the exact nature of the increases in new business registration, the evidence

presented here – which is consistent across almost all case studies - suggests that policy

makers should focus on promoting significant business entry reforms such as the introduction or

improvement of a one-stop shop for business registration. The larger the reform – in so far it

impacts the ease of starting a business – the larger the impact.

Impact estimates obtained with the synthetic control and the proximity control

approaches are very similar (see graph 18), even though these methodologies can result in very

R² = 0.99807

0.0%

50.0%

100.0%

150.0%

200.0%

0.0% 50.0% 100.0% 150.0% 200.0%

Impact eskmate using Synthe

kc Con

trols

(%)

Impact eskmate using Proximity Controls (%)

Graph 18. Comparing impact esGmates using Proximity and SyntheGc Controls


42

different linear combinations of control countries. The average prediction error between the two

methodologies is around 3.2 percentage points for the 12 case studies, compared to an average

impact of about 54%. These results suggest that both methods are interchangeable and offer

equally accurate ways of measuring impact at the aggregate level.

Synthetic control and proximity control share a number of features that make them

attractive methodologies for certain types of impact evaluations:

• They enable quick evaluations at the aggregate level which are difficult to achieve and very

expensive using alternative methods, in particular Randomized Control Trials which are

limited by external validity issues;

• They do not have extensive data requirements: minimum data requirements are two

observations in time – one before, and one after the intervention – and complete data on a

number of variables for the treatment region and at least two control regions (however, the

more data is available, the more likely it is that the methods work);

• Contingent on data availability, they can be carried out post-intervention and do not require

a lot of pre-treatment planning;

• As we have shown in this paper, they are easily replicable to other regions/countries,

thereby ensuring cross-country comparability;

• They are transparent: both clearly outline the weights assigned to each control region and to

each variable in order to obtain the control region;31

• They are falsifiable and lead to exact inference: a number of tests enable the researcher to

check the consistency and validity of the results, leading to exact inference.

Yet there are some fundamental differences between the two approaches. The synthetic

control methodology relies on a least squares minimization algorithm that assigns weights to

control regions and selected variables, such that the resulting linear combination of control

regions best fits the reference region on the selected variables before the treatment. This has a

number of consequences:

• The number of possible variables that can be used to match the treatment and control

region are limited - if more than 10 variables are included, for example, researchers will find

that the algorithm often fails due to non-convexity problems;

31 See Abadie et al. (2007).


43

• The synthetic control approach assigns weights to variables as well as control regions,

which means that the weights assigned to variables are different each time the method is

applied to a different case study. This limits the comparability of results when applied to

several different case studies, as the observed impact will not only be the result of different

control regions, but also a different combination of explanatory variables.

• The synthetic control methodology only produces one “alternative history”, or control region,

which is the region that best matches the reference region on the dependent variable and

the explanatory variables. As a result, the only tests that can be conducted to check the

validity of the resulting impact estimates (as we show in Section 3), involves applying the

synthetic control methodology to the Placebo region itself (the falsification test) as well as all

the countries in the donor pool (MSPE test), or to iteratively eliminate countries from the

donor pool to check the sensitivity of the results to changes in the donor pool. If the donor

pool is small, then the number of possible tests is limited.

• It is possible for the resulting synthetic control region to be a combination of multiple

countries (the limit is the number of countries in the donor pool), which can often be very

different in nature, leading to biases caused by interpolating across regions with very

different characteristics.32 In order to avoid interpolating across regions with different

characteristics, it is not always evident which metrics or rules to use to eliminate a control

region from the donor pool.

The proximity control approach solves some of these issues:

• It works best when Proximity is calculated using as many variables as possible (in this case

32, compared to only 7 for the synthetic control), as the resulting similarity measure is based

on more information. On the other hand, this means that proximity control require more data

than synthetic control, which in practice can be a problem.

• The weights assigned to each variable (in this case equal) can be kept constant across case

studies, thereby ensuring better comparability.

• The underlying idea behind proximity control is that there is not only one “alternative

scenario” but many. The paths that best match the reference country (in this case we

selected the 12 alternative controls that best predicted new business registration in Rwanda)

all contain information worth being used, as long as each of these are “good enough”

predictors of the reference country before the intervention. The more diverse the

32 Ibidem.


44

composition of these “alternative scenarios” in terms of the control regions that contribute to

them, the more valuable the information that is captured. Also, these “alternative scenarios”

lead to a distribution of impact estimates, enabling the researcher to test the sensitivity of

the proximity control to changes in composition.

• And lastly, by construction, the proximity control ensures that regions in the donor pool are

the most similar to the reference region on the measure of interest. Control regions are

ranked based on their similarity to the reference region and are included or excluded on that

basis. In this case, we only include in the donor pool the 5 countries that have the most

similar business environment to the reference country and that do not have a one-stop shop.

The rule for inclusion or exclusion is very clear.

These differences make the two approaches good complements for each other.

6. Conclusion

In this paper we use the synthetic control and proximity control methodologies to

measure the impact of introducing a one-stop shop on new business registration in twelve

different countries. We show how, using easily available datasets, it is possible to conduct

testable and comparable measurements of the impact of an investment climate reform in a cost-

effective way.

Typically, research on the impact of investment climate reforms has either focused on

cross-country comparisons, which face inference problems, or country-specific impact

evaluations, which face external validity issues and tend to be very expensive and time

consuming (see Bransmetter 2010). The synthetic control and proximity control methodologies

offer a third approach, making inference possible while enabling cross-country comparisons. In

chapter five, for example, we use both country-specific impact measurements and variation in

cross-country outcomes to establish the link between the introduction of a one-stop shop, the

depth of the related reforms, and the resulting increase in new firm creation.

Moreover, the synthetic control and proximity control methodologies enable impact

evaluations at the aggregate level. For example, Bransmetter (2010) measures the impact of

the introduction of a one-stop shop on new firms creation in Portugal by comparing a group of

counties that did introduce the one-stop shop and a group of counties that did not introduce it –

but the study does not answer whether or not the one-stop shop had a large impact on new firm

creation at the national level. Likewise, Bruhn (2008) measures the impact of a one-stop shop


45

on new firm creation and employment in Mexico by using a variation in the timing of the

introduction of the reform in selected municipalities. This is not an aggregate measure of the

impact of the one-stop shop on new business creation in Mexico, but rather a measure of the

average increase in new firm’s creation on the specific municipalities.

One of the main objectives of this paper is to make the case for a wider use of the

synthetic control and proximity control methodologies as cost-effective alternatives to measure

the impact of private sector development reforms. The World Bank Group, regional

development banks and the IMF should and can make much more extensive use of these

methodologies to test the impact of large-scale reforms or projects. Not only do they provide

cheap, quick, comparable and testable results, but they also have a wide array of possible

applications. The synthetic control methodology has been used to measure the impact of

terrorism on economic growth in the Basque Country (see Abadie et al, 2003), the effectiveness

of a tobacco regulatory reform in California (see Abadie et al, 2010), the impact of a World Bank

health care program on aggregate health indicators in Peru (see Parodi et al, 2008), as well as

the cost of the German re-unification on GDP per capita (Abadie et al, 2011). The proximity

control methodology, calculated on the basis of export similarity networks, has been used to

measure the cost of the political and economic crises in Kenya, Ivory Coast and Indonesia.33

The paper also introduces a new way of looking at Doing Business data by introducing a

measure of the similarity of the business environment of a pair of countries. Typically, analysis

using Doing Business data is done within one country (looking at where the country ranks

across variables) or across countries (looking at a specific variables of interest). However, there

is additional information to be inferred by looking at measures of Doing Business similarity

between country-pairs. Every year, the Doing Business team at the World Bank Group gathers

data from over 180 countries and 32 different variables, totaling about 4,800 data points. By

aggregating these 32 variables into one single variable of similarity between pairs of countries,

the resulting number of data points in a single year increases to 32,22034. Likewise, looking at

the similarity levels of triplets of countries, the number of data points would increase to

5,725,160; and so forth. In the same way, data on pairs of countries captures much more

information on the business environment than analysing the single indicators of the Doing

Business database. By showing a clear link between the performance in the Doing Business

indicators and new business registration, we infer that countries with similar business

environments also have very similar levels of new business registration (see Graph 8),

33 See Gathani et Stoelinga (2013). 34 180x179=32,220.


46

This has practical implications for policy makers. While Rwanda was ranked 67th in the

Doing Business 2010 report on the overall ease of doing business, it was also more similar to

Burkina Faso (ranked 147th) than it was to Macedonia (FYROM, ranked 32nd). These differences

and similarities between countries can guide policy makers towards better targeting policies and

understanding how “reform packages” – as opposed to individual reforms aimed at moving up

the rankings – will impact their position in the Doing Business similarity network. Furthermore,

comparisons also create an additional motivation for governments to change; while Rwanda can

pride itself for being one of world’s top reformers several years in a row, we showed that in 2010

its business environment was still too similar to Malawi, Burkina Faso and Paraguay, which all

ranked well below the 120th place in the Doing Business report.

Finally, we showed that introducing a one-stop shop for business registration is an

effective policy to increase the size of the private sector and that the deeper the extent of the

reform, the greater the impact. For countries where the one-stop shop leads to large

improvements in the time, cost, procedures, and minimum capital required to start a business,

this type of reform can mark a true turning point. That is how in Rwanda, Tajikistan and Belarus,

the introduction of the one-stop shop led to more than doubling the creation of new businesses

after only one year in operation.


47

Bibliography

Abadie, Alberto and Gardeazabal, Javier. 2003. “The Economic Costs of Conflict: A Case Study

of the Basque Country,” American Economic Review, vol. 93(1), pages 113-132, March.

Abadie, Alberto, Diamond, Alexis, and Hainmueller, Jens, 2010. “Synthetic Control Methods for

Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program,”

Journal of the American Statistical Association, vol. 105(490), pages 493-505.

Abadie, Alberto, Diamond, Alexis, and Hainmueller, Jens, 2011. “Comparative Politics and the

Synthetic Control Method,” MIT Political Science Department Research Paper 2011-25.

Alcázar, Lorena, and Jaramillo, Miguel. 2011. “Panel/tracer Study on the Impact of Business

Facilitation Processes on Microenterprises and Identification of Priorities for Future Business

Enabling Environment Projects in Lima, Peru.” Mimeo. Grade. Peru, Lima. June.

Bennett, A. and C. Elman. 2006. “Qualitative Research: Recent Developments in Case Study

Methods.” Annual Review of Political Science. 9: 455-76.

Branstetter, Lima, and Taylor, Venancio. 2010. “Do Entry Regulations Deter Entrepreneurship

and Job Creation? Evidence from Recent Reforms in Portugal” NBER Working Paper 16473,

October.

Bruhn, Miriam. 2008. “License to Sell: The Effect of Business Registration Reform on

Entrepreneurial Activity in Mexico,” World Bank Policy Research Working Paper 4538, January.

Ciccone and Papaioannuou. 2007. “Red Tape and Delayed Entry”, Journal of the European

Economic Association, vol.5, no.2-3, pp.444-458.

Doing Business database, 2005-2012. The World Bank Group.

Fearon, J.D. 1991. “Counterfactuals and Hypothesis Testing in Political Science.” World Politics.

43(2): 169-195.


48

Gathani, Sachin and Stoelinga, Dimitri. 2013. “Export Similarity Networks and Proximity Control

Methods for Comparative Case Studies,” Forthcoming, Journal of Globalization and

Development.

Investment Climate Advisory Services, 2010. “How many stops in a one-stop shop? A review of

Recent Development in Business registration,” Flagship report, The World Bank.

Kaplan, Piedra, Seira. 2007. “Entry Regulation and Business Start-Ups: Evidence from Mexico,”

World Bank Policy Research Working Paper 4322, June.

Klapper, Leora, and Love, Inessa, 2010. “The impact of business environment reforms on new

firm registration,” Policy Research Working Paper Series 5493, The World Bank.

Klapper, Leora and Love, Inessa, 2011. “The impact of the financial crisis on new firm

registration,” Economics Letters, vol. 113(1), pages 1-4, October.

Motta, Marialisa, Ana Maria Oviedo, and Massimiliano Santini. 2010. “An Open Door for Firms.

The Impact of Business Entry Reforms.” Viewpoint Note No. 323, The World Bank Group.

Washington, DC. June.

Newey, Whitney K. and McFadden, Daniel. “Large Sample Estimation and Hypothesis Testing,”

in Robert F. Engle and Daniel L. McFadden, eds. Handbook of econometrics, Vol. 4. New York:

Elsevier Science, 1994, pp. 2111–245.

Rand, John and Torm, Nina. 2012. “The Benefits of Formalization: Evidence from Vietnamese

Manufacturing SMEs,” World Development Vol. 40, No. 5, pp. 983–998.

Rubin, D.B. 1990. “Formal Modes of Statistical Inference for Causal Effects.” Journal of

Statistical Planning and Inference. 25: 279-292.

Sims, Christopher A. and Zha, Tao. “Error Bands for Impulse Responses.” Econometrica,

September 1999, 67(5), pp. 1113–55.

World Bank Group Entrepreneurship Snapshots (WBGES). World Bank, 2010.


49

Annexes35

35 The “Estimated Impact” is the average estimated impact among both methodologies.

Albania Year of Reform: 2006

Year of Impact: 2007

Notes

Synthetic Controls

Proximity Controls

Impact estimates Year 1 Year 2 Year 3

Estimated Impact (%) 55% 39% -27%

Estimated Impact (#firms) 676 1039 1437

Indicator Albania Synthetic Albania Proximity Control (6)

GDP per capita (constant USD2000) 1488.8 1386.4 2539.7

Agricture (%GDP) 23.1 14.3 13.3

Industry (%GDP) 21.0 29.1 29.9

Services (%GDP) 55.9 56.6 56.9

Trade balance (%GDP) -22.6 -11.4 -4.2

Gross fixed capital formation (%GDP) 24.0 20.5 20.7

Urbanization (% Population) 44.8 32.5 41.8

MSPE Synthetic Albania Proximity Control (6)

Albania 84.32 134.73

Minimum MSPE Controls 0.56 0.39

Maximum MSPE Controls 6.09 2.20

Proximity Control: Sri Lanka (58.7%), Argentina (21.1%), Phil ippines (15.87%), Uruguay (4.3%)

Synthetic Control: Guatemala (55.3%), Sri Lanka (44.7%)

Donor pool: 6 countries; GDP per capita >USD1000 & <USD2500

Donor pool: 5 countries; Peru and Bolivia eliminated due to impact

0

500

1000

1500

2000

2500

3000

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Num

ber o

f new

firm

s

New Business Density in Albania and its Proximity and Synthetic Controls (2000-2009)

Albania

Synthetic Albania

Placebo

Proximity Control


50

Belarus Reform 2007


Notes

Synthetic Controls

Proximity Controls


Estimated Impact (%) 103% 44% 17%


Indicator (mean 2001-2005) Belarus Synthetic Belarus Proximity Control (606)


Agricture (%GDP) 10.4 14.2 12.1

Industry (%GDP) 40.1 32.5 34.1

Services (%GDP) 49.5 53.3 53.8




MSPE Synthetic Belarus Proximity Control (606)

Belarus 828.05 920.63



Proximity Control: Phil ippines (75.73%), Argentina (23.3%), Moldova (0.83%), Uruguay (0.12%)

Synthetic Control: Philippines (68.3%), Pakistan (20%), Argentina (11.7%)

Donor pool: 25 countries; minimum GDP per capita 300

Donor pool: 5 countries; Bolivia eliminated from pool due to impact

0

1000

2000

3000

4000

5000

6000

2003 2004 2005 2006 2007 2008 2009

Num

ber o

f new

firm

s

New Business Density in Belarus and its Proximity and Synthetic Controls (2003-2009)

Belarus

Synthetic Belarus

Placebo

Adjusted Proximity Control


51

Canada Year of Reform: 2005


Notes

Synthetic Controls

Proximity Controls


Estimated Impact (%) 9% -1%

Estimated Impact (#firms) 14177 17196

Indicator (mean 2001-2005) Canada Synthetic Canada Proximity Control (166)


Agricture (%GDP) 2.1 1.3 3.3

Industry (%GDP) 31.5 19.2 26.0

Services (%GDP) 66.4 79.5 70.7

Trade balance (%GDP) 4.3 4.3 5.4



MSPE Synthetic Canada Proximity Control (166)

Canada 2.06 3.36



Proximity Control: Hong Kong (47.6%), Malaysia (32.3%), Italy (14.2%), Austria (5.8%)

Synthetic Control: Italy (50.7%), Hong Kong (49.3%)


Donor pool: 5 countries; Peru and Ireland eliminated due to impact

0

50000

100000

150000

200000

250000

2000 2001 2002 2003 2004 2005 2006 2007

Num

ber o

f new

firm

s

New Business Density in Canada and its Proximity and Synthetic Controls (2003-2009)

Canada

Synthetic Canada

Placebo

Proximity Control


52

Denmark Year of Reform: 2006


Notes

Synthetic Controls

Proximity Controls


Estimated Impact (%) 16%

Estimated Impact (#firms) 3335

Indicator (mean 2001-2005) Denmark Synthetic Denmark Proximity Control (122)



Industry (%GDP) 25.3 23.0 23.4

Services (%GDP) 72.6 71.3 74.8

Trade balance (%GDP) 5.7 -0.1 6.6



MSPE Synthetic Denmark Proximity Control (122)

Denmark 22.44 14.65



Proximity Control: Austria (42.83%), Hong Kong (37.8%), Iialy (12.2%), Malaysia (7.1%)

Synthetic Control: Croatia (39.6%), Uruguay (33.4%), Hong Kong (25%), Argentina (1.9%)


Donor pool: 5 countries; Peru and Ireland eliminated due to impact

0

5000

10000

15000

20000

25000

30000

35000

2000 2001 2002 2003 2004 2005 2006 2007

Num

ber o

f new

firm

s

New Business Density in Denmark and its Proximity and Synthetic Controls (2000-2009)

Denmark

Synthetic Denmark

Placebo

Proximity Control


53

Georgia Year of Reform: 2006


Notes

Synthetic Controls

Proximity Controls




Indicator (mean 2001-2005) Georgia Synthetic Georgia Proximity Control


Agricture (%GDP) 20.0 20.0 23.3

Industry (%GDP) 24.6 28.6 23.9

Services (%GDP) 55.4 51.4 52.8




MSPE Synthetic Georgia Proximity Control

Georgia 18.10 16.71



Proximity Control: Moldova (59.5%), Pakistan (31.2%), Armenia (7.8%), Uruguay (1.4%)

Synthetic Control: Moldova (55.5%), Thailand (37%), Ethiopia (7.4%)

Donor pool: 11 countries; maximum GDP per capita 2500

Donor pool: 4 countries; Bolivia eliminated from pool due to impact

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Num

ber o

f new

firm

s

New Business Density in Georgia and its Proximity and Synthetic Controls (2003-2009)

Georgia

Synthetic Georgia

Placebo

Proximity Control


54

Kyrgyztan Year of Reform: 2008


Notes

Synthetic Controls

Proximity Controls




Indicator Kyrgyztan Proximity Control Proximity Control (83)


Agricture (%GDP) 23.9 24.8 17.7

Industry (%GDP) 42.2 47.7 54.5

Services (%GDP) -16.8 -19.8 -27.8




MSPE Synthetic Kyrgyztan Proximity Control (83)

Kyrgyztan 288.13 164.73



Proximity Control: Moldova (52.6%), Malawi (18.5%), Niger (14.6%), Ethiopia (14.3%)

Synthetic Control: Cambodia (52.7%), Moldova (37%), Armenia (10.3%)

Donor pool: 6 countries; max average GDP per capita USD2000

Donor pool: 5 countries; optimal sample size: 83

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Num

ber o

f new

firm

s

New Business Density in Kyrgyztan and its Proximity and Synthetic Controls (2000-2009)

Synthetic Kyrgyztan

Proximity Control

Placebo

Proximity Control


55

Netherlands Year of Reform: 2007


Notes

Synthetic Controls

Proximity Controls




Indicator (mean 2001-2005) Netherlands Synthetic Netherlands Proximity Control (28)



Industry (%GDP) 24.2 29.4 26.2

Services (%GDP) 73.6 65.6 72.2




MSPE Synthetic Netherlands Proximity Control (28)

Netherlands 3.40 0.37



Proximity Control: Austria (55%), Italy (18.8%), Switzerland (14%), Hong Kong (12.1%)

Synthetic Control: Argentina (45%), Italy (35.2%), Ireland (11.1%), Hong Kong (8.7%)

*Note: Ireland dropped from Proximity Control due to impact


Donor pool: 4 countries; Ireland eliminated due to impact

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

2000 2001 2002 2003 2004 2005 2006 2007 2008

Num

ber o

f new

firm

s

New Business Density in Netherlands and its Proximity and Synthetic Controls (2003-2009)

Netherlands

Synthetic Netherlands

Placebo

Proximity Control


56

Oman Year of Reform: 2006


Notes

Synthetic Controls

Proximity Controls




Indicator (mean 2001-2005) Oman Synthetic Oman Proximity Control (468)



Industry (%GDP) 54.2 33.9 31.5

Services (%GDP) 43.8 57.6 47.8


Gross fixed capital formation (%GDP) 18.1


MSPE Synthetic Oman Proximity Control (468)

Oman 36.62 26.69



Proximity Control: Pakistan (66.26%), Malaysia (19.93%), Armenia (13.8%)

Synthetic Control: Argentina (88%), Malaysia (12%)

Donor pool: 4 countries; GDP per capita >USD4000 & <USD15000

Donor pool: 3 countries; Peru and Thailand eliminated due to impact

0

500

1000

1500

2000

2500

3000

3500

4000

4500

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Num

ber o

f new

firm

s

New Business Density in Oman and its Proximity and Synthetic Controls (2000-2009)

Oman

Synthetic Oman

Placebo

Proximity Control


57

Rwanda Year of Reform: 2009


Notes

Synthetic Controls

Proximity Controls




Indicator (average 2000-2008) Rwanda Synthetic Rwanda Proximity Control (12)


Agricture (%GDP) 36.7 35.9 33.5

Industry (%GDP) 13.8 19.9 16.9

Services (%GDP) 49.5 44.2 49.6




MSPE Synthetic Rwanda Proximity Control (12)

Rwanda 388.21 163.16



Proximity Control: Malawi (72.37%), Niger (14.45%), Ethiopia (11.11%), Croatia (2.06%)

Synthetic Control: Cambodia (40.5%), Malawi (32.6%), Ethiopia (26.9%)

Donor pool: 7 countries; max average GDP per capita USD$1000


0

500

1000

1500

2000

2500

3000

3500

2003 2004 2005 2006 2007 2008 2009

Num

ber o

f new

firm

s

New Business Density in Rwanda and its Proximity and Synthetic Controls (2003-2009)

Rwanda

Synthetic Rwanda

Placebo

Proximity Control


58

Senegal Year of Reform: 2007


Notes

Synthetic Controls

Proximity Controls


Estimated Impact (%) 45% -18%

Estimated Impact (#firms) 561 824

Indicator (mean 2001-2005) Senegal Synthetic Senegal Proximity Control (117)


Agricture (%GDP) 15.6 33.7 18.5

Industry (%GDP) 24.2 19.9 28.2

Services (%GDP) 60.2 46.4 53.2




MSPE Synthetic Senegal Proximity Control (117)

Senegal 398.29 110.38



Proximity Control: Pakistan (58.4%), Bolivia (28.8%), Argentina (12.1%), Algeria (0.6%)

Synthetic Control: Ethiopia (60.1%), Sri Lanka (36.3%), Armenia (3.6%)


Donor pool: 5 countries; Nigeria eliminated from pool due to impact

0

200

400

600

800

1000

1200

1400

1600

1800

2000

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Num

ber o

f new

firm

s

New Business Density in Senegal and its Proximity and Synthetic Controls (2003-2009)

Senegal

Synthetic Senegal

Placebo

Proximity Control


59

Tajikistan Year of Reform: 2008


Notes

Synthetic Controls

Proximity Controls




Indicator (mean 2001-2005) Tajikistan Synthetic Tajikistan Proximity Control (9)


Agricture (%GDP) 24.8 29.5 37.4

Industry (%GDP) 33.4 19.9 15.3

Services (%GDP) 41.9 50.6 47.3




MSPE Synthetic Tajikistan Proximity Control (9)

Tajikistan 119.89 129.47



Proximity Control: Niger (55.6%), Ethiopia (32.2%), Moldova (12.2%)

Synthetic Control: Malawi (46.4%), Sri Lanka (24%), Phil ippines (20.9%), Cambodia (6.8%), Moldova (1.8%)



0

200

400

600

800

1000

1200

1400

1600

1800

2000

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Num

ber o

f new

firm

s

New Business Density in Tadjikistan and its Proximity and Synthetic Controls (2003-2009)

Tajikistan

Synthetic Tajikistan

Placebo

Proximity Control


60

Tunisia Year of Reform: 2006


Notes

Synthetic Controls

Proximity Controls


Estimated Impact (%) 33% -7% 21%


Indicator Tunisia Synthetic Rwanda Proximity Control (99)



Industry (%GDP) 29.4 38.5 32.1

Services (%GDP) 60.1 50.8 58.0

Trade balance (%GDP) -2.5 0.9 -3.2



MSPE Synthetic Tunisia Proximity Control (99)

Tunisia 76.27 75.84



Proximity Control: Austria (58.05%), Armenia (25.3%), Sri Lanka (16.6%), Panama (0.05%)

Synthetic Control: Philippines (54.5%), Algeria (24.7%), Brazil (13%), Latvia (5.8%), Thailand (1.9%)

Donor pool: 17 countries; max average GDP per capita USD5000

Donor pool: 5 countries; Peru eliminated due to impact

0

1000

2000

3000

4000

5000

6000

7000

8000

2001 2002 2003 2004 2005 2006 2007 2008 2009

Num

ber o

f new

firm

s

New Business Density in Tunisia and its Proximity and Synthetic Controls (2001-2009)

Tunisia

Synthetic Tunisia

Placebo

Proximity Control

innovative techniques to evaluate the impact of private ......the government of rwanda recently...

Documents