final report evaluation of psom/psi 1999-2009 and mmf

FINAL REPORT EVALUATION PSOM/PSI 1999-2009 AND MMF For the Netherlands Ministry of Foreign Affairs Department of Sustainable Economic Development (DDE)

Triodos Facet

P.O. Box 55, 3700 AB Zeist

The Netherlands

T +31 (0)30 6933 766

F +31 (0)30 6923 936

E [email protected]

W www.triodosfacet.nl

Author: Triodos Facet BV, the Netherlands Date: 14 July 2010 Photo: Employees returning to work after lunch, Maranque Plants PLC, Ethiopia; N. Stam 2010. Disclaimer: This report has been prepared with the utmost care, based on sources of information deemed reliable by the

consultant. However, Triodos Facet does not assume any legal liability or responsibility for the accuracy, completeness, or usefulness of the information disclosed.

‘THE MISSION OF TRIODOS FACET IS TO CONTRIBUTE TO SUSTAINABLE ENTERPRISE DEVELOPMENT’

TRIODOS FACET FINAL REPORT PSOM/PSI AND MMF EVALUATION 1999-2009 TABLE OF CONTENTS

TABLE OF CONTENTS

LIST OF ABBREVIATIONS

EXECUTIVE SUMMARY

1 INTRODUCTION..................................................................................................................................... 1

1.1 BACKGROUND TO THE EVALUATION .................................................................................................................. 1 1.2 EVALUATION OBJECTIVES AND RESEARCH QUESTIONS ........................................................................................... 1 1.3 METHODOLOGY AND ORGANISATION OF THE EVALUATION .................................................................................... 3 1.4 STRUCTURE OF THE REPORT ............................................................................................................................ 6

2 BACKGROUND TO THE PSOM/PSI PROGRAMME .................................................................................. 7

2.1 RECONSTRUCTION OF POLICY DEVELOPMENT FOR PSOM/PSI .............................................................................. 7 2.2 FINDINGS ON EARLIER EVALUATIONS (MTR AND EVALUATION 2005) ..................................................................... 9

3 FINDINGS ON THE PSOM/PSI PROGRAMME ........................................................................................ 11

3.1 THE EVD AS IMPLEMENTING AGENT FOR PSOM: ITS ORGANISATION AND COSTS .................................................... 11 3.2 FINDINGS ON THE WORK OF THE EVD: SELECTION AND MONITORING AND SUPPORTING SERVICES .............................. 16 3.3 FINDINGS ON PROJECTS FINANCED BY PSOM ................................................................................................... 26

4 ANALYSIS OF EVD AS IMPLEMENTING AGENTS AND THE PSOM/PSI PROGRAMME ............................. 37

4.1 THE EFFICIENCY AND EFFECTIVENESS OF THE EVD IN SELECTING AND MANAGING PROJECTS ....................................... 37 4.2 THE EFFICIENCY, EFFECTIVENESS AND RELEVANCE OF THE PSOM/PSI PROJECTS ..................................................... 40

5 EVALUATION OF THE MMF .................................................................................................................. 45

5.1 THE BACKGROUND TO THE MATCHMAKING FACILITY.......................................................................................... 45 5.2 FINDINGS ON INPUTS, OUTPUTS AND OUTCOME ............................................................................................... 45 5.3 ANALYSIS OF THE EFFICIENCY AND EFFECTIVENESS OF THE MMF PROGRAMME ....................................................... 52

6 CONCLUSIONS AND ISSUES THAT REQUIRE ATTENTION ....................................................................... 56

ANNEXES Annex 1 Terms of reference Annex 2 Evaluation matrices Annex 3 Sampled completed and stopped projects for PSOM/PSI and MMF Annex 4 Key results of 32 projects visited Annex 5 List of interviewees in the Netherlands As separate reports: Annex 6 Ghana field report Annex 7 Vietnam field report Annex 8 Ethiopia field report Annex 9 Mozambique field report Annex 10 Indonesia field report Annex 11 Suriname field report

TRIODOS FACET FINAL REPORT PSOM/PSI AND MMF EVALUATION 1999-2009 ABBREVIATIONS

LIST OF ABBREVIATIONS

APSOM/APSI Independent Advisory body for PSOM/PSI

BEMO Beleidsmemorandum (policy document used by DDE)

CBI Centrum ter Bevordering van Import uit ontwikkelingslanden

CL Cluster leader (management position in Agentschap NL)

CLOM Independent Advisory body for PSOM/PSI (now known as APSI)

CSR Corporate Social Responsibility

DDE Directie Duurzame Economische Ontwikkeling (Department of Ministry of Foreign Affairs)

EVD Agency for international business and cooperation (now: Agentschap NL)

FEM Financieel Economisch Medewerker

IOB Inspectie Ontwikkelingssamenwerking en Beleidsevaluatie

LDC (MOL) Least Developed Country

MMF Matchmaking Facility

MoU Memorandum of Understanding

MTR Mid-term review

PA Project Advisor

PSOM Programma Samenwerking Opkomende Markten

PSI Private Sector Investment programme

PUM Programma Uitzending Managers

SPA Senior Project Advisor

ToR Terms of reference

UM Unit Manager

TRIODOS FACET FINAL REPORT PSOM/PSI AND MMF EVALUATION 1999-2009 EXECUTIVE SUMMARY - I

EXECUTIVE SUMMARY The Programme for Cooperation with Emerging Markets (PSOM) was established by the Dutch Minister for Development Cooperation at the end of 1998. In 2008 PSOM came to an end and in 2009 a new, but similar programme was set up: the Private Sector Investment programme (PSI). The objective of PSOM/PSI is to contribute to poverty reduction by stimulating sustainable investments in innovative businesses in selected developing countries. PSOM/PSI aims to encourage Dutch and foreign entrepreneurs to establish investment projects in developing countries in a joint venture company with local entrepreneurs. The intention is to encourage investment project that would not otherwise have been carried out because of the high product/market risks. The programme provides a subsidy equal to 50% or 60% of the project costs, depending on the country. The Matchmaking Facility (MMF), which helps forge partnerships between companies, is a possible preparatory step to submitting a PSI proposal. The MMF started as a pilot programme in 2006 and subsequently became a regular programme. Its objective is to match entrepreneurs in developing countries with Dutch firms and then help them build a sustainable relationship which will hopefully lead to a joint investment or another form of trading relationship. As the name suggests, the programme supports matchmaking between companies. It can award a €5,000 subsidy through consultants who facilitate the forging of a partnership. Both programmes are implemented by the Agentschap NL (formerly EVD), the agency of the Dutch Ministry of Economic Affairs for international business and cooperation, through an assignment by the Minister for Development Cooperation. Both programmes are within the remit of the Sustainable Economic Development Department of the Ministry of Foreign Affairs (DDE) In 2002, a mid-term review of the PSOM programme was conducted, followed by an evaluation in 2005. At the time of that evaluation the number of completed projects was limited and therefore many impacts were still just ‘envisaged’ or ‘to be achieved’. The purpose of this evaluation is to examine whether the PSOM/PSI projects carried out between 1999 and 2009 and MMF activities between 2007 and 2009 have indeed delivered the expected results. It therefore has a strong element of accountability, but the results of the evaluation may also be used to improve the current policy and implementation guidelines for PSI and MMF. The basic research questions used in this evaluation relate to an assessment of the efficiency, effectiveness and relevance of the PSOM/PSI and MMF programme and projects as well as the efficiency and effectiveness of the EVD as the agency which implements both programmes. To carry out this evaluation, the consultants studied the PSOM/PSI and MMF-related information that was available at the EVD, and interviewed EVD, DDE and embassy staff as well as the members of the advisory committee. A desk study covered 60 completed PSOM projects in six pre-selected countries: Ethiopia, Ghana, Indonesia, Mozambique, Surinam and Vietnam. Visits were made to 32 of these projects. In addition, 25 PSOM projects that stopped prematurely were covered through telephone interviews and desk study. For MMF, the databases were analysed, 11 completed project files were analysed, MMF staff and two consultants were interviewed and 97 of the 457 applicant firms participated in an internet survey PSOM/PSI programme Since its start in 1998, PSOM has gone through four phases, with five related policy memoranda (BEMOs). Neither the main objectives, nor the implementation modalities and criteria have changed over time. Follow-up investments, effects on the chain of work and income-generating activities and the absence of a negative impact on gender-related issues or the environment, have always been considered to be indicators of success. Adjustments made in the programme relate to the number of countries, which increased from eight in 1999 to 53 in 2009. Since 2003 the criteria for development impact include job creation, an increase in income, knowledge transfer and establishing linkages with local small and

TRIODOS FACET FINAL REPORT PSOM/PSI AND MMF EVALUATION 1999-2009 EXECUTIVE SUMMARY - II

medium-sized companies. In 2007 the contribution to the business climate dialogue was added as an objective and in 2009 the strengthening of the local private sector became one of the expected indicators of success. The total available subsidy budget between 1999 and 2009 was €391 million. Since 1999, a total of 656 PSOM/PSI projects were approved in 56 countries. Of these projects, 205 have been completed, 275 are ongoing and 176 have stopped, 120 of which prematurely. Most of the applicants are Dutch, male and the owner of a small or medium-sized business. Most projects are in Africa (46%) and relate to the agricultural sector (62% of the funds disbursed). In total €331 million was committed of which €151 million was disbursed to projects running until the end of 2009. The average subsidy per completed project is €495.215 which is 55% of the average project cost of €908.509. The total implementation costs for the entire evaluation period was €26 million, half of which was spent in the past three years. The PSOM/PSI application and approval process has been quite efficient.

- 84% of the targeted number of projects were approved over the last three years - For the period in which all projects were completed (1999-2003), 69% of the projects were still

operating at the completion date, while the target was 66%. The effectiveness of PSOM/PSI projects is good: two thirds of the projects that were visited are effective. The average company created 81 jobs with a subsidy contribution of €6,130 per job. Innovation has also been achieved, mostly by introducing new hardware, which constitutes 67% of the PSOM subsidy. Knowledge transfer to 96 people per project has taken place, which means that the subsidy contribution equals €5,158 per trainee. ‘Outgrowers’ represent an important outcome in agricultural projects; the average number per project with outgrowers was 1,342. The definition of outgrower needs refinement in order to assess PSOM’s contribution to outgrower development. If the outgrowers are included in the knowledge transfer figures, an average of 237 people were trained per project. The relevance of PSOM/PSI projects is good: two thirds of the visited projects are relevant because they created considerable follow-up investments and achieved higher levels of sales and jobs. The average multiplier of the PSOM subsidies is 1.22 when projects are completed and the field study suggests that for some projects this figure increased considerably after completion. Based on the field study, we estimate that of all projects approved and completed, 57% resulted in a lasting enterprise seven to ten years after the approval date. The surviving companies visited have seen an average increase of 31% in employment after the project completion date. The multiplier for follow-up investment varies between 0 and 20. The effects beyond the individual projects such as SME linkages, catalytic effects on the sector and the contribution to the business climate dialogue were limited except for in the horticultural sector in Ethiopia. This lack of linkages is mainly due to the fact that about 55% of the projects are of the re-exporter type. A typical company in this category imports inputs which are used to produce a product that is then exported. Linkages with local enterprises are in such cases limited. Sector effects and contribution to the business climate dialogue are more recent objectives and the responsibility for achieving these effects is beyond the responsibility of the beneficiaries and the EVD. Local private sector stakeholders and the Dutch embassies could contribute to the realisation of these effects, but their involvement in PSOM/PSI is limited except for isolated cases as in Ethiopia. All PSOM projects were additional ex ante based on the EVD definition: all involve risk and were not commercially fundable. According to the findings of the consultants, 53% of the projects are fully additional ex-post, 41% are additional in specific aspects because it is likely that some of the outputs and outcomes would also have been achieved without the PSOM subsidy and two projects were not additional. The EVD as implementing agent for PSOM/PSI The assessment of the efficiency and effectiveness of the EVD as the implementing agency of the PSOM/PSI programme involved an assessment of the major tasks of the EVD. These tasks are selecting and monitoring projects and providing overall management of, and support to, the programme. Efficiency assessments were based on the organisational set-up, including the staff and management of the PSOM/PSI unit and costs related to implementation. The effectiveness assessment was based on the

TRIODOS FACET FINAL REPORT PSOM/PSI AND MMF EVALUATION 1999-2009 EXECUTIVE SUMMARY - III

results of the work of the EVD, and included assessing the quality of the selection process, the project portfolio, as well as monitoring and reporting. Over the period 1999-2009, €26 million were spent as implementation costs, which equal 17% of the disbursed programme cost. Between 2006 and 2009 these costs increased from €3.1 million to €5 million while the amount of subsidy disbursements remained stable at €24 million each year. This increase is mainly due to increased overhead (explicit in budgets, or encapsulated in increasing fees), and not an increase in the actual workload. Apart from workload increases, wage cost increases and the increase in legal costs, implementation in 2009 was 9% more expensive than in 2006. Overall, the percentage of overhead in the total cost has increased over the last five years when PSOM became a large programme, as compared to the first five years, when PSOM was much smaller. We therefore conclude that the efficiency of the EVD as the implementing agency has declined and that no economies of scale have been achieved. As a measure for the efficiency of the selection process, we found that the cost per contracted project has increased in recent years, while the approval ratio has also increased. The time spent to follow the steps of the current selection procedure is close to the norm applied by the EVD. Therefore, we rate the selection process as efficient organisationally, but overall as only partially efficient, considering the EVD cost price increases. The project development and selection phase also comprises hidden additional cost, because approximately 50% of the applicants contract a consultant to develop the project, for which the costs are often hidden in the subsidy. In terms of the output achieved by the EVD, we have assessed the number of applications, projects contracted, and approval ratios over time. The available budgets increased from €30 million per year in 2003 to €70 million per year in 2009 and the number of eligible countries increased from 21 in 2004 to 51 in 2009. The number of applications received in the first six years (1999-2004) was 867 applications, against 801 applications in the past five years. The number of contracted projects increased, because the approval ratio increased from an average of 30% between 1999 and 2004 to 50% in the period 2005-2009. One plausible explanation for the increased approval ratio is improved quality of the applications; this could be the result of the intake interviews and information workshops organised by the EVD for potential applicants, as well as increased experience of consultants. The role of the advisory committee APSOM/APSI is positive: the four members assure quality, consistency and feasibility of the proposed projects, but the overlap with the work of the PSI cluster leaders creates some inefficiency. The role of the consultants is ambiguous; they generate applications and ensure smooth implementation of the projects, but are a hidden cost in addition to the EVD implementation cost. Although the assistance from consultants is considered a necessity by most applicants, it is not always appreciated. The selection process is effective in terms of generating a portfolio of successfully completed projects; 69% of the approved projects continued up to the completion date. The costs of the projects that stopped prematurely are low with only 7% of the disbursed programme funds used for these 120 projects. The selection process is less effective in terms of utilising available budgets and maximising development effects. The current appraisal process does not require the EVD to choose between projects on the basis of their development effects. In addition, the PSOM/PSI communication strategy designed to increase the number of quality applications was found not to be very effective: the increase in funds used for communication has not resulted in substantially more interest from the business community. The management of ongoing projects is efficient, since the costs of management have decreased over time for the same output, and the activities necessary according to the procedures in place are implemented within a relatively small number of days. Project management and monitoring is effective in ensuring that the hardware part of the subsidy is correctly spent according to the rules and contract. With respect to the monitoring of the project outcome in terms of employment, knowledge transfer, turnover and follow-up investments the current monitoring system is limited, since the information is mainly provided by the beneficiaries. The very low reductions in the disbursed subsidy amounts for the projects that didn’t stop suggests that all results were achieved, which is surprising in view of the fact that PSOM/PSI was designed as a programme for high-risk ventures.

TRIODOS FACET FINAL REPORT PSOM/PSI AND MMF EVALUATION 1999-2009 EXECUTIVE SUMMARY - IV

One explanation could be that the results which are formulated in the contract are too soft; another explanation could be that monitoring is not effective. The current reporting on the PSOM/PSI portfolio is not very efficient, as two similar reports are produced each year. The elaborate reporting on development impact, which started in 2004, was discontinued in recent years. The EVD support services include the legal department and administrative systems. With respect to the legal procedures that are needed in case of objections by companies, we conclude that these services are provided efficiently and effectively. No claim has yet been acknowledged and the cost is reasonable compared to the overall implementation cost. Anti-fraud procedures are in place, and there are only a few cases per year. The administrative systems are inefficient and only moderately effective. Most systems that are needed for administrating the programme are outdated and employees are widely dissatisfied with the systems in use, as well as the hardware. Efficiency gains could be achieved with improved systems. However, the current systems do generate the necessary information and do not hamper the execution of the programme. The Matchmaking Facility The MMF programme has received 457 applications for matchmaking between 2007 and 2009. Half of these applications come through embassies, which are the main source of pipeline for MMF. A total of 277 applicants were screened positively and for 174 of these companies matches were found. Two thirds of the matches were found by consultants and one third by the MMF unit in the EVD. In total 150 voucher trajectories started of which 35 were finalised. Most vouchers were provided to companies in Central and Eastern Europe (35%) and Africa (29%) and to companies active in the agricultural sector (40%). Total costs for MMF in the period 2007- 2009 were €1,266,793; two thirds was spent on implementation costs and one third on subsidies for consultancy services. The efficiency of MMF is low and its overall results do not meet the proposed targets: the number of applications and disbursements for voucher trajectories is less than expected and the implementation cost per voucher was planned to be €3,543 while the actual costs were €5,779. However, during the MMF implementation period, targets were adjusted, so the achievement of targets is improving. In order to enhance efficiency, the MMF unit has gradually transferred the match-making tasks to consultants who are not (yet) paid for this task. The MMF unit’s core tasks are now to promote, manage and monitor the matchmaking programme. The effectiveness of MMF is promising. Up to March 2010, 14 of the finalised trajectories have led to a positive outcome, eight PSI proposals for joint investment were approved and six trade relationships were established. If the remaining 115 ongoing voucher trajectories are equally successful, another 60 positive outcomes can be expected, but these would require disbursement of additional implementation and programme costs. The conclusion with regard to the MMF programme is that a new policy framework for the matchmaking facility needs to be developed. The adjustments in targets, the evolution in the role of the MMF unit and consultants, new collaboration with other parties such as PUM and CBI and the varying expectations at the level of the applicants, justify such a new policy framework. Issues for attention The key issues that could improve PSI programme implementation by Agentschap NL are related to a review of costs and overhead (increases), an increased choice in applications based on development effects, and a review of the monitoring methods. With respect to efficiency, we suggest a review of the overheads and general costs of the EVD; as we have estimated, the costs of overhead (both budgeted and within the fees) increased significantly over the last five years. Overheads as a percentage of overall implementation cost are now higher than at the beginning of the programme.

TRIODOS FACET FINAL REPORT PSOM/PSI AND MMF EVALUATION 1999-2009 EXECUTIVE SUMMARY - V

Within the PSI unit, some improvements can be made to increase efficiency: for example, the early de-selection of applications that are eligible but nonetheless unlikely to qualify; and a review of the task distribution between APSI and the PSI cluster leaders in order to ensure quality and consistency. With respect to effectiveness of the selection, we believe the main challenge for Agentschap NL should be to widen the choice of eligible and relevant applications. In our view, the assessment of the development relevance is not as rigorous as the assessment of the partners and the business plans. With a wider choice, the existing ranking could be applied, and the development impact of PSI could be improved. To increase the number of applications for the current PSI programme, the EVD could develop structural cooperation with other organisations that are working with potential applicants and recipients, such as CBI, PUM and others, including private sector. Efforts to cooperate with other organisations to date have been limited to dialogue and cooperation agreements, and have not resulted in joint action or tangible results. The monitoring of results relies too much on the principle of “trust”, which is not appropriate when problems arise that affect the key results upon which the payments depend, such as employment and turnover targets. Here we suggest reviewing whether the targets set in the contract reflect the ambitions in the business plans and whether companies receiving subsidies should not be requested to submit an audited account for the final year of the project, so that results can be verified independently. PSI project-related issues that require further attention include a better definition of the necessity and role of joint venture partners and better definition of key result areas such as innovation, trainee, outgrowers, additionality and market distortion. A common understanding between Agentschap NL and the Ministry of Foreign Affairs about these criteria would make the assessment of PSOM/PSI results more objective. Finally, we recommend Agentschap NL to modernise its administration systems in order to improve efficiency and information quality, which will free resources for the staff to focus on core tasks. In order to make a proper assessment of market distortion, innovation and additionality of PSI applications, sufficient time and expertise needs to be made available in each country. We therefore suggest that PSI only operates in countries where embassies commit the required time, (hired) capacity and funds to the programme and whereby this involvement is not voluntary. This might reduce the number of countries where PSI is active but it might also enhance the number of eligible applications and the development effects per country. Another issue that requires further attention is that although PSOM/PSI projects can contribute to sector effects that encompass the individual project level and the business climate dialogue, to achieve these effects, it is necessary to involve local sector stakeholders and the embassies, private sector lead donors or related organisations. Finally, we think that Agentschap NL and the embassies can play a catalysing role in shaping Corporate Social Responsibility (CSR) at sector level, based on experiences with individual PSOM/PSI projects. This would require more in-house or hired capacity in this area at all implementation levels.

TRIODOS FACET FINAL REPORT PSOM/PSI AND MMF EVALUATION 1999-2009 1

1 INTRODUCTION

1.1 Background to the evaluation

The Programme for Cooperation with Emerging Markets (PSOM) was established by the Dutch Minister for Development Cooperation at the end of 1998. In 2008 PSOM came to an end and in 2009 a new similar programme was set up: the Private Sector Investment programme (PSI). The objective of PSOM/PSI is to contribute to poverty reduction by stimulating sustainable investments in innovative businesses in selected developing countries. PSOM/PSI encourages Dutch and foreign entrepreneurs to start an investment project in developing countries in partnership with local entrepreneurs. The aim is to encourage investment project that otherwise would not have been achieved because of high product/market risks. The Matchmaking Facility (MMF), which helps forge partnerships between companies, is a possible preparatory step prior to submitting a PSI proposal. The MMF started in 2006 as a pilot programme and continued afterwards as a regular programme. Its objective is to match entrepreneurs in developing countries and Dutch firms and then help them build a sustainable relationship which will hopefully lead to a joint investment or lasting trade relationship. The budget for the MMF in the period 2007 - 2010 is €2.8 million. Both programmes are implemented by the Agentschap NL (formerly EVD), the agency of the Ministry of Economic Affairs for international business and cooperation, which is within the remit of the Minister for Development Cooperation. In 2002 a mid-term review on the PSOM programme was conducted, followed by an evaluation in 2005. Both evaluations were conducted by Ecorys. At the time of that evaluation the number of completed projects was limited and therefore many impacts were still just ‘envisaged’ or ‘to be achieved’. This evaluation will therefore cover the entire period from 1998 until 2009 in order to be able to assess to what extent the investment of PSOM projects worked as a catalyst to develop a sector or a market in a country. The terms of reference for this evaluation are enclosed in Annex 1.

1.2 Evaluation objectives and research questions

The purpose of the evaluation is to examine whether PSOM/PSI projects and MMF activities that have been completed have indeed delivered the expected results and had the required impact. It therefore has an element of accountability (‘prove’). However, the results of the evaluation may also be used to enhance the current policy and implementation guidelines for PSI and MMF (‘improve’). It is important to note that the evaluation was intended to review the current status of projects that were completed during the evaluation period. The findings of the evaluation are described in this report, which shows to what extent the funds invested in PSOM/PSI and MMF have produced results that meet the objectives of the Minister for Development Cooperation. A second purpose of the evaluation is to examine the efficiency and quality of the work of the EVD. The emphasis of the evaluation was on the efficiency, effectiveness and relevance of the PSOM/PSI and MMF programmes, as well as on the efficiency and effectiveness of the EVD in implementing the programmes.

The research questions used to evaluate the efficiency, effectiveness and relevance of the PSOM/PSI and MMF projects are:


Efficiency

How many and what kind of investment projects with knowledge and technology transfers have been achieved through PSOM/PSI, at what cost and with which resources and systems?

How many and what type of matchmaking activities have been achieved by MMF, at what cost and with which resources?

Effectiveness

How many and what type of jobs have been created by the PSOM/PSI supported companies? And how many outgrowers have been contracted?

To what extent has support for the companies led to innovations?

To what extent have the PSOM/PSI projects contributed to knowledge transfer (type and outreach)

How many sustainable trade relationships and PSI proposals have been achieved as a result of MMF activities?

Relevance

To what extent have the PSOM/PSI supported companies contributed to job creation and sustainable private sector development (creation of financially (sales), institutionally (JV) and environmentally (CSR) sound) companies)?

How many and what type of follow-up investments have been made by the PSOM/PSI supported companies?

Which improvements in the private sectors concerned (chain and catalyst effects) have been achieved thanks to the PSOM/PSI projects?

Additionality

Commercial viability/Payback period (when did you start making a profit?)

Was the project bankable?

Other sources of finance available?

Would entrepreneurs have realized the project without PSOM ?

These research questions were translated in the result matrices which are included in Annex 2. As can be seen from this table, the additionality question relates to the projects completed and will therefore be based on an ex-post assessment of the additionality, which is different from the ex-ante additionality assessment done by the EVD. In the ex-post situation the consultants assess whether the output and outcome could have been achieved without PSOM subsidy, whereas in the ex-ante assessment the EVD assesses whether the project is risky and whether the project can be financed commercially. For evaluating the efficiency and effectiveness of the EVD in implementing the programmes the research questions for the assessment of the process and performance of the EVD and DDE are:

Efficiency

How many and what kind of projects were approved by the EVD and at what administrative cost?

What type and quality of monitoring and reporting is carried out by EVD?

What is average throughput time, quality and ‘success rate’ of the EVD appraisal process?

Effectiveness

How often and what type of interaction is there between those involved in the operational and strategic level of the PSOM/PSI and MMF programme?

To what extent is the monitoring and planning cycle guided by internal learning and policy insights?


To what extent have evaluations and feedback led to adjustments in the programme implementation?

1.3 Methodology and organisation of the evaluation

In order to evaluate the PSOM/PSI and MMF programmes the consultant followed the logical steps of a typical evaluation exercise:

1) Fine tuning of the research questions

2) Drafting the result matrix and formulating indicators for input, output, outcome and impact

3) Sampling projects to determine the information to be collected

4) Collecting data for each result area through desk research, interviews and field visits

5) Data analysis to assess efficiency, effectiveness and relevance

6) Formulating an opinion based on the findings and analysis The evaluation method is based on gathering information on the basis indicators formulated for each of the result levels (input, output, outcome and impact). Subsequently, the relationship between these indicators at the different result levels is assessed or calculated, in order to obtain information on efficiency, effectiveness and relevance. The details of this approach are described in our proposal as well as in the Inception report. The client for this evaluation assignment is represented by a reference group consisting of: Sabine Blokhuis – DDE; Ton Negenman - DDE; Fred van der Kraaij - Policy and Operations Evaluation Department - IOB; Michiel van Erkel - DZO; and Hidde van der Veer - EVD PSOM/PSI Unit manager as an observer. The lead consultant met five times with the reference group prior to the production of this final report. Meetings that took place were a briefing meeting, a meeting on submission of the Inception report, two meetings to discuss the draft field visit reports on Ghana and Vietnam, and a final meeting to discuss the draft final report. The team who produced this evaluation consisted of five Triodos Facet consultants: Hedwig Siewertsen –Team leader and field visit to Ghana and Ethiopia; Sonke Buschmann – Assessment EVD and field visit to Vietnam and Indonesia; Gerrit Ribbink – Field visits to Surinam and Mozambique; Nienke Stam – Desk research and field visit to Ethiopia and Vietnam and Joep Vonk – Desk research and Field visit to Mozambique and Indonesia. The team was complemented by a local consultant in each of the countries visited (for the names, kindly refer to the country reports). The PSOM/PSI evaluation consisted of the following activities:

1) Preparation research framework, indicators, interview guidelines and work programme 2) Desk study on 60 completed projects 3) Telephone interviews and desk research on 25 stopped projects 4) Assessment of the EVD organisation through desk study and staff interviews 5) Field visits to 32 projects in six countries 6) Preparation of draft field reports 7) Comments and feedback from EVD project advisers on field reports 8) Data analysis, interpretation and reporting

The MMF evaluation consisted of the following activities:

1) Preparation of research framework and indicators 2) Review of MMF documents, products and data analysis 3) Internet survey among all MMF applicants 4) Interviews with MMF staff and MMF consultants 5) Reporting


Preparing the research framework and indicators as well as the work programme and sampling method were described in the Inception report, which has been discussed with the reference group. The evaluation frameworks are included in Annex 2. The people met during the field visits are listed in the annexes of each field visit report.

1.3.1 The sampling method used by PSOM/PSI

For the evaluation of PSOM, the client selected six countries for a field visit on the basis of the following criteria:

a) geographical coverage across Africa, Latin America and Asia b) at least five completed projects in a country; c) status: partner country of the Netherlands in 2009; d) distribution over income classes: two LDCs, two LICs, one LMIC, one UMIC; e) geographical distribution within Africa.

In accordance with these criteria the countries selected for field visits were Ghana, Mozambique, Ethiopia, Indonesia, Vietnam and Surinam. We used systematic sampling to create a sample of 60 projects from the total population of 84 projects in the six research countries that were completed as per 31/12/2009. The sample size per country is in each case five projects per field visit, plus one ‘spare’ field visit project. In addition, a number of projects were selected for desk study only. The desk study selection per country is proportionate to the total number of completed projects per country. Table 1: Overview of the evaluation sample PSOM/PSI 1999-2009 (number of projects per country)

Country Completed projects Field visit Desk study Total sample

Ethiopia 12 5 4 9

Ghana 15 5 6 11

Indonesia 29 5 13 18

Mozambique 12 5 4 9

Suriname 5 5 0 5

Vietnam 11 5 3 8

Total 84 30 30 60

In order to create the sample, the 84 completed projects were sorted by country, year started, and sector. In sequence, we skipped every Xth project in the list to eliminate the projects that will not be included in the sample. For example, in Ethiopia 3 of the 12 projects were skipped: in effect, every fourth project was thus eliminated. The results of the sampling are shown in Annex 3. The total number of projects that stopped because of problems is 120 since the start of PSOM/PSI. As part of their 2005 evaluation Ecorys interviewed people in the projects that stopped which started in or before 2005. For this evaluation we have therefore selected a sample from the 46 stopped projects that date from 2006 or later. Of these 46 projects we randomly selected 25 projects for telephone interviews and desk research. For the results of the sampling, see Annex 3.

1.3.2 Sampling method MMF

To assess the quality of the MMF products (output) we selected 11 projects from a total of 33 successfully completed MMF trajectories. After sorting the completed projects in alphabetical order, every third project was selected. The internet survey was conducted among all 457 companies that requested support from MMF. The two MMF consultants to be interviewed were randomly selected after shortlisting them according to the number of MMF cases handled. For the results of the sampling, see Annex 3.


1.3.3 Data collection PSOM/PSI

To address the questions related to the functioning of the EVD, the consultants interviewed representatives of all functional groups in the EVD to collect information, observations of staff members and consulted reports and other data on the PSOM programme. In addition, essential documents and reports were studied and assessed. Those interviewed in the Netherlands are listed in Annex 5. The interviews conducted during the field study are mentioned in the country reports. All interviews were conducted confidentially and all interviewees cooperated with the evaluators on a voluntary basis. Some of the sensitive information obtained during the interviews has been used by the consultants in their assessment of the results, but this information has not been literally disclosed in the text in order to maintain confidentiality. In all, there were 26 interviews. The research method involved conducting semi-structured, explorative interviews with staff members. in order to gain insights into the workings of the EVD and to discuss findings and assessments with the staff on a number of key issues emerging from the interviews. The key findings have been discussed with PSOM/PSI management to verify their validity. However, it should be noted that an organisational assessment such as this cannot be solely based on reported figures and other “hard evidence”; instead, the assessment takes into account what EVD staff believes, what we have seen in the field, and what we consider logical and reasonable. The information available on the entire population of PSOM/PSI projects is kept in Excel sheets and the EVD management information system ‘BAS’. The financial and development impact data have been provided in an Excel sheet. The Excel sheet used by the EVD, which contains all the financial and project-related information, is formatted in such a way that the EVD cannot use it to produce pivot tables for analysis purposes. The consultants have adjusted the design of the sheet, enabling them to analyse the existing data for the purpose of this evaluation. For the 60 completed projects, the first batch of information was obtained by means of a data query from the EVD’s BAS softeware system. This query generated data on 51 of the 60 sampled projects. Furthermore, the consultants went though the paper files of the projects in order to complement the data query. By examining the EVD’s archives the consultants were able to retrieve paper files for 54 completed projects. For five projects the paper files were missing and for one project the paper file could not be obtained as it was in use for legal reasons at the time of the desk study

1. When studying the paper files,

the consultants found the applicants’ business plans, the defined results in the contracts between applicant and EVD, CLOM and embassy recommendations, and the final reports particularly helpful. It should be noted, however, that the available files differ considerably in terms of the amount and quality of information. In this respect we noted the following:

The files of the more recent projects (starting from 2003) are most complete.

Projects started before 2003 lack information on development impact which is understandable as the development impact that was expected at the time was not clearly defined and therefore not monitored.

Information in the files is not always consistent. For instance, information on proposed development indicators in the final report sometimes differs from data available in applicants’ business plans. Furthermore, within the BAS system, the development impact achieved also show inconsistencies as the numeric data reported in certain fields do not always match narrative descriptions elsewhere.

Overall, the files of the sampled projects from Surinam, Vietnam and Ethiopia are more complete than those from Indonesia, Ghana, and Mozambique. This is not surprising if you consider the starting dates of the projects: the selected projects from Surinam, Vietnam and Ethiopia all started in 2003 or later; the samples of Indonesia, Ghana and Mozambique also include a number of older projects.

1 The file which was in use by the legal department concerns PSOM05/GH/24. The five missing paper files are PSOM00/GH/3/2, PSOM99/GH/1/7, PSOM99/GH/2/7, PSI01/RI/1/5 and PSOM01/MZ/1/5. According to the EVD these files might have gone missing as a result of the transfer of files from Senter to the EVD. Fortunately, some information on these missing projects could be obtained by consulting annual CLOM qualification reports available in other files.


Given the limitations listed above, the data reported in this evaluation should be treated with some care. During the field visit, the consultants assessed the actual figures, but the triangulation of desk study data was difficult for older projects, because information in the file was unavailable or outdated.

1.3.4 The collection for the MMF research

MMF data are very complete and files are well organised, with easy access to products and information. The electronically kept data are stored in an Excel sheet, unfortunately in such a way that it was not possible to analyse the data. The consultants modified the format of the datasheet in order to analyse the data for the purpose of this evaluation.

1.4 The structure of the report

The structure of the report differs slightly from the proposed table of contents in the Inception report. It was found to be more logical to separate the assessment on the functioning of the EVD as implementer of PSOM/PSI from the findings and analysis of the completed projects. This way, the consultants were more able to follow the terms of reference. The next chapter reconstructs the policy development that has taken place on PSOM/PSI and it summarises the findings and conclusions of the earlier evaluation. Chapter 3 presents the findings of the evaluation, first those that relate to the EVD as implementing agency and then those that relate to the projects. Chapter 4 analyses the efficiency and effectiveness of the EVD followed by an analysis of the efficiency, effectiveness and relevance of the PSOM/PSI projects. Chapter 5 comprises the evaluation of MMF and in chapter 6 the consultants draw conclusions and identify issues for attention. Annexes 5 to 9 contain the six country reports each of which consists of a core report and data sheets on each project visited.


2 BACKGROUND TO THE PSOM/PSI PROGRAMME

2.1 Reconstruction of policy development for PSOM/PSI

The PSOM/PSI programme went through the following phases: Table 2: Phases of the PSOM/PSI programme 1999-2009

Phase and name Period Budget # countries

1st

phase PSOM 1998- 2001 €37.9 million 8

2nd

phase PSOM & PSI (Indonesia)

PSI: 2002 -2007 PSOM: 2003 -2006 plus 3 year to finalise projects = 7 years Budget ceiling of €30 million per year

PSI: €17 million PSOM: €112 million

2002: 11 2003: 17 end 2003 : 21

3rd

phase PSOM (extension of 2

nd phase)

2004- 2006 plus 3 years to finalise projects = 6 years

€45.9 million

42

4th

phase PSOM-EZ and PSOM-OS together since 2005

2007 -2010 plus 3 years to finalise projects = 6 years Budget ceiling of €55-60 million per year

€235 million 53

5th

phase PSI 2009 -2010 plus 3 years to finalise projects = 5 years Budget ceiling of €70 million per year

€140 million 51

Despite the fact that the PSOM/PSI programme is now in its fifth phase, some parameters have been quite stable over the years. Stable parameters include:

The percentage of subsidy for each project (now between 50% for developing countries and 60% for least development countries (LDC)/fragile states; was 66.7% and 80% in earlier phases)

The maximum amount per project (up to €898,000 for LDCs and €1.5 million for other countries).

The implementation cost percentage; around 7.5% of the committed budget. Payments have been made on disbursed funds since phase 4 (7.5% in phase 4 and 10% in phase 5). See also 3.2.2.

The Memorandum of Understanding (MoU) between the Government of Netherlands and of the recipient country. In the MoU, the priority sectors and regions are specified. Up to 2008 the MoU was necessary to comply with EU tender rules. Since 2009, the MoU has been signed to confirm the development cooperation aspect of PSI and to involve the embassies in the programme.

PSOM-financed goods and services are transferred to the local counterpart (the locally registered company) in the recipient country.

For each phase the Ministry of Foreign Affairs prepared a so-called ‘Beleidsmemorandum’ (BEMO), which describes the motivation for the budget allocation. These documents, together with the ‘Bepalingen en criteria’, are the main policy documents describing the expected effects and impact of the PSOM/PSI programme. According to the BEMO dated 28 August 1998 the objective of PSOM is to motivate Dutch companies to invest in innovative pilot projects or in lasting trade relationships in developing countries. The recipient company can be a 100% subsidiary of the Dutch company. The project should be considered to be additional, although the criteria for additionality were developed at a later stage. The three indicators of success (‘Bepalingen en criteria’, 21 August 1998) were:

1) Visible follow-up investments made after the pilot project by the created company or applicant firm without using government support

2) Creating effects in the entire chain of work and income generating activities in that sector 3) No negative impact on the poor, the environment or the position of women and, where possible,

positive environmental effects should be pursued.


These objectives and success factors have remained valid throughout the five phases. In 2003, a new policy guideline ‘Bepalingen en criteria’ was developed. The PSOM objectives mentioned in these guidelines are:

1) A PSOM project should lead to Dutch investments in the recipient country or to a sustainable trading relationship between a Dutch and a local firm.

2) PSOM projects should lead to follow-up investments by the concerned companies or to an upscaling of the trade relationship (without government support)

3) The PSOM projects as well as the follow-up investments should support pro-poor economic growth such as job creation, increase in income (employees, contract farmers and others), knowledge transfer and linkages with local (SME) companies.

The conditions for PSOM projects are:

No negative effects on women and environment. Projects with positive effects on women, environment or CSR should be prioritised.

To fit into the local environment: proven technology, no need for highly specialised management capacity or labour that is not available locally.

To contribute to knowledge transfer: a combination of technical assistance, knowledge transfer, training, market research and hardware should be proposed to obtain the subsidy.

To comply with policies on CSR (anti-corruption, social and environmental impact).

To be finalised within two years.

To be additional (according to ‘Ecorys method’) and experimental. Since 2003, some countries were ‘untied’, i.e. applicant firms no longer need to be Dutch companies and the payments to the applicant firms are made on the basis of results and not on the basis of invoices. Only the hardware costs are based on actual costs justified by invoices. Senter is responsible for making sure that the development criteria are part of the results that validate payments. In the fourth phase covered by the BEMO of 2007, the PSOM programme was enlarged following the positive outcome of the evaluation conducted in 2005. In order to make growth possible, the following measures were taken:

the programme became untied for an additional 19 countries

the number of countries was extended to 53

the matchmaking facility was introduced. An additional objective was added to the PSOM programme: ‘The PSOM programme experiences with investments are fed back to policy makers, thus providing an active contribution to improving the business climate’. The criteria for selection as described in the 2003 guidelines remained unchanged. Some quantitative targets for success were formulated:

Number of approved projects per year (targeting 110/year for 2007 up to 130 / year in 2010)

100 jobs/project

> 60% of projects successful

Higher income (no definition or further specification was provided)

Trained personnel (no definition of trained staff was provided)

Links with local SMEs. In 2008 PSOM was stopped, because of legal problems. The programme could no longer be implemented in the form of an assignment framework, because it had all the characteristics of a subsidy according to a court ruling. A new programme was set up: the Private Sector Investment programme (PSI). The BEMO covering this phase includes the same objectives and success factors as prior phases but also mentions strengthening of local private sector as a factor contributing to poverty alleviation. The MoUs with national counterpart organisations were to be maintained. New elements are:

The PSI programme is open for 51 countries with more focus on poor countries. The list includes 23 countries for which the PSI support is untied

Simplification of subsidy percentage and project size: 50% subsidy with a maximum project budget of €1.500,000 and for fragile states 60% subsidy, with a maximum project budget of €1,500,000


2.2 Findings on earlier evaluations (MTR and the evaluation in 2005)

In 2002, a mid-term review was conducted by Ecorys. A total of 32 projects were assessed, nine of which were completed. In 2005, an evaluation was conducted by Ecorys. This evaluation included an assessment based on 45 completed projects and 22 projects that had stopped pre-maturely. The summarised conclusions of these reviews were as follows: Table 3: Main findings and conclusions of the 2002 PSOM mid-term review

Indicator Mid-term review 2002 (actual and anticipated

results) Evaluation 2005

Average size of projects €824.000

Average PSOM subsidy spent per stopped project

€75.000

% of funds disbursed for stopped projects 14.4%

Own contribution project partners 30%-50%

Multiplier (private money including follow-up investments compared to PSOM subsidy)

1.5 1.46

Job creation/project 40-60 147

PSOM subsidy/job €10.000 --

Investment/job €9.218

People trained per project 71

Observed follow-up investments/project N/A €275,000 (54% of PSOM contribution)

Additionality 41% not additional

Sector effects No demonstration of technology

Development effects Mainly agricultural sector Mainly agricultural sector

After the mid-term review in 2002, the EVD introduced the monitoring of impacts (employment, knowledge transfer and outgrowers). For each project, these indicators have since then been included in the proposal and monitoring efforts throughout the project cycle and spin-off phase. From 2003 onwards, screening of additionality was also enhanced. Ecorys used the following criteria for additionality:

commercial viability (low)

payback period (long)

bankable (not). In addition, complaints from (Dutch) entrepreneurs about market distortion and unfair competition led to stricter screening of additionality and market distortion. The overall conclusions of the 2005 report were positive: PSOM projects scored ‘fair’ on efficiency and effectiveness and ‘good’ on relevance.

The efficiency was fair because even though projects were mostly completed within the proposed budget and time frames, the budget was underutilised, which had a negative effect on efficiency.

The effectiveness was fair because the cost per job created was low, but the follow-up investments fell short compared to the expected investments.

The relevance was good (3.7 on a scale of 5) because technology transfer, income effects and indirect employment are important development impacts that were observed.

The evaluation suggested that the following issues should receive further attention:

The expansion of the PSOM budget will result in an absorption challenge or underutilisation of budget

The financing gap limits companies to grow the projects beyond the PSOM project phase

The need to improve the meagre quality of financial projections in the PSOM proposals

To open the possibility for replicating PSOM innovation in other regions within the same country


To enhance the contribution of PSOM in business climate dialogues

The importance of the business partners’ commitment to developing the project. The Minister reacted to the 2005 evaluation in a letter to the Dutch Parliament dated 3 August 2006. She stated that interest in PSOM was increasing and that it was a good decision that PSOM had been enlarged in terms of budget and number of countries eligible for PSOM. The letter also mentions that the mid-term review (MTR) in 2002 found that the evaluated projects would not have been completed without the PSOM subsidy. This is strange because the additionality was found to be low in the MTR (59% of projects were found to be additional) and as a consequence the screening of this point improved in the years following the MTR. The 2005 evaluation did not assess the additionality of the PSOM projects. The Minister also mentioned that the evaluation team overlooked the fact that PSOM projects feed the EVD and embassies in their dialogue with the World Bank and the local government on business climate issues. However, the evaluation reports mentions (on page 38) that PSOM projects have not played a major role in enhancing the business climate and stimulating the private sector. The evaluation report continues to say that the embassies are not using PSOM in their dialogue with national governments. In a letter to parliament the Minister proposed to increase the budget for PSOM and to increase the number of eligible countries. As the structural underutilisation of PSOM budgets is an issue, the following accompanying measures were proposed:

the MMF was introduced to help entrepreneurs in developing countries find a business partner in the Netherlands. This partnership could lead to a PSOM proposal

the programme was untied for more countries, i.e. non-Dutch companies became eligible as applicant firms

the number of countries was extended to 53 countries. Within the EVD the following changes were implemented after the 2005 evaluation:

More emphasis on the screening of the partners involved in the venture. The main reason for failure of projects was problems with one (or more) of the partners in the projects. Weak financial performance, wrong expectations, miscommunication etc. could be avoided if the partners have already worked together and know each other.

More attention to the financial strength and contribution of project partners and greater potential for follow-up investments


3 FINDINGS ON THE PSOM/PSI PROGRAMME

3.1 The EVD as implementing agent for PSOM, its organisation and costs

In the following section we briefly describe the EVD as the organisation responsible for implementing the PSOM programme, and its management structure. We then look at the implementation costs over the period 1999-2009, discuss the way budgets were used and analyse key trends.

3.1.1 Organisational embedding of the PSOM/PSI programme

Senter Internationaal, as a part of Senter, originally managed the implementation of the PSOM/PSI programme. In 2003, Senter Internationaal, with the PSOM programme, merged with the EVD, which in turn has recently become Agentschap NL. Currently, the PSOM programme is hosted in the “To Develop” division of Agentschap NL, in which four units and 15 programmes of the Ministry of Foreign Affairs are implemented. The PSI unit is currently managed by a unit manager (UM), who oversees four clusters, one for each continent the programme is active in. Each unit is managed by a cluster leader (CL), who oversees several (Senior) Project Advisors, who carry out the main work on project selection (ontwikkeling) and management (beheer). The PAs are supported by financial administrators (FEMs), project assistants (PMs) and secretarial staff (ASMs). In addition, the PSI unit draws on resources within Agentschap NL (and previously the EVD) for human resources, internal control, legal services and communication support services. Currently, the PSI unit has 43 staff members (with 33.6 FTE working on PSOM/PSI), who handle a portfolio of 275 projects in 52 countries (end of 2009); in 2006, 28 people were employed in the PSOM-OS unit (220 projects in 52 countries open for PSOM), and in 1999, only nine staff members were employed with a portfolio of only 23 starting projects in eight countries. In addition to the PSOM-OS programme, the current staff manages several smaller facilities, which are, however, not subject of this evaluation.

3.1.2 Management and organisation

The PSOM unit manager’s tasks – besides general management – are mainly to ensure that the PSOM subsidy is spent in a legally correct way, and “to generate development impact through passionate entrepreneurs”. According to the UM the most important value added by the staff is to select qualifying and promising proposals that generate employment and economic growth in the PSI countries. More practical, the main challenge is to ensure that the budgets are disbursed correctly, and on time; that tender procedures are implemented within the time and instructions given, and that internal procedures and systems are modernised for better efficiency and effectiveness. The CLs have a diverse set of tasks, alongside general management of their unit, or cluster. Although the tasks are not the same for each CL, and the estimated time for each task differs as well, the tasks generally include:

Quality assurance during the selection procedure; i.e. co-appraisal with the PAs, which is done to ensure consistency, objectivity and the institutional memory is utilised.

Quality assurance for managing projects; largely consisting of signing all payment orders for projects; PAs are not allowed to sign and effect these, in line with standard procedures in governmental organisations, everything has to be counter-read and signed by CLs (and for higher amounts by UM, or DT), in addition to the signatures from IC (for amounts over €100,000).

“Trouble shooting” is another task that CLs naturally have, which includes managing staff, and any problems that may arise in relation to staff.


In addition to the above tasks (which to a certain extent relate to all CLs), each CL has specific, PSOM/PSI-wide tasks, such as overseeing the overall tender procedure, implementing changes to the tender instructions, and all related documentation (such as that used more recently for CSR), being responsible for other smaller facilities which the PSOM/PSI manages, or develops, and general management tasks.

In the interviews, no fundamental objections to the current organisation have emerged; however, some respondent have voiced concerns about the following topics:

“Long signature lines”; i.e. it would be more efficient to have less people involved in counter-checking payments or letters where additional checking by CLs adds relatively little value.

Co-reading tender applications by CLs is appreciated, but this could also be done by the more senior PAs. Three of the four CLs have long-standing experience within PSOM, but SPAs also have sector-specific knowledge and experiences to offer.

Several PAs feel that they have to spend a great deal of time on work for which they are overqualified, such as secretarial work and other administrative activities.

3.1.3 Internal learning

As mentioned above, the PSOM programme was evaluated in 2002 and 2005, and several conclusions were drawn along with issues for attention. The EVD has reacted to these findings on the following issues:

Partner screening: since 2005, the screening of partners for their financial and technical/organisational ability has been intensified

Financial projections have been refined and improved

Local partners have been concluded to be the main cause of stopped projects, and the policies with respect to local ownerships have been adapted to allow for applicant-owned local firms, if desired.

In addition, we have observed that the EVD produces and continues to up-date manuals for developing and managing projects, which are by now very detailed and contain the EVD’s experience on how to deal with all kinds of situations that have occurred and been dealt with in the past. Overall, we noticed that most learning is embodied in the experiences that the staff members have accumulated over the years. The informal culture of exchange ensures that learning from problems, and other features of PSOM projects, is applied in practice. On a more critical note, there is not much portfolio analysis done with the rich database of projects the EVD has accumulated, although this may not be crucial for internal learning. The issues that evaluations have pointed out, but have not led to much action are follow-up finance for PSOM projects (see below), and enhancing the contribution of PSOM in business climate dialogues.

3.1.4 Overall implementation cost

The norm agreed by EVD and DDE for implementation costs is 5% of the total budget per year and 6% of the (projected) programme cost (over the expected implementation period of four years). This norm has been maintained in the quotations over the period, except for in the early years, when the implementation cost included high start-up cost vis-à-vis a relative small portfolio. It should be noted, however, that this norm is based on expected annual implementation cost (i.e. usually very close to the actual cost), compared to the multi-annual expected programme cost, which typically comes in 20-25% lower than expected. The reason for this is that a number of projects stop during implementation due to the high-risk nature of doing business in developing countries (this is described in more detail below). This typically reduces the amount of funding actually put to use by around 20-25%. In retrospect, the actually disbursed programme costs of €151 million required €26.1 million of implementation cost (17%) during the period from 1999-2009.


As an appraisal of efficiency, we have compared the actual (i.e. achieved) programme cost per year to the actual implementation cost per year (see Table 1 below). The individual values do not necessarily explain increases or decreases in efficiency, however: since most of the cost relates to developing new projects, a growing implementation cost budget should lead to higher programme costs in subsequent years. Therefore, the trend in costs is the indicator to consider. Table 4: Disbursed programme cost compared to achieved implementation cost, 1999-2009 (in €1000s)

Year/ X €1000

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Total

Programme cost (achieved)

- 2,110 5,248 10,047 10,996 7,931 17,156 24,982 23,691 24,547 24,416 151,125

Implementation cost (achieved)

519 592 1,060 1,120 1,674 2,041 2,558 3,122 4,027 4,524 5,022 26,402

Applications received

115 126 180 77 126 243 127 158 208 95 213 1,668

Source: Annual reports (Senter, EVD) to DDE 1999-2009 In the course of the ten-year period from 1999-2009 we detect two trends:

The first trend covers the period until 2004 when PSOM was still a “small” programme, after which both the budget and the number of partner countries increased significantly. Implementation costs increased and the costs for completed programmes increased with a time delay.

From 2005 on, the disbursement for programme costs grew to a stable figure of €24 million, while the trend for the implementation costs also moved upward (overall by 41%).

One reason for this was the cancelled tender in 2008 (due to external reasons), which led to higher costs for lower numbers of developed projects in 2008 and fewer projects to manage in 2009. However, there was no strong “catch-up” effect in 2009, when only five more applications were received and processed than in 2007 (213 vs. 208). The EVD considers that the financial crisis has had a major impact on both applications and the costs of completed programmes in 2009; so the number of applications could be seen as good, under these circumstances.

To net out the components of the implementation cost increase of the PSOM-OS programme we have made an analysis to isolate the cost increase factors from each other. Overall, the cost increases in absolute terms (2005-2009) stem from three separate factors:

1) the increase in daily fees charged by the EVD for its services 2) increases in volume of the programme (more applications, contracts and project portfolio), and 3) increases in several overhead costs (general, communication, out-of-pocket expenditures (OOP)

and legal costs). As an illustration, we have compared two similar years, namely 2006 and 2009

2; in both years a similar

level of programme cost was spent, in a similar number of countries (open for PSOM), and the EVD norm for project development and project management has remained constant. Of course, the project portfolio is larger in 2009 than in 2006: 55 more projects to manage, and 18 more projects contracted. However, the implementation costs have also risen by about €1.9 million (38%). In the table below we analyse where the cost increase in the implementation cost came from and we present the main contributors at the bottom of the table.

2 The same test has been carried out comparing 2006 and 2007, and 2007 and 2009; the results are not that different, nor do they show better efficiency. 2008 has not been used (and hence not the whole data series from 2006 to 2009) because in 2008 a tender was cancelled due to external reasons, which distorts the picture significantly.


Table 5: Comparison of PSOM/PSI programme implementation costs 2006 and 2009 in €

Item 2006 2009

Commitments (realised) 45,056,646 67,410,067

Programme cost (realised) 24,981,553 24,416,363

Implementation cost (realised) 3,122,451 5,021,723

Project portfolio

Projects under management 220 275

Applications processed 158 213

Projects developed 83 101

No. of countries (open) 53 52

Variable cost (€)

Development 1,094,435 1,796,073

Management 925,072 1,468,537

Norms (days)

Development 18.5 18.5

Management 13.0 12.9

Overhead (€)

General (basisinspanning), including €107,497 for legal overhead in 2009 280,916 627,952

Out-of-pocket expenditures (total) 664,621 698,079

Communication (basis+ part of OOP expenses) 463,706 381,574

Legal (direct cost, part of OOP expenses) 18,122 153,367

Cost drivers, EVD implementation cost 2006-2009 Cost increases

% of total increase

Absolute difference in implementation cost (minus OOP differences) 1,947,122 100%

Overhead (basisinspanning and communication cost increase; of which 61% due to direct and indirect legal cost)

507,118 26%

Increased workload 776,488 40%

Unit of cost increase (fees and volumes changes of fee categories) 663,516 34%

Source: EVD Annual reports 2006, 2009 Note that we have calculated the absolute difference in implementation cost, leaving out differences in OOP expenditure, with the exception of the overhead items communication (voorlichting) and legal cost. This allows a fair comparison of overheads and variable cost between the years. Applying the same calculation to 2005, 2006, and 2007 yields a similar result for the three years. The fee increases and overhead increases explain 77% (2005), 60% (2006) and 80% (2007) of the cost increase compared to 2009. The additional workload accounts for the smaller part, namely 23% (2005), 40% (2006) and 20% (2007). Note that 2008 has not been taken into account, since the second tender was cancelled for external reasons and would significantly distort the results.

Table 6: Cost drivers, compared to EVD implementation cost in 2009

Item 2005 2006 2007

Absolute increase in implementation cost, compared to 2009, in € (excl. OOP differences)

2,223,587 1,947,122 1,468,995

Due to: € in % of total

increase € in % of total

increase € in % of total

increase

Additional workload 506,900 23% 776,488 40% 290,110 20%

Overhead increases 729,560 33% 429,256 22% 456,784 31%

Fee increases 987,127 44% 741,378 38% 722,101 49%


From above table we can conclude that during the whole period from 2005 to 2009, 44% of the cost increases are the result of fee increases, 33% due to overhead increases, and 23% are the result of additional workload (caused by more projects carried out and managed per year). To analyse this pattern more closely, we look at the comparison of 2006 to 2009 in more detail:

Overheads One noticeable factor is the increase in general overhead (basisinspanning), which has more than doubled between 2006 and 2009. This increase is due to a strong increase in legal overhead cost (which comes in addition to the direct legal expenditure), which grew from nil to €107,497 in 2009. Overall, increased legal cost explains 61% of the total (budgeted) overhead variation (incl. direct legal expenses). In all, the variation in (budgeted) overhead expenses explains 22% of the total cost increase. EVD explained in the annual report that the general overhead cost was over average because of time spent in 2008 and 2009 on transforming PSOM into PSI (new guidelines and both internal and external documents, architecture of database and legal advice). Overall, total overhead cost (i.e. including legal costs, communication and information workshops) as a percentage of the total EVD budget for implementation is now higher than in the first year (1999). Although there was no consistent increase over the years, the overhead percentage in the last five years was higher in the last five years than in the first five years. Note that there is also a significant amount of overheads encapsulated in the fees (which are both in the general overhead and in the development and management budgets); this overhead comes in addition to the percentages given below. Table 7: Overhead cost as a percentage of total implementation cost, 1999-2009 (in €1000s)

Year/ x 1000

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Total overhead 172.1 243.1 150.2 343.3 670.2 596.7 746.9 1,196.1 1,321.5 2,001.5 1,861

As % of total cost

33% 41% 14% 31% 40% 29% 29% 38% 33% 44% 37%

Workload

Another natural cost increase is the consequence of a larger portfolio in 2009; more projects were carried out and managed. To assess the effect of this increased workload, we divided the total development cost by the number of developed projects and applied the cost per developed project in 2006 to the (higher) number of developed projects in 2009. The same was done with the number of projects under management. This describes the increase in cost exclusively due to the larger portfolio (and not the increases in EVD cost), the volume of work effect. This effect accounts for 40% of the total cost variation.

Fee

Adding the overhead variation and the workload variation therefore explains 62% of the cost increase, while the remainder is (necessarily) due to increases in the unit cost of the EVD, i.e. the daily fee rate variation per category, and changes in the volume of days in each fee category. This increase contributes 38% to the overall cost increase. Overall, 7.98% (-points) of the 38% increase in the total implementation cost between 2006 and 2009 is explained by an increase in programme overheads, and by 14.44% (-points) in EVD unit cost increase. If we subtract the effect of increased legal cost (which is largely outside the control of EVD), the overall increase was about 17.6% (-points) due to increases in overhead cost. In other words, the EVD’s cost has increased by about 17.6% within three years, net of changes in the workload. That is on average about 6% per year, approximately half of which could be explained by inflation and wage increases, half by the increasing cost of overhead for the EVD and the PSOM programme. Therefore, you could say that without the overhead increases at the EVD, the overall cost of EVD in 2009 would be lower by about 9%, or by approximately €450,000

3.

3 Note that the absolute amount of the fees as charged by the EVD were not the subject of this research; therefore, cost efficiency here is only assessed on the basis of year-to-year changes (i.e. it is relative, rather than absolute).


3.2 Findings on the work of the EVD: selection and monitoring and supporting services

The EVD carries out its task essentially by providing two services: one is selecting new projects (this is called development or ontwikkeling), while the other is managing and monitoring these projects (called management or beheer). In support of these two core business processes, the EVD employs a range of support services, such as communication and marketing, legal services, and human resources. Finally, the EVD budgets for out-of-pocket expenses (for travel, external consultants) to carry out its activities. Below we discuss the EVD’s key services, selection, management and supporting services in order to arrive at an assessment of the efficiency and effectiveness of the EVD service in the following chapter.

3.2.1 Selection

Since 1999, the methods, processes and tender instructions have changed nearly every year, to accommodate new countries in the portfolio, incorporate new topics (such as CSR) and develop new tools to improve the processing of applications. Below we discuss the main aspects of the selection process, which is called “development” in EVD terms.

3.2.2 Selection results: commitment and utilisation of the budget

PSOM-OS has been operational since 1999, in five different phases; the total maximum budget available for commitment to projects was €391 million (until 2009). The EVD managed to commit 85% of that amount to projects, while 15% remained unused. The trends in the utilisation of budgets are shown in the table below: after a high commitment in the first phase, the subsequent budget expansions were not fully absorbed, and are significantly under budget. The most recent tender, however, has revealed a higher commitment level; this might be due to some deferred applications from the cancelled second 2008 tender. Table 8: Budget utilisation per PSOM phase, 1999-2009

PSOM-OS programme phase

No. of projects

o/w ongoing

Budgets available (until 2009)

Committed Utilisation

(commitments)

Total PSOM 1999-2002 135 - €40,100,000 €51,548,417 129%

Total PSOM 2003-2006 281 60 €174,900,000 €134,632,595 77%

Total PSOM 2007-2010 138 113 €106,000,000 €77,570,704 73%

Total PSI/PSI Plus 2009-2010 102 102 €70,000,000 €67,410,067 96%

Total PSOM-OS 1999-2009 656 275 €391,000,000 €331,161,783 85%

The utilisation (or realisation) of committed budgets describes how many euros committed were actually spent on projects. Up to end of 2009, €208 million were “achieved”, which means either “spent” (€152 million) or “withdrawn” (€56 million). Withdrawn means that, if a project stops, the remainder of the budget is returned, and not paid out; in addition to that, completed projects have not always received, or even requested, the full amount (in total about €2.5 million of the total withdrawn). In the table below we can see that the utilisation percentage differs per phase (see the last column, which describes the actual realisation, or the maximum possible, assuming all open commitments were spent). This has to do with the fact that for all phases, except PSOM 1999-2002, there are still ongoing projects. Therefore, the percentages will decrease due to prematurely stopped projects and unspent budgets in ongoing projects.


Table 9: Realisation of committed budgets, 1999-2009

PSOM/PSI phase Achieved Withdrawals Received

(end 2009)

Open commitment (end 2009)

(Max. possible) Utilisation, in % of committed

budgets

Total PSOM 1999-2002 €37,932,060 €13,616,356 €529,921 - 74%

Total PSOM 2003-2006 €93,535,220 €28,630,299 €10,804 €12,467,076 79%

Total PSOM 2007-2010 €20,039,984 €14,014,989 - €43,515,731 82%

Total PSI/PSI+ 2009-2010 €399,514 - - €67,010,553 (100%)

Total PSOM-OS 1999-2009 €151,906,858 €56,261,644 €540,725 €122,993,360

The table below provides an overview of the achieved results, in terms of applications received/processed, contracts awarded and resources spent (here, only the cost specifically for development has been noted, exclusive of overheads). Table 10: Selection PSOM-OS projects, 1999-2009

Year 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008* 2009 Tot.

Applications received

115 126 180 77 126 243 127 158 208 95 213 1,668

Projects developed4

26 23 34 20 31 78 65 83 88 50 102 600

Approval ratio 23% 18% 19% 26% 25% 32% 51% 53% 42% 53% 48% 36%

Development (total cost, p.a.)

291,315 128,414 144,318 250,012 448,556 776,521 1,014,920 1,094,435 1,428,811 1,017,694 1,796,073

Development costs per contract, in €

11,204 5,583 4,245 12,501 14,470 9,955 15,614 13,186 16,236 20,354 17,609

*) Note that in 2008 one of the two tenders was cancelled, explaining the drop in applications and contracts, and cost increase per contract.

We see the following trends:

1) The approval rate has significantly increased; until 2004, the average approval rate (applications

to contracts) was 24%, while from 2005-2009 the ratio was around 50%. Note, however, that in 1999-2002 a prequalification system was applied: companies only needed to submit a brief concept note to qualify (=low threshold), and only then developed a full plan (for a modest payment). If we include the pre-qualified projects in the projects that were developed, the average approval rate in the period 1999-2004 was 30%. Besides the change in the application procedure in 2002, there are two possible explanations for the increased approval rate. One is that the applicants have become more proficient, and that experience of involved consultants, and EVD information workshops and other communication (intake services offered by EVD used by 62% of the applicants) have resulted in a “better population” of applicants and improved quality of proposals. The other possibility would be that because budgets have grown and applications have not, more projects were approved to compensate for this lower application number per available budget.

4 Projects developed are the projects approved (656) minus the projects that did not pass the pre qualification phase (56).


2) The cost of selection (per contract) has grown over the years. In the same period the “norm” for

the number of days that the EVD spends on development dropped from 20.5 to 18.5 days per contract; therefore the cost increases are purely the result of fee increases. Note that this only relates to the direct cost of development; in addition, there is a proportion of the general overhead and other development-related costs.

3.2.3 Selection method and process

The EVD organises two tenders per year, each taking 18/19 weeks to complete (i.e. within 18/19 weeks of the closure date for preparing a contract). Project advisers uniformly stated that this activity is very intensive and requires almost all of their time. Management tasks in the tender period are reduced to a minimum and caught up with after the tender period is over. This of course varies per PA: some have large country portfolio, with many applications, others have less. The selection method includes three major subjects:

1) Appraisal of the applicant: Here the financial strength of the company is scrutinised in order to assure that (a) the company is a serious business and committed to the project, (b) has the resources to fund its part of the project, and (c) is not in a perilous financial state. This is the result of a learning process, which showed that many projects run aground if the applicant firm is not committed and not financially stable, or strong enough. EVD maintains strict ratios as criteria for the selection, which are threshold criteria for approval. Most of the applicant assessment is done by financial staff, and based on audited year accounts (which are, however, often outdated, since they are usually not available for the past year). If there are doubts, guarantees, or letters of intent from banks are demanded. PAs visit and talk to the company management to ensure that there is full commitment and a realistic understanding of the risks, as well as experience with the proposed business activity.

2) Appraisal of the business plan: FEM and PAs jointly assess the business plan in terms of its commercial viability, soundness of concept, and risks and operational viability. The financial analysis is done by FEMs, using internal benchmarks and methods to assess whether or not the commercial viability is strong enough, but not too high to be eligible for subsidy. PAs check the content of the plan; here, most of the appraisal relates to checking the inner consistency of the business plan, comparing it to other similar proposals in the PSOM archives, discussing with sector-experienced PA colleagues, and – as a final resort – requesting external expertise. The result is a ratio sheet on business ratios and the financial standing of the applicant. In addition, a market analysis and a financial assessment are made.

3) Assessment of development relevance: PAs assess the proposal for a range of criteria such as long-term economic activity (i.e. sustainability and expansion of the investment), employment effects, chain effects, knowledge transfer, environmental effects, CSR, and gender. There are no minimum levels for e.g. employment creation or knowledge transfer. However, a ranking system has been developed which ranks project according to their developmental relevance. This ranking has been in place since 2007. Since the total value of the applications that are suitable (in terms of the business plan and the financial standing of the applicant, and generally meeting the development criteria) has always been lower than the maximum budget available, the ranking was never needed.


The selection process consists of the following steps: Table 11: Key steps in PSOM-OS selection process

Step Description of Activity Responsible

staff member

Working hours,

estimate by respondents

1 Completeness check; all received documents are reviewed to check that all administrative and other documentation is in the dossier

PA 1

2 Preparing a letter to confirm receipt, or that documents are missing PA 0.5

3 Second completeness check; where required PA 1

4 Reading of project proposals; checking of content with internal sources, checking of consistency

PA and CL 8+4

5 Financial appraisal; checking financial figures in the business plan, cash flow analysis, and financial position of applicant, ratio sheet applicant firm, requesting additional data

FEM 6

6 Internal appraisal form PA, FEM 1.5+1

7 Verification; if required use of internal/external expertise, planning of visit to applicant (+/- in 80% of the cases), and the recipient (nearly all the time)

PA 4

8 Visit to applicant (NL), incl. travel time PA (FEM) 6

9 Visit to recipient (in the course of the selection mission carried out), incl. travel time

PA 12

10 Reporting PA 2

11 Reassessment in team meeting PA, CL, FEM 0.5+0.5+0.5

12 Additional discussions/questions to applicant, recipient (25% of the cases)

PA 2

13 Preparation of the APSOM/APSI report, content (PA) and financial part (FEM), checked by CL

PA, FEM, CL 6+4+2

14 Presentation to APSOM/APSI PA 0.5

15 Contract preparation, incl. payment schedule PA, FEM 3+3

The above summary of the process is of course the “ideal” case, in which there are no issues, problems, or additional activities, such as using external advice, and the like. An unusual case can easily absorb 50% or more time, and this is typically the case with roughly a quarter or a third of the applications. PAs estimate that their workload is roughly 50-60% related to developing projects (some say it is an “almost continuous activity”), another 30% to management, the remainder for internal and miscellaneous matters. This is not surprising: the above table based on estimates of the interviewees suggest that the contracting would be done in 69 hours, or about 8.5 days, while the norm is 18.5 (which also includes 3.5 days for ASM, PM and IC). This is of course an estimate based on a few interviews. In reality, it seems – as several PAs have stated – a great deal of time is spent on other, internal matters related to the selection process. Also, the appraisal process relies still much on non-digital information, slowing down the preparation of analyses. The methods have clearly been refined over the years; as a measure of that, the final APSOM/APSI advice contains 43 fields in 2009, as opposed to only 18 in 2005 (the volume also increased, from 3 to now 7-8 pages). More attention is now also paid to the development aspects than before and new aspects such as CSR are addressed as well. However, in essence the approach and result are very similar. To some PAs, the length of the form, and the effort required, is not in line with the added value.

Role of APSOM

The final step, approval by the APSOM committee, is mostly a formality in the view of the PAs interviewed. The main function of the APSOM is – in their own words – to assure that the EVD has “done its homework properly”. In other words, the value is that an external body takes another close look at what has been decided, and scrutinises various crucial aspects when interviewing the PA responsible. The APSOM members only review the final document submitted, which is a summary of all the documents submitted by the applicant. On the basis of this document alone, an independent assessment would be difficult to make. In general, the APSOM committee challenges one or two cases per year. In our desk review the


CLOM/APSOM advice were available for 49 projects. In all available cases the APSOM advice was either positive (48 cases) or neutral (one case). However, the Unit Manager, who acts as the secretary to the APSOM/APSI, highly values the advice from the APSOM/APSI. Both in terms of the depth and wealth of the discussions based on the banking, industry and developmental experience of the individual members combined as well as in terms of a final check on the assessment by EVD. In many instances additional conditions are set for the administrative decisions.

Approval rates

By rule of thumb, PAs, CLs, and PSOM/PSI management estimates that of a given amount of applications, around 20-25% do not meet one or more of the required “binding” criteria (such as being incorporated for at least two years, or positive equity), or cannot provide requested documentation, and are dismissed in the beginning of the process. Another 20-25% are administratively compliant and meet the criteria but propose a business concept that that has substantial issues that may prevent PSOM/PSI to fund the project. The remainder, 50-60% are very likely to be granted a subsidy. We note here, that EVD does not seem to have much “choice”; as mentioned above, the typical application-to-contract ratio was around 50% in the last five years. Development criteria are scrutinised to a lesser extent, judging from a selection of APSOM advice over the years; as mentioned there are no minimum levels (e.g. of employment or chain effects) but mostly non-binding criteria (such as “must be innovative”, or “must have a CSR policy”). An important aspect is whether the subsidy would not be market-distorting, all the more so since competitors can appeal against the granting of a subsidy. To test whether or not the project is “additional” (this word does not appear on the APSOM form), innovative aspects and risks are described. In the current definition, a project is additional if there is no commercial finance available the new business is innovative (which assumes there will be no market distortion), and is a risk-bearing undertaking. According to an agreement with DDE, multiple applicants are accepted, as long as it is not the same project in the same country, and the number/volume of multiple applicants is lower than 66%, or two thirds. EVD usually reports on the number of multiple applicants in most annual reports to DDE. The picture is that typically 10-12% of the applicants per tender “have not been involved with another PSO/M project before”. In addition, there are also multiple recipients who we have noted in the field visits. The number of these, as well as how many projects these multiple applicants and recipients have received (two, three, or more), is not reported in the annual report, and thus not known.

The use of consultants

On average, 33% of the project budget is spent on TA. The TA components consist of two essential elements: project management and technical training courses and related activities. The ratio of these two components is not known. In the sample of 60 projects we found that 32 projects have made use of consultants. From the visited projects we have understood that the Dutch consultants often take care of the progress reporting and administrative matters of the project, while technical training courses and the like are carried out by industry specialists or local consultants, depending on the task. Typically, the consultant providing the project management services writes the business plan or assists significantly in the application preparation, in return for a share in the TA budget. On average 10% of the subsidy is used as fee for these services, although this amount is not as such specified in the project budget. These costs could be seen as additional implementation cost (inachtnemingskosten), in addition to the overall implementation cost. If the sample (n=60) is representative, a total of approximately 5% of the overall committed budget would need to be added to the EVD implementation cost to arrive at the full cost of implementation. A customer satisfaction survey carried out in 2009 emphasised that many of the applicants feel they need a consultant to successfully apply for a project. Applicants feel obliged to use consultants, because the procedure is not clear to them. Many applicants mentioned they would not do so if the application procedure was more understandable. Applicants feel that being forced to use consultants is an obstacle to apply. Another finding is that the use of consultants might be an indication of reduced additionality because in some visited cases the consultants approached entrepreneurs who were already planning to

jager.hans

Cross-Out


start a new venture. In these cases, the PSOM subsidy was a welcome additional source of capital but it was not a precondition for starting the venture.

PSOM management and project advisors have a hesitant view towards the use of consultants (regarding the project management and acquisition services); on the one hand, consultants help with identification of new projects, write proper proposals, make correct requests for payments and deliver usable progress reports. This increases the pipeline and approval rate, and makes management less time consuming. On the other hand, the EVD feels that consultants add to cost at the expense of the recipient, may develop projects for the sake of consultancy work, and pre-select or filter information about applicants, the business plans and development effects in such a way that an objective appraisal by EVD is more difficult (EVD is “forced to approve”). The fact is, however, that consultants have been used widely in the PSOM programme. Some interviewed applicants were critical about the role of consultants: too much of the content in the proposals comes from the consultant and is not really ‘owned’ by the JV partners. This was confirmed during the field visits, where some of the proposed development effects (such as working with outgrowers) were never taken seriously by the JV partners. The consultant included these elements in the proposal to ensure a higher ‘hit rate’ at the EVD. Finally, remarks were made on the fact that some of the consultants work more on behalf of the applicant (instead of the JV) because they are paid by the applicant who is the recipient of the subsidy. This had a negative impact on some JV relationships.

3.2.4 Management (or monitoring of project portfolio)

In addition to selecting projects, EVD manages and monitors ongoing projects (this is called beheer); below we describe the key findings of this service.

Time and budget allocation for management

The key management issue is to ensure the correct, but also timely disbursement of funds; as the tables above show, the realisation (i.e. actual disbursement) has flattened out in the last four years at a level of around €24 million, despite increasing commitments. The main reason for this is the delays which most projects incur; while projects are planned to last 2½ years (or 3 years if in agriculture), they frequently require a year or more extra time. Of the projects that were approved in 2005, 23% were still ongoing at the end of 2009. Also, the main expenditure is hardware (around 65% of the subsidy/budget), which can only be spent when all licenses, partner discussions, premises and other matters are resolved. The latter is, according to most PAs, the main cause of delays in expenditure. There is, however, no evidence that longer project duration delivers better or worse results. In recent years, PSOM management has increasingly focused on keeping track of the applicants and the payment schedules that were originally agreed. PAs have to report to management whenever a projected result (disbursement) needed to be delivered, but was delayed. There is also an elaborate manual for project management, which describes what the standards are for all staff involved and how frequently problems can occur, or need to be resolved. Overall, the costs of managing projects have fallen in relative terms and stayed more or less stable in absolute terms (considering normal cost increases) in the period 1999-2009. Management costs have become smaller relative to development costs, as the ratio shows. In 2009 management costs were only 30% of the development cost (only direct cost, excluding out-of-pocket expenditure and overheads), while in the years before 2005 management costs were between 44% and 68%. The portfolio has increased over the years, from 31 to 275 projects under management. However, we note that the management cost is paid annually; therefore, if a project is delayed by a year, (total) cost per project goes up. We have no information on the size of the management cost per completed project to assess whether the cost has indeed decreased or not.


Table 12: Management cost per project per year for the period 1999-2009

Item 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Management cost per year (total, x €1000)

220 111 579 612 718 809 925 1,234 1,623 1,469

Projects under management

31 57 89 69 109 178 220 258 238 275

Management cost, per project, €

7,111 1,955 6,508 8,873 6,583 4,546 4,205 4,785 6,819 5,340

Cost ratio, mgt to devmt

44% 54% 68% 59% 58% 30% 32% 29% 34% 30%

Note that the above numbers on projects under management is based on the reporting in the annual reports, which uses projects under management at the end of the reporting period, not the average number of projects under management. However, for the trend the relative numbers are of importance, not the absolute. The “norm” for management per project per year has remained nearly constant; from 13.5 days to 12.9 days per year per project.

Procurement

About two-thirds of the PSOM subsidy consist of “hardware” (machinery, buildings) and one-third of “software”, or TA in EVD terminology. To check whether the hardware is not over-priced, each hardware item over €25,000 requires a “declaration of market conformity” from a valuation company. In addition, the responsible PA conducts a physical inspection on site to verify that the goods are actually delivered and installed according to the agreement. We have noted during the field visits that this indeed takes place in all projects. TA expenditure is generally proposed in the business plans/applications and negotiated during the conclusion of the contract; where fee rates are excessive, and underlying support documentation is not convincing, reasonable prices are agreed with the applicant.

Monitoring results

Since 2003, the EVD concludes a contract with the applicants in which mile stones, with clearly defined results, are agreed. Payments are made on the basis of achieved results. Therefore, apart from procurements, the PAs’ task is to verify that the results are achieved, and payments can be made. For this there are predominantly two sources of information for the PA:

1) The progress reports; in which the sub-results are reported and the documentation for the defined means of verification is delivered to the EVD

2) Visits to the projects and the applicants. The EVD has produced a standard form in which the milestones and means of verification are recorded and agreed; these cover not more than five milestones, with several sub-results (which are subject to negotiation). These include:

Copies of legal documents and financial commitments (bank guarantees etc) of the project partners in the beginning of the project

Various plans for engineering, training programmes, building permits and the like

Protocols of receipt confirming that hardware has arrived and supporting materials that it is operational

Depending on the negotiation with the applicant, and the possibilities of a company, certifications for work safety and e.g. organic production are requested

Realisation of production/sales results and employment figures is also monitored, by means of copies of contract, payrolls, and other recipient company materials.

The final 20% of the subsidy are only paid out by the EVD if the hardware is delivered and verified, and the sales or production targets (‘results’) have been achieved. The reports delivered on sales, production and employment result are no longer based on audited company accounts or accountants declarations, as was the case until 2003. Until then, these report were required because of the disbursement system that was based on actual payments made by the joint venture companies.


There is monitoring of the development results in terms of employment and knowledge transfer and (more recently) CSR, but proof of indirect development effects, such as chain effects and outgrowers is usually monitored less stringently. PAs and/or recipient companies have no effective tool available to make many of these assessments; hence reporting is limited to the visible, more obvious effects.

The EVD approach to monitoring is characterised by the “high trust” principle; i.e. PAs try to build a relationship of trust in order to be fully and better informed on problems and other issues concerning the project. PAs mostly believe that during the implementation phase the EVD has few tools available to interfere; once the contract is signed, the only sanction is to terminate the project, for which there need to be very strong reasons. Gaining information against the desire of the applicant, or local JV partner is very difficult. If PAs suspect wrongdoing or fraud, a procedure can be started to put the project on hold, and investigate the alleged fraud (see the chapter on legal services below). Since introducing result-based disbursements, the EVD can reduce payments if the agreed results are not achieved (or re-negotiated during the project). This, however, occurs sporadically as can be seen in the table below: Table 13: Withdrawals from completed projects for the period 1999-2009

Item Amounts (EUR)

Original commitment, completed projects 103,604,387

Final payment, completed projects 101,519,147

Total withdrawn from completed projects 2,085,240

Withdrawals as % of commitment 2.01%

Average per year, total 260,655

Average amount withdrawn per completed project 10,171

In total, payments to completed projects are about 2% lower than the original committed amount. PAs state that generally, reductions are rare; in most cases lower disbursements with completed projects are the consequence of an activity that the applicant decided not to carry out, a machine that was not bought (or found more cheaply), or valuations of hardware that have produced a lower than expected amount. In the database, very few completed projects exist with a substantial reduction; in total, only 4 projects have seen their final amount been reduced by about 20% of the project budgets; most others are relatively small reductions. This would mean that only three projects in the last ten years have not achieved the sales and production targets, and six companies had reductions of over 10% of their budget. In our sample of 60 projects, we have found that the proposed sales targets were not attained by 34% of the projects. This should have led to significant reductions, but it has not because nothing is done if the applicant can make a case that the targets are not met due to external circumstances (i.e. market situation). The limited reduction in disbursements is either due to the fact that the results in the application were negotiated down once the application “came through”; or that the underperformance on results was attributed to external circumstances, or have not been sanctioned.

3.2.5 Support services

In order to deliver its key outputs – developing and managing PSOM projects – the EVD has budgets for several support services. We describe the following in more detail: communication, legal services and procedures, and administrative systems.

Communication

The communication budgets and the services the EVD has in place aim to assure that potential PSOM applicants are aware of the subsidy and apply if the planned venture fits the criteria. The stated operational objective of the communication strategy is “brand” recognition. This mainly entails promoting the PSOM/PSI programme in mass media and business magazines, nearly always as free publicity (i.e. placing articles about PSOM and PSOM projects in media without payment). In addition, the PSOM part of the larger Agentschap NL website is maintained and brochures and newsletters are produced and disseminated. Another part of the communication effort is conducting information workshops and events; the aim of these is to promote PSOM directly to interested business people, to explain in more detail what the


programme can do, and how to apply for it. The EVD organises information workshops before each tender in the Netherlands, and also in PSOM partner countries (mostly in cooperation with embassies), to encourage local entrepreneurs to get involved. Both efforts (communication and voorlichting) are summarised in the table below in terms of cost and results. Table 14: Communication cost in € and numbers of applications for the period 1999-2009

Cost item 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008* 2009

Total comm. expenses

15,811 14,203 14,667 27,116 66,227 170,174 188,677 463,706 159,024 410,681 381,574

Comm. cost per project

687 1,775 367 1,043 2,208 1,870 2,775 5,392 1,828 8,214 3,778

Applicants per country

28.75 15.75 20.00 6.42 10.50 11.57 3.18 2.98 3.92 1.86 4.10

*) 2008 is a special case; there was only one tender for external reasons, so the results per application and project are much higher as a consequence, and should not be interpreted as a part of a trend. The communication budget has increased substantially, also in terms of cost per project, and cost per country. Despite this, however, the number of applications per country (or the overall number of applications) has not significantly increased in recent years. Since PSOM has been enlarged from a small programme in 2004 to a programme with 40-50 countries, the number of applications per country has fallen substantially. The communication and marketing efforts have not been able to reverse that trend. One reason is that the country list over the years comprised more and more countries such as Moldova, Nepal or Madagascar with very small economies and a short history of attracting FDI, while large countries such as India and China have been removed from the list of eligible countries. The consumer satisfaction survey of 2009 points out that the information on the tenders and the application procedure is difficult to access and understand. The PSOM/EVD website which contains the information is rated lowest of the information media available. Even so, applicants prefer to use the internet site as their main source of information. Nearly 60% of applicants have come to know PSOM not via the communication means offered, but through networks (30%) and consultants (29%).

Legal services

The legal department of Agentschap NL (JZ) provides support in two legal procedures the EVD has in place for cases of:

1) appeals by applicants, competitors of applicants or other involved parties 2) cases of (suspected) fraud by applicants or recipients 3) general advice during the appraisal period.

The appeal procedure is largely the consequence of the Hanseland case, in which a court determined that PSOM is a subsidy and not a grant, and therefore Public Administrative Law applies. Since this verdict, applicants and other parties with relevant interests can appeal to administrative decisions made by the EVD. This has led to significantly increased legal cost as can be seen below. Table 15: Legal costs per year in € for the period 1999-2009

Item 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Legal Cost, per year, EUR

3,523 1,405 7,241 14,584 12,341 13,531 18,122 34,266 117,581 260,864

Before 2008, legal costs were negligible (1% of budget or less), but these have increased to about 5% of the total implementation budget. The legal costs before mainly related to cases of applicants who were obliged to return advances, but did not, i.e. fraud. The results of the appeals are as follows (for 2009 only): For PSI: 14 appeals, of which one was retracted, one was inadmissible and 12 were dismissed. Of these, six appeals concerned unsuccessful applicants who challenged binding criteria, or the reason for denial; three were appeals from local competitors, and one was inadmissible because of no established interest.


For PSOM-OS there were six appeals, all of which were dismissed. Of these, four were about reductions in payments, while one concerned an unsuccessful applicant and one a competitor. Generally, JZ advises PSOM/PSI to ensure that documentation (e.g. letters of rejection to unsuccessful applicants) is motivated in line with the legal requirements, in order to minimise the number appeal cases. However, so far, no case has led to a financial obligation of EVD, and the frequency of cases is judged by JZ to be “fairly average”. With respect to fraud, the management manual has a clear instruction how PAs can start a procedure if they suspect fraudulent behaviour by an applicant. However, the legal department could not provide any information on the number, type and outcome of any of these procedures, citing confidentiality constraints. The UM indicated, however, that on average two to three serious fraud cases per year occur. Interviewed PAs estimate that investigations into alleged fraud in “nine out of ten cases” deliver no result since either no case could be made or the possibilities for recovering the damages are too limited in the local legal and institutional environment.

Administrative systems

The EVD has several key administrative systems: to file relevant documentation, administer budgets, generate management information, and for time registration. Here, employees have – across the board – a similar opinion, namely that the systems are outdated, cumbersome, and fragmented. The main programme (BAS) PAs and FEMs work with is an EVD-wide application, which is used for project administration, but is far from being an electronic dossier; some information can be stored in the programme, while e.g. incoming letters can only be stored in the paper archive. Essential information, such as the project proposal, or application form cannot be stored together in the main administrative system, BAS. Management information, for example for the monitoring of payments and liabilities has to be manually generated in an Excel sheet, financial information, or information on HR (time registration) comes from other systems. PSOM management is aware that that the systems are “beyond their economic and at the end of their technical lifespan”; most PAs see impractical administrative systems as the main reason for inefficiency, especially during peak time (i.e. tenders). There is also a common concern about the quality of the office hardware, such as computers, copy machines and the like; these, too, are seen by many as outdated, unreliable and time-consuming to use.

3.2.6 Reporting

In reporting the Ministry of Foreign Affairs, EVD reports bi-annually on the results of each tender as well as on ongoing projects. This information is used by the Ministry staff to answer questions or provide information on PSOM-financed projects. These reports have become very large, considering the portfolio and budget of the programme. To compile the reports, two PMs spend two to three weeks to produce the reports. Considering the fees for PMs, and for the PAs who have to deliver the content, this is a significant expense for each report. The reporting to DDE is very data-intensive, whiles the analysis, especially on the development impact is short, and delivers few new insights. We also noticed that the added value of the semi-annual report that is in the same format as the annual report is very small; most project descriptions do not change and numbers vary only in line with the expected. Reporting on development aspects is now reduced to a number of paragraphs, without analysis, while in 2004 this was covered in much greater detail, and included self-analysis. However, DDE values the reports, considering the high demand for information from various stakeholders, such as embassies and other organisations, and therefore needs semi-annual reports. A process is underway to replace the report with a database that can be consulted in “real time”. In addition, the EVD produces a printed booklet after each tender in which abstracts of the winning projects are shown. These booklets cost almost the same amount of effort from PMs (and significant printing cost), and are published on the website, and send to the embassies by mail in the partner countries, for display and use in information events. As of recent, EVD management has decided to make the production of this booklet cheaper (reduced number and black and white printing).


3.3 Findings on projects financed by PSOM

3.3.1 Inputs other than those provided by the EVD

The main input for implementing the PSOM/PSI programme is the work done by the EVD which is described in the previous section. In order to relate the inputs for the PSOM projects to the outputs, in this section we present detailed findings on the following inputs:

1) From the DDE, which is the department in the Ministry of Foreign Affairs that develops the policy and strategy of PSOM/PSI

2) From embassies, which play a role in the approval and monitoring of PSOM/PSI projects.

The Ministry of Foreign Affairs (DDE)

The budget holder of the PSOM/PSI programme is the Dutch Ministry of Foreign Affairs’ Directorate for Sustainable Economic Development (DDE), which comes within the responsibility of a cluster called ‘knowledge and capacity development’. As described in the description of the context of the PSOM/PSI programme (2.1), the underlying basic principles and objectives of the programme have not changed since the start. The main change in focus took place in 2003, when the indicators for measuring development impact were introduced. After new indicators (‘bepalingen en criteria’) were produced, no major innovations were brought into the programme except for increased attention to Corporate Social Responsibility (gender and social and environmental impacts). As the PSOM/PSI programme is a demand-driven programme, DDE has a reactive attitude. It does not actively create policies to direct the programme in certain sectors or countries. In addition, the lack of information on development impact and results obtained through the PSOM/PSI financed projects makes active policy development difficult. DDE has outsourced the implementation to the EVD (Agentschap NL) and its main role is therefore to monitor the implementation (mainly on input and output) and to deal with arbitration issues (mostly complaints from enterprises, NGOs or embassies). DDE participates in the appraisal process as an observer and has bi annual strategic meetings with EVD and the APSOM/APSI. Overall, DDE is satisfied with the implementation by the EVD. There are, however, concerns about:

the cost-effectiveness of the EVD

the internal learning capacity of the EVD

a lack of reporting on development impact (jobs created, follow-up investments, outgrowers, knowledge transfer).

Dutch embassies

Up to 2007 the embassies had a veto on PSOM proposals. Since 2007 they have had an advisory role and the advice of the embassy is important but not binding or decisive. The advice given by the embassies relates to additionality, innovation, potential sector effects and CSR aspects. Embassies also play a role in monitoring projects, especially if there are few projects in a particular country. In most of the countries visited the embassy staff accompanied the EVD during monitoring visits. Their contribution was ‘voluntary’. If the staff wanted to offer advice they were invited to do so; if not, this did not hamper the approval process.

In practice the embassy’s role in PSOM/PSI varies according to:

The capacity at the embassy: are the required FTEs available for economic cooperation?

The motivation and capacity of people working in the embassy: do the staff assigned have knowledge about private sector development?

The strategic development plans of the embassy: is the private sector included in the country strategic plan or not? Is private sector support delegated to another ‘lead donor’?

The relationship between EVD staff and embassy staff: do the EVD project advisors have a close relationship with the embassy staff? Do they conduct joint monitoring and promotion missions?


During the field visit the level of involvement of embassies in practice was assessed (this is described in the country reports). The level of involvement varies considerably, with the embassy in Ethiopia being the most active and those in Ghana and Mozambique being the least active in PSOM/PSI. A PSI customer satisfaction survey among embassy staff in 2010 concluded that the embassies are generally very satisfied with the collaboration with PSOM/PSI project advisors, and with their expertise. Overall, most embassies feel to have enough time to make recommendations, but a significant number do not have the capacity in place to answer the requests. Where these capacities are in available, embassies would like to have a less marginal role and more involvement with ongoing and future PSI projects. Also, having greater insight into the selection procedures and the reasons why projects have not been selected would be appreciated.

Embassy qualification advice was available for 42 projects in our sample of 60 projects. The embassies’ qualifications are heterogeneous with 32 “positive” cases, four “neutral cases”, and six “negative”, “negative, unless…” or “negative/neutral” cases. In the table below we present the main reasons the embassies gave for the negative advice which eventually did not lead to the rejection of the projects.

Table 16: Main reasons for negative embassy advice Project code Project title Reason

PSOM04/RI/13 Upgrading Technical Software Development Maturity Level in Indonesia

Project too ambitious

PSOM06/RI/22 Aloe Vera Processing on Bali -medicine against unemployment-

Lack of innovation

PSOM04/ET/14 Addis Roses Ethiopia Partners’ may not be interested in development effects

PSOM05/SR/1 Improving the export chain for high-quality products from Surinam

Reputation of local manager

PSOM05/SR/4 Development of the feed industry in Suriname by innovated new technology for aqua and pet foods

Market distortion

PSOM06/SR/04 Spang Makandra NV – Suriname e-media development studio

Market distortion

Local counterpart/national authorities

The role of the local counterpart organisation in PSOM/PSI as described in the BEMOs is ‘Identifying priority sectors and regions, promoting the programme and general monitoring’. The main formal contribution of the local counterpart is the signing of the MoU. With the change to a subsidy programme in 2008, this formality has become a deliberate choice of the Ministry of Foreign Affairs mentioned in the BEMO of the 2009-2010 PSI programme. As of March 2009, DDE has 31 valid MoUs (for which the validity period had not yet expired) and 21 expired MoUs. From the field visits we conclude that in practice the MoU and the role of the local counterparts is negligible. In some countries the local counterpart organisation is not at all involved in PSOM, in other countries only courtesy visits and occasional contacts take place.

3.3.2 Outputs: what have the inputs produced?

In this section we present the major findings on the outputs of PSOM/PSI, which are: 1) The project portfolio: completed and stopped projects 2) The characteristics of the projects 3) The number of people trained as a result of the PSOM/PSI project

From the EVD database the following output information has been produced: Table 17: Overview of funds disbursed in € for the period 1999-2009

Status Number of projects in % of total Funds disbursed until end of 2009 In % of total

Completed 205 31% 101.519.147 66.8%

Ongoing 273 42% 39.423.581 26.0%

Stopped 178 27% 10.968.092 7.2%

Total 656 100% 151.910.820 100%


The average PSOM subsidy for completed projects (n=205) is €495,215. The average total project cost amounted to €908,509 per project (n=59), slightly higher than the amount found during the 2005 evaluation (€824,000). On average, the PSOM/PSI programme committed to subsidise 55% of the total project costs. In the table below it can be seen that the number of approved projects has not yet met the targets as defined in the policy document (BEMO). In the past three years the approvals against targets were as follows: Table 18: Overview project approval targets and realisation 2007, 2008 and 2009

Year Approved projects Target Realisation

2007 87 110 79%

2008 (one tender was cancelled) 50 57 88%

2009 101 120 84%

Stopped projects Over the entire evaluation period, a total of 120 PSOM projects (20%) stopped prior to completion because of problems. There are 58 other projects that stopped, but that happened because the prequalification phase ended unsuccessfully

5 (56 cases) or because projects were put on hold (two cases).

During the field visit it was found that three of the visited 32 projects were no longer operating after being completed (see cases for background). If we consider the period where all projects were either completed or stopped (1999-2003), we see that an average of 31% of the approved projects stopped. Table 19: Overview of stopped and approved projects for the period 1999-2009

Item 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

No. of stopped projects 8 8 10 6 10 21 11 22 19 5 0

No. of approved projects

26 23 34 20 31 78 65 83 88 50 102

Ratio stopped/approved

31% 35% 29% 30% 32% 27% 17% 27% 22% 10% n/a

Ongoing projects 0 0 0 0 0 5 15 39 68 45 102

Ongoing as % of approved

6% 23% 47% 77% 90% 100%

The 20% stopped projects used 7% of the disbursements until end of 2009 (total amount: €10 million); this is down from the 14.4% found during the 2005 evaluation. On average €86,515 PSOM subsidy was spent per stopped project. This is slightly higher than the €75,000 found in 2005. The highest number of stopped projects is in China (13 on 44) and Uganda (11 on 37). We have conducted a small survey of the stopped projects. The main reasons for which projects ended prematurely are: problems between partners (seven occurrences), and financial constraints (six occurrences). Problems between the project partners relate to a lack of mutual trust, or misperceptions regarding the partners’ financial commitments during the inception phase. In Ethiopia, the division of shares and the fact that the subsidy was allocated to the applicant (Dutch) firm created friction between JV partners. Financial constraints related to the financial condition of the recipient firm, but also related to the reduced availability of external finance due to the financial crisis affected quite a number of projects. In four cases, the economic downturn itself was mentioned as a reason for early project termination.

The business environment is also a constraining factor. In five cases, bureaucratic problems regarding the purchase of land, the obtainment of concessions or the forming of a joint venture were cited. In two other cases, the project partners felt that the investment climate was not stable or secure enough to continue the project.

5 Up to 2005 PSOM was implemented in two phases: a prequalification phase (to elaborate a business plan) and an implementation phase.


At times, projects are ended prematurely due to changes in the priorities or strategic direction of the applicant firm. In two cases, the acquisition of one of the project partners by another firm affected the project. In both cases the overtaking firm decided to stop the PSOM project. Finally, in three cases the withdrawal of one of the project partners was cited as a reason.

The above conclusions are in line with the results from the 2005 evaluation on the same research question. The 2005 study also pointed to financial problems, the business environment (investment climate) and problems between partners as main reasons. In addition, the 2005 study also specifically mentioned problems with setting up the joint venture as a frequently cited reason. Untied projects The number of untied projects as of the end of 2009 was 41 (n=656), i.e. 6% of the total number of approved projects. Most untied projects started in 2009 (26) which is 25 % of the 102 approved projects of that year up from 8% of the approved projects in 2008. As of end 2009, four untied projects were completed, 35 are ongoing and two projects stopped. It is too early to draw conclusions on untied projects because the number is limited and most projects are in an early stage of development.

Characteristics of projects - Continents and sectors The largest numbers of projects were in Africa (46%) but in terms of disbursements, Asia and Africa are almost equally represented. The analysis per sector of the PSOM subsidies disbursed and the number of projects approved (n=656) shows that most approved projects and disbursement were in agriculture (46% and 62%, respectively). The largest numbers of approved projects were proposed in Ethiopia (47), China (44), Indonesia (42), Ghana (38) and Uganda (37). - Business model proposed The classification of projects according to type of business model gave the following result:

Model Description % (n=60)

Exporter model Developing the local market for the applicant: The joint venture enterprise is created to support the sale of the applicant’s core product, together with locally made parts and services

10%

Re-exporter model The joint venture produces inputs or final products for the applicant who sells in the Dutch/EU market

55%

Migration/Relocation model

The joint venture enterprise is the result of relocation of the activities from the country of the applicant to the country of the joint venture

0%

Globalisation model The joint venture enterprise is intended to become an integral part of the global operations of the applicant

26%

Most projects (33 out of 60) can be characterised as “re-exporter models” whereby the applicant company creates a production facility in a developing country which exports its products back to the applicant’s country. Examples are the flowers from Ethiopia, which are sold against inter-company price to the applicant and then sold through the Dutch auctions, the agricultural production in Mozambique, which is exported and the assembly industry in Vietnam, whereby the assembled product is re-exported to the Netherlands. The advantage of this model is that the Dutch applicant firm knows the export market and is close to where the money is made. The driver for starting a re-exporter type of business is often the favourable climate and/or availability of cheap labour allowing the production of agricultural or industrial goods in more efficient (cheaper) way. Inter-company pricing is found to be an important issue in the re-exporter model: if the applicant firm has most of the JV shares and determines the buying price, this often means that the JV company makes low profits. This will also impact the tax that the company in the developing country pays (although this effect may only become visible after some years, since most newly established international joint ventures enjoy a tax holiday of 3-10 years).


Other business models are the globalisation model (16 cases) which is mostly found in Ghana and Surinam and the exporter model (6 projects). The migration model was not encountered in the evaluation sample.

- Share in ownership of local partner in joint venture In 38 of the completed projects information on the division of shares between the applicant and the recipient was available. On average, the local partner holds a minority share of 35% in the joint venture. On only seven of the 38 cases the local partner holds a majority share. In addition to the 38 cases, the desk study revealed seven cases in which the local partner is a subsidiary of the applicant firm and another seven cases in which no joint venture was formed and the beneficiary was an existing company. In eight cases no information was available.

From the field visits it became clear that the type and level of involvement of local JV partners differs from country to country. In Ghana, Vietnam and Ethiopia, the vast majority of companies are Dutch-owned and managed and the created JV are in many cases 100% subsidiaries of the Dutch companies or Dutch individuals. In Surinam, Mozambique and Indonesia the local JV partner have minority shares and are involved in managing the JV company. In Mozambique the particularity is that in all cases both JV partners were found to be non-Mozambican, the local partner were in these cases foreigners living in Mozambique. - Size of the firm and gender The typical PSOM/PSI applicant is male (100 %, n = 52) and is running an SME (94 %, n = 53). In our random sample of 60 projects, two of the applicant firms (Masterfoods Veghel and Unilever Research & Development) are large firms. One applicant (LNB International Feed B.V.) is a subsidiary of a large firm (Nutrivest N.V.). The gender of the applicant or recipient firm does not play a role in the appraisal process.

- Training The average number of reported in-company trainees per PSOM project in our sample is 96 trainees (n=53) at project completion, where an average of 69 trainees were proposed by the applicant. The outputs are therefore higher than was projected. The 2005 evaluation found an average of 71 people trained per project so the average has increased since then. The actual number of trainees in the projects visited stands at 165. This increase is due to increased employment in Ethiopia and Vietnam.

Often, the number of trained staff equals or exceeds the number of jobs created because if staff turnover is high, new staff needs to be trained. If one would also consider larger training schemes to outgrowers as part of the project outputs, the average number of trainees would be 237

6.

From the field visits the following observations can be made on the training:

Most training is on the job and continues to be provided for new staff because it is necessary for implementing the job

Training on the job is often informal and given by colleagues or direct supervisors

Formal training, resulting in a training certificate to diploma, has a stronger effect on the employability and career opportunities of staff. Especially in the case of technical training courses (handling machines and equipment), employability of the trainees is often enhanced.

The number of staff trained may not be the most adequate indicator to assess knowledge effects of PSOM projects as it does not provide an indication of the type of training and training intensity. The files do not contain information on the type of training (formal training provided by certified trainer or educational institution vs. informal/on the job by co-workers) and the training intensity (hours of training received per employee) but this information might improve the usefulness of this indicator.

3.3.3 Outcome: what are the effects of projects?

In this section we present the major findings on the outcomes of the PSOM/PSI financed projects. We present findings on direct job creation, knowledge transfer, innovation and outgrowers.

6 For this figure we have used the dataset concerning 196 projects because the outlier (PSOM05/ET/2 with 30,000 cooperative members) is excluded. Including this outlier would increase the average number of trainees per project to 390 while in fact the PSOM financed training was provided to trainers (ToT).


From the figures provided in the annual report 2009 we found the following key indicators for PSOM/PSI outcome effects of completed projects (n=197): Table 20: Key indicators and findings for all completed projects in the period 1999-2009

Indicator Findings

Direct jobs created per project 81

Knowledge transfer (people trained) per project 388

Outgrowers (n=70 and n=69)7 per project with outgrowers 1.751 or 1.342

Source: PSOM annual report 2009 Direct employment: gender disaggregated The average PSOM/PSI project generates 81 direct jobs (n=197), which is considerably higher than the proposed average of 55 direct jobs (n=55), but lower than the average found in 2005, which was 147 jobs /project. On average, roughly half (47%, n = 41) of the direct jobs created is for females, slightly under the average proposed proportion of female workers of 56% (n=29)

8.

In the table below we present the results in terms of job creation by comparing the data collected through the life time of the created company. For 25 companies we have the number of jobs proposed in the application, achieved at project completion, expected for the spin-off phase and the number of jobs found during the field visit. The results are as follows: Table 21: Overview employment creation, findings of evaluation sample

Item (N=25) Jobs proposed Jobs achieved at

completion Jobs expected during spin off

Jobs found during field visit

Ethiopia 134 346 412 496

Ghana 42 30 129 8

Indonesia 49 79 76 46

Mozambique 48 47 94 23

Suriname 9 13 7 13

Vietnam 53 75 149 103

Total 62 122 166 150

In most cases the number of jobs achieved at completion exceeds the number of jobs proposed. It is also clear that the realisations in Ethiopia and Vietnam exceed the proposed numbers of jobs and that growth in jobs continues.

During the field visits the following observations were made regarding direct employment:

In the countries where follow-up investments are high the increase in jobs is more significant. This is especially true for the labour-intensive sectors (garment production, horticulture) in Vietnam and Ethiopia.

Most of the employment created is low skilled manual work in industry or agriculture.

The EVD does not classify employment creation, so each job is equally relevant. During the field visits the consultants found that a differentiation between jobs that are created in rural remote areas in Ethiopia and employment creation in highly industrialised booming areas in Vietnam would make sense when assessing development relevance of projects.

In Mozambique, Indonesia and Ghana the expectations for the job creation in the spin-off phase appeared to be too optimistic; this is explained by the fact that some of the businesses have not yet been able to grow due to not yet functioning value chains (regular supply of raw materials or a stable market). Examples are Novel Development and Sitos cocochar in Ghana and Produsola and Gouda Gold in Mozambique.

In Surinam employment growth is limited because projects have only recently been completed.

7 The outlier in Ethiopia consisting of two coffee unions with 30,000 members has been excluded in the second figure 8 Note: if one would only compare the proposed and actual gender figures for projects for which both percentages are available the proportion would read: female jobs proposed = 57 %, female jobs achieved = 53% (n = 26).


Innovation For 52 of the 60 completed projects the type of innovation could be mentioned. In 17 projects, the innovation can best be described as the launch of a new product in the recipient country, in 23 cases a new technology was used to produce an existing product and in four cases a combination of a new technology and a new product was found. Finally, in eight cases a different type of innovation was proposed. These other types of innovation were often related to new marketing concepts, or the up-scaling of existing products. In one third of the cases (7 out of 21) for which the embassies’ opinion on the innovative character could be derived from the files, the embassies were of the opinion that the proposed project lacked an innovative character. The screening of innovativeness appears to be quite complex especially in large countries and when technical aspects are involved. As the interviewee in the Indonesian Embassy put it: we can only assess the innovative character by checking whether “it is a product you see or wouldn’t see on the shelf very often.” During the field visits the following observations were made regarding innovation:

In some cases the innovations were actively promoted by the JV company: demonstrations (bitumen in Surinam), lectures in University (horticulture) or exchange visits (rose varieties) were mentioned during the field visits as ways of promoting the innovation.

During field visits several projects were found to be less innovative than presented in the business plan. A new technology (‘ready to eat packaging’) or product (shrimp feed) was proposed but not introduced.

The “level of application of the innovation” is also an issue that needs to be taken into account. In some projects only a minor share of the production could be qualified as innovative, but the vast majority of the PSOM subsidy was used to buy assets for the regular, non-innovative production process.

Innovation does not necessarily have impact on other actors in the same sector. When most inputs are imported, and most outputs are exported to the Dutch applicant (the re-exporter model), the local sector and companies cannot benefit from the ‘innovation’.

In some cases the innovation was not successful because of competition from similar products (java eggs in Indonesia) or because time was too short to develop the innovation into a business case (sea food meals in Indonesia).

Outgrowers According to the annual report figures provided by the EVD a total of over 122,000 outgrowers have been reached through 70 PSOM/PSI projects. There is one outlier in the dataset: in Ethiopia one PSOM project involves 30,000 members of coffee cooperatives. If this case is not considered, there were an average of 1,342 outgrowers per project. The definition of an outgrower in the PSOM programme is quite broad: apart from the outgrowers contracted by a nucleus farm (traditional definition of an outgrower), the outgrower numbers used by the EVD also include members of (already existing) cooperatives and unions, collectors of products (who work during a limited number of days per year), and smallholders that sell to the PSOM-financed company. Due to these very different types of ‘outgrowers’ it is difficult to provide quantitative results in this area. We refer to the individual case studies for in-depth information on effects at outgrower level. In the field visit sample of 32 projects, 17 projects had proposed an outgrowers scheme. In some cases, specific numbers of outgrowers were mentioned, whereas in other cases it was just mentioned that a scheme would be carried out. In 11 cases, outgrower schemes were developed, with the number of outgrowers varying between ten (for seedlings production) to 10,000 (sesame outgrowers) and 30,000 (coffee smallholders). In six of the 17 visits, the proposed outgrower scheme was not successful. The main examples are in Ethiopia, where four of the six visited projects did not achieve the proposed outgrower scheme because high-tech floriculture is not suitable for outgrowers. The same happened in Vietnam with a floriculture project.


During the field visits, the following conclusions with respect to outgrowers were formulated:

The role of the PSOM projects in developing the outgrower schemes differ largely. In some cases the outgrowers associations or cooperatives already existed (coffee producers in Ethiopia), whereas in other cases the outgrowers had to be organized from scratch.

Some outgrower schemes have benefited from development support other than PSOM. In these cases it was acknowledged by interviewees that developing outgrower schemes requires a great deal of time and money, which was not anticipated in the PSOM budgets.

Some interviewees stated that they included the outgrowers scheme in order to make their application more attractive, knowing that such schemes would not be feasible, given their high-tech production methods.

In order to assist the PSOM applicants in developing the outgrowers’ schemes, the EVD together with other partners has produced a manual for outgrower schemes.

3.3.4 Impact: which effects have been achieved beyond the project?

In this section we present the major findings on the impact of the PSOM/PSI-financed projects. We present findings on:

Follow-up investments

Sales

Income effects and working conditions

Environmental effects

SME linkages (backward and forward)

Catalytic effects on the sector

Identifying issues related to the business climate Follow-up investments The most important ‘proof of the pudding’ of PSOM projects according to the BEMOs, are follow-up investments. The EVD measures all follow-up investment made in the PSOM-subsidised venture, as long as funds do not come from donors or governmental organisations. The findings are presented below: Table 22: Average follow-up investment proposed, achieved, and expected (all amounts in €)

Item (N=41) Proposed Realised at completion Expected during

spin off

Average follow-up investment 1,065,357 576,257 1,054,927

Ethiopia 1,276,739 1,167,778 1,736,667

Ghana 1,350,000 599,600 993,600

Indonesia 1,195,000 410,350 997,000

Mozambique 283,429 53,575 319,143

Suriname 540,000 676,250 540,000

Vietnam 1,557,500 489,167 1,381,667

On average, at completion of a PSOM project a follow-up investment of €576,257 was achieved (n=41). This is lower than the proposed average follow-up investment at project start (€1,065,357) but higher than the amount found during the 2005 evaluation (€275,000). It appears that PSOM applicants are overly optimistic on the rhythm of follow-up investments; this observation was also made in 2005. The findings on the field visit (n=22) on follow-up investments is that the actual average follow-up investments for the 22 projects is €1,872,318. This outweighs the proposed follow-up investments. Conclusion is that it takes more than the actual two years of the spin-off phase to achieve the proposed follow-up investments. The average multiplier for the visited projects is four, i.e. the follow-up investments equal four times the PSOM subsidy. As the variations between the figures are very large a closer look into the data is needed: The field findings include ten cases where the follow-up investments are less than proposed (two to ten


times less), nine cases where the follow-up investments exceed by 4 to 5 times the proposed investments and three cases where no follow-up investments were achieved. Follow-up investments exceeded expectations in Ethiopia and Vietnam. Especially the floriculture businesses in Ethiopia have achieved large extensions (multipliers of 15) during and after the spin-off phase.

Sales figures The data gathered in the PSOM files with respect to expected sales figures were difficult to compare because they describe expected turnover, expected sales in units or expected total value of sales per year. Sales figures are presented in the table below. To ensure a meaningful comparison of the figures, the table only includes the 37 cases for which the proposed, actual and expected figures could be obtained from the files. Table 23: Average cumulative value of sales during PSOM projects: proposed, realised and expected (all

amounts in €)

Item (N=37) Proposed Realised Expected after pilot

Average sales 958,577 845,475 2,757,970

Ethiopia 814,111 1,489,538 3,187,222

Ghana 576,750 82,648 6,299,420

Indonesia 655,556 287,333 2,380,556

Mozambique 277,222 166,442 1,709,533

Suriname 676,667 1,592,667 693,333

Vietnam 2,706,667 1,530,583 2,400,000

With an average realised sales value of €845,475, the PSOM projects are in line with the proposed average sales value of €958,577 (n=37). In addition, many entrepreneurs take a rather optimistic view with regard to expected sales figures, reaching an average of €2,757,970 towards the end of the spin-off phase. This figure is distorted by the very optimistic expectations of Novel Development in Ghana, which estimated a turnover of €23 million, whereas the actual sales during the field visits were around €100,000. The actual sales of the Ethiopia horticultural firms on the contrary exceeded the expected sales of €3 million.

The findings related to sales during the field visit (n=28) show that actual average sales for the 28 projects were €1,648,821. This is less than the proposed sales during the spin-off of these 28 projects, which was €2,866,875. If Novel Development is left out, then the expected sales are €2,092,524, which is closer to the achieved level of sales found during the field visit.

Employment and Income effect The employment effects beyond the project phase are important. The average number of jobs in the 31 visited projects was 130 jobs per company – up from an average of 110 jobs achieved at project completion. If the three stopped projects are taken out of the average the average become 144 jobs per company. This high average is mainly thanks to the labour-intensive horticultural and industrial project in Ethiopia and Vietnam, respectively.

According to the files, in all 46 projects for which information is available, employees receive or will receive a salary on or above minimum wage plus secondary benefits. This was confirmed during the field visits. In most cases market trends were followed and in cases where there is pressure on the labour market the income effects were higher (Vietnam, Ethiopia). From these findings we can conclude that job creation and income effects are related effects, whereby the number of jobs is a more reliable indicator than income. The outcome of focus-group discussions in Vietnam and Ethiopia on the impact achieved through enhanced income was that in Ethiopia income contributes to enhanced empowerment of young women and increased access to basic services, while in Vietnam the acquisition of assets (motorbikes, house) was mentioned as the main impact.

Environmental effects In all but one of the cases for which information on environmental effects could be obtained, (as described in the proposal and monitored and reported on by the EVD) the effect is either neutral or positive. Many reports also mention compliance with emission rules, the limited use of fertilisers, and/or adequate waste


management methods. Most horticultural projects in Ethiopia have opted for some type of certification in the area of good agronomic practices; these standards have now been adopted by the Ethiopian Horticultural Association and made a condition for export licences. Only one flower project had negative environmental effects. The EVD urged the company to look for a more sustainable method for soil disinfection.

During the field visits, no environmentally hazardous effects were encountered. However, an in-depth assessment of the true environmental impact of the PSOM supported companies (groundwater effects of flower cultivation, soil pollution in container cleaning and health effects of garment production) was outside the scope of work of the consultants. We also observed in the reports that the texts on environmental aspects are copied from the proposal without further indications that the EVD has verified or assessed the environmental impacts. As this often requires technical expertise, it might be difficult for EVD staff to do in-depth environmental assessments while monitoring the projects.

Linkages with SMEs The realisation of chain effects has been an objective of PSOM since the start of the programme. However, this indicator of success is not systematically collected as is the case for direct jobs, trainees and outgrowers. In the final report (at project completion) of each project there is a section on chain effects. Of the 60 files studied, 25 reports make mention of backward linkages (e.g. SMEs supplying raw materials, or services) and in seven cases examples of forward linkages (e.g. SMEs involved in processing and transportation of end product, or SMEs making use of the end-product) were found.

During the field visit, the following observations were made with regard to SME linkages:

Especially in the re-exporter model (55% of the cases), there are limited possibilities for SME linkages with local SMEs because the inputs are mostly imported (seeds, components, fertilisers, equipment) and the end product is exported.

During the project implementation the SME linkages are more significant than after project completion because most projects include construction, which is often done by local SMEs.

We found few cases where a deliberate decisions was made to include local SMEs: NEDDECK in Vietnam uses SMEs to produce components of their products, Java Eggs sources eggs from local small traders and Tunas buys their products from 2,000 cottage industries.

Catalytic effects : wider knowledge sharing and sector effects In about one third (18 out of 60) of the project files, there is evidence of a wider level of knowledge sharing beyond the training of employees or the training of project-related outgrowers. Examples of wider knowledge sharing found in file and field are:

Lecturing or research at educational institutes: six cases

The influence of government policy: two cases

Training for farmers (beyond outgrower schemes): six cases

Training of suppliers or retailers: three cases

Other non-specified seminars: one case. In Ethiopia we identified numerous catalytic effects due to PSOM projects: the horticulture sector has developed tremendously in the past ten years and PSOM projects have played a catalytic role in technology transfer (greenhouses, drip irrigation and substrate culture), introducing standards (environmental and social) and export logistics (airport facilities and forwarding companies). In other countries these effects have not been identified. The reason for this may be that projects are more diverse or relatively new (Surinam), the embassies are less active (Ghana, Mozambique) and or the economies are large (Indonesia and Vietnam).


Corporate Social Responsibility Within DDE and the EVD developing practices and guidelines for CSR has been an important issue since 2009. The consultants proposed a categorisation of corporate social responsibility as presented below. It was quite difficult to retrieve this type of information from the files, as no reporting on these aspects is required. In the table we summarise some field findings on corporate social responsibility in practice, which might provide input for further development of guiding principles in this area.

Category Definition and Found examples Occurrence

Category 1: Corporate charity (liefdadigheid)

Non-systematic donations to social projects (sponsoring of events or making contributions to school desks for example)

Most projects proposed some type of charity to make the proposal more attractive. Implementation is not systematic but in the visited companies charity is seen as ‘good business practice’ and a condition for being accepted in the community

Category 2: Engagement with the community (Maatschappelijk Betrokken Ondernemen)

Systematic contributions to community improvement projects (payment of drinking water infrastructure, sponsoring of school fees)

Few companies have firm commitments on charity or sponsoring. Most interviewees prefer to decide on a case by case basis

Category 3: Corporate Social Responsibility (Maatschappelijk Verantwoord Ondernemen)

Efforts made to make (parts of) the core production process of the company more sustainable (certifications, waste management)

Especially in the agricultural and textile sector the projects have implemented some type of certification to make the business process more sustainable. These efforts are often encouraged by the EVD staff

Category 4: Sustainable Enterprise (Duurzaam Ondernemen)

The company’s core business is production of a sustainable product

There are some examples of sustainable projects: ecological tourism, organic fruits (bananas and mangoes), sustainable oils (aloe vera and allanblackia)


4 ANALYSIS OF THE EVD AS IMPLEMENTING AGENTS AND THE PSOM/PSI PROGRAMME

4.1 The efficiency and effectiveness of the EVD in selecting and managing projects

Below we present our assessment of the key research questions, based on the findings presented in the preceding chapters. We have grouped these assessments in terms of efficiency and quality (=effectiveness) and, according to the main processes within EVD, selection and managing projects

4.1.1 Overall EVD efficiency: no economies of scale and increasing overheads

Implementation efficiency in terms of cost has not improved, despite the larger size of the programme. Therefore, no economies of scale have been achieved so far. Implementation costs have been 17% of the cost of completed programmes; in other words, it required €26.4 million to achieve disbursement of €151.1 million. However, this does not account for the work delivered to commit resources for future disbursement. We notice, however, that in the last four years actual disbursements have stayed flat, while implementation costs have increased significantly. The EVD has taken on more overhead-related tasks (CSR policy, for example) and has thus become more expensive, irrespective of the effects of inflation, wage increases and changes in workload; an estimated 9% of the implementation cost in 2009 (compared to 2006) is the consequence of increased overheads in fees and budget. We therefore conclude that cost efficiency has declined. Budgeted overhead as a percentage of the total implementation cost was higher in the last five years than in the first five years; considering the fact that the programme budget has increased, it would be reasonable to expect progressively lower overhead percentages.

4.1.2 Assessment of the efficiency of selection: selection is expensive but well organised

Overall, in our view the EVD is efficient – to a limited extent – in selecting (or developing) new projects because the direct cost of developing a project has increased, reducing efficiency relative to earlier years, even though less time is spent per application. In addition, in our sample we found that over half of the applicants (feel the need to) use consultants to successfully apply; the cost of this could easily add a similar amount of TA cost – paid from the PSOM subsidy – to the cost of the EVD for selection. The organisation of the selection process is efficient in our view; the current norm for the number of days for developing one contract is very reasonable, considering the challenges. The EVD has developed an efficient process for appraising applications; in our view there are only some minor areas where EVD could consider introducing some changes:

The external “double-check” of the applications carried out by the APSOM committee is important and a must because they are the advisor of the Minister. Their work is efficient (low cost), but this element of quality assurance and objectivity is also done by the CLs who co-read applications with PAs. This co-reading is important for quality, as it ensures that the application criteria are applied in a consistent fashion. We believe, however, that to improve efficiency, reducing one of the two would save time and cost. Either senior PAs can also take up this role as a second pair of eyes and/or APSOM/APSI could reduce their role to broadly assessing regular cases, and more deeply assessing doubtful cases.

Another minor matter is that each application that meets the basic criteria, but has significant flaws in the view of the PAs, is nonetheless fully processed; in our view and that of many PAs this is inefficient, considering the fact that a significant number of the applications are such cases. De-selecting these cases earlier would improve efficiency.


4.1.3 Assessment of the effectiveness of selection: good portfolio but underutilisation of budgets and limited choice

In our opinion, the selection process is effective in terms of generating a portfolio of successfully completed projects, but less effective in terms of utilising available budgets and maximising development effects. The reasons for this are as follows:

The outcome of the selection process can be measured in terms of the number of failed/stopped projects; here, we consider that the overall number of projects that stopped prematurely (31% for the years in which there were no ongoing projects, 20% for the total portfolio) is significant, but not bad considering the nature of the PSOM programme and the willingness to finance risky ventures. If we include the findings from the field visits where four of the 32 visited projects were no longer operational, this makes the average stopped project rate 43.5%: 31% stopped prior to completion see above + 12.5% (4 out of 32) stopped after completion. Most projects are stopped before large expenditures are made: only 7% of the disbursed funds are paid to these stopped projects.

The EVD has utilised, or allocated 85% of the available budgets to projects; €331 million has been committed to projects, of a total available budget of €391 million. The trend for utilisation has been downward since 1999; however, only the last tender of 2009 has revealed a higher commitment (96%) than the average of the years before (73%-77%). We consider that this is also due to a missed tender in 2008. The absolute number of applications has not increased, so that high utilisation resulted from a high approval rate (47%).

The selection method and process has matured and been refined; and implementing the process is done professionally and with enthusiasm. This counts for the assessment of applicant and recipient, as well as for the assessment of the business plans. With regard to development relevance, there are no minimum levels, and scrutiny is noticeably lower than with the partner and business checks, in our view. The criteria for development relevance are criteria that would usually apply to any company, such as generating employment, knowledge transfer or having chain effects through suppliers. Equally, additionality is (implicitly) assessed by considering risk, innovation and availability of bank finance. These too are features that nearly always apply in the setting of a PSOM partner country, where risk is high, bank finance usually utopian, and most products can be seen as innovative, depending on the definition.

In our view, the main challenge the EVD faces as an implementing agency is to generate a large enough stream of applications in order to have more choice. While budget absorption is not an aim in itself, more choice would be desirable in our opinion, as development aspects could then also become a criteria for selection. The ranking of development relevance of projects is in place, but never had to be applied, as there have always been fewer applications than budget. Having more choice in turn would allow selecting projects on their relative development relevance more than is possible now. EVD considers that (full) budget utilisation is not its objective; firstly, PSOM/PSI is a demand-oriented programme, where it is the responsibility of the applicants to come forward with ideas and business plans, while the EVD’s role is to process applications and manage projects. Secondly, the increase in budgets in 2007 and beyond has been a political decision, to demonstrate the connection between business and development cooperation, and was not based on a needs assessment, or absorption capacity. While we agree with this statement, we would like to make two qualifications: first, the growing cost for communication indicates that the EVD is trying to generate applications, but as shown, does not achieve this objective. Secondly, underutilisation does not result in low effectiveness as such, but the fact that there is too little choice – in our view – does not generate the best possible development results, as argued above. The solution to a low number of applications should be an effective communication strategy; however, according to our assessment this is not (yet) in place. The considerable communication budgets are spent either on mass media-type activities (these are broadcast too widely; only a tiny proportion of the target group will be interested to know about investing e.g. in Africa), or on information workshops (these are too narrow; they address only the concerns of potential applicants who have already been informed). These have not resulted in more choice among the applicants. The EVD is taking action in order to be more


easily found on the internet (search engine optimisation), and the PAs are also spending time in-country in order to popularise the programme.

4.1.4 Assessment of the efficiency of management: the monitoring costs per project are decreasing

The management (monitoring) of projects is generally efficient; cost for management has decreased over time, and the procedures in place are implemented with a relatively small number of days.

4.1.5 Assessment of the effectiveness of management: effective in terms of correct spending of the subsidy, but no validation of reported outcome and impact

The EVD’s monitoring visits and tools to ensure that the hardware element of the subsidy is correctly spent are in our view sufficient to ensure that the PSOM subsidy is spent according to the rules and the contract. Also, surveys among applicants have shown that the EVD PAs are highly regarded and their support is considered very important for the success of the project. Therefore, we consider that this aspect is very effective.

With respect to monitoring the project results in terms of employment and turnover we believe that the current monitoring system is only effective to a limited extent, and can be significantly improved. While a significant amount of time is spent on collecting information via progress report from the applicants, the value of the obtained information is not necessarily high. Monitoring of the results, as agreed in the performance contract, is based mostly, if not exclusively on the information provided by the applicant, and based on the “high trust” principle. Although this reliance on formal reporting is a logical consequence of the administrative rules applied when subsidies are granted, we see that the PAs have very limited information other than that of a party to the contract. Significant payments depend on achieving turnover and employment figures. Even during field visits, a PA will be challenged to ascertain real figures and facts, unless the deviation is very large. As an indicator, the EVD reports that only 2% reductions are not paid out or recovered from completed projects, which suggests that the results-based contract delivers a near-100% result, which is at odds with high-risk nature of doing business in developing countries, for which the subsidy is – at least in part – granted. The fact that 25% of the projects stop before completion again suggests that doing these projects must be risky, but the reduction percentage suggests that once a project has passed the early critical stages it is nearly completely successful. The low level of reductions suggests that either the results in the performance contracts lack ambition, or that the monitoring information is not always objective, or both. EVD interprets the low reductions as follows: EVD applies a policy whereby reductions are not made if the causes of the underperformance are “outside the enterprise”, such as adverse market developments, or any other cause that is not directly a fault of the business. We cannot verify which of the reasons applies, but either way, in our view the result monitoring instrument is not very effective: if applicants can achieve the results anyway, the concept of a performance contract (introduced in 2003 instead of a service contract, i.e. inspanningsverplichting) does not mean very much if it is not enforced. With respect to the reporting on development results, such as sector and chain effects, monitoring information is problematic; here, the applicant, or local manager can only make an educated guess, and is unlikely to collect information, or even conduct deeper research into the true nature of the project’s developmental relevance.

4.1.6 The efficiency and effectiveness of supporting services

With respect to the legal procedures for in case of objections of companies, we conclude that these services are provided efficiently and effective; no claim has yet been acknowledged, and the costs are still reasonable compared to the overall implementation cost.

With respect to the adequacy of legal procedures in case of suspected fraud we cannot make an assessment since no information regarding the nature, frequency or results of these procedures could be disclosed.


The administrative systems are mostly inefficient and only moderately effective. Most systems essential for programme administration are outdated and employees are widely unsatisfied with the systems, as well as with the hardware. However, the systems in use are generating the necessary information and are not hampering the execution of the programme as such. Better and more in-depth management information, as well as more efficient systems for staff to work with would most likely also have a positive impact on effectiveness, as management will then be better informed and staff can spend more time on doing their real work.

Reporting to DDE and other stakeholders takes place, but the quality of the reporting is only fairly satisfactory: a great deal of data is provided which is useful to respond to information requests, but limited policy relevant and managerial information is distilled; the reports are not produced very efficiently. The booklets for the embassies, considering the high cost involved, are worth reviewing, which has been done recently.

4.2 The efficiency, effectiveness and relevance of the PSOM/PSI projects

A summary table with the findings on additionality, effectiveness and relevance from the field visits can be found in Annex 4.

4.2.1 Efficiency, including the additionality of PSOM/PSI project

Under efficiency of PSOM/PSI projects we assess the outputs that were created with the used inputs. A summary table on the findings during this evaluation (covering 11 years) and the findings of the former evaluation (covering six years) as well as the target described in the BEMO are presented below:

Table 24: Overview of PSOM/PSI outputs 1999-2009, including targets, findings of the 2005 evaluation and findings of the 2010 evaluation

Indicator Findings in 2010 Findings in 2005 Target

Number of approved projects 656 267 110 in 2007 up to 130 in 2010

Number of completed projects 205 47 N/A

Number of stopped projects 120 24 N/A

Completed/ongoing rate of approved projects (success rate)

82% 91% > 66%

Average size of projects €908,509 €824,000 <1,500,000

Average subsidy amount €495,215 N/A <50% of project size>

(or <60%)

Trainees 96 71 N/A

In 11 years, 205 projects have been completed with an average subsidy of €495,215. As of end 2009, 82% of all the approved projects were either completed or ongoing; i.e. 82% of the projects. In these projects an average of 96 people were trained. That translates to a subsidy of €5,158 per trainee. The success rate is not going down as one might think based on the above figures. Between 2005 and 2009 some projects that were approved and were still ongoing between 1999 and 2004, stopped in that period. If we look at all 134 projects that were approved after prequalification between 1999 and 2003

9 , we have

only those projects that as of the end of 2009 were either completed or stopped. Of these 134 projects, 92 projects were successfully completed meaning a success rate of 69%. This is close to the informal rule within the Ministry of Foreign Affairs that two thirds of the projects should be successfully completed. In terms of projects approved the efficiency is lower than expected: the average realisation between 2007 and 2009 is 84%

10.

9 Starting from 2004, there are still a number of approved projects that are on going. 10 240 projects were approved against a target of 110 (2007), 57 (2008) and 120 (2009).


Additionality is also part of the efficiency assessment. The EVD uses the following definition:

1) Could the project be financed by commercial funding, at the time of the PSOM application? 2) Is the project risky?

Based on this definition, the evaluation concludes that all projects were additional ex ante, for the simple reason that it is impossible to establish in retrospect whether the projects were too risky for commercial funding at the time of the application, in particular if the commercial funding was limited to bank finance. However, this definition of additionality is considered too limited by the IOB and the consultants. In our technical proposal we mentioned the following criteria for the ex post assessment of additionality:

1) Commercial viability/Payback period (when did you start making a profit)? 2) Was the project bankable? 3) Other sources of finance available? 4) Would entrepreneurs have carried out the project without PSOM?

With this broader definition, the availability of bank funding is only one of the criteria. Other sources of finance, including the applicant and/or recipient themselves, as well as counterfactual evidence are also considered. The main question is: Would the output and outcome of the project have been achieved without a PSOM/PSI subsidy? The results of this assessment are as follows: Table 25: Project scoring on additionality as found during field visits (in no. of projects)

Item (N=32) Additional

(+) Additional in specific aspects

(+/-) Not Additional

(-)

Ethiopia 2 3 1

Ghana 3 2

Indonesia 2 3

Mozambique 5

Suriname 3 2

Vietnam 2 3 1

All countries 17 13 2

According to this definition, 53% of the projects are fully additional, i.e. the companies, jobs, sales, outgrowers, knowledge effects and follow-up investments would not have been achieved without the PSOM subsidy. Another 41% is additional in specific aspects, based on the assessment that the investment may have taken place, but in a different timeframe, on a different scale or in a different way, possibly with less development effects. Finally in two cases – one in Ethiopia and the other in Vietnam – the evaluators concluded that there was no additionality. The main reasons why some projects are less or not additional are that:

Similar companies already existed so that risks were limited (e.g. LNB Surimix in Suriname and Tall flowers in Ethiopia).

Government policies were such that other companies were investing in similar ventures at the time, without being subsidised (e.g. ECS in Vietnam).

The applicant firm and/or the recipient firm had enough experience and/or funds to be able to take the (limited) risk. In all these cases the firms indicated themselves that they would have started the project without PSOM subsidy (e.g. Aloe Vera Indonesia).

The projects that were completed do not conform to the proposed project: the risky part of the proposal was not implemented, but the subsidy was not withdrawn (e.g. Lafto Roses and Ethio Vegfru in Ethiopia).

We conclude that nearly all the PSOM/PSI projects are at least partially additional, but only about half of the projects are fully additional. For the future, we recommend that a common definition of additionality should be agreed between Agentschap NL and the Ministry of Foreign Affairs including IOB to facilitate the assessment of additionality.


4.2.2 The effectiveness of PSOM projects

The effectiveness of a project is measured by the health of the company translated in employment, knowledge transfer, outgrowers and innovation. The table below provides an overview of the effectiveness of the projects as found during the field visits. Table 26: Project scoring on effectiveness as found during field visits (in no. of projects)

Item (N=32) Effective

(+) Effective

to a limited extent (+/-) Not Effective

(-)

Ethiopia 5 1

Ghana 1 3 1

Indonesia 2 1 2

Mozambique 3 1 1

Surinam 4 1

Vietnam 6


In two thirds (21) of the cases PSOM subsidies have been effective: the PSOM subsidy has facilitated the creation of healthy companies where jobs have been created, innovation has taken place and knowledge has been transferred. The less effective projects are those that are struggling to survive and where the expected jobs and growth have not (yet) been achieved. For the recently completed projects in Mozambique and Surinam the situation can therefore still improve in the coming years. The four projects that are rated ‘not effective’ are those that were in operation at completion of the project but that were no longer in operation when the field visit took place. The table below provides the figures found from the desk research on the 60 completed projects: Table 27: Project scoring on indicators: direct jobs, outgrowers and trainees

Project cost Direct jobs Outgrowers11

Trainees

Ethiopia €821,863 244 15,008 304

Ghana €828,447 39 1,041 65

Indonesia €978,593 51 608 56

Mozambique €838,935 39 3,714 34

Surinam €826,365 13 30 24

Vietnam €1,077,985 75 216 74

All countries €908,509 79 2,843 97

The table indicates that the Ethiopian projects were most successful as Ethiopia scores highest on each of the indicators jobs, outgrowers and trainees and has the lowest average project cost. Surinam, in turn, scores lowest on effectiveness because the human resource-related indicators such as direct jobs, outgrowers and trainees

12 are lowest.

If we compare the output and outcome figures for the entire population (n=197) we come to the following effectiveness indicators:

11 Average for those projects that have outgrowers. 12

Note: The average reported follow-up investment and sales in the tables slightly differ from the data reported in the earlier tables , due to the fact that in the latter tables only “complete” cases have been included for which proposed, realized, and expected data were available. In the earlier tables, all cases for which achieved figures have been included.


Table 28: Effectiveness indicators when applying sample findings to entire population

Indicator Findings 2009 Findings 2005 Target (since 2007)

Direct jobs created per project 81 147 100

Investment13

per job created €11.145 €9.218 N/A

Knowledge transfer (people trained) per project

237 71 None

Outgrowers (n=70 and n=69) 14

1.751 or 1.342 per project N/A N/A

PSOM subsidy disbursed per outgrower €293 or €389 per outgrower N/A N/A

As we can see, the number of jobs created (81) is lower than the number in 2005 and slightly below the target of 100 jobs per project. The average amount of investment per job created increased slightly since the 2005 evaluation. The PSOM subsidy (55% of the investment) spent per job created for the entire evaluation period was €6,130. In terms of effectiveness, agricultural projects perform best: they create more direct jobs and trainees and are the only sector that develops outgrowers. As 46% of PSOM-approved projects are in the agricultural sector, these clearly contribute to the enhanced effectiveness of the programme.

4.2.3 The relevance of PSOM projects

In this section we assess the relationship between the outcome and impact. We assess the follow-up investments and sales that have been achieved after project completion and assess the contribution of the project to sector development and chain effects. The table below provides an overview of the effectiveness of the projects, as found during the field visits. Table 29: Overview of the effectiveness of the projects as found during field visits (in no. of projects)

Item (N=32) Relevant

(+) Relevant

to a limited extent (+/-) Not Relevant

(-)

Ethiopia 5 1

Ghana 2 2 1

Indonesia 2 1 2

Mozambique 3 1 1

Suriname 4 1

Vietnam 6


Over two thirds of the PSOM subsidised projects have been relevant: follow-up investments have been made and sales are good and growing. These two indicators tell us that these businesses are sustainable and continue to create jobs and add value to society (through tax payment and foreign exchange generation). The SME linkages and catalytic effects found during the field visits were limited. One reason is that most PSOM projects are re-exporter models where limited linkages are made. In sectors where more PSOM projects have been financed and where the government and/or the embassies have played an active role in bringing business climate issues to the table (floriculture in Ethiopia, harbour infrastructure in Ghana) the effects of PSOM projects on that specific sector at large have been important.

13 Investment is calculated on the basis of an average of 55% PSOM subsidy per total investment amount 14 The outlier in Ethiopia with 30,000 outgrowers has been excluded in the second figure.


In figures the relevance found in the desk study (n=60) is as follows: Table 30: Project relevance as found during desk study

Item (N=60) Project cost Follow-up investment Sales

All countries €908,509 €606,145 €787,724

Ethiopia €821,863 €1,167,778 €1,489,538

Ghana €828,447 €599,600 €117,227

Indonesia €978,593 €566,958 €496,750

Mozambique €838,935 €53,575 €142,664

Suriname €826,365 €676,250 €1,592,667

Vietnam €1,077,985 €485,625 €1,354,785

The table indicates that the Ethiopian projects were again most successful as Ethiopia scores highest on each of the indicators and has the lowest average project cost. Because the projects in Mozambique have been completed recently, they have generated less follow-up investment. The average multiplier (follow-up investment per €PSOM subsidy) is 1.22 but with a large variation between 0.07 and 28. This is considerably higher than the 0.55 found during the 2005 evaluation. Details can be found in the project datasheets. In terms of sales achieved, on average, a project cost of €908,509 generated a follow-up investment of €606,145, and sales results of €787,724. Overall, the "Industry"-sector is most successful and agriculture comes in second place. The energy/environment projects are most successful when taking the realization of follow-up investment into account. The results in terms of corporate social responsibility are mixed because this is a very recent topic in the appraisal of PSOM projects. In the past it was important that some kind of corporate charity was undertaken and that laws and treaties were respected. Corporate responsibility in main business processes became important in a later stage. The EVD, through PSOM, has played an important role in the promotion of certifications and standards (social, health, and agronomic practices), code of conducts and good practices in different sectors in varying countries. As these efforts are not systematised it is difficult to provide quantitative information on the results but the datasheets provide information on these realizations. For the applicant firms the role of the EVD in encouraging corporate responsibility has often been the main driver for adopting better business practices. In terms of gender we can conclude that job creation has been an important driver for empowerment of young women in Ethiopia and Vietnam. Female management has been rare and the fact that none of the PSOM applicants is female means that in the promotion of PSOM there might be room for improvement in attracting female entrepreneurs.


5 EVALUATION OF THE MMF

5.1 The background to the Matchmaking Facility

The Matchmaking Facility (MMF) started in 2006 with a pilot programme (in eight countries

15 plus the

Balkans) and continued after a positive evaluation as a regular programme. The programme originated from the need felt by embassies and PSOM for matchmaking for entrepreneurs from developing countriese. The embassies wanted to have a facility to which they could refer inquiring entrepreneurs and the EVD wanted to increase the number of PSOM applications.

The target group of the MMF is formed by entrepreneurs in developing countries with a concrete investment or trade idea who are searching for a Dutch investment or trade partner. The objective of the MMF is to match an entrepreneur in a developing country with a Dutch firm in a sustainable relationship. The relationship is established in terms of a joint investment in the developing country (not necessarily a PSI project) or in terms of developing a trade relationship. MMF intends to collaborate with PUM, CBI, BiD Network and other programmes where possible. In 2007 MMF worked in 39 countries. In 2008, this number increased to 43, because countries formerly covered by the IntEnt Foundation were added to the list. In 2009 the total number of countries where MMF was active increased to 44. Some countries were removed from the list by the Minister (Ecuador, El Salvador and Sri Lanka), while others were added (Kosovo, Namibia, Sierra Leone, Senegal) because the embassy agreed to participate or because of a decision by the Minister.

The matchmaking consists of two main services: 1) Identifying Dutch partners based on the needs of the applicant company and 2) facilitating the collaboration by a consultant using a voucher worth €5,000. The company in the

developing country can buy certain, predetermined services delivered by a consultant using this voucher.

5.2 Findings on inputs, outputs and outcome

5.2.1 Inputs

In this section we describe the process, tools and systems developed by the MMF unit to manage the matchmaking. We also identify the budget and costs related to the facility.

How does it work? The programme consists of a phased approach involving the following steps:

1) An entrepreneur in a developing country fills a form to request matchmaking services 2) The embassy assists, screens and interviews the applicant and provides advice on the eligibility 3) The MMF team screens the file and decides whether the applicant is eligible on the basis of a set

of criteria 4) If the application is accepted, the MMF unit starts the search for partners in the Netherlands and

contacts them 5) Once potential business partners have been identified, a voucher letter is issued to the applicant 6) The applicant firm can choose with which consultant it wants to work, in some cases MMF sends

a consultant list to choose from; in other cases the MMF unit recommends a specific consultant who is already involved in the process of matchmaking.

The actual matchmaking activity (no.4) was initially implemented by the MMF unit. They publish the names of the positively screened applicant firms on the website of MMF in order for Dutch companies to react on the requests for partnership. As the programme evolved it turned out that consultants were more

15 Burkina Faso, Colombia, Egypt, Kenya, Mali, Pakistan, Peru, Rwanda (countries of limited interest to Dutch enterprises or with no PSOM applications or with no local joint venture partners in PSOM applications).


efficient in matchmaking then the MMF unit, thanks to their network in the Netherlands. Little by little, the matchmaking activity shifted from the MMF unit to the consultants. Since 2010, the MMF unit has published names of positively screened applicant firms on line, after which the consultants can start to identify three potential Dutch partner businesses. The consultant who identifies three suitable matches and communicates these to MMF gets the voucher. It is estimated by MMF and consultants that one third of the matches are found by MMF and two thirds by consultants. This division (33%/66%) is confirmed by the internet survey. The voucher, which is worth €5,000, provides access to the following consultancy services:

1) Preparing a quick scan on the country, sector and company of the applicant 2) Preparing and facilitating a visit by the applicant firm to potential business partners in the

Netherlands including a report on this visit 3) Preparing a follow-up plan with commitment form applicant and Dutch firm 4) Producing an evaluation report six months after the visit has taken place.

MMF staff, embassies The MMF unit, which is housed in the EVD, consists of four people: a manager, two programme advisors and a financial/administrative member of staff. The profile of the programme advisors includes an academic level with experience of international cooperation and good language and communication skills. For the work performed until 2009 this profile has suited the work of a matchmaker. However, with the transfer of the matchmaking activities to the consultants this profile might need to be reconsidered. The programme management and monitoring of consultants and pipeline development through the embassies or other channels seems to have become the main tasks of the MMF unit. The embassies differ in their capacity and motivation to promote MMF and to assist in receiving and screening the applicants. The embassies in Central and Eastern Europe have been much more active in the promotion of MMF than embassies in other countries. As MMF is dependent on the pipeline developed by the embassies, most applications come from these countries. A lack of capacity or the existence of other priority areas, were mentioned by the embassies visited (in Ethiopia, Ghana and Vietnam) as reasons for underutilisation of the MMF. The quality of services by staff involved in MMF was rated as satisfactory by the respondents to the internet survey: the embassy staff were rated 3.27 (on a scale of 5), the EVD staff 3.11 and the consultants 3.08. Consultants and partners In 2007, the consultants eligible for implementing voucher trajectory were invited to submit their interest and bid according to European tender procedures. A total of 42 consultants expressed an interest. From them, 31 were contracted and grouped per country for which they would be eligible as implementer. This eligibility per country has been banned as per 1 January 2010. As of today a total of 28 consultants are included in the MMF database. A total of 21 consultants have provided services under a voucher trajectory. Half of the trajectories are implemented by three consultants: Kubita (21%), Alliance Plus (13%) and KB Agro (14%). These consultants work with sector specialists or other networks to do the matchmaking. The motivation to join MMF for the consultants is that they expect to get follow-up work from their matchmaking activities. Follow-up activities might be training, technical assistance or project management when a PSI request is submitted. For Kubita the results have been quite disappointing: no follow-up work came as a result of the MMF. Alliance Plus focuses more on PSI applications and for them MMF is a useful tool in their pipeline development for PSI. PUM’s core product is providing technical advisory services to companies in developing countries, including matching companies with the EU or Dutch market. The collaboration with PUM has not yet been successful; very few applications came from the ‘PUM countries’ and as most PUM consultants prefer to work in developing countries and do not always have an active network in the Netherlands, their contribution to MMF is limited. A new pilot to improve this collaboration starts in 2010.


CBI’s core product is to link businesses from developing countries to the European market. This support also includes matchmaking with Dutch trade partners. CBI was therefore also interested in implementing the MMF programme, but it was EVD that was contracted by DDE. The collaboration between MMF and CBI started to take off in 2009 with the promotion of MMF for CBI clients and exchange of client information and services requests. It is too early to present the results of this collaboration. MMF budgets and costs The costs for MMF are twofold: implementation costs (staff of EVD) and programme costs (payment for vouchers). - Budgets The 2007 approved budgets for the MMF regular activities were made, but in July 2008 and January 2009, new adjusted budgets were proposed because the results were not in line with the 2007 budget . In 2008 management and network time was added to the budget and the hours for screening increased from 1 to 4 hours per applications. The budget for programme costs was reduced because fewer applications than projected were received. The new budgets are as follows: Table 31: Approved MMF budgets for the years 2007, 2008 and 2009

New approved budgets Y 2007 Y 2008 Y 2009 Total

Implementation costs €165.582 €340.008 €380.035 €885.625

Programme costs €165.000 €375.000 €479.000 €1.019.000

Total budget €330.582 €715.008 €859.035 €1.904.625

The budget in hours for the MMF team was as follows: Table 32: Budget in hours for the MMF team at EVD per average month in 2007, 2008 and 2009

Activity Budget 05/2007 Budget 07/2008 Budget 2009

Screening applicants 1 hour per unit 4 hours 3 hours

Matchmaking 16 hours per unit 16 hours 16 hours

Monitoring: 4 hours per unit 4 hours 4 hours

Management 100 hours 240 hours ??

Networking - 160 hours ??

- Realisation The actual costs for the MMF in the period 2007 - 2009 were as follows (based on annual reports): Table 33: Actual costs for the MMF in 2007, 2008 and 2009

Costs 2007 2008 2009 Total

Implementation costs €108.662 €358.294 €399.837 €866.793

Programme costs disbursed 0 €57.000 €343.000 €400.000

Total €108.662 €417.302 €742.837 €1.266.793

Commitments made €55.500 €216.500 €551.000 €823.000

Systems, formats and tools used by MMF unit For the entire process of MMF a set of document formats, forms and standard contracts exist. The forms and letters are comprehensive and formal. The length and content of the documents might make it difficult for some applicants to understand its precise content. For its matchmaking activity the MMF unit has several sources of information: the network of the EVD (PSI applicants, companies that request information, participants to other programmes), a rolodex address book, contacts with branch organisations and internet. There is no electronic database of the contacts that were found in earlier matchmaking efforts although developing an IT application was planned for 2009. This has not yet materialised. Systematic recording of the contact details of potential partners is a task for the consultants but is not done at the level of the MMF unit. As matchmaking has mostly been done by consultants since 2010, further systematisation of knowledge at EVD level is now less important.


The files of the products of the voucher trajectories are very accessible, both on paper and electronically. However, the information is not gathered systematically in the voucher products (quick scans, for example, contain sector information that could be useful to other companies) because this information is not in line with the type of information that is provided on the EVD websites. For monitoring the applications, disbursements and results, two Excel sheets are used where all information is gathered. Unfortunately, the format of these data does not allow for quick production of management information related to the programme. However, with minor adaptations to the spreadsheet it can be used for analysis and management purposes. The use of pivot tables could enhance the usefulness of the data collection process.

5.2.2 The output of MMF

The projected output of MMF is that:

A match between entrepreneurs from developing countries and one or more Dutch companies is established

Both parties show a willingness to collaborate in the short term in, for example, knowledge transfer, trade and exchange visits (not necessarily make an investment).

Quantity of outputs The proposed outputs for the MMF programme are part of the offer that the EVD prepares for the service contract with DDE. The proposed outputs in these offers are presented below: Table 34: Proposed outputs for the MMF programme in 2007, 2008 and 2009

Regular MMF activity: Offer 07/2007 Offer 05/2008 Offer 01/2009

Number of tenders 1 in 2007, 3 in 2008, 3 in 2009

3 3

Applications per year 600 (200/round) 300 (100/round) 300 (100/round)

Positively screened companies 200 150 150

Successful matches 100 75 75

Trade missions: 1 5

- Total visiting companies N/A N/A 100

- Total vouchers N/A N/A 35

- Applications The main source of information for the applicants is the Dutch Embassy: over 50% of the respondents to the internet survey heard from MMF through the Dutch embassies. Most applicants also took other initiatives to identify business partners; internet searches (57%) and contacts with branch and sector organisation (31%) were mentioned by the respondents. The motivation to apply for the 97 applicants who responded to the internet survey was:

64% to identify a business partner for developing a project

54% to identify an investor

37% to find a market for their product and

20% to identify a supplier of equipment.

These motivations are very different in nature: some applicants want to sell their products in the Netherlands, while others want to come to the Netherlands to buy equipment. These different motivations determine the character of the matchmaking and visit programme (a ‘sales’ or a ‘buying’ visit). The output realisation in terms of received MMF applications was as follows: Table 35: MMF applications received in 2007, 2008 and 2009

Round 2007 2008 2009 Total

Round 1 52 92 58 202

Round 2 79 52 131

Round 3 + CBI 65 59 122

Total 52 236 169 457


The average number of applications per round was 65. This is considerably lower than the planned 200 (in 2007) or 100 (in 2008 and 2009). The trend in number of application was downwards in 2008 but became stable in 2009 at close to 60 per round. The explanation of the decrease in 2008 might be that the ‘pool’ of interested companies from the start of MMF has been depleted. Another explanation is that the embassies have become stricter in pre-screening after having received feedback from the MMF unit. This means that not all interested companies are supported in submitting an application. This is confirmed by the increasing percentage of applicants who are screened positively (48% in 2007 and 64% in 2009). A final explanation might be a decreasing motivation at the level of the embassy, which came out in the embassies interviewed during the field visit (Ghana, Ethiopia and Vietnam). These interviewees found the matchmaking process too cumbersome and time consuming. The nine embassies interviewed by MMF in 2009 indicated that it is difficult to reach out to the right type of companies for the facility. - Screening From the 457 applicants who requested support, 277 have been screened positively. This is 60% and similar to the rate found during the pilot phase (59% positively screened). This 60% is considerably more than the 33% (2007) or 50% (2008 and 2009) which were mentioned in the EVD offer to DDE. This might be explained by the fact that the pre-screening by the embassies is better than anticipated. - Matches The MMF programme has found matches for 174 of the 277 positively screened companies. According to the respondents of the internet survey, two thirds of the matches were proposed by a consultant and one third by the MMF unit. This is confirmed by interviews with consultants and the MMF unit. 75% of the respondents found the number of matches sufficient. The quality of the matches was perfect for 15% of the respondents and partially satisfactory for the remainder of the respondents. - Vouchers From the 277 positively screened companies a total of 150 have received a voucher as of the end of 2009 (54%). This number can still increase as for 51 applicants the matchmaking is ongoing. This is higher than the 30% found in the pilot phase. As of end 2009, of the 150 voucher trajectories 35 have been finalised and 115 are ongoing. In summary, the output in terms of number of applicants is less than expected but the outputs in terms of screening and matchmaking are better than expected. This means that the pipeline development turned out to be more difficult than planned but that the screening process and matchmaking are more effective than planned. Table 36: MMF applicants per phase and per region for the period 2007-2009

Neg. screening Voucher Ongoing search Other Total

Africa 36 44 7 16 103

Asia & Mid East 38 28 5 14 85

C&E Europe 88 52 27 27 194

Latin America 18 26 12 19 75

Total 180 150 51 76 457

In the category ‘Other’, different reasons for non-continuation of the MMF trajectory are grouped: either no partners have been found (52) or the applicant company has withdrawn (4) or withdrawn after a voucher letter was sent (20). Regional differences in the application process are:

Most applications and vouchers were from C&E Europe (42% and 35%, respectively) and from Africa (22.5% and 29%, respectively).

45% of the applications from C&E Europe companies were negatively screened as against 20% of the Latin American applications.


The highest percentages of vouchers were submitted to African applicants (43% of applications lead to a voucher) and the lowest to C&E European applicants (27% of applications lead to a voucher).

In terms of sectors, 25% of the applications relate to agricultural projects while 40% of the vouchers relate to agricultural projects. So the matchmaking is more successful for agricultural projects than for other sectors. The remaining large sector is food and food stimulants, with 14% of the applications and 12% of the vouchers. For the other sectors the numbers are too small to assess whether MMF is more or less successful in matchmaking these sectors. Fund commitment and disbursement The programme costs in MMF consist of the €5,000 voucher trajectory that is disbursed in four tranches (3x €1,500 and 1 x €500). In terms of funds committed and disbursed the situation as per end 2009 is as follows: Table 37: MMF vouchers and value of vouchers for the period 2007-2009

Category No. of vouchers Amount in Euro

Committed voucher funds 150 €750.000

Disbursed voucher funds: 109 €400.500

- Closed & fully disbursed 30 €150.000

- Closed & partially disbursed 3 €6.000

- Closed & voucher cancelled 12 €21.000

- Ongoing vouchers 64 €223.500

Trade missions In 2008, two matchmaking events for incoming trade missions from Mozambique and Afghanistan were organised and in 2009 the same was done for a mission from Yemen. This means that half of the planned six missions have been achieved. As it was not clear who should be financing the trade missions (Min. of economic affairs or Min. of foreign affairs), this activity came to a standstill. During the 2008 trade mission from Mozambique, five Mozambican companies were matched with 20 Dutch companies. During the trade mission from Afghanistan the EVD matched 17 Afghan companies with more than 40 Dutch companies. One trade relationship resulted from the Afghan mission, other results are yet unknown. During the 2009 mission from Yemen, 57 Dutch companies were present as potential partners for the 15 Yemeni businesses. No MMF vouchers were issued following the trade missions. The consultant who did the follow-up of the trade mission is not expecting any concrete results (PSI or trade) from the trade mission. Quality of voucher trajectories Of the 33 completed voucher trajectories, 11 files have been randomly selected for a qualitative assessment by the consultant (see Annex ….for sample). The products of the voucher trajectory are:

1) Quick scan 2) Visit programme report 3) Follow-up plan 4) Evaluation

- Quick scan The content of the quick scan varies according to the consultants who prepared it. Some quick scans focus on the mutual expectations and a description of the applicant company, while others focus more on the country, economic situation and sector in the country of the applicant firm. The earlier quick scan had a high copy & paste character, but with the shift to a more tailor made ‘getting to know each other’ document this has improved. 40% of the 42 respondents in the internet survey found that the quick scan presented their companies and the sector they work in perfectly. Another 40% found the quick scan useful for the potential Dutch partner. The survey revealed that the quick scan is not used for other purposes then for submission to the Dutch partners.


- The visit programme The screened visit programmes were all very comprehensive, informative and easy to read. A total of 21 internet survey respondents came to the Netherlands. They spent on average €3,200 for such visits. More than half of the respondents would not have undertaken the visit without MMF support. The purpose of the report is proof that the field visit has taken place. - Follow-up report This report summarises the discussions held during the field visit and points out the commitments of both companies. The consultant drafts the report and gets approval of both the applicant and the Dutch firm. The purpose of the document is to provide an overview of follow-up actions for both parties in case cooperation seems possible. It is not very clear why this document is necessary: if the companies are interested in further collaboration they should have taken notes themselves. It is also possible that the applicant firm has been talking to competing companies, in which case sharing the entire report with all visited companies is not desirable. The report might be useful as a baseline situation for further monitoring of results, but that would require systematic data collection and processing. - Evaluation The evaluation report is useful for monitoring the progress and results of the voucher trajectory. It therefore mainly serves the purpose of assessing the accountability of the MMF programme. The countersigning of this report by all companies involved is again cumbersome: only if continued collaboration between the applicant and Dutch company is anticipated can such endorsement be reasonably asked from them. The analysis of the evaluation reports indicates that in many cases it is considered too early to confirm whether the voucher trajectory has been successful. It seems that the 12- month period between the visit and the evaluation that was proposed earlier should be reconsidered.

5.2.3 Outcome

The projected outcome of MMF is that within a year after the match, the matched companies should have established a collaboration which can lead to a common investment in a developing country. This common investment might be in the form of a PSI project but not necessarily. During the pilot phase (from the beginning of 2006 to April 2007) five of the 18 voucher trajectories (27%) led to intentional or actual collaboration. As a result of the 2007-2009 matchmaking activities, nine companies have applied for PSI up to February 2010. One company from Egypt has applied for PSI following MMF efforts but without using a voucher. Two companies applied twice for a PSI subsidy, so the total number of PSI applications was 12 in February 2010. Table 38: PSI applications made by MMF supported applicants in 2007, 2008 and 2009

Continent MMF applicants 2007 MMF applicants 2008 MMF applicants 2009 Total

Africa 2 2 4

C&E Europe 6 6

Latin America 1 1

Asia & Mid East 1 1

Total PSI applications 0 9 3 12

Eight of the 12 PSI applications have been approved: four in Africa, one in Latin America, one in Middle East and two in Central and Eastern Europe.


Other information

16 on achieved outcomes is as follows:

The annual report of 2009 states that six businesses have established a trading relation. So that would make a total of 15 successful (9 PSI + 6 trading) outcomes. In percentage this means that 43 % of the finalised voucher trajectories have led to a positive outcome and that 3% of the applicants have obtained a positive outcome up to now. These percentages may of course increase when more voucher trajectories are finalised.

From the 97 respondents in the internet survey (applicants to MMF), 4 entrepreneurs mentioned that a concrete business relationship or sales transaction resulted from their participation in the MMF programme.

Of the 21 respondents in the internet survey that visited the Netherlands, 15 made business contacts, three got access to the EU market, one increased its sale and one bought equipment from the Netherlands.

Another, not measurable outcome is the learning effect of the visit programmes for the participating companies mentioned by the consultants and the MMF unit staff.

5.3 Analysis of the efficiency and effectiveness of the MMF programme

5.3.1 Efficiency

In order to evaluate the efficiency of MMF we reviewed:

Budget against realisation (in money and time allocation)

Ratio between implementation cost and programme cost

The cost per output (vouchers) Budget against actual In the following table we present the realisation rate of the MMF programme outputs. The actual outputs are compared with the projected outputs. Table 39: MMF actual outputs compared to DDE estimates 2007, 2008 and 2009

Output DDE estimate Actual % realised

Applications 2007 200 52 26%



Positively screened 2007 67 25 37%



Successful Match making 2007 33 13 39%



Vouchers 2007 33 9 27%

Vouchers 2008 75 68 91%

Vouchers 2009 75 73 97%

From this table it can be seen that in terms of meeting the targets, the MMF programme has improved from 2007 to 2008 and slightly deteriorated from 2008 until 2009. The improvement between 2007 and 2008 is largely thanks to the adjusted targets and because the programme took time to take off. Most projects in 2008 are the result of activities conducted in 2007. As said earlier in 2009 the inflow of applications declined and this was apparently not anticipated when the planning was made.

16 The MMF unit is not paid to collect outcome information, so it is not sure whether all result are captured here.


If we relate the outputs to the budgeted hours to be used for the activities we see the following results: Table 40: MMF output related to budgeted hours for the period 2007-2009

Activity No. of units

Budgeted hours per activity

Total hours in theory

Hours reported in annual report for regular

matchmaking

Screening of applicants 457 4 1.828

Matchmaking 277 16 4.432

Voucher monitoring 150 4 600

Management and networking 3 years 900 900

Total 7,760 1,500 (2007) + 4,231 (2008) + 4,858 (2009) = 10,589

This table does not take into account the fact that 115 voucher trajectories have not yet been finalised. i.e. the four monitoring hours have not yet been fully used. Another issue is that the MMF unit does one third of the matchmaking and that the other two thirds are done without costs by consultants. This would reduce the theoretical 4,432 hours to about 1,500 hours spent on matchmaking. From the table we can conclude that the MMF staff does use hours for other activities than those budgeted in the offer submitted to DDE. Ratio implementation/programme cost In this section we compare the proposed and actual programme and implementation costs as well as the planned and actual costs per unit spent for programme purposes. Table 41: MMF programme costs and implementation costs estimated and realised

Item Estimate made

by DDE Realised % Realised

Programme cost 2007 €165.000 0

Implementation cost 2007 €165.582 €108.662 66%

Ratio 2007 1 --

Programme cost 2008 €375.000 €57.000 15%


Ratio 2008 0.91 6.29 693%

Programme cost 2009 €479.000 €343.000 72%


Ratio 2009 0.79 1.17 147%

Total ratio 2007-2009 0.87 2.17 249%

From the table we can conclude that MMF is a labour-intensive and slow disbursing programme. Over the entire evaluation period the cost of spending 1€in vouchers is €2.17 instead of the proposed €0.87. The programme costs show an under spending and the implementation costs a slight overspending in 2008 and 2009 compared to the budgets presented to DDE. The deviation between the planned and realised figures is decreasing over time. The MMF programme shows an improving ability to estimate its efficiency. As the MMF unit provided until at least end of 2009, the matchmaking services, the ratio cannot be compared with other subsidy management contracts implemented by the EVD. The output of the MMF unit is not only its disbursement of funds but also the number of matches realised by the staff. Cost per output The efficiency can also be measured by comparing the number of vouchers issued and the implementation costs as of the end of 2009. The ratio tells us the implementation cost per voucher issued.


Table 42: Vouchers issued and implementation costs estimated and realised

Estimate made

by DDE Actual % Realised

No. of vouchers issued in 2007 33 9 27%


Ratio 2007 5.018 12.073



Ratio 2008 4.533 5.269



Ratio 2009 5.067 5.477

Total average ratio 2007-2009 4.839 5.779 163%

It can be seen that the planned average implementation cost per voucher during the entire period covered by the evaluation was €4,839 per voucher and that the actual average implementation cost per voucher is €5,779. The gap between the estimated and actual cost per voucher issued is becoming smaller each year. The activities covered by the voucher do not include matchmaking and monitoring results after the follow-up phase has been completed. Until now consultants have delivered these services for free to the MMF unit, this has enhanced the efficiency. It is questionable if they are willing to continue to do so in the future. If the MMF unit is now going to spend most of its time on promotion (via embassies and other organisations), programme management and monitoring, it should make the costs per unit disbursed comparable to other programmes. Thereby it is important to realise that the disbursement of (only) €5,000 is very labour-intensive, hence the process will be expensive if calculated per unit spent. In summary, we conclude that the programme costs have been overestimated while the implementation costs have been underestimated. Said otherwise DDE has received fewer outputs for more inputs. The efficiency is therefore lower than expected. Additionality Under efficiency we also look at the additionality of the programme. If we compare MMF to similar programmes we see that matchmaking services are also provided by PUM, CBI, the Dutch embassies and private consultants. CBI focuses on access to the European markets and PUM on knowledge transfer. The Dutch embassies and private consultants also do matchmaking as a response to a demand without MMF support. According to the MMF staff and interviewed consultants, the MMF niche in terms of target group is the companies that are not advanced enough for the CBI programme and that are not in need of PUM services. In order to provide statements on additionality it would be worth studying the different target groups of CBI and MMF in more detail because the desired outcome for the applicant companies to both programmes is quite similar and there might overlap: in both programmes companies from developing countries seek to develop their companies by establishing a business relationship with European companies and markets. If additionality is assessed as the counterfactual situation, the internet survey revealed that 12 of the 21 companies that conducted a visit to the Netherlands would not have made the visit without MMF support. Another finding was that 60% of the applicant firms were already doing business internationally when they applied to MMF.

5.3.2 Effectiveness

Although the MMF unit has not budgeted time to collect information on the outcome of its work, MMF staff with the help of consultants do follow up the results obtained after the voucher trajectory. If we compare the ratio between finalised voucher trajectories and PSI applications, called the ‘PSI success rate’, we get the following result:


Table 43: PSI success rate: Finalised voucher trajectories resulting in PSI applications Year Vouchers provided Finalised vouchers PSI applications PSI success rate

2007 MMF applicants 9 9 0 0%

2008 MMF applicants 68 24 9 37.5%

2009 MMF applicants 73 2 2 100%

Total 150 35 11 + 1 without voucher

31%

When we compare the PSI approval rate of MMF applicants with the average approval rate of PSOM/PSI applications, we see that for MMF 67% of the applications have been approved (8 out of 12) compared to an PSOM/PSI average of 50%. If we compare the inputs with the outcome we see that up to end 2009 for each of the eight approved PSI proposals which came through MMF, €108,349 (€866,973 divided by 8) was spent on implementation cost and €50,000 (€400,500 divided by 8) on programme costs. This would mean €168.349 per successful PSI application. It is, however, expected that more of the 115 ongoing voucher trajectories will yield a PSI application in the coming period. In order to estimate the costs if all vouchers have the same success rate as the first 35 completed vouchers, we consider the average implementation cost per voucher euro. This cost is €2.17. With the average PSI success rate of 31% (one third of trajectories lead to PSI) then the total MMF costs of having a successful PSI application will amount to €51.130. This calculation excludes the outcome other than PSI (the six business relationships mentioned earlier in the outcome). It is difficult to assess the value for money generated by the MMF. One way of comparing is to look at another way of developing PSI proposals (i.e. to make use of the services of a consultant without involving MMF). In that case, the work of consultants in relation to PSI contracts involves matchmaking, proposal writing and monitoring the implementation of the PSI project. Most of the consultants work on a ‘no cure no pay’ basis. If a PSI proposal is won, the consultant receives on average 10% of the approved budget as a fee. On average this means that the consultants receive €50,000 per successful proposal. This fee is not only for preparing the proposal but also for writing reports during implementation, paperwork and, in some cases, technical assistance or training.


6 CONCLUSIONS AND ISSUES FOR ATTENTION The overall results of the PSOM/PSI programme – based on the 32 projects included in the field visits – suggest that the programme has been effective in generating jobs and follow-up investments after project completion, albeit with marked differences between the six selected countries. The most successful countries in this respect are Ethiopia and Vietnam. There are strong variations between countries as well as between projects in the same country, which makes the average figures found in the field study difficult to use. The actual number of jobs varies between five and 1,000 and the actual multiplier for follow-up investments between 0 and 20. If we take into account 197 of the 205 projects that have been completed worldwide, the average number of jobs that have been created per project is 81, with an average subsidy contribution of €6.130 per job. Based on the sample of 60 completed projects, the average follow-up investment at completion was €576,526, which amounts to a multiplier effect of 1.2 with respect to the PSOM subsidy. However, it should be noted that many of these projects have only been finalised quite recently so that more follow-up investments can be expected as we have seen in the field study. Most (57%) of the finalised projects still exist five years later. In terms of effectiveness the conclusion is that two thirds of the visited projects are effective because they have created a significant number of jobs with a limited subsidy contribution per job. Innovation has also been achieved, mainly through the introducing hardware, which represents 67% of the PSOM subsidy. Significant knowledge transfer has taken place: 96 people per project with a subsidy contribution of €5,158 per trainee. Outgrowers are an important outcome in agricultural projects. The numbers vary between ten and 10,000 per project, although the definition of outgrower needs refinement in order to be able to assess the PSOM contribution to outgrower schemes. In terms of relevance the conclusion is that two thirds of the projects are relevant based on the amount of follow-up investments, growing sales and ongoing job creation. We estimate on the basis of the field visits that 57% of all approved and completed projects still exist seven to ten years after the date of approval. On average, the companies that still exist and were visited now have 30% more jobs than when the project was completed. The catalytic effects on specific sectors have been limited, except for in the horticultural sector in Ethiopia and in some isolated cases in other countries. Neither the applicants nor the EVD are responsible for these effects, hence no concerted efforts are made to achieve effects beyond the individual projects. Agentschap NL, embassies and local private sector stakeholders could play a role here as has been shown in Ethiopia. In most countries the involvement of the local stakeholders in PSOM/PSI is currently very limited and the involvement of embassies strongly depends on the country. The implementation efficiency of the EVD is decreasing, as a result of increasing costs. Due to changes in workload and other factors, the implementation costs have increased significantly in the past three years. This increase is caused by higher overhead and higher fees. Part of the overhead increase is explained by higher legal costs in 2009. Overall, the percentage of overhead in the total cost has become higher in the last five years, as PSOM became a much larger programme. Thus, we can conclude that the economies of scale that were to be expected have not materialised.

The selection process is well organised and effectively implemented. However, the rising costs of the service and the significant TA cost related to project development reduce the efficiency. The process is effective in terms of generating a portfolio of successfully completed projects, but less effective in terms of utilising available budgets and maximising development effects. The number of new applications is low and approval rates are increasing, limiting the discretionary space to select the most relevant projects. To be able to choose from more applications, EVD’s communication strategy has to change. So far the rising cost for communication has not resulted in substantially more interest from the business community.

Project management is efficient; the cost of management has decreased over time and established procedures are implemented within a limited number of days. Project management is partially effective.


We consider that the EVD’s monitoring visits and tools are an effective way of ensuring that the PSOM subsidy is spent according to the rules and the contract. However, we believe that the current monitoring system is not effective for assessing the results in terms of outcome and impact (employment, turnover, follow-up investments or jobs) as most of the information comes from the applicants. As a result subsidy amounts are rarely withheld or reduced, despite significant deviations in the actual project implementation. This suggests that the established targets are too soft or that monitoring is ineffective, or both.

We consider that the legal procedures for dealing with complaints from companies are efficient and effective. No claim has yet been acknowledged and the costs are reasonable compared to the overall implementation cost. The administrative system is inefficient and only moderately effective. The management information system used for monitoring the programme is outdated and on the whole employees are unsatisfied with the system, as well as the hardware. However, the system does generate the necessary information and does not hamper the implementation of the programme.

As to the efficiency of the PSOM-financed projects, we observe that in the past three years the number of approvals was 84% of the number anticipated in the ‘Beleidsmemorandum’. Over the entire programme period, 85% of the available budget was committed. The success rate of 69% of approved projects that operate at least until project completion meets the minimum target of 66%. All PSOM projects were ex-ante additional, based on the EVD definition of additionality as “risky and not commercially fundable”. In our view, 53% of the projects were definitely additional, whereas in the other half of the cases it is quite likely that some outputs and outcomes would also have been achieved without the PSOM subsidy.

EVD-related issues that require attention include:

1) With respect to efficiency, we suggest a review of the overheads and general costs of the EVD, which have both increased significantly over the last five years - in absolute and in relative terms - and are now higher than at the start of the programme (as a percentage of overall programme costs).

2) Within the PSI unit, some improvements can be made to increase efficiency. These include the early rejection of applications that are eligible but nonetheless unlikely to qualify and reviewing the distribution of tasks between APSI and CLs when ensuring quality and consistency.

3) With respect to effectiveness of the selection, we believe the main challenge for the EVD should be to widen the choice of eligible and relevant applications. In our view, the assessment of the development relevance is not as rigorous as the assessment of the partners and the business plans. With a wider choice, the existing ranking on development relevance could be applied more rigorously, which could lead to the selected projects having a greater impact.

4) To achieve a wider outreach of the programme, the EVD needs to develop structural cooperation with other organisations working with potential applicants and recipients, both in the Netherlands (e.g. CBI, PUM or AECF) and in the recipient countries. We believe that so far the efforts in this sense have been mainly limited to dialogue and establishing cooperation agreements, but have not resulted in joint action and tangible results.

5) Monitoring results relies too much on the principle of “trust”, which is fine when projects go (reasonably) well, but not when there are problems which affect the key results, such as employment and turnover targets, upon which the final payments depend. Here we suggest two types of monitoring and evaluation:

Reviewing whether the agreed targets reflect the goals set out in the business plans, and

Requiring companies that have received PSI subsidies to submit an audited account for the final year of the project, so that results can be verified independently. PAs cannot carry out an independent verification because of their role as advisors to projects.

6) Finally, we recommend that the EVD modernises its administration system in order to improve efficiency and free up resources for the staff to focus on core tasks.


PSOM/PSI project-related issues that require further attention include:

1) Currently all eligible and viable PSI applications are approved. An enhanced marketing and pipeline development effort or a reduction in the budget allocation for PSI would allow approval of projects to depend on the development ranking of the applications.

2) In order to reduce the energy and funds spent on issues related to the creation of a joint venture, this requirement could be dropped and the choice as to whether to involve a local entrepreneur in establishing the new venture could be left to the applicant firm.

3) The results that lead to disbursement of PSI subsidies should include development-related indicators in order to give Agentschap NL the means to enforce the realisation of projected development effects.

4) A refinement in the definition of innovation, trainee, outgrowers, additionality and market distortion as described in this report and a common understanding between Agentschap NL and the Ministry of Foreign Affairs about these criteria would make the assessment of PSI results more objective.

5) For a proper assessment of the market distortion, innovation and additionality of PSI applications, sufficient time and expertise needs to be made available at the country level. We therefore suggest that PSI only operates in countries where embassies can commit the required time and capacity to the programme and where the embassy involvement is not voluntary. This may reduce the number of countries where PSI is active, but it will enhance the number of eligible applications per country.

6) In some cases, PSI projects can contribute to the business climate dialogue and sector effects. To achieve these effects, which go beyond the individual projects, it is necessary to involve local sector stakeholders and the embassies or private sector lead donors.

7) Agentschap NL and the embassies can play a catalytic role in shaping corporate social responsibility at sector level, through specific PSI projects. This requires more capacity in this area at both levels.

The conclusion about MMF’s performance during the period from 2007-2009 is that an investment of €1,266,793 has led to 457 applications, which in turn have led to 150 voucher trajectories, 35 of which have been finalised. Of the finalised trajectories, 14 have led to a positive outcome. If the same success rates were to apply to the ongoing 115 trajectories, 60 positive outcomes could be achieved but additional implementation and programme costs would be necessary. These results do not meet the proposed targets. In view of the changed role of the MMF unit, consultants and embassies, as well as the varying expectations of the applicants, and the fact that the MMF unit has made various adaptations to the programme, it is recommended to develop a new policy framework for matchmaking between entrepreneurs in developing countries and the Netherlands. MMF-related issues that require further attention in relation to the consultants include:

1) The objective of MMF needs to be defined more clearly in order to meet the expectations of both the Dutch enterprises and those in the developing country. A clear distinction should be made between similar programmes funded by the Ministry of Foreign Affairs. Does MMF aim to help companies in developing countries acquire (investment) goods, technology and knowledge from Dutch companies or is the main objective access to the Dutch market? Both motivations currently exist, but they require different methods of matchmaking.

2) The important deviations between the pilot project results, the planned and the actual efficiency and effectiveness of the MMF programme would in our view need a reaffirmation by DDE that the MMF programme is an effective and efficient instrument for supporting sustainable economic development.

3) The dependency on the embassies as the main pipeline developers for MMF applications makes the programme vulnerable. If a particular embassy does not provide input for MMF, it becomes difficult to achieve results in that country. It should be considered whether embassies should be the sole promotion and screening partner for MMF applications. The collaboration with PUM is one way to address this issue; others might be developed in future.

4) The main activities needed for the success of MMF are matchmaking, follow-up and writing PSI proposals. In its current design, MMF does not provide funding for these activities. It may be an interesting option to support these activities with subsidies provided under MMF.


5) The changing character of the work of the MMF unit (from matchmaking to pipeline development, programme management and monitoring) may require a different type of staffing. This should be considered when a follow-up of the programme is planned.

6) The current system of voucher distribution among consultants is not ideal. It seems fairer if, after an expression of interest, the consultants are selected by either the applicant firm or through a ‘draw’ system.

60

ANNEXES Annex 1 Terms of reference Annex 2 Evaluation matrices Annex 3 Sampled completed and stopped projects for PSOM/PSI and MMF Annex 4 Key results of 32 projects visited Annex 5 List of interviewees in the Netherlands As separate reports: Annex 6 Ghana field report Annex 7 Vietnam field report Annex 8 Ethiopia field report Annex 9 Mozambique field report Annex 10 Indonesia field report Annex 11 Suriname field report

ANNEX 1: Terms of reference

Annexe to tender document for open EU award procedure PSOM/PSI and MMF evaluation 2009

1

Terms of reference

PSOM/PSI and MMF evaluation 2009

Ministry of Foreign Affairs

Sustainable Economic Development Department (DDE)

The Hague

October 2009


2

Contents

1 Background PSOM/PSI and MMF

2 Background of the evaluations

3 Objectives of the evaluation and methodology

3.1 Purpose

3.2 Research questions

3.3 Evaluation matrix

4. Scope and planning of the evaluation

4.1 Scope

4.2 Planning

4.3 Inception Phase

4.4 Field Work

4.5 Draft Final report

5. Reference group

6. Reporting

7. Indication of time schedule

8. Budget


3

1. Background PSOM/PSI and MMF

The Programme for Co-operation with Emerging Markets (PSOM) was established by the Minister for

Development Co-operation at the end of 1998 with an initial budget of 41 million euro for the period 1998-

2001. The programme supports initial investments in innovative business-to-business ventures in a number of

developing countries. Over time the programme was enlarged, because if its success, both in scope and budget

and has presently an annual budget of 70 million euro. In 2008 PSOM was stopped, because of judicial

problems. The programme could no longer be implemented in the form of an assignment, because it had all

the characteristics of a subsidy. A new programme was set-up: the Private Sector Investment programme

(PSI). The PSI programme is presently open for 49 countries.

The objective of PSOM/PSI is to contribute to poverty reduction by stimulating sustainable investments in

innovative businesses in selected developing countries. PSOM/PSI intends to encourage Dutch and foreign

entrepreneurs to start an investment project in developing countries in partnership with local entrepreneurs.

The programme aims at triggering an investment project that otherwise would not have been realised because

of the high product/market risks.

The Match Making Facility (MMF) is closely linked with the PSI as it assists in forging partnerships between

companies and is a possible preparatory phase for submitting a PSI proposal. The MMF started in 2006 with a

pilot programme and continued afterwards as a regular programme. The target group of the MMF is formed

by entrepreneurs in developing countries with a concrete investment or trade idea that are searching for,

preferably, a Dutch investment or trade partner. The objective of the MMF is a match between a searching

entrepreneur in a developing country and a Dutch firm into a sustainable relationship with view on a joint

investment or development of a trade relation, not necessarily a PSI project. The budget for the MMF in the

period 2007 – 2010 is 2.8 million euro. The programme is open for the same countries as PSI.

Both programmes are being implemented by the EVD, the agency of the ministry of Economic Affairs for

international business and cooperation, through an assignment by the minister for Development Cooperation.

Planning and reporting for the two programmes is done jointly. The EVD however has two separate

implementation units for the programmes. Because of the close link of the PSOM/PSI and the MMF and its

execution by the same agency, the EVD, it was decided for efficiency reasons to cover the evaluation of both

programmes in one evaluation exercise.

In 2005 the PSOM was evaluated by Ecorys. The evaluation consisted of a file research and a field study. For

the file research 47 completed project files were studied elaborately. Then, 22 of these studied projects in five

countries were selected to be visited in the field. The countries visited were Ghana, Thailand, Indonesia,

Tanzania and Mozambique.

The upcoming evaluation will address the PSOM/PSI programme since its beginning, thus covering the

period 1998 – 2009. The MMF evaluation period covers the period from its start in 2007 until end 2009.

The following PSOM/PSI periods can be distinguished (see also table 2):


4

1998-2001 PSOM first phase

2002-2004 PSOM second phase

2005-2007 PSOM-EZ and PSOM-OS together, third phase

2007-2010 PSOM fourth phase

2009-2010 PSI first phase

Table 1. PSOM/PSI projects per region

Region No. of

ongoing

projects*

Completed and

stopped projects

until December

2008**

Africa 114 120

Eastern Europe 25 3

Asia & M-E 67 106

Latin-America 45 13

Total 249 242

* Source: PSI 1st semi annual report 2009

** Source: PSOM/PSI Annual report 2008

Table 2. PSOM/PSI periods

Period No countries Total budget

available for

commitments

Countries

1 1998-2001 8-9 € 40.9 million {China}, ({India}), {Egypt}, {Ghana},

({Ivory Cost}), ({Zimbabwe}), {South

Africa}, ({Brazil}), [Cuba], [Thailand],

[Mozambique], [Tanzania], [Uganda]

2 2002-2004 16 € 120 million plus [Ethiopia], [Indonesia],

[Sri Lanka], [Vietnam], [Bolivia], [Peru]

3 2005-2007 All partner countries

4 2007-2008 All partner countries

5 2009-2010 52 € 140 million 49 countries

{……} original eligible countries; ({……}) countries for which the programme was stopped; [……] countries that were added to the programme

Information about PSOM/PSI and MMF can be found on the website: www.evd.nl/psi

2. Background of the Evaluation

Since the start of the PSOM/PSI program over 500 projects have been started. The number of projects that

have been completed is more than 160. Close to 250 projects were ongoing on July 31 2009 and new projects

are being added. Around 80 projects have been prematurely stopped, due to various reasons.

http://www.evd.nl/psi


5

The MMF programme started, after a pilot phase, as of 1 September 2007. More than 3 application rounds

have been held. Hundreds of interviews have been organised with Dutch counterparts. Also incoming trade

missions from Mozambique, Afghanistan and Yemen have been matched.

This evaluation will look into the efficiency and effectiveness of the EVD in implementing the PSOM/PSI

and MMF programmes. Further, the efficiency and effectivity of the PSOM and MMF programmes will be

studied. The evaluation will look also into the relevance of both the PSOM and MMF programmes. A number

of special issue research questions need to be investigated. The results of the evaluation must be

representative for the entire group of completed PSOM projects and MMF activities.

3. Objectives of the evaluation and methodology

3.1 Purpose

The purpose of the evaluation is to examine whether the PSOM/PSI projects and MMF activities that

are completed indeed have delivered the expected results and impact. The findings of the evaluation

may require DDE and the EVD to make changes in the current policy guidelines and implementation

of the PSI and MMF programmes.

An overarching question for the evaluation of the PSOM program is not only what the direct effects

are of the project, but also to what extent the investment projects worked as a catalyst to develop a

sector or a market in a country.

3.2 Basic research questions

The emphasis of the evaluation will be on efficiency, effectiveness and relevance of the PSOM/PSI

and MMF programmes and on the efficiency and effectiveness of the EVD in implementing the

programmes.

Basic research questions are:

1. Efficiency of the EVD in implementing the PSOM/PSI and MMF (input → output).

2. Effectiveness of the EVD (quality of the output).

3. Efficiency of the PSOM/PSI and MMF programme (input → outcome).

4. Effectiveness of PSOM/PSI and MMF, i.e. determine if and to what extent the completed

PSOM/PSI projects and MMF activities have achieved intended results (output→outcome).

5. Relevance of the PSOM/PSI and MMF, i.e. do the completed PSOM/PSI and MMF projects have

the desired development effects? (outcome→impact).

3.3 Evaluation matrix

Based on the PSOM/PSI and MMF appraisal documents the following result chain evaluation matrices have

been developed. The objectives and indicators in the matrices need to be verified and elaborated by the

evaluation team in the inception phase and will form the basis of the evaluation.


6

Table 3 PSOM/PSI Evaluation matrix

Objectives-Means Performance Indicators

INPUT

Funds for pilot

investments.

Policy (DDE).

Implementation

capacity (EVD).

no. of tenders;

no. of proposals;

amount of programme funding spent;

amount of implementation costs spent;

implementation costs per proposal;

implementation costs per project under

implementation.

OUTPUT

Quality quantity of

the services of the

EVD

Quantity and quality

of investment

projects.

quality and quantity of the EVD appraisal and

selection process of new projects;

quality and quantity of the monitoring of the EVD

of running projects;

no. of contracts/grants awarded and declined;

no. of successfully ended projects;

no. of stopped projects;

additionality of the financing/innovative character

of implemented projects;

OUTCOME

Employment.

Technology catch-

up/innovation

no. of direct jobs created;

no. of trained people;

no. of indirect jobs created;

innovativeness of technology, production method,

product for the sector;

costs per job created.

IMPACT

Follow-up

investments

Catalyst for change

of sector or market,

Continuity of

investment projects

Development effects

amount of follow-up investments realised;

no. of years of existence of companies;

no. of companies that have adopted innovation;

% growth of sector;

increase of income;

environmental, gender and other effects

chain effects

Table 4 Evaluation matrix of MMF

Objectives-Means Performance Indicators

INPUT

Funds for MMF

Policy (DDE)

No. of rounds;

No. of applications;

No. of applications screened positively;


7

Implementation

capacity (EVD) Quality of information on applicant;

Amount of programme funds spent;

Amount of implementation funds spent;

Costs per search.

OUTPUT

Contact between

companies

No. of vouchers emitted for regular MMF and for incoming missions;

No. of incoming trade missions;

No. of interviews per incoming trade mission;

No. of companies found by EVD;

No. of companies found by consultant.

OUTCOME

Partnership/cooperation

No. of successful matches leading to a cooperation relationship within

one year after issuing of voucher and one year after visit program;

Costs per successful match.

IMPACT

Private sector

development in OS-

partner countries

No. of successful matches leading to joint investment proposals

3.4 Detailed research questions

Detailed research questions for PSOM/PSI:

1. How efficiently an effectively is PSOM/PSI implemented by the EVD?

Number of proposed projects per tender;

Quality of assessment of proposals;

Number of awarded projects per tender;

Average time spend on project appraisal, contracting, managing and monitoring and up-scaling

effects of the programme;

Number of successfully completed projects;

Number of stopped projects and reasons for stopped projects;

Amount of money spend on prematurely stopped projects;

Quality of administrative management;

Quality of monitoring of and reporting on projects by EVD;

Adequateness of legal procedures in case of objections by companies;

Adequateness of procedures in case of corruption and fraud incurred by the project partners;

Role of the Advisory Commission programme for Co-operation with Emerging Markets/Private

Sector Investment Programme (APSOM/APSI).

2. What contribution does PSOM/PSI have in developing the private sector through innovative projects?

Direct employment created;

Number of staff trained;

Number of contract farmers and/or indirect staff;

Realised growth of turnover;

Realised follow-up investments;

Innovativeness and additionality of the projects.


8

3. What are the development effects of projects financed by PSOM/PSI?

Increase of income of employees and contract farmers linked to the project;

Labour standards, wages, working conditions, quality of employment at the project site including

possible positive demonstration effects to similar companies in the same sector/region;

Effects on the gender composition of the staff employed; other gender effects;

Environmental effects of the investments;

Relation between ownership (shares) and development effects;

Financial sustainability of the joint-venture or partnership established between the project

partners;

Observed change in the sector or market that can be attributed to the PSOM/PSI project; other

contributing factors to the develoment of the sector or chain in which the project operates;

Rating table with development effects per project.

4. To what extent is the operating environment for PSOM/PSI changing?

Access to follow-up financing with local and international financing institutions;

Changes in investment climate in the partner countries;

Consequences of the present financial and economic crisis.

Detailed research questions for MMF:

1. How efficiently and effectively is de MMF implemented by the EVD?

Quality/depth of search for matching;

No. of companies identified by EVD;

No. of companies identified by consultant through voucher.

2. What contribution has the MMF programme made in developing the private sector?

3. How relevant is MMF for the private sector development in the OS partner countries?

Number of successful matches leading to some kind of cooperation (i.e. trade, knowledge sharing,

investment);

Number of matches that led to PSOM/PSI proposals;

Number of matches leading to extra trade or investments;

Number of jobs created.


9

4. Scope and planning of the evaluation

4.1 Scope

The PSOM/PSI evaluation will cover a representative sample of projects and countries involved in the

programme. Specific questions that will be treated will vary according to the nature of the projects

(stopped or completed) and will be elaborated in the inception report (see paragraph 4.2.1.). Countries

to be included will take account of the experiences gained with the previous PSOM evaluation of

2005. Table 5 shows the number of completed and stopped projects from 1999 until 31 December

2008.

Table 5 Completed and stopped projects until 31 December 2008

Year

Region

99 00 01 02 03 04* 05 06 07 08 Completed

and stopped

projects

Africa

- - - 7 16 16 18 12 28 23 120

Eastern

Europa

- - - - - - - - - 3 3

Asia

- - - 3 4 10 7 21 29 32 106

Latin-

America

- - - 3 6 4 13

Total

- - - 10 20 26 25 36 63 62 242

Total

cumul.

- - - 10 30 56 81 117 180 242

Source: PSOM/PSI 2 Annual report 2008

The evaluation of the MMF activities will include all the activities in the period 1 September 2007 – 31

December 2009.

4.2 Geographic coverage

Whereas the evaluation will cover all countries where PSOM/PSI and MMF projects have been taken place, it

will concentrate on a number of countries (6), selected on basis of the following criteria:

a) geographical coverage over regions Africa, Latin-America, Asia, Eastern Europe;

b) at least 5 completed projects in a country;

c) status: partner country of the Netherlands in 2009;

d) distribution over income classes: 2 LDCs, 2 LICs, 1 LMIC, 1 UMIC;

e) Geographical distribution within Africa.


10

Given the distribution of PSOM/PSI projects per region, 1998-2009 (Table 1), the countries selected include

three African countries, two Asian countries and one Latin American country. Since the ongoing IOB

evaluation of the Netherlands’ policy with respect to the Balkan also includes a case study on Private Sector

Development, the present ToR will not include a (Eastern) European country.

In accordance with the aforementioned criteria the countries selected for field visits are:

In Africa: Ghana, Mozambique and Ethiopia

In Asia: Indonesia and Vietnam;

In Latin America: Surinam.

4.3 Planning

The PSOM/PSI and MMF evaluation will consist of three phases:

inception,

in-depth-studies (including field work in partner countries),

debriefing/reporting.

4.3.1 Inception phase

The evaluation team will be briefed by the Ministry of Foreign Affairs and the PSOM/PSI and MMF team at

the EVD. The evaluation will be accompanied by a reference group established and chaired by the Ministry of

Foreign Affairs. The evaluation team will start the evaluation at the EVD office in The Hague with the

examination of relevant background materials and documentation. The main outcome of this phase will be the

elaboration on and development of a detailed evaluation matrix as presented in chapter 3 of this ToR.

For the assessment of the effectiveness and relevance of the programme a desk study will be performed on 60

completed PSOM projects concentrated on the six countries that will be visited. For each country to be visited

5 projects will be selected from the sample of 60 reviewed in the desk study. For the assessment of the

effectiveness an efficiency of implementation by the EVD an extra sample of stopped and running projects

will also be viewed by the evaluators, through desk study and interviews. No client satisfaction survey is

required as this will be done separately by the EVD. The results will be available at the end off the first

quarter of 2010.

During the inception phase the evaluation team will have the opportunity to discuss the details and logistics of

the subsequent phase of the field work in consultation with the DDE, the Embassies concerned and the EVD.

At the end of the inception phase the consultants are expected to submit an inception report, which will

contain:

Detailed evaluation matrix, evaluation criteria and methodology;

Detailed proposal for the field studies to be undertaken;

Questionnaire for the survey to be conducted for the projects in the 6 countries;


11

Detailed schedule of work, including field studies in the 6 countries and proposed projects to be

evaluated, and list of main persons to be interviewed;

Any further data requirements from the EVD in order to fulfil the project;

Proposed table of contents for a draft version of the final report, including a brief overview of

suggested structure and contents of each chapter. Suggestion of lay-out of the project data sheet per

project visited.

The inception report will be discussed with the reference group. A final agreement on the details of the

evaluation methodology needs to be reached between the reference group and the evaluators.

4.3.2 Field work

The field studies will concern visits to the selected PSOM/PSI projects in the 6 developing countries, to the

Netherlands Embassies and the Government counterpart ministries or institutions in recipient countries. The

evaluation team will brief the DDE and the EVD on the data it has collected after the first country visit(s).

The team will present one or more- for each country visited – interim reports based on the field visits to the

reference group for comments.

4.3.3 Draft Final report

The team will present its draft final report for comments to the reference group within four (4) weeks of the

completion of the field visits. The reference group will discuss the draft report not later than three (3) weeks

after receipt. The final evaluation report will be submitted ten (10) days after the team will have received the

comments of the reference group.

5. Reference group

The reference group will meet with the consultant to discuss the inception report and the draft final report.

The reference group will consist of:

Sabine Blokhuis – DDE;

Ton Negenman - DDE;

Fred van der Kraaij - Policy and Operations Evaluation Department - IOB.;

Michiel van Erkel - DZO;

As an observer: Hidde van der Veer - EVD PSOM/PSI Unit manager.

6. Reporting

The reporting must be done according to the following requirements:

Inception report (electronic version and five hard copies);

Interim report based on the first field visits (electronic version and five hard copies);

Draft final report (electronic version and 10 hard copies);

Final report (electronic version in Word/Excel and PDF format, CD ROM and 20 hard copies).


12

All reports will be written in English. The draft final report and final report must contain an executive

summary not exceeding 5 pages, a main report with the main findings (30 pages) and annexes including data

per project that has been visited as well as summary tables.

7. Indication of time schedule

November 2th 2009 opening of bidding period

December 17st 2009 closing of bidding period

January 11th 2009 contracting and start evaluation

February 26st 2010 inception report

March 4th 2010 discussion inception report

March - April- May 2010 field studies

June 14th 2009 draft final report

July 1st 2010 discussion draft final report

July 12th 2010 final report

8. Budget

The total amount of the financial bid for the evaluation must be including travel and all other costs, excluding

VAT.


13

5. Award criterion: quality (weighting factor 60%)

# Award criteria

Ass

ess

ed

on

:

We

igh

tin

g f

ac

tor

Su

b-w

eig

htin

g

fac

tor

1 Organisation and staffing 50%

1 Structure and composition of the team and clear description of the roles and

responsibilities of the key experts 5%

2 Expertise and experience of the team leader (on the basis of CV’s), evaluated

on the basis of qualifications and skills, general professional experience and

specific professional experience.

18%

3 Expertise and experience of the senior expert (on the basis of CV’s), evaluated



12%

4 Expertise and experience of the local experts (on the basis of CV’s), evaluated



15%

2 Approach and methodology 35%

5 Appreciation of the terms of reference, highlighting of the pertinent issues in the

assignment and presentation of innovative additions.

10%

6 Clear and complete explanation of the approach and methodology to answer the

evaluation questions as listed in the terms of reference.

25%

3 Workplan and timetable 15%

7 Clear and consistent presentation of main activities of the assignment, their content and

duration, and delivery dates of the reports 15%

ANNEX 2: Evaluation Matrices

Annex 2 - Evaluation Matrices PSOM PSI

EVALUATION FRAMEWORK: PSOM/PSI

Relation to be

measured

Result Level ID Performance Indicators Source: Desk-study and interviews NL Source: Telephone interviews Source: Local

consultant

Source: Field-study Research method

1.1Follow up investments realised in Euro

(expected, realised at closing, current)

EVD reports on 60 sample: expected and at

closing

Other 30 completed: follow up

investment current

Interview manager follow up investment

current

Data analysis and

interviews

1 2 Sales (expected, realised at closing, current)EVD reports on 60 sample: expected and at

closingInterview manager: sales current

Data analysis and

interviews

1 3 Working conditions Assessment & Employees Focus group discussion

1.4 Income effects realisedEVD reports on 60 sample: expected and at

closingEmployees Focus group discussion

1 5 Environment and gender effects EVD reports on 60 sample Assessment during project visit Observation and interview

1.6

1.6.1 Innovation and knowledge effect in sector EVD reports on 60 sample Sector-report Interviews with sector-stakeholders

1.6 2 Chain effects for SMEs (linkages created) EVD reports on 60 sample Sector-report Interviews with sector-stakeholders

1.7 Identification of business climate issues Interviews DDE Interviews HMA and sector stakeholders

Relevance

2.1 Innovation proposed by JV realised Bus plan, APSI and HMA opinion on 60

sampleTriangulation during interview manager Observation

2 2Number of direct jobs created expected and

realised (gender disaggregated)

EVD reports on 60 sample, Final report.

For entire population annual reportTriangulation during interview manager

Data analysis and

interviews/observation

2 3 Application of knowledge by trainees Employees Focus group discussion

2.4 Number of outgrowers (realised)EVD annual report for entire population Triangulation during interview manager Data analysis

Effectiveness including sustainability

3.1 No. of companies created Excel sheetData analysis stopped,

granted, declined and cross

tabulation

3 2 Business model proposed EVD reports on 60 sample, Final report Data analysis and cross

l3 3 % ownership of local partner in joint venture EVD reports on 60 sample, Final report Interview manager

Data analysis and cross

tabulation

3.4 Gender of manager joint ventureOther 30 completed: gender

managerInterview manager Observation

3 5 Applicant company is SME or not EVD reports on 60 sample, Final report Calculation share SME

3.6No. of people who completed training

(expected and realised)EVD reports on 60 sample, Final report Employees, interview manager

Data analysis & Focus

group discussion

3.7No. of applications for PSOM/PSI and quality of

appraisalBAS, APSI interviews Interview HMA

Data analysis stopped,

granted, declined and cross

tabulation

3 8Quantity, quality and use of EVD monitoring

reportsReports, interviews with DDE staff Interview HMA

Document assessment and

interviews

OUTCOME

PSOM/PSI as catalyst for:

PSOM/PSI supported

companies lead to

innovation, enhanced

knowledge and job

creation

Sustainable private sector

development

Creation of innovative

joint venture companies in

developing countries

IMPACT

OUTPUT

Annex 2 - Evaluation Matrices PSOM PSI

EVALUATION FRAMEWORK: PSOM/PSI

Relation to be

measured

Result Level ID Performance Indicators Source: Desk-study and interviews NL Source: Telephone interviews Source: Local

consultant

Source: Field-study Research method

Efficiency Including additionality

4.1Amount spent on PSOM/PSI grants and division

between TA and hardware

Excel sheet: committed and disbursed and

60 completed sample: TA/hardware

division

Data analysis, trends and

cross tabulation

4 2Monetary and in kind contribution of

promoters in joint venture companyExcel sheet: theoretical contribution

Other 30 completed: what was own

contributionInterview manager: own contribution

Data analysis and cross

tabulation

4 3Time spent by EVD on appraisal process and

legal procedures (complaints or fraud)

Implementation proposal EVD & interviews

with EVD staffAnalysis of throughput time

4.4Amount spent on EVD implementation (agency

fee)

Implementation proposal EVD & interviews

with EVD staffData analysis and trends

4 5 Quantity and quality of staff at EVDOrganizational chart, CV's and Client

satisfaction researchAnalysis

4.6

4.6.1 Embassy EVD reports on 60 sample: HMA advice

and interviews at EVDInterview HMA

Document review,

interviews

4.6 2 APSOM/APSIEVD reports on 60 sample: APSOM advice.

Annual reports, evaluation report,

interview APSI

Document review,

interviews

4.6 3 DDE interviews DDE, BEMO, policy documentsDocument review,

interviews

4.7 Systems to enhance internal learning EVD interviews, instruments and systems Analysis

4 8Systems to determine needs and priorities of

private sector in developing countriesMemoranda of Understanding

Interview HMA and national counterpart

organisations

Document review,

interviews

4 9Systems to promote, plan, monitor and

evaluate the PSOM programme

Appraisal, Monitoring and marketing

system of EVD and DDE

Appraisal of evolution of

systems

Other issues Performance Indicators Source: Desk-study and interviews Source: Telephone interview Source: Field-study Research method

Payback period (when did

you start making a profit)Files (sample 60 completed projects)

Triangulation: Manager interviews

and audited accounts

Data analysis and

interview

Bankable proposal

(yes/no)Files (sample 60 completed projects) Manager and bank interviews

Data analysis and

trends

Other sources of finance

available?Business plan Telephone interviews other 30 completed

Sector stakeholders and bank

interviews

Data analysis and

trends

Reasons stopped Files (25 stopped) Telephone (25 stopped) Analysis

PSOM money invested in

stopped projects (% of

total disbursed)

Excel sheetData analysis and

cross tabulation

Continuity JV & positive

effectsTelephone (25 stopped) Interviews

Strategic contributions to the programme of:

Additionality

Stopped projects

INPUT

Funding - budgeting -

monitoring-accounting -

staffing - learning -

management

Annex 2 - Evaluation Matrices MMF

EVALUATION FRAMEWORK: MATCH MAKING FACILITY MMF

Relation to be

measured

Result Level ID Performance Indicators Source Research

method

1.1No. and size of joint investments that are the result

of MMFFiles and interviews with beneficiaries

Document

assessment and

interviews

1.2Recognition and specific proof that MMF has acted

as catalyst for the creation of a joint investmentInterviews with EVD staff and beneficiaries

Interviews,

anecdotical

evidence

Relevance

2.1Specific proof that MMF activities led to creation of

partnerships

Files at EVD and verification through

interviews with beneficiaries and Embassies

Document

assessment and

interviews

2.2Quantity and quality of partnerships created

through MMF supported activities

Files at EVD and verification through

interviews

Document

assessment and

interviews

Effectiveness including sustainability

3.1 No. of vouchers emitted Monitoring system EVD Data analysis

3.2 No., type and quality of trade missions Monitoring system EVD and interviews Data analysis

3.3No. of company matches identified by EVD and

consultant respectively

Monitoring system, file study and

interviewsData analysis

3.4Quantity, quality and use of matchmaking facility

products (studies, company profiles, matchmaking..)

Reports, interviews with beneficiaries and

consultants

Document

assessment and

interviews

Efficiency Including additionality

4.1 Amount spent on MMF grants (vouchers) Finance departmentData analysis

and trends

4.2Quantity and quality of MMF related databases

(Consultants, Dutch companies)Databases, interviews Assessment

4.3Time spent by EVD on MMF appraisal and

monitoringProject files

Analysis of

throughput time

4.4Amount spent on EVD and Embassy implementation

(agency fee)

Assignment letters from Ministry of Foreign

Affairs, programme budgets

Data analysis

and trends

4.5Quantity and quality of staff at EVD and in

Embassies for MMFOrganizational chart and CV's Analysis

4.6Strategic contributions of DDE and Embassies to the

programme

Meeting minutes and correspondence

between DDE and EVD

Document

review and

interviews

4.7

Systems to enhance internal learning and to

determine needs and priorities of private sector in

developing countries

Interviews and background material

Document

review,

interviews

4.8Systems to plan, monitor and evaluate the MMF

programmeMonitoring system of EVD

Interviews,

systems

assessment

INPUTFunding - budgeting - monitoring -

staffing - learning - management

IMPACT Sustainable private sector development

OUTCOMESustainable business partnerships

created as result of MMF

OUTPUT

Match making activities: business

support for local company, match

making, trade missions

ANNEX 3: Sampled completed and stopped projects for PSOM/PSI

and MMF

TRIODOS FACET PSOM/PSI AND MMF EVALUATION 1999-2009 ANNEX 3

ANNEX 3

Sample 60 completed PSOM projects (all projects below= desk study, light blue= also field visit, green=spare for field visit) Country Sector Year Project number Applicant Company

ETHIOPIA Agriculture 2003 PSOM03/ET/6 Flodac B.V.

ETHIOPIA Agriculture 2003 PSOM03/ET/12 Beekenkamp Beheer Maasdijk

ETHIOPIA Agriculture 2003 PSOM03/ET/14 Linssen International

ETHIOPIA Agriculture 2004 PSOM04/ET/8 Van Oers Import B.V.

ETHIOPIA Agriculture 2004 PSOM04/ET/5 C.G. Zuurbier & Co. Rozenkwekerijen B.V.

ETHIOPIA Agriculture 2004 PSOM04/ET/14 G. van der Deijl Noordwijk b.v.

ETHIOPIA Industry 2004 PSOM04/ET/11 Trento Engineering & Protobouw B.V.

ETHIOPIA Agriculture 2005 PSOM05/ET/3 Hamer Houdstermaatschappij B.V.

ETHIOPIA Agriculture 2005 PSOM05/ET/2 Trabocca B.V.

GHANA Energy/environment 1999 PSOM99/GH/2/7 Wijma International

GHANA Agriculture 1999 PSOM99/GH/1/7 Comma

GHANA Transport/Infra 1999 PSOM99/GH/3/2 Nepostel

GHANA Energy/environment 2000 PSOM00/GH/2/3 Stroomwerk

GHANA Transport/Infra 2000 PSOM00/GH/3/2 Unicontrol Comm.

GHANA Agriculture 2001 PSOM01/GH/1/7 Global Green

GHANA Agriculture 2003 PSOM03/GH/17 Sitos Commodities & Logistics (SCL) BV

GHANA Agriculture 2003 PSOM03/GH/1 Unilever Research & Development

GHANA Agriculture 2004 PSOM04/GH/16 Marcel Vermaat Holding BV

GHANA Agriculture 2005 PSOM05/GH/24 MDK Plants & Decorations B.V.

GHANA Transport/Infra 2005 PSOM05/GH/21 Morcon B.V.

INDONESIA Industry 2001 PSI01/RI/2/8 Wijma

INDONESIA Agriculture 2001 PSI01/RI/1/8 ENZA Zaden

INDONESIA Agriculture 2001 PSI01/RI/1/5 Westkust Holland

INDONESIA Industry 2002 PSOM02/RI/2/5 G.Spilt en Zonen

INDONESIA Agriculture 2002 PSOM02/RI/1/1 Masterfoods Veghel

INDONESIA Agriculture 2003 PSOM03/RI/15 Bartels Stek

INDONESIA Agriculture 2004 PSOM04/RI/6 ForesTrade Europa B.V.

INDONESIA Agriculture 2004 PSOM04/RI/4 Smaragd B.V.

INDONESIA Agriculture 2004 PSOM04/RI/15 W.G. Den Heijer bv

INDONESIA Agriculture 2004 PSOM04/RI/12 Westkust Holland Export B.V.

INDONESIA Energy/environment 2004 PSOM04/RI/8 Shell International Renewables

INDONESIA ICT 2004 PSOM04/RI/13 QnAp iT

INDONESIA Transport/Infra 2004 PSOM04/RI/14 Slagboom Beheer bv

INDONESIA Agriculture 2005 PSOM05/RI/26 Agriom Beheer B.V.

INDONESIA Agriculture 2005 PSOM05/RI/1 PCM Laan Castricum Beheer BV

INDONESIA Industry 2005 PSOM05/RI/22 Ropac Holding B.V.

INDONESIA Agriculture 2006 PSOM06/RI/22 Heusden Veste B.V.

INDONESIA Industry 2006 PSOM06/RI/01 Norma B.V.

MOZAMBIQUE Agriculture 2001 PSOM01/MZ/1/7 Sunsmile Trading

MOZAMBIQUE Agriculture 2001 PSOM01/MZ/1/4 DO-IT

MOZAMBIQUE Agriculture 2001 PSOM01/MZ/1/3 T&D Colours&Comm

MOZAMBIQUE Agriculture 2001 PSOM01/MZ/1/1 Flodac

MOZAMBIQUE Tourism 2001 PSOM01/MZ/2/1 Fortunatura

MOZAMBIQUE Transport/Infra 2001 PSOM01/MZ/1/5 Comos

MOZAMBIQUE Agriculture 2004 PSOM04/MZ/9 David Sole P/L

MOZAMBIQUE Agriculture 2004 PSOM04/MZ/3 Zimflora (pvt) Ltd

MOZAMBIQUE Agriculture 2004 PSOM04/MZ/13 Retzlaff Farm Pvt Ltd

SURINAM Agriculture 2005 PSOM05/SR/4 LNB International Feed B.V.

SURINAM Agriculture 2005 PSOM05/SR/1 Amar Import en Export VOF

SURINAM Industry 2005 PSOM05/SR/21 Finish Profiles Beheer B.V.

SURINAM Transport/Infra 2005 PSOM05/SR/24 AP&G Consultancy N.V.

SURINAM Other 2006 PSOM06/SR/04 Us Media Holding BV

VIETNAM Agriculture 2003 PSOM03/VN/4 East-West International B.V.


Country Sector Year Project number Applicant Company

VIETNAM Industry 2003 PSOM03/VN/3 Kwintet KLM Kleding N.V.

VIETNAM Industry 2004 PSOM04/VN/9 Freetex

VIETNAM Agriculture 2005 PSOM05/VN/3 EcO2 B.V.

VIETNAM Agriculture 2005 PSOM05/VN/21 Calla Bulbs International B.V.

VIETNAM Industry 2005 PSOM05/VN/4 Ned-Deck Marine B.V.

VIETNAM Industry 2006 PSOM06/VN/26 ECS Electronics B.V.

VIETNAM Other 2006 PSOM06/VN/24 Wassenburg & Co B.V.

Sample 25 stopped PSOM/PSI projects

Country Sector Year Project number Company

AFGHANISTAN Transport/Infra 2006 PSOM06/AF/01 Tradepoint Europe BV

BOSNIA-HERZEGOVINA

Industry 2006 PSOM06/BA/22 Aqua Team B.V. (Tempus Fugit Groep B.V.)

BRAZIL Tourism 2007 PSOM07/BR/21 Intercapital B.V.

CAPE VERDE Industry 2006 PSOM06/CV/21 De Gouwe Interhold B.V.

CHINA Agriculture 2006 PSOM06/CH/03 Agriprom Stalmatten B.V.

CHINA Agriculture 2006 PSOM06/CH/01 Dutch Organic International Trade (DO IT) B.V.

CHINA Transport/Infra 2007 PSOM07/CH/01 Het Anker B.V.

ETHIOPIA Agriculture 2006 PSOM06/ET/21 Finflower Afrika BV

GEORGIA Transport/Infra 2006 PSOM06/GE/01 Womy Equipment Supply

GHANA Agriculture 2007 PSOM07/GH/04 Anova Holding BV

INDIA Energy/environment 2006 PSOM06/IN/21 Holland Mineral Water Machinery B.V.

INDIA Industry 2006 PSOM06/IN/24 Spyker Cars N.V.

INDONESIA Industry 2007 PSOM07/RI/22 Excellent Products bv

MACEDONIA Industry 2007 PSOM07/MK/01 Ethics Group BV

MOROCCO Industry 2007 PSOM07/MA/23 CEBO International B.V.

PERU Agriculture 2008 PSOM08/PE/03 Brouwers Equipment B.V.

PHILIPPINES Industry 2006 PSOM06/PH/01 Focus Human Resources and Consultancy B.V.

RWANDA Energy/environment 2007 PSOM07/RW/21 Solar Construct Ltd.

SURINAM Other 2007 PSOM07/SR/22 Wijma Kampen BV

THAILAND Agriculture 2007 PSOM07/TH/21 Ter Haar Ornamental Flower BV

UGANDA Energy/environment 2006 PSOM06/UG/03 R.V. Siemons Holding B.V.

UGANDA Other 2007 PSOM07/UG/21 Almex B.V.

VIETNAM Industry 2007 PSOM07/VN/02 Wartsila Propulsion United B.V.

VIETNAM Transport/Infra 2006 PSOM06/VN/23 TNT Express Worldwide NV

ZAMBIA Agriculture 2008 PSOM08/ZM/02 Agro Technical Supplies b.v.


MMF completed voucher projects (sample for product analysis= in light blue)

Country Year Project number Company Consultant

Albania 2007 MMF/07/AL/01/V1 AlbaMilk Eurocon

Albania 2007 MMF/07/AL/02/V3 BellaConfex HB Berenschot

Albania 2008 MMF/08/AL/11/V6 Sejega Food Processing shpk Addventures

Albania 2008 MMF/08/AL/04/V5 Shpresa shpk Q-Point

Albania 2008 MMF/08/AL/01/V3 Zoo-Al Bushat shpk KB Agro

Armenia 2007 MMF/07/AM/03/V1 Artur ev Edita LLC Larive International

Armenia 2008 MMF/08/AM/01/V2 Agat 777 LLC Larive International

Benin 2008 MMF/08/BJ/01/V1 (APAP) Ass. Int.pour la Promotion Agropastorale

Kubita

Benin 2008 MMF/08/BJ/02/V2 Isocel Sarl La'Met/Interliaise

Bolivia 2007 MMF/07/BO/02/V2 IAS Kubita

Bosnia H. 2008 MMF/08/BA/20/V4 Inox AL Addventures

Bosnia H. 2008 MMF/08/BA/12/V2 Nobil d.o.o. KB Agro

Bosnia H. 2008 MMF/08/BA/28/V5 Smrcak d.o.o. Addventures

Bosnia H. 2008 MMF/08/BA/15/V3 Telalovic d.o.o. Addventures

Bosnia H. 2008 MMF/08/BA/06/V1 Zedex doo Addventures

Burkina Faso 2008 MMF/08/BF/02/V1 La Generale des Operations SA EPM Consulting

Egypt 2008 MMF/08/EG/11/V1 Hashem Brothers KB Agro

Egypt 2008 MMF/08/EG/18/V3 Hashem Essential Oils KB Agro

Georgia 2008 MMF/08/GE/05/V1 Microfinance Organization Alliance Group Kubita

Guatemala 2007 MMF/07/GT/01/V1 IVAGRO Plants Ideavelop/Reflex

Guatemala 2008 MMF/08/GT/02/V1 Agroindustrias G.M., S.A. Kubita

Macedonia 2008 MMF/08/MK/02/V3 Agripro Valandovo d.o.o. KB Agro

Mongolia 2008 MMF/08/MN/01/V1 Monos Group Larive International

Mozambique 2008 MMF/08/MZ/07/V4** Mozsharing, Ida Alliance Plus

Mozambique 2008 MMF/08/MZ/04/V3** Tendas de Mocambique Lda Alliance Plus

Nicaragua 2008 MMF/08/NI/01/V1 Hidroponicas de Nicaragua S.A. Ideavelop/Reflex

Nicaragua 2008 MMF/08/NI/02/V2 Matagalpa Tours Ideavelop/Reflex

Peru 2008 MMF/08/PE/01/V1 Agricola Pampa Baja HB Berenschot

Philippines 2007 MMF/07/PH/04/V1 IPM Construction HB Berenschot

Rwanda 2008 MMF/08/RW/05/V2 KTS Kigali Tourism School Kubita

Rwanda 2008 MMF/08/RW/06/V1 Sefik Kubita

Senegal 2008 MMF/08/SN/03/V2 Saphs EPM Consulting

Zambia 2007 MMF/07/ZM/01/V1 CHC Commodities EPM Consulting

ANNEX 4: Key results of 32 projects visited


ANNEX 4

This table provides an overview of the key findings of the 32 visited projects. Detailed information on each visited project is available in the country reports.

Country Sector JV name Effectiveness Relevance Additionality

Ethiopia Agriculture Maranque Plc + + +/-

Ethiopia Agriculture Linssen Rose Plc + + +

Ethiopia Agriculture Lafto Roses Plc + + -

Ethiopia Agriculture TAL Flowers Plc + +/- +

Ethiopia Agriculture Ethio Vegfru Plc +/- + +/-

Ethiopia Agriculture Florensis Ethiopia Plc + + +/-

Ghana Transport/Infra Unicontrol Commodity Ghana + + +/-

Ghana Agriculture Novel Development +/- +/- +

Ghana Agriculture Sitos coconut derivates - - +

Ghana Transport/Infra WATCH +/- + +/-

Ghana Energy/environment Greencoal +/- +/- +

Indonesia Agriculture Tunas Indoland + + +

Indonesia Agriculture No JV (Den Heijer) - - +/-

Indonesia Agriculture Java Egg Specialities + + +

Indonesia Agriculture Miraflora - - +/-

Indonesia Agriculture PT Aloe Vera Bali +/- +/- +/-

Mozambique Agriculture Sunsmile Mozambique + + +

Mozambique Tourism Flamingo Bay + + +

Mozambique Agriculture Gouda Gold Mozambique + + +

Mozambique Agriculture Good luck Mozambique - - +

Mozambique Agriculture Produsola +/- +/- +

Suriname Agriculture Caribbean Foam Technology + + +/-

Suriname Industry Vasco Profiles NV +/- +/- +

Suriname Transport/Infra BE&C (Bitumen Emulsions & Cutbacks)

+ + +

Suriname Agriculture L&B Surimix + + +/-

Suriname Other Spang Makandra + + +

Vietnam Industry Song Ha company + + +/-

Vietnam Industry JV Flexcon + + +/-

Vietnam Agriculture Bonnie Farms + + +/-

Vietnam Agriculture ECO2 (Vietnam) + + +

Vietnam Industry NDM (Vietnam) + + +

Vietnam Industry ECS Ltd (Vietnam) + + -

ANNEX 5: List of interviewees in the Netherlands


ANNEX 5

People interviewed in the Netherlands

Name Function Date of interview

Baneke, F. APSOM/APSI member 1 March

Blokhuis, S. Head DDE/NB, unit responsible for PSOM/PSI 19 February

Borger, J. Project Advisor Agentschap NL 6 May

Buchem, S.M. van Cluster Leader Agentschap NL 27 April

Doorakkers, D. Financial Economic Co-worker Agentschap NL 12 May

Dumitru, A. Financial Economic Co-worker Agentschap NL 12 May

Duthler, J. Legal Services Agentschap NL 27 April

Freitas de Sousa, L. de Financial Economic Co-worker Agentschap NL 6 May

Haak, K.M. Project Advisor Agentschap NL 6 May

Hamers, K. Cluster Leader Agentschap NL 27 April

Harting, M. Commercial Advisor Agentschap NL 27 April

Hartsema, E. Project Advisor Agentschap NL 7 May

Hilgeman, H. Cluster Leader Agentschap NL 7 May

IJzermans, S. Chairman APSOM/APSI 1 March

Jongma, S. Management Team Member Agentschap NL 1 June

Kappers, S. Director Kubita consultants 27 May

Keijzer, T.J. APSOM/APSI members 1 March 2010

Kok, J. Project Advisor Agentschap NL 12 May

Kooij, M. van der Human Resources Agentschap NL 27 April

Kraaij F.P.M. van der Inspectiedienst Ontwikkelingssamenwerking en beleidsevaluatie

7 June

Krull, B. Manager Match making Facility Agentschap NL 9 February, 27 May, 7 June

Nahon, C. Project Co-worker Agentschap NL 6 May

Nannings, J. Project Co-worker Agentschap NL 6 May

Negenman T. Senior Policy Advisor, DDE 2 Februari

Oost, K.J Legal services Agentschap NL 27 April

Paalman, B. APSOM/APSI member 1 March

Poortvliet, P. Cluster Leader Agentschap NL 7 May

Ridder, M. Project Advisor Agentschap NL 7 May

Roodenburg, W. Director, Alliance Plus 7 June

Valstar, M. Project Advisor Agentschap NL 28 April

Veer, H. van der Unit Manager PSOM/PSI Agentschap NL 31 March, 3 June, 14 June

Verreijt, F. Project Secretary Agentschap NL 28 April

Veul, J. Senior Policy Advisor DDE 19 February

Vieveen, P. Financial Economic Co-worker 28 April

Vink, M. Project Advisor Agentschap NL 28 April

Zacht, S. Project Advisor Agentschap NL 7 May

final report evaluation of psom/psi 1999-2009 and mmf

Documents