final report on the evaluation of the matra training pro- gramme … · 1.3 matra training...

Final Report on the evaluation of the Matra Training Pro-gramme for European Cooperation (MTEC) and its prede-cessors SPAN Consultants 10 July 2008

Report on the evaluation of MTEC

Table of contents

TABLE OF CONTENTS 2

ABBREVIATIONS AND ACRONYMS 5

SUMMARY 6

INTRODUCTION 9

Acknowledgements 11

1. POLICY DEVELOPMENT 12

1.1 Introduction 12

1.2 General Matra policy framework/context 12

1.3 Matra training programmes 13

1.4 Policy with respect to eligible countries 14

1.5 Policy with respect to course and academic programme topics 16

1.6 Summary of evaluation findings 16

2. ADEPT / MTEC MANAGEMENT 18

2.1 Introduction 18

2.2 Division of responsibilities between the parties involved 18

2.3 Planning 19

2.4 Selection of training providers 20

2.5 Programme Monitoring & Evaluation 22

2.6 Reporting and consultations on Programme implementation 23

2.7 Financial management 24


3. SHORT COURSES AND ACADEMIC PROGRAMMES: IMPLEMENTATION 29

3.1 Introduction 29

3.2 Courses and academic programmes implemented 29

Report on the evaluation of MTEC 3.3 Responsiveness to needs 32

3.4 Training programme quality 35

3.5 Quality assessment by trainees 37


4. SHORT COURSES AND ACADEMIC PROGRAMMES: TARGET GROUPS/TRAINEES 41

4.1 Introduction 41

4.2 Getting applications – marketing and recruitment 41

4.3 Selection and selection criteria 43

4.4 Number of participants – by year, course or academic programme (male/female) 44


5. NETWORKING 49

5.1 Introduction 49

5.2 Networking initiatives during the training 49

5.3 Networking and the EVD/CROSS database 50

5.4 Networking with the Netherlands 50

5.5 Networking at national level and East-East contacts 52


6. PROGRAMME ‘IMPACT’ 56

6.1 Introduction 56

6.2 Impact for the individual trainee 56

6.3 Capacity development 57

6.4 Goodwill 60


7. SUMMARY OF CONCLUSIONS AND RECOMMENDATIONS 62

7.1 Relevance 62

7.2 Effectiveness 63

7.3 Efficiency 65

7.4 Impact 67

Report on the evaluation of MTEC 7.5 Recommendations 68

Report on the evaluation of MTEC

5

Abbreviations and acronyms ADEPT Accession-oriented Dutch European Proficiency Training

Programme DZO/UM

Directie Zuidoost en Oost-Europa, Afdeling Uitvoering Matra

ENP European Neighbourhood Policy EU European Union EVD

Agency for International Business and Cooperation (Economische Voorlichtingsdienst)

FLEX Matra Flex for short-term incidental interventions HOB How to Operate in Brussels IAC International Agricultural Institute IOB Inspectie Ontwikkelingssamenwerking en Beleid-

sevaluatie IPS Unit Internationale Publieke Samenwerking (part of EVD) ISPA Instrument for Structural Policies for Pre-Accession KAP Kleine Ambassade Programma (Small Embassy Pro-

gramme) KvW Keuringdienst van Waren (Food Inspection) LNV Ministerie van Landbouw, Natuur en Voedsel (Ministry of

Agriculture, Nature and Food) Matra Maatschappelijk Transformatieprogramma MDF Management for Development Foundation MES Matra voor Europese Samenwerking (Matra for Euro-

pean Cooperation) MOP Matra Opleidingen Programma MPAP Matra Pre-Accession Program for project support to

Candidate member states MTEC Matra Training Program for European Cooperation (Ma-

tra training programma voor Europese Samenwerking) NEA Transport Onderzoek en Opleiding NEI Nederlands Economisch Instituut (predecessor of

ECORYS) NSPH Netherlands School of Public Health OCW

Ministerie van Onderwijs, Cultuur en Wetenschappen (Ministry of Education, Culture and Science)

RVD Rijksvoorlichtingsdienst SF Structural Funds ToR Terms of Reference V&W

Ministerie van Verkeer en Waterstaat (Ministry of Trans-port, Public Works and Water Management)

VWS

Ministerie van Volksgezondheid, Welzijn en Sport (Ministry of Health, Welfare and Sport)

WUR Wageningen Universiteit Research Centrum (Wagenin-gen University Research Centre)

Summary Report on the evaluation of MTEC

6

Summary The first chapter ‘Policy Development’ describes the overall Matra policy context as the frame-work for the various training programmes that are the subject of this evaluation. It subsequently provides some detail on the key characteristics of ADEPT and MTEC in terms of aims and pri-mary target groups and the policies pursued with respect to the selection of beneficiary countries and training themes. The main findings in terms of policy making are that the policy aims of MTEC have not been made operational and neither specific results nor indicators have been identified at programme level. Political choices with respect to the selection of eligible countries are not always consistent, well understood, nor well communicated. The current broad range of eligible countries moreover requires training providers to adapt their training to very diverse tar-get groups, influencing the quality of the training. Furthermore, the evaluation has found a lack of inter-relationship between the MTEC programme and other Matra instruments. Chapter two ‘ADEPT/MTEC Management’ is related to the issue of ‘efficiency’ and concerns is-sues of Programme management, procedures and financing. It starts with a brief description of management of ADEPT before going into more detail on the current state of affairs, based on the Convenant that was concluded in 2005. The chapter subsequently goes into issues of programme planning, and management of programme implementation. Particular attention is paid to the role of the Netherlands embassies and line Ministries and the process of selection of training provid-ers. The chapter also deals with programme level monitoring and evaluation and reporting issues as well as programme funding and management costs and the financing of short courses and academic programmes. Main evaluation findings are that the EVD/CROSS basically complies with its responsibilities under the Convenant of 2005 in terms of monitoring and reporting al-though no comprehensive and consistent overview of expenditures over the years could be ob-tained. Moreover, EVD/CROSS’ role in terms of providing policy advice to the Ministry of For-eign Affairs is found to be limited and their country monitoring visits seem to add little value to the end of training reports of the training providers. Relations between Ministry of Foreign Af-fairs and EVD/CROSS are functioning well as is confirmed by both sides, though a key concern have been the delays in approval of annual plans and in the transfer of programme funds to EVD/CROSS. Regular meetings between EVD/CROSS and Ministry as well as the training pro-viders are held. These are appreciated by all concerned, even though follow-up to these meetings is not always ensured. The evaluation further shows that the role of Netherlands line Ministries in shaping the directions of the Programme remains very limited. In terms of selection of training providers, the evaluation finds that tendering procedures are not consistently used. In relation to the budget, it is found that the lion share of the budget for the academic programmes is set aside for accommodation, travel, and costs of living of the students. By and large, the available re-sources are sufficient, though remarks have been made about the lack of flexibility to use certain budget lines. The criteria and procedure for the use of the budget for strengthening of sister insti-tutions are also ill defined. The third chapter ‘Short Courses and Academic Programmes: Implementation’ first of all ad-dresses the question: which training courses and academic programmes were effectively imple-mented over the years and by whom? It then addresses the indicator ‘themes are priorities for the countries concerned’ by paying attention to issues of responsiveness to training needs in the Ma-tra partner countries as a key element for ensuring training relevance. Attention is paid to mechanisms used to assess these needs and to the number of applications received as key indica-tors for responsiveness. The chapter subsequently goes into the issue of training quality, based


7

on indicators such as clarity of training objectives, appropriateness of the structure of the train-ing, training methodologies, etc. It finally provides some succinct information on the apprecia-tion of the training by the trainees, which was also identified as indicator of the programme qual-ity. Summarised, the chapter finds that the short training courses organised under MTEC have responded to perceived training needs in the Matra partner countries. Several mechanisms have been used by EVD/CROSS to gauge these needs and the number of applications received out-weighed the number of places available. It is further shown that the short training courses that have been provided over the years meet with the criterion of ‘providing quality training’. How-ever, with the decline in the number of participants from new and potential member states and increased participation of the neighbouring countries, the relevance of the training appears to be-come an issue. Due to the different training needs of these very different groups, some courses have been found to be too difficult for some and too easy for others. With regard to the academic programmes, the academic quality has been appreciated, with some variation. However, while MTEC students participate with other ‘regular’ (international) students and are not treated differ-ently, serious questions have been raised about the quality and appropriateness of the special ‘MTEC’ components (personal development, leadership, intercultural communication and change management and the social programmes). Chapter four ‘Short Courses and Academic Programmes: Target Groups/Trainees’ pays atten-tion first of all to the process of recruitment and selection of applicants for the courses and aca-demic programmes and the selection criteria used. It subsequently focuses on the key question: were the selected participants the intended ones, i.e. for short courses – senior public officials and for academic programmes – ‘high potential, recent university graduates and young public administrators’? Findings indicate that selection criteria for the short courses were indeed clear and available to the potential applicants. They have been applied with the necessary vigour, es-pecially when the number of applicants exceeded the number of places available by far. Efforts have been made, by all key stakeholders to bring the courses and academic programmes to the attention of interested parties in the Matra partner states. In addition, informal communication channels have played a key role. Data available show that for the short courses the aim of reach-ing senior and mid-level government officials has been achieved with at least half of the 3,887 course participants. It is noted that 2/3rd of participants were female and participation of people from outside the capitals has been limited. Furthermore, in the different Matra countries, there has been a considerable concentration on a limited number of key ministries and other state bod-ies, therefore creating the necessary critical mass. Ensuring a balanced representation of the dif-ferent Matra partner countries has not always been possible, one reason being that the countries vary considerably in terms of ‘size’ (population, bureaucracy, etc.) and in terms of priorities. Fi-nally, participants in the academic programmes are generally young, with most of them probably just completing another type of higher education. Participation of junior civil servants in the aca-demic programmes is limited. The aim of equally reaching promising students and young profes-sional in the public sector is therefore not realised. Chapter five ‘Networking’ deals with the importance of networking as a secondary objective of the MTEC programme. To assess this networking aspect, this chapter focuses on the networking initiatives of training providers and the use of the EVD/CROSS database. Furthermore, it pro-vides details on the networking in the Netherlands, with training providers, line ministries and embassies, and on the east-east networking relations. The evaluation finds that some initiatives aimed at future networking are already taken during the training courses but that these do not necessarily translate into intensive networking after the training. Nevertheless, the evaluation shows that some contacts are maintained, and then particularly among former participants (East-


8

East networking). To maintain contact, a majority of participants applauds the EVD/CROSS da-tabase – however not very often is it actually used. Complicating factor is that about 1/3rd of the e-mail contacts in the database no longer appears to work. Networking with institutions in and of the Netherlands (training providers, ministries and embassies) occurs but is not particularly fre-quent. In all cases, it is evident that networking takes different shapes and is of variable intensity. It is not evident that recent initiatives, such as linking the MTEC alumni database with the Dutch Alumni Business Society funded by the Ministry of Economic Affairs, will stimulate networking among the MTEC alumni. Moreover, privacy issues in establishing such links would clearly come up.

Chapter six ‘Programme Impact’ focuses on questions of ‘impact’, i.e. the use that was made of the training once the participant in the short course or academic programme had returned to her/his home country. In line with the key features and aims of the different programmes, impact is distinguished at three levels; a) the individual trainee – career elements, employment, b) the employer of the trainee – ‘institutional enhancement’ and c) ‘image building’ and goodwill for the Netherlands. The main evaluation findings on impact in terms of personal and professional development indicate that a majority of the respondents of an online questionnaire were in the same professional position as before attending the course and most continued to work with the same organisation. However, more than half of the trainees indicate that participation in the training had enhanced their career prospects and nearly all respondents considered that they had been able to apply knowledge and skills relevant to their jobs. In terms of institutional develop-ment of government institutions in the Matra partner countries, the evaluation indicates a focus of participants on a limited number of key ministries and state bodies. Furthermore, the majority of participants have undertaken a range of initiatives to spread knowledge and skills gained from the training. However, little has happened so far in terms of strengthening of sister institutions, which is one of the elements of the academic programmes. Finally, training in the Netherlands appears to have had a positive influence on people’s perception of the country and the policies that it pursues. The last chapter ‘Summary of Conclusions and Recommendations’ summarises the main conclu-sions of all chapters whereby the evaluation findings, in line with the Terms of Reference, have been regrouped under the headings of relevance, effectiveness, efficiency and impact. This chap-ter ends with recommendations regarding policy, short courses and the academic programme, networking and the role of the line ministries. With regard to the management of the programme, recommendations are given on monitoring and evaluation, tendering, planning, reporting and consultations.

Introduction Report on the evaluation of MTEC

9

Introduction This report reflects the outcomes of the evaluation of the Matra Training Programma voor Eu-ropese Samenwerking (Matra Training Programme for European Cooperation, MTEC) pro-gramme and its predecessor ADEPT, the Accession-oriented Dutch European Proficiency Train-ing Programme as commissioned by the Netherlands Ministry of Foreign Affairs, South-East and Eastern Europe Division, Afdeling Uitvoering Matra (DZO/UM)1. The evaluation covers the im-plementation of these programmes in the period 1999 to 2007.

The evaluation was carried out in the period March - May 2008 by a team2 contracted by DZO/MU on 18 February 2008 (ref. DZO/UM-0104/2008) through SPAN Consultants, The Hague. The team was supported by a ‘Begeleidingscommissie’ comprising (former) staff mem-bers of the Ministry of Foreign Affairs. As per the Terms of Reference, aims of the evaluation were as follows: ‘To inventorise the effectiveness of the programme in the context of its objectives – includ-

ing an appreciation of the central assumptions of the MTEC Programme in practice and the extent to which it has been able to offer good quality education in themes that are relevant to the participating countries’3

An evaluation of the efficiency of agreements and procedures for the implementation of the Programme, with special attention for more process-oriented questions and recommenda-tions, as well as ‘the guidance by the Ministry of Foreign Affairs, alignment and cooperation between DZO/UM, EVD, line ministries and embassies and the procedures used for identifi-cation, implementation, monitoring and evaluation of the courses and programmes

To formulate recommendations for the future, which also need to include suggestions for other training modalities that the current formula of courses of three weeks or academic stud-ies of one to two years in the Netherlands’4 .

In addition, the Terms of Reference asked for attention to issues that exceed evaluation at effec-tiveness and efficiency levels and can be situated, given the intervention logic of the Pro-grammes, at the level of impact and relevance. Key topics warranting attention were in particu-lar the following: The contribution of the courses to ‘improved functioning of Government authorities, in par-

ticular as far as relations between state and civil society/citizens are concerned’ and the es-tablishment of a ‘critical mass for change within government institutions’5

1 The evaluation was also to deal with the Matra Opleidingen Programma (MOP) that was implemented simultaneously with ADEPT. However, in the course of the the evaluation it became clear that in the files of the Ministry of Foreign Affairs, only little information on MOP could be lo-cated. Information on e.g. the people trained (who they were, in what numbers, from where, etc.), the subjects in which they were trained, costs, etc. is not available. For this reason it was decided not to exclude an analysis of MOP from this evaluation report. Some general information on MOP can be found in Annex 2. 2 Team members were Regina Engels, Geert Edelenbosch and Paul G. De Nooijer. 3 “Het in kaart brengen van de doeltreffendheid van het programma in de context van de doelstellingen [effectiviteit] – inclusief een appreci-atie van de “uitwerking van de centrale assumpties van het MTES programma in de praktijk en in hoeverre het in staat is geweest kwalitatief goed onderwijs aan te bieden op voor de deelnemende landen relevante thema’s” 4 Het formuleren van aanbevelingen voor de toekomst”. Bij dit laatste gaat het ook met name ook om het formuleren van “suggesties voor andere trainingsmodaliteiten dan de huidige formule van drie weekse trainingen dan wel academische studies van één tot twee jaar in Ne-derland” (Terms of Reference, page. 4).


10

The contribution of the courses ‘to development of a network with/creation of a goodwill (‘draagvlak’) for the Netherlands in the partner countries and the role of the CROSS database in this respect

The contribution of the Programme to ‘capacity and network development in the partner countries’, e.g. by contributing to the establishment of a ‘ critical mass for change within government institution’

The structure of the report is as follows:

Chapter 1 deals with issues related to policy development and the policy context of MTEC and its predecessors. It particularly deals with programme objectives6, and the policies pursued with respect to eligible countries and participants as well as topics of the different training courses and programmes.

Chapter 2 concerns programme management, procedures as well as financial and cost is-sues, with attention for topics such as the division of responsibilities between different key stakeholders, i.e. the Ministry of Foreign Affairs, EVD/CROSS, Netherlands training providers, etc.

Chapter 3 provides information on the implementation of the ADEPT and MTEC courses and academic programmes. It pays attention to issues such as processes of course/programme design, implementation, monitoring and evaluation as well as the views of the trainees and representatives of the Netherlands training providers on training quality, relevance and effectiveness.

Chapter 4 concerns the ADEPT and MTEC target groups and participants, describing who they were, and where they were from and procedures for their recruitment and selec-tion

Chapter 5 deals with the ‘secondary objective’ of networking, i.e. the establishment of re-lations among former ADEPT and MTEC trainees and between these former trainees and institutions of the Netherlands (East-East and East-West networking)

Chapter 6 addresses issues of impact, with impact defined at three main levels, i.e. the individual trainee, and his or her employer (institutional strengthening) and image build-ing for the Netherlands

Chapter 7 contains the main conclusions and recommendations of the Evaluation Team and is based on the main conclusions reflected at each of the foregoing chapters.

The appendices to the report contain amongst others the Terms of Reference for the evaluation, a list of documents used, and an overview of the people interviewed and/or consulted, both through individual and group sessions. In addition, they include information on the evaluation methodology and key statistical data which have provided the basis for the analysis presented in this report as well as a summary of the key characteristics of MOP. These appendices have been added in a separate folder. As agreed with the ‘Begeleidingscommissie’, the report was prepared in English. An effort has been made to translate texts from Dutch into English as accurately as possible. Some key policy texts are, nevertheless, incorporated into the report in their original version to do better justice to the views expressed.

5 De bijdrage van de cursussen aan “een verbetering van overheidsfunctioneren, met name waar het gaat om relaties tussen overheid en maatschappelijk middenveld/burgers” en aan het creëren van een “kritische massa voor verandering binnen overheidsinstellingen” (Terms of Reference, page 4). 6 Unfortunately, information on MOP in the archives of the Ministry of Foreign Affairs turned out to be very limited; as a result, focus in the evaluation has necessarily been on ADEPT and MTEC.


11

Acknowledgements

The evaluation team would like to express its appreciation first of all to the members of the ‘Be-geleidingscommissie’ for their valuable inputs and suggestions. Secondly, we would like to thank all persons who have devoted their time to respond to our questions and complete the on-line questionnaire, and/or participated in the focus groups and/or interviews that were held in Roma-nia and Turkey. Particular thanks are also due to the staff of EVD/CROSS. This report represents the views of the members of the Evaluation Team and does not necessarily reflect those of the Ministry of Foreign Affairs. The Team finally bears sole responsibility for the contents of this report. The Hague, 1 July 2008

Chapter 1: Policy Development Report on the evaluation of MTEC

12

1. Policy Development

1.1 Introduction

This chapter first of all describes the overall Matra policy context as the framework for the vari-ous training programmes that are the subject of this evaluation. It subsequently provides some detail on the key characteristics of ADEPT and MTEC in terms of aims and primary target groups and the policies pursued with respect to the selection of beneficiary countries and training themes. The main findings of the evaluation are finally summarised in section 1.6.

1.2 General Matra policy framework/context

The Programme ‘Maatschappelijke Transformatie’ (Matra) was established by the Dutch Parlia-ment on 17 November 1993. The Matra Programme aims to contribute to the transformation process in Central and Eastern European countries towards a modern democratic state of law and a multiform society. In the initial years of the Programme, emphasis was on ‘civil society’ and local authorities, but increasingly the need was felt for support of central Government, particularly among the EU ac-cession countries. For this reason, and because the central authorities in these countries would be future partners in an expanded EU, a ‘Pre-accession’ facility, the Matra programma voor Eu-ropese Samenwerking (Programme for European Cooperation (MEC)) was added in 1998 to the already existing Matra for Good Governance programme. This facility was established to sup-port candidate member states in preparing for EU accession in a broad range of topics (including e.g. approximation of legislation, public sector reform, justice and legislation, social policy and employment conditions, environmental policy, education, health and housing policies, institu-tional development and implementation of EU rules and regulations). Following a series of evaluations of different Matra components it was decided in 2004 to strengthening the relation between policy making and policy implementation and to link a pro-posed further differentiation of policy vis-à-vis the new member states, candidate member states and ‘EU neighbours” with a strong ‘simplification’ of policy implementation. In line with this the number of instruments under MEC was reduced and MEC management was handed over to one institution, i.e. the Agency for International Business and Cooperation (‘Economische Voor-lichtingsdienst’) of the Ministry of Economic Affairs (EVD; Letter 21/9/2004). Since 2005 there have been two main Matra programmes, i.e. Matra for Good Governance and Matra for European Cooperation of which the Matra Training Programme for European Coop-eration is one of the instruments7. According to the EVD Annual Plan 2007 (page 4), MEC aims at:

Establishing strategic cooperation and coalitions with the new EU member states Pre-accession support to candidate member states

7 Other instrument are: the Matra Pre-Accession Programme for project support to Candidate member states (MPAP), Matra Flex for short-term incidental interventions (FLEX) and the Matra National Contact Point for EU twinning, to support the Dutch contribution to Multilateral pro-grammes for twinning (Twinning).


13

Preparation for partnership in the extended EU of the candidate member states as well as

Promotion of good ‘neighbourhood’ with the Eastern and Southern neighbours of an expanded European Union.

1.3 Matra training programmes

There have been three training programmes under the overall Matra programme, i.e. up to the end of 2004, the programmes ADEPT and MOP and, united since 2005, in the Matra Training for European Cooperation. The main characteristics of ADEPT and MTEC are described in the following paragraphs. These training programmes find their origin in the consideration that the transition to a more market oriented and democratic society and membership of the EU, required strengthening of (semi) governmental and non-governmental institutions in Central and Eastern European countries. This required both theoretical knowledge and practical skills and a change in public sector mentality8.

1.3.1 ADEPT ‐ Accession‐oriented Dutch European Proficiency Training Programme Aim of ADEPT was to train senior civil servants (mid-level and higher) that were employed with relevant Ministries and (semi) governmental organizations in the candidate member states. Sup-plementary aim was the creation of East-West and East-East networks9. Emphasis was on ‘inten-sive, practice oriented trainings modules’ in English that covered topics in which the Netherlands had specific expertise. They were conducted in the Netherlands by Dutch institutions, offered several times, preferably during the summer. Application was in principle through the Nether-lands embassy while final selection of participants was done in the Netherlands. Programme management was entrusted to CROSS.

1.3.2 MTEC As of 2005, ADEPT and MOP were merged into the Matra Training for European Cooperation programme, as a facility that could be used for ‘Matra- partnership, Matra Pre-accession en Ma-tra-neighbourhood’. Main reasons for this merger were to increase efficiency of the different programmes and to reduce the number of courses and academic programmes that were provided. As mentioned in the Terms of Reference for this evaluation, main objective of MTEC is to offer training courses and academic programmes to civil servants for capacity building and creating sustainable networking relations with the Netherlands. A secondary objective is institutional strengthening and capacity building of related education institutions (‘sister institutions’) in part-ner countries’10. 8 Jaarverslag ADEPT 1 april – 31 december 2002, page 1: ‘De overgang naar een pluriforme democratische samenleving met een markteco-nomie in de landen in Centraal - en Oost - Europa en, in het verlengde daarvan, hun aanvraag voor lidmaatschap van de Europese Unie (EU), heeft een immense vraag gegenereerd naar steun bij opleiding en training op tal van terreinen. De beoogde toetreding tot de EU levert vooral binnen de overheden capaciteitsproblemen op. Immers, ingevolge de zogenaamde ‘Kopenhagen criteria’ dienen de landen aan de verplich-tingen van het lidmaatschap te kunnen voldoen, inclusief de beoogde politieke, economische en monetaire unie. Ze dienen het acquis com-munautaire over te nemen, hetgeen betekent dat nationale wet- en regelgeving aangepast moet worden aan die van de EU, alsmede de be-schikking te hebben over een goed opererende overheid en rechterlijke macht. Dit houdt in dat er grote behoefte is aan het versterken van be-staande en oprichten van nieuwe instellingen en het trainen van mensen die werkzaam zijn bij de (semi-)overheid en non-gouvernementele organisaties (NGO’s). Er is naast theoretische kennis en praktische vaardigheden een enorme mentale omslag nodig van de gehele bevolking en het ambtenarenapparaat. Gezien enerzijds de grote vraag naar goed opgeleide ambtenaren en anderzijds de druk op de ambtenaren die zich met accessie bezighouden, waardoor deze niet voor lange tijd kunnen worden vrijgemaakt voor training, is de gedachte ontstaan om door middel van korte, praktijkgerichte trainingen aan deze vraag te voldoen. Door deze trainingen gedurende meerdere jaren aan te bieden zal op den duur de capaciteit bij de (semi-) overheidsinstellingen worden vergroot om aan de verplichtingen van het lidmaatschap van de EU te kunnen voldoen. Op basis van bovenstaande overwegingen is het zogenaamde ADEPT - programma tot stand gekomen’. 9 Evaluatie ADEPT 99-02 page 3 10 In exceptional cases, representatives from the private sector or NGOs could be eligible ‘en als dit met het oog op de implementatie van het acquis noodzakelijk wordt geacht’ (letter of 10 July 2007).


14

MTEC consists of two different types of training:

Training courses on transformation related issues (duration: 2-3 weeks), particularly concerning the preparation and finalisation of EU-accession.

Academic programmes in areas of good governance. These programmes can last from 12 to 24 months (the period required to obtain an MA degree)11. The aim of these academic programmes is to train high potential recent university graduates and young public administrators on themes relevant for good governance and to create long lasting networks between the participants of these programmes.

Programme development is further shaped through the annual so-called ‘jaarplanaanschrijving’ (i.e. correspondence providing policy guidelines for the following year). In this correspondence, the Ministry is to indicate the main features of MEC, including MTEC, for the coming year, in-cluding related policy changes and priorities on the basis of an annual inventory of priorities of the Netherlands line Ministries. The Ministry furthermore provides advice on the emphasis to be put in the implementation of the various MEC instruments. According to a Note of the Ministry of Foreign Affairs of 2004: ‘Through the introduction of an annual planning cycle for the entire Matra programme for European Cooperation, it will be possible to link policy and policy imple-mentation ….’. On the ‘jaarplanaanschrijving’ as a policy and planning instrument, see further section 2.3 below.

1.4 Policy with respect to eligible countries

At the start of ADEPT in 1999, the eight countries that were then eligible for support were di-vided into four so-called “front runner” states, i.e. the Czech Republic, Estonia, Hungary, Slove-nia and four “ runner-up” states, i.e. Poland, Latvia, Lithuania, Slovakia. This policy changed in 2000 with an equal focus on all 12 candidate member states, when Romania, Bulgaria, Cyprus and Malta were invited to join the EU. Except for Romania and Bulgaria, all these countries officially became a new EU member state in May 2004 (the so-called new member states), when Turkey and Croatia became candidate member states and Serbia and Montenegro potential candidate member states and with that eligi-ble for Matra support. Participants of the new member states remained eligible for participation in the MTEC training courses until the end of 2006; phasing out of this eligibility went in paral-lel with the phasing out of the EC’s pre-accession programmes. Up to the end of 2007, MTEC distinguishes the following five groups of eligible countries:

New member states (Romania and Bulgaria) Candidate member states (Croatia and Turkey and, since 2007, Macedonia) Potential member states (Serbia and Montenegro) Eastern neighbours – i.e. Belarus, the Russian Federation and Ukraine and, since

2007, Moldavia12 Southern neighbours – i.e. Jordan and Morocco

11 The Convenant (Annex 5) is not consistent in terms of the duration of the academic programmes, referring to 12 to 36 months on one occasion and to a maximum of 18 months on another. 12 to 24 months is current practice. 12 According to the letter of the Ministry of Foreign Affairs of 7 February 2007 (ref DZO/UM-0137/2007), ‘In 2007 kunnnen kandidaten uit Moldavie zich aanmelden als deelnemer voor MTEC korte cursussen. Witrussische kandidaten kunnen zich zowel voor de korte cursussen als voor de lange opleidingen aanmelden’. See also the letter of EVD of 30 March 2007 (Ref CROSS 07-0070):’Na uw brief is door u beslo-ten dat Moldavie aan zowel het cursus als het academisch programma van MTES zal deelnemen’.


15

Romania and Bulgaria, that joined the EU in January 2007, will remain eligible for participation in MTEC until 1 January 2010. Civil servants and students from these two countries can apply for one of the academic programmes of MTEC latest for the study year 2009-2010. In the case of the candidate member states and the potential candidate member states support will be con-tinued until accession. With respect to the eastern neighbours, the programme follows the EC’s European Neighbourhood Policy (ENP) and the Joint Action Programmes that were concluded in 2003. With respect to the southern neighbours, a decision was made to deploy the Matra pro-gramme for strengthening of bilateral relations and for strengthening of civil society in the Arab neighbouring countries South of the EU13. For these southern and eastern neighbours, eligibility for participation in MTEC can only be determined after consultation with DZO/UM; the contents of a programme or course determines whether participants from these countries are eligible for participation14.Until now, only Morocco and Jordan were declared eligible in 2004 for strength-ening of bilateral relations – and for short-term MTEC interventions. According to a Ministry of Foreign Affairs Notitie of 2004, these countries were selected because the ‘transformation per-spective’ was relatively favourable. The evolution of the eligibility of countries under ADEPT/MTEC is shown in table 1. Table 1: Overview status of countries and eligibility in ADEPT/MTEC (1999-2007)

1999 2000 2001 2002 2003 2004 2005 2006 2007 Czech Republic TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP AP Lithuania TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP AP Hungary TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP AP Poland TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP AP Slovenia TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP AP Slovakia TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP AP Latvia TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP AP Estonia TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP TP+AP AP Malta TP+AP TP+AP TP+AP TP+AP AP Cyprus TP+AP TP+AP TP+AP TP+AP AP Romania TP TP TP TP+AP TP+AP TP+AP TP+AP Bulgaria TP TP TP TP+AP TP+AP TP+AP TP+AP Turkey TP TP TP TP+AP TP+AP TP+AP TP+AP Croatia TP TP TP TP+AP TP+AP TP+AP TP+AP Serbia TP TP TP TP Montenegro TP TP TP TP Ukraine STC STC STC Russian Federation STC STC STC Belarus STC STC STC Moldavia STC Macedonia15 TP+AP Jordan STC STC STC Morocco STC STC STC

Candidate Member State New Member state Potential member state Eastern neighbour Southern neighbour TP = Training Programme AP = Academic programme (until 1999

academic training under MOP) STC = Selected training courses

Political choices as regards the countries that are eligible for MTEC support are the prerogative of the Ministry of Foreign Affairs.

13 ‘De centrale doelstelling van Matra-nabuurschap is: Harmonisatie van de voor de overheden in de ten (zuid-)oosten en zuiden van de EU gelegen buurlanden geldende normen met de Europese normen voor democratie en de rechtsstaat. ...in de relatie met de geselecteerde buur-landen (wordt) gestreefd naar samenwerking met het oog op capaciteitsopbouw van instituties die behoren tot het democratische bestel, als-mede naar een helder en positief imago van Nederland in deze landen. Alleen landen met een redelijk transformatie-perspectief komen in aanmerking’. 14 See also the letter of the Ministry of Foreign Affairs of 14 December 2007 (ref DZO/UM-1016/2007):’MTES staat in 2008 bovendien – afhankelijk van het soort cursus – ook open voor kandidaten uit Rusland, Witrusland, Moldavie en Jordanie’ – Morocco seems to have dis-appeared from the list. 15 Macedonia, which finds itself in a transition phase from ODA to non-ODA status is a particular case. MTEC is the only Matra programme that can be applied; at the same time, Macedonia is still eligible for the Netherlands Fellowship Programme (NFP).


16

In some cases, these choices have neither been well argued nor easily ‘sold’ to the outside world, including representatives from the countries concerned (e.g. Bosnia Herzegovina). Interviews held indicated that the choices made are also not well understood by the Netherlands line minis-tries, for whom other countries may have higher priority (e.g. the Ministry of Finance) and who have difficulty in explaining why certain neighbouring countries were not eligible. EVD/CROSS finds itself in a similar position in this regard. Moreover, the combination of very different countries, from the Russian Federation to Jordan, has evidently had implications for the short training courses: training providers have had to ad-just their programme to increasing number of very diverse countries, a difficult balancing act which has resulted in too easy training for some and too difficult for others.

1.5 Policy with respect to course and academic programme topics

DZO/UM strives for a balanced division of programmes/courses over the various countries and Matra themes. These latter themes are the following:

Legislation/law Communication and media Governance/public safety/police Culture Strengthening environmental NGO’s Welfare Strengthening environmental authorities Health Public housing Labour and social policy Education Human rights/minorities

As will be evident from the data on courses and academic programmes presented in chapter 3, MTEC and ADEPT have covered some of these themes, others not.

1.6 Summary of evaluation findings

Main conclusions in terms of policy making can be summarised as follows:

The policy aims of MTEC (and its predecessors) have not been made operational, Aims are formulated in terms of training and education activities to be developed and development of networking as a secondary objective. There has been little articulation of what purposes training, education and networking would serve and for whom. There has also been little reflection about the appropriateness of short-term training for building up sustainable networks that would also be relevant to the Netherlands line ministries.

Neither specific results nor indicators have been identified at Programme level and in this respect evaluation findings are similar to those of the IOB evaluation ‘Enlarged Europe Policy. Evaluation of the Dutch Policy concerning the Acces-sion of Central European Countries to the EU 1997-2003’ (see e.g. page 63 and 65 of Annex 1 of the Country Case Study Romania’ of May 2005).

Political choices have been made in the selection of eligible countries; these choices are neither always well understood nor well communicated to stake-holders involved in the Programme. They also do not seem to be very consistent. with respect to the eligibility of Morocco.

The current broad range of eligible countries requires training providers to adapt their training to very diverse target groups even though for eligible countries like Jordan, Morocco, and Malta, the number of participants has been negligible (see


17

further chapter 4). Moreover, increased diversification, has had implications for the quality of the training that could be provided which has been too easy for some, too difficult for others.

While according to the Convenant that was concluded between the Ministry of Foreign Affairs and EVD in 2005, ‘EVD aspires for an efficient implementation of Matra for European Cooperation and realises where possible cost savings through an integrated utilisation of the available instruments’, the inter-relationship between MTEC and other Matra instruments is not well elaborated, even though these ‘other instruments’ may include the training of civil servants in topics that are also covered under MTEC. At the same time it is noted that CROSS has requested EVD colleagues to inform people informed in other Matra projects about MTEC courses that could be relevant16.

16 Annual Report 2007, page 35.

Chapter 2: ADEPT / MTEC Management Report on the evaluation of MTEC

18

2. ADEPT / MTEC management

2.1 Introduction

This chapter is related to the issue of ‘efficiency’ and concerns issues of Programme manage-ment, procedures and financing. It starts with a brief description of management of ADEPT be-fore going into more detail on the current state of affairs, based on the Convenant that was con-cluded in 2005. The chapter subsequently goes into issues of programme planning, and man-agement of Programme implementation. Particular attention is paid to the role of the Netherlands embassies and line Ministries and the process of selection of training providers. The chapter also deals with programme level monitoring and evaluation and reporting issues as well as pro-gramme funding and management costs and the financing of short courses and academic pro-grammes. A summary of the main evaluation findings is presented in section 2.8.

2.2 Division of responsibilities between the parties involved

2.2.1 Historical division In 1999, the ‘coordination of ADEPT courses’ was entrusted to CROSS, established in 1992 as the ‘centre of expertise’ of the Netherlands Min-istry of Education, Culture and Science. CROSS was responsible for financial management of ADEPT as well as programming and annual planning exercises, including the identification of training themes in consultation with the candi-date member states and line Ministries in the Netherlands. CROSS had furthermore specific responsibilities with respect to the management of individual courses funded under ADEPT (see box).

2.2.2 Current division according to the Convenant For the management of MEC, which includes MTEC, an agreement (‘Convenant’) was con-cluded between the Ministry of Foreign Affairs and the Agency EVD, i.e. the Department for In-ternational Business and Cooperation, of the Ministry of Economic Affairs on 19 April 2005 (signed on 20 and 26 April 2005). The Agreement went into force retroactively as of 1 January 2005 and is valid, according to Article 8.4 for a period of five years, i.e. up to 31 December 2009, with funding secured for the first three years of the agreement, i.e. up to the end of 2007. Within EVD, the EVD/CROSS unit deals with MTEC. The Convenant, and its annexes, provides information on the policy framework and objectives of MEC and the different MEC instruments and highlights the obligations of both the Ministry and EVD and procedures to be followed in the management of MEC, including MTEC. In brief, the roles of the different stakeholders in programme management can be described as follows. Based on the Convenant, the Ministry of Foreign Affairs is to play a role in terms of:

CROSS responsibilities for ADEPT courses

Drafting of Terms of Reference and initiate tendering procedures

Selection of potential implementing institutions through the assessment of proposals (technical and financial) of implementing institutions together with relevant line Ministries

Drafting of contracts Dissemination of information on the courses through

brochure and website Management of applications and monitor selection of

participants Monitoring/attending of courses to guarantee their

quality and develop a sense for themes and target groups

Evaluation of courses together with implementing insti-tutions and with participants (and possibly their superi-ors), some time after the course in the candidate member states.

Settling of courses (financial, contents)


19

Setting the annual policy and planning framework, indicating the main policies and changes in accents in policy with respect to MTEC, determining priorities per theme, based on consultations with the Netherlands line ministries and providing instructions on ‘accents to put in the implementation of the different MEC pro-grammes’

Providing the overall financial framework for MEC and its instruments According to the Convenant, main responsibilities of EVD are the following:

Selection and tendering of training programmes Ensuring adequate and careful application of the procedures for the different MES

instruments. Undertaking of adequate monitoring and accompaniment of projects in the recipient

countries (with the exception of the instrument of Twinning). The role of the Netherlands embassies in running MTEC is not detailed in the Convenant but can be described in the following terms (either current or in the past when Matra was still operational in the country concerned): (a) dissemination of information on the training courses and academic programmes; and (b) monitoring the application procedure. According to a ‘Notitie’ of the Ministry of Foreign Affairs of 2004: ‘Coherence between policy and implementation implies a commitment from the line ministries to ensure sufficient quality content. Their involvement is of importance to identify pre-accession projects and to advice on Matra grant applications’. The Convenant also refers to an annual inventory of priorities of the line ministries. At the same time, the ‘Notitie’ already warns that ‘(insufficient) capacity within the line ministries could restrict their opportunities to optimally use the Matra potential’17. How the above stakeholders play their role in the different phases of programme implementa-tion, i.e. from programme level planning, programme level management, programme level moni-toring and evaluation and reporting is assessed in the following sections of this chapter. An overview of the individual line ministries’ involvement can be found in annex 4.

2.3 Planning

Findings of the evaluation demonstrate that in line with the Convenant a formal ‘jaarplanaan-schrijving’ is prepared each year. There is furthermore some exchange of correspondence and bi-lateral contact on the basis of these annual plans. Based on the Ministry’s ‘jaarplanaanschrijving’, EVD prepares annual plans for MEC as whole, including MTEC, and submits these to the Ministry of Foreign Affairs. According to the Conve-nant, this annual plan is to be a policy and monitoring instrument in which operational objectives for MEC as a whole and for the individual MEC instruments as well as the intended results are described. For MTEC, the annual plans provide an indication of the specific courses that will be implemented in the following year and some considerations that are at the basis of the proposed trainings and countries that ought to be eligible. With a focus on concrete activities (courses and 17 ‘Samenhang tussen beleid en uitvoering veronderstelt daarnaast een hernieuwd committent van de zijde van de vakdepartementen, die im-mers een belangrijke rol spelen bij zowel beleidsvorming als bij de uitvoering. Zo is het niet voorstelbaar dat het partnerschap met de nieu-we lidstaten en het nabuurschap met de buurlanden voldoende inhoud zal kunnen krijgen zonder de volledige inzet van tenminste een aantal van de vakdepartementen. De betrokkenheid van de vakdepartementen is verder van belang bij de identificatie van pre-accessie-projecten, terwijl de advisering over projecten in het kader van de Matra- subsidieregeling evenmin kan worden gemist. Knelpunten in de capaciteit kunnen voor sommige departementen een beperking vormen om de mogelijkheden binnen de programma-onderdelen ten volle te benutten’


20

academic programmes), there is little in terms of ‘policy’ or reflection on policy issues (such as e.g. the implications of participation of neighbouring coutries), ‘operational objectives’ or ‘in-tended results’. The annual planning exercise thus suffers from the same deficiencies as the overall policy framework. The Ministry of Foreign Affairs is to approve the annual plans after a discussion with EVD and/or give additional guidelines and instructions once DZO has determined its own annual plan. In its recent annual MES reports, EVD/CROSS has indicated that Ministerial approval of its an-nual plans is rather late for them to adequately plan the courses and to make sure that the right people can attend them. As a result of the late approval, only a few courses are organised before the summer holidays, often with very late or too late official approval, and most courses are or-ganised in the second half of the year (seven of the MTEC courses took place before, 24 after the summer holidays). For the same reason, courses held before the summer, sometimes started only on the basis of an informal agreement. This has created problems as not all (financial) require-ments were clearly determined beforehand and implied a financial risk for the training provider which had to pre-finance part of the costs, especially the costs of international travel, accommo-dation, etc. To resolve this problem, at the end of 2006 EVD/CROSS started preparation of the annual plans earlier, and was able to submit the annual plan for 2007 in December. Nevertheless, the annual plan for 2007 was only approved in April 2007.

2.4 Selection of training providers

2.4.1 Short courses Within the framework of ADEPT, tendering of courses was not obligatory but did take place in practice. Examples include:

the ISPA/Environment training course, for which ECORYS was contracted in 2002

the course ‘How to Operate in Brussels’ for which Clingendael was contracted On the other hand, many other courses were not tendered and involvement was ‘by invitation only’:

For the training on ’Strategic Planning and Realisation of Infrastructure’, NEA was invited to prepare a proposal for the Ministry of Foreign Affairs

The Academie voor Overheidscommunicatie was nominated by the Ministry of General Affairs, based on its training experience for civil servants from Central and Eastern Europe

Wageningen International (former IAC) took over the organisation of the course ‘Food Safety’ from the Voedsel en Waren Auroriteit at the request of the Ministry of Agriculture

CROSS asked the Trimbos Instituut to organise the Drug Policy course based on its expertise in this area.

While tendering is to be practiced under MTEC, this procedure is not always followed:

E.g. though it is mentioned in the EVD/CROSS Annual Plan for 2007 that for the course on development cooperation ‘the implementing agency will be selected ac-cording to a tender procedure’, a letter from the Ministry of Foreign Affairs dated 25


21

April 2007 states that, based on a discussion involving several Ministry departments, the course will be implemented by MDF rather than tendered.

In 2007 one of the new courses organised was Common Agricultural Policy of the EU at the request of Ministry of Agriculture. The contract was directly given to Wageningen Business School, as they had been involved, together with the Ministry, in developing the idea for the programme18.

It is understood that so-called ‘herhaalcursussen’ do not need to be tendered in case evaluation of the course is positive but that this rule is not always put into practice. Irrespective of whether tendering procedures are followed, the information on ADEPT and MTEC shows that implementation of short courses has been entrusted to a limited number of public and private institutions (see further chapter 3). In case tendering takes place, it is also understood that a short-list of institutions is approached by EVD/CROSS in addition to publication of the tender on the EVD/CROSS website. It is nei-ther clear to what extent this affects the competition nor what the criteria are for getting on this short-list since the ‘basic requirements’ mentioned with respect to the academic programmes (see further below) do not seem to apply for the short courses that are not registered in CROHO, the ‘Centraal Register Opleidingen Hoger Onderwijs’. In those cases when it is clear that only one specific institute in the Netherlands has the knowl-edge and capacity to organize the short courses, omission of tendering is understandable since it is unlikely that this would lead to an increase in the number of potential service providers. It is however not evident that this is in compliance with EU and Dutch legislation on tendering for the provision of services. Moreover, the reasons behind the choice to tender or not are only briefly touched upon in EVD/CROSS reports. There seems to be no well-argumented strategy for whether or not to ap-ply the tender rules as a result of which the procedures of selecting training providers become not transparent for (potential) training providers.

2.4.2 Academic programmes For the implementation of the academic programmes a tendering procedure was used and all 13 Dutch universities received an invitation in writing to submit an expression of interest that should meet certain basic requirements19. Universities that met these requirements were then in-vited to submit a comprehensive proposal following instructions provided by EVD/CROSS. 18 Letter EVD of 30 March 2007 (ref CROSS -07-0070):’In het jaarplan is opgenomen dat de uitvoerder van deze cursus via een aanbeste-dingsprocedure wordt bepaald. Het betrokken vakdepartement, het Ministerie van LNV, heeft de EVD aangegeven zeer te hechten aan uit-voering van de cursus door de Wageningen Business School (WBS). LNV heeft sinds 1985 nauw contact met het Post Hoger Landbouw On-derwijs (de voorloper van WBS) samengewerkt bij de ontwikkeling en uitvoering van deze cursus … en wil deze samenwerking graag bin-nen MTES-verband voortzetten’. 19 Basic requirements for participation in tender process are the following: The provider is a Netherlands public or special university, as referred in to in the 'bijlage behorende bij de Wet op het hoger onderwijs en wetenschappelijk onderzoek', onder punt a. en punt b.; The provider is capa-ble of implementing the programme himself. Use may be made of sub contractors. Under all circumstances, the organiser is responsible for the quality of the services provided; The provider is capable of receiving and training of candidates from the target countries and able to provide ac-companiment and accommodation (adequate physical and teaching infrastructure, presence and availability of qualified staff); The provider has ample experience in offering LLM trainings for the target countries; In view of the sustainability of the programme, institutions preferably have a twinning relation with sister organisations in the partner countries. A clear description of the aims of the training that is part of the programme, in the light of the Matra programme objectives. Other basic conditions a training provider needs to provide are the following: The training provider guarantees that the programme will be implemented, also in case a limited number of students will participate; The training provider guarantees that the degree, the diploma or the certificate that is promised to a participant upon successful completion of the training, will be issued; The doorlooptijd of the training is a maximum of 12 months; The programme is registered in the CROHO, the ‘Centraal Register Opleidingen Hoger Onderwijs’;The provider guarantees that candidates can follow the programme as proposed without additional costs. It should be clear before-hand which costs are covered by the tuition fee and which should be paid by the candidate him/herself; The tuition fee for a Matra student may


22

Competition was not particularly fierce at the start of MTEC. The University of Amsterdam, that has its own English language Master Programme in International Law since 1998 and the Uni-versity of Wageningen, involved since 1999 with the support to MOP students faced no competi-tors and EVD/CROSS signed an agreement for respectively the education of 2 (one year) and 3 (two years) batches. In 2006, only two universities submitted an application for each of the three academic programmes that were tendered (Public Administration: Twente and Erasmus universi-ties; Public Health: Maastricht and Free University of Amsterdam; European Studies: universi-ties of Maastricht and Twente). Proposals were submitted to EVD/CROSS and a panel comprising two EVD/CROSS staff and two independent experts, the composition of which varied depending on the training subject, as-sesses the proposals20. The selection criteria were included in the Terms of Reference and a ten-der selection form was used which is based on the selection criteria, a mix of financial and tech-nical criteria, with maximum scores for each criterion (see table 2). The selected providers were presented to the Ministry of Foreign Affairs for approval. Table 2: Maximum evaluation scores for proposals Scoring criteria No. of points Sustainability 8 Quality of social programme and personal development programme 15 Quality of training 40 Budget 30 Recruitment and selection of participants 7 Sub-total 100 Suggestions supplementing ToR or further elaboration 5 Total 105

2.5 Programme Monitoring & Evaluation

In terms of Programme monitoring and evaluation (as opposed to internal course evaluation, which is done by the training providers), EVD/CROSS representatives undertake the following:

Pay visits during the implementation of the short courses and academic programmes: attend opening and closure sessions and/or in case certain parts of the training have encountered problems in the past and/or new elements in the courses and/or when line Ministries are involved in course implementation.

Are present at meetings with embassies and representatives of line Ministries that are involved in training course implementation.

Undertake country visits for discussions with former trainees, national coordinators, embassy representatives, etc. These visits take place frequently though not always annually as planned.

The added value of attending open and closing ceremonies as part of EVD/CROSS’ M&E re-sponsibilities is not well understood. The evaluation team appreciates at the same time the ad hoc under no circumstances be higher than the amount charged for a non-Matra student; The provider guarantees that it will be candidates all allow-ances to which they are entitled; The provider may conclude cooperation agreements with sister institutions in the Matra countries in order to en-able these institutions, through transfer of knowledge, to arrange for such training themselves in the future. 20 The requirements for tender dossier have been stipulated in the following terms: A clear description of the ‘end terms’ a candidate must comply with to obtain the diploma or certificate; A complete training programme (including curriculum), including test/exam moments; The name of the training coordinator (including CV); A list of trainers (including CVs) that will be involved in the implementation of various programme compo-nents; A description of the facilities that are offered for accompaniment of the candidate and the way in which the institution will handle housing issues; A description of evaluation instruments; A completed and initialled form concerning the selection criteria; A completed and initialled form concerning the selection criteria; and A completed and initialled form concerning the ‘gunningscriteria’ as well as a completed budget.


23

visits during training course implementation as these served to verify whether problems signalled in previous editions of the course were addressed. It is not clear whether the visits were system-atically recorded. The country-level meetings with former trainees focus on topics that are directly related to the training itself – expectations of the course and whether these were met, views on the course in general and what ought to be changed, content of the course, immediate effect of the training on the country, as well as any after-training contacts with other participants. Notes on these meet-ings with former trainees are written. However, these notes add rather little to the end-of course assessments and internal evaluation reports that are prepared by the Netherlands training insti-tutes. Notes on the interviews per participant or groups of participants are grouped per course and sent to the Netherlands training institutes. Reports analysing the outcomes of these field vis-its interviews are not produced.

2.6 Reporting and consultations on Programme implementation

2.6.1 Programme level reporting As far as progress reports are concerned, EVD/CROSS prepared very detailed reports on ADEPT. The reports provided general information on ADEPT, followed by a very detailed de-scription of each course in terms of contents, structure, information on participants as well an as-sessment, conclusions and recommendations for each course. In addition, the reports contained general conclusions and recommendations for ADEPT as a whole. Annexes could furthermore include the results of interviews that were held during the country visits. Understandably, the Ministry of Foreign Affairs indicated at the time that these reports should be shorter. With respect to MTEC, since 2005 reporting is done through the half-yearly and annual reports that are prepared by EVD/CROSS on MEC as a whole in accordance with the Convenant. On MTEC, these reports contain briefs of a few pages on and short assessments of the individual courses. In addition, information on MTEC is included in the country briefs that together make up more than a third of the reports. Reflection on policy related issues has been limited and the Ministry of Foreign Affairs has indicated that more analyses and more policy level inputs would be desirable. Primarily because the various MEC instruments are handled by different EVD units, reporting is not always in time21. In line with the Convenant, which provides considerable detail with respect to financial reporting requirements22, EVD ensures that financial reports are accompanied by an audit certificate with auditing being done by the Accountancy Department of the Ministry of Economic Affairs.

2.6.2 Meetings Twice a year, in conjunction with the progress and financial reports, meetings are held between the Ministry of Foreign Affairs and EVD on MEC – including MTEC. The agenda for such

21 ‘Tijdigheid: ook deze rapportage is later opgeleverd dan afgesproken’ (Conclusies en afspraken voortgangsoverleg Matra voor Europese Sa-menwerking, EVD en DZO/UM, 19 October 2006). According to the Convenant, deadlines for the submission of reports are six weeks after the end of the 2nd quarter for the report on the first six months and 31 March of the next year for the annual progress and financial report.. 22 Financial reports will contain information on: 1. Available resources at the start of the reporting period; 2. A specification of current and ex-pected projects and per project an indication of verplichtingenbedrag, the amount of expenditures up to the reporting period, the amount of ex-penditures realised during the reporting period, the total amount of realised expenditures, an estimate of expected expenditure for the future per quarter and the closing data of on-going projects; 3. The balance of available financial resources at the end of the reporting period. The financial reports are to be accompanied by a request for an advance (‘bevoorschottingsverzoek’).


24

meetings are prepared by EVD. As per the Convenant, notes on the meetings are prepared by the Ministry. In addition, ad hoc and informal meetings and contacts take place on a regular basis. Once or twice a year, meetings are also held with representatives of the training providers in the Netherlands. Separate meetings are held with training providers of the short courses and with the universities implementing the academic programme (the first one in 2007) to exchange ideas and learn from each other’s experiences. Both the universities and training providers have indicated that these meetings are a valuable initiative although these do not necessary lead to the imple-mentation of the suggestions made. Meetings on MEC are also held between the Ministry of Foreign Affairs, EVD and the Nether-lands line ministries (‘Interdepartementaal Overleg Matra Europese Samenwerking’). Notes on the meetings indicate that they primarily serve to exchange factual information on the different MEC instruments, including MTEC.

2.6.3 Reporting by Netherlands training providers Netherlands training providers prepare elaborate reports on the individual courses as a contrac-tual obligation according to their contracts with EVD/CROSS. There is no uniform format for this kind of reports. It is understood that in recent years EVD/CROSS has increasingly formalised the reporting re-quirements and that, certainly in comparison with the ADEPT years, reporting requirements are more clearly defined. Interviews indicate that some providers do not object against more uniform reporting, for others this is less opportune given their own existing internal reporting formats. Basically, the course reports provide the same type of information: recruitment and selection processes, participants’ profiles, information on the course programme, participants’ assessment of the course and an evaluation by the course provider. Some further streamlining of reporting remains necessary; e.g. an annex of 73 pages with short quotes on the assessment of presenta-tions would seem excessive (ECORYS, report on Structural Funds course May-June 2007). The reports of the Netherlands training providers are sometimes sent to the ministry of Foreign Affairs, though there is no consistency in this respect. The training providers are not informed about how their reports are used by EVD/CROSS, and in what way the reports are used as a source for policy evaluation and/or policy making. Furthermore, training providers have indi-cated that they would find it useful to have insight into the reports of other training providers.

2.7 Financial management

2.7.1 Programme financing According to the Convenant, the Ministry of Foreign Affairs is to make available programme funding for a period of three years (2005-2007) with an extension period of two years for pro-jects started in 2007. The period for which funding was made available thus differs from the life-time of the Convenant itself. After a first advance paid upon signature of the Convenant, pay-ments are made on the basis of approved semi-annual and annual reports and estimated expendi-tures for the next six months. EVD is obliged to remain within the programme estimates and to inform the Ministry of Foreign Affairs in case expected expenditure lag behind or increase sig-nificantly (> 15%). After receipt and approval in writing of the progress report and financial re-port by the Ministry, the final volume of programme funding is determined. According to the


25

Convenant, funds are to be transferred to an interest bearing account; interest accrued on this ac-count will be used for undertaking activities within the framework of this agreement. The programme level financial data provided to the evaluation team lead to the following obser-vations. First of all, it is difficult to get a comprehensive picture of actual expenditures per course over the years – as data are not always consistent and reflect the training providers but not necessarily the courses (data 1999-2001) or gave information on expenditures only, without in-dication of course provider and course. This also applied for data provided on 2005. No up-to-date picture could be generated about the actual expenditures on the academic programmes (2005-2007). Headings are used for the columns on expenditures that are not consistent for the various years which make it difficult to understand how much money was actually spent23. An issue observed as regards programme funding is that transfers from the Ministry of Foreign Affairs to the accounts of EVD/CROSS take place often only in May. This is due to the late ap-proval of EVD/CROSS’ annual plan and has had implications for the timing and organisation of the courses as mentioned above in section 2.3. Moreover, until recently, the entire budget for MTEC was reserved for a set number of courses and academic programmes that had been approved by the Ministry of Foreign Affairs. In recent times, this practice was discontinued and some funding was set aside to be able to respond to more ad hoc training requests24. This would seem to be a good initiative, especially since not all financed allocated are effectively used each year, as it would permit responding to specific and urgent training needs that are identified by the Netherlands embassies in the Matra partner coun-tries.

2.7.2 Programme management costs EVD/CROSS receives Programme funding for its role in MTEC management; the same was true for CROSS with respect to ADEPT. Some more detailed data on EVD/CROSS management funding (budget and expenditures) for the years 2005 to 2007 is provided in table 3. Table 3: Management resources available and used (2005-2007)

2005 2006 2007

Budget Expendi-ture % Budget

Expen-diture % Budget

Expen-diture %

Basisinspanning' 77,833 57,641 74.1% 64,446 61,745 95.8% 49,133 34,727 70.7% Ontwikkeling en beheer 208,995 180,257 86.2% 207,999 231,363 111.2% 238,557 239,582 100.4% Inzet stafdiensten 3,035 4,373 144.1% Out-of-pocket kosten 14,000 5,960 42.6% 14,000 6,462 46.2% 12,500 12,460 99.7% Maximum amount available 303,863 248,231 81.7% 286,445 299,570 104.6% 300,190 286,769 95.5%

The ‘basisinspanning’ refers to expenditures incurred with respect to ‘beleidsadvisering’ (policy advice), reporting, communication and public relations (‘voorlichting’) and other costs. Costs for project development and management, (‘projectontwikkeling en –beheer’) relate to e.g. the preparation of annual plans and tendering of courses and ‘out of pocket expenses’ to the costs of country visits and communication materials. Management or overhead costs as a percentage of total Programme costs have been variable over the years. The average was 5% for the period 1999 to 2004 and 8% in the years 2005-2007. In

23 For purposes of illustration only, the financial data we have been able to collect and structure are provided in Annex 4 to this report. Due to the fact that data are incomplete and difficult to relate to what was actually financed (e.g. in terms of numbers of participants, staff costs, etc.), we have refrained from a detailed analysis. 24 See the Annual Plan for 2007 page 14: ‘Daarnaast houdt CROSS graag de mogelijkheid open om extra, specifieke trainingverszoeken .. te faciliteren die niet in dit jaarplan zijn opgenomen. Voorstellen tot extra activiteiten zullen apart aan DZO/UM worden voorgelegd’. Similar observations are made on page 34.


26

the light of international experience, this percentage is low in the early years and reasonable at present given the efforts made in running MTEC on behalf of the Ministry of Foreign Affairs.

2.7.3 Funding of short courses Training providers include in their proposal a budget, which, if necessary, is adjusted in consul-tation between EVD/CROSS and the training provider. The contract concluded between the two parties is on the basis of this financial proposal. Following contract signature, the implementing institution receives an advance of 80%. Upon completion of the course, EVD/CROSS requires the institution to present its invoice together with an audit certificate (‘accountantsverklaring’). In the interviews, some training providers in-dicated that the financial conditions were insufficiently known when preparing a budget, which later on resulted in complications when submitting the accounts. They also highlighted the issue of delays in transfer of funds mentioned above. Financial information on some of the short courses indicates that costs vary from course to course, depending on factors such as:

Whether the course is new or a repeat of an earlier course – in the first case, considerably higher preparation and development are incurred

Where the course is held – with training in the ‘Randstad’ being more expensive than elsewhere

The costs of fees of (external) experts. The information also indicates that costs of international travel and board and lodging of partici-pants consume a major share of the total budget. An indication of the costs incurred for several MTEC short courses in 2007 is provided in table 4; based on the number of participants in these courses, costs per participant were € 5,911 for the Structural Funds course, € 5,412 for the How to Operate in Brussels training and € 4,896 for the Food Safety course. Table 4: Expenditures for selected MTEC short courses (2007)

Structural Funds % HOB % Food Safety % Staff Management 9,180.00 3.10% 4,000.00 1.51% 27,306.40 12.12%

Team leader 22,000.00 7.44% 18,952.00 7.15% 23,240.00 10.32% Assistant team leader 4,120.00 1.39% 19,200.00 7.24% 27,750.40 12.32% Training, preparation, evaluation

79,941.00 27.02% 43,850.00 16.53% 14,180.15 6.30%

Study tour guidance 4,416.00 1.49% 0.00% 0.00% Sub total personnel 119,657.00 40.45% 86,002.00 32.43% 92,476.95 41.06%

Training materials Training materials 3,938.10 1.33% 17,500.00 6.60% 8,506.50 3.78% Training rooms, coffee, tea

14,382.13 4.86% 98,582.55 37.17% 3,985.29 1.77%

Computers 3,073.10 1.04% 10,029.75 3.78% Communication 3,750.00 1.41% Auxiliary ICT 400.00 0.14% Survey 618.07 0.21% Accountant statement 0.00 2,900.00 1.09% 1,000.00 0.44%

Travel costs Participants 20,220.03 6.83% 21,013.27 7.92% 21,587.09 9.58% Trainers 910.40 0.31% 600.80 0.23% 722.72 0.32%

Insurance 1,847.50 0.62% 901.70 0.34% 1,489.71 0.66% Logistics Excursions 8,851.25 2.99% 16,728.61 6.31% 18,460.39 8.20% Drinks, diners, further expenses 33,234.44 11.23% 7,000.00 2.64% Costs of accommodation Costs (50 persons) 88,117.60 29.78% 76,459.50 33.95%

Costs trainers 600.88 0.20% Other 206.90 0.08% 555.42 0.25%

Total 295,850.50 265,215.58 225,243.57


27

2.7.4 Funding of academic programmes Once a proposal for an academic programme has been selected, a framework agreement is signed between EVD/CROSS and the university. Each framework agreement stipulates the re-quirements for the selection of students and finances available. The maximum budget for a 2-year agreement (2 times a one-year master programme) is € 1,200,000, the maximum for a 3-year framework agreement (2 times a two-year master programme) € 1,740,000, irrespective of the topic of the programme. These amounts include €15,000 for supporting cooperation with a ‘sister institution’. The universities are asked to exhaust the budget as much as possible by pro-viding as many scholarships for this amount. The universities are in general satisfied with the overall budget though there have been com-ments about the lack of flexibility with respect to the use of certain budget lines. An issue in this respect has been the financing of students who for valid reasons were unable to complete the programme in time. There is furthermore uncertainty as regards the budget line for support to ‘sister institutions’ which the universities can include in their proposals but for the utilisation of which they apparently need to seek separate approval from EVD/CROSS.

2.8 Summary of evaluation findings

In terms of programme management, the evaluation findings can be summarised as follows:

The responsibilities of EVD/CROSS in relation to MTEC differ little from those of CROSS under ADEPT, ranging from assessment of needs to management of courses/programmes and financial management. EVD/CROSS basically complies with its responsibilities under the Convenant of 2005 in terms of monitoring and reporting – both on Programme implementation and finances although it turned out to be difficult to obtain a comprehensive and consistent overview of ADEPT and MTEC expenditures over the years. Moreover, its role in terms of providing policy advice to the Ministry of Foreign Affairs is found to be limited.

In terms of monitoring, in-country monitoring visits are regularly carried out; however, they appear to add little to the end-of training reports of the training providers. Moreover, EVD/CROSS attends some training sessions; the added value of this is not fully evident, except when monitoring training elements that were recommended to be improved on earlier occasions.

The EVD/CROSS progress reports, semi annual and annual, provide some detail on MTEC to allow the Ministry of Foreign Affairs to monitor what is happening in general in terms of Programme implementation. Relations between Ministry of Foreign Affairs and EVD/CROSS are functioning well as is confirmed by both sides, though a key con-cern have been the delays in approval of annual plans and in the transfer of programme funds to EVD/CROSS.

Regular meetings between EVD/CROSS and Ministry as well as the training providers are held. These are appreciated by all concerned, even though follow-up to these meet-ings is not always ensured.

The training providers provide ample information on the implementation of the individ-ual MTEC courses. Further streamlining of reporting in order to avoid an overload with detail is recommended; there is no consistency in terms of providing the Ministry of For-eign Affairs with copies of the reports. Training providers would furthermore welcome an exchange of reports and a clearer idea of how their reports are used.

While the Ministry of Foreign Affairs is responsible for policy making, as observed in chapter 1 this role is not played with the requisite vigour and current policy is insuffi-


28

ciently operational, justified and lacks clear objectives, results and indicators. With a fo-cus on concrete activities (courses and academic programmes), the annual ‘jaarplanaan-schrijving’ of the Ministry suffers from the same deficiencies. Moreover, annual plans are approved only later in the year; this affects the planning and implementation of train-ing courses with some 75% taking place after the summer and has contributed to compli-cations in financial management at the level of the training providers.

Regular exchanges also take place between EVD/CROSS and other line Ministries that are supposed to play a role in different stages of the Programme, from identification of needs to involvement in course implementation. The evaluation shows that the role of Netherlands line Ministries in shaping the directions of the Programme remains very lim-ited.

In terms of selection of training providers, the evaluation finds that tendering procedures are not consistently used. Irrespective of the procedures used, it is evident that implemen-tation of short courses has been entrusted to the same limited number of public and pri-vate institutions. The reasons to tender or not are only briefly touched upon in EVD/CROSS’ reports and there seems to be no well-argumented strategy on this issue. Procedures of selecting training providers are not fully transparent for (potential) training providers as some are approached by EVD/CROSS while others have to get information from the EVD/CROSS website. The criteria for getting on the short-list of potential train-ing providers are not clear, since the ‘basic requirements’ mentioned in the Convenant with respect to the academic programmes do not seem to apply for the short courses that are not registered in the ‘Centraal Register Opleidingen Hoger Onderwijs’. In those cases when it is clear that only one specific institute in the Netherlands has the knowledge and capacity to organize the short courses, omission of tendering is understandable since it is unlikely that this would lead to an increase in the number of potential service providers. It is however not evident that this is in compliance with EU and Dutch legislation on ten-dering for the provision of services.

In terms of Programme funding and budget utilisation, it has been difficult to obtain sys-tematic data on expenditures over the years. This has affected the extent to which the evaluation has been able to address questions of efficiency, though it is evident that a ma-jor part of the funding was used to finance costs of stay, (international) travel and per di-ems. Until recently, all available funding was programmed. This practice was discontin-ued in 2007 in order to be able to reserve some funds which would permit responding to ad hoc training requests from the Matra partner countries. Financial data on selected courses indicates that costs per participant vary (ranging between € 4,896 and € 5,911), depending on whether the course is new or a repeat of an earlier course, where the course is held and the level of fees of (external) experts. Like for the short courses, the lion share of the budget for the academic programmes is set aside for accommodation, travel, and costs of living of the students. By and large, the available resources are sufficient, though remarks have been made about the lack of flexibility to use certain budget lines. The cri-teria and procedure for the use of the budget for strengthening of sister institutions are ill defined.

Chapter 3: Short Courses and Academic Programmes: implementation Report on the evaluation of MTEC

29

3. Short courses and academic programmes: implementation

3.1 Introduction

This chapter first of all addresses the question: which training courses and academic programmes were effectively implemented over the years and by whom? It then addresses the indicator ‘themes are priorities for the countries concerned’ by paying attention to issues of responsive-ness to training needs in the Matra partner countries as a key element for ensuring training rele-vance. Attention is paid to mechanisms used to assess these needs and to the number of applica-tions received as key indicators for responsiveness. The chapter subsequently goes into t

final report on the evaluation of the matra training pro- gramme … · 1.3 matra training...

Documents