an overview of the findings for the black sea basin programme a project funded by the european union

An overview of the findings for the Black Sea Basin Programme

A project funded byThe European Union

Black Sea Basin Operational Area

BSB Programme Situation and Initial Assessment

Delays in commencing the BSB programme.Main challenges relate to:• Size and Diversity of the programme area

and the relatively small financial allocation;• A complicated geo-political situation involving

many countries, numerous long standing conflicts / challenges between different participating countries;

Situation and Initial Assessment• The need to develop experience (its a new

programme) and develop effective models of cooperation in the BSB region;

• The financing from Turkey (IPA) is based on annual financing agreements with Turkey, which creates issues related to timings and the expiry of the agreement;

Situation and Initial Assessment • The requirement (national) for a paper

submission (with stamp) of applications and other documents required in the application process. Attempting to find a way to allow electronic submission would be important, but will require efforts of all parties to find a solution;

• While the high number of applicants is very encouraging, the number of applications creates an administrative challenge, especially given the delays experienced by the programme.

Situation and Initial Assessment• Technical assistance provided to the

programme by RCBI and Interact is perceived as positive, although the resources were inadequate to meet the needs of the Black Sea Basin, given the size and diversity of the programme.

• Currently, a comprehensive BSB strategy does not exist, which contributes to a strategic void and provides additional complications to direct financial resources to specific challenges.

General CommentCBC is complicated

Conducting a Mid-term Evaluation is complicated

But with sufficient information and a solid evaluation framework (defined in a

programmes OP / JOP) it is possible to complete a good Evaluation

But this was not the case, so based on opinions

According to the EC Monitoring Guidelines – Programmes are assessed on seven (7) Assessment Criteria

Assessed based on qualified opinions of PMS / Stakeholders

Relevance

“To what extent is each individual programme in line with the objectives

indicated in the ENP Strategy Paper and needs of the programme area?”

At this current stage of implementation, this MTE is intended to confirm continued

relevance of the objectives for the stakeholders, given possible changes in the

wider environment.

Relevance - FindingsA significant component of Cross Border

relevance can depend on the choices at programme level in the definition of the eligibility criteria and selection criteria.

Not all cross border mechanisms and as a consequence, not all CBC projects, can aim to

the same type of actions and results. Targeting and limiting priorities are critical

components to ensuring relevance at the Programme Level

Relevance - FindingsIn general, it is clear that significant issues

relating to the high degree of demand / competition have a negative impact on overall

operational viability of the individual programmes. A main conclusion and

recommendation with respect to relevance is specifically related to the

issue of curbing demand and competition.

Efficiency

Efficiency specifies how well the various activities implemented in an individual programmes or in the instrument as a whole (ENPI-CBC) transform available

resources (inputs) into intended outputs in terms of quantity, quality and timeliness.

* Approved within regular CfP/Large scale projects; ** Involving the ENPI financial component only

Opinion of the PMS Opinion of project beneficiaries (lead partners) ‘Integrated’ evaluation

Opinion of the PMS Opinion of project beneficiaries (lead partners) ‘Integrated’ evaluation

Opinion of the programme Opinion of project beneficiaries (lead partners) Integrated evaluation

Efficiency – Aggregate ResponseWith respect to the aggregated responses

for the BSB programme it is clear that both PMS and those beneficiaries polled believe that the project selection process

is, for the most part, efficient.

EffectivenessEffectiveness in general specifies the degree to

which the achieved outputs have provided planned outcomes (benefits) and contributed

to the programme purpose/specific objective(s).

Effectiveness criterion involves the assessment of individual programmes’ outputs which have been delivered, outcomes produced and their contribution to the programme purpose/specific objective(s).

EffectivenessUnfortunately difficult to assess, as

planned/expected outputs and outcomes with a corresponding set of effective output and

outcome indicators were only formally defined in the Baltic Sea Region (BSR)

Programme.

BSB did not elaborate Output Indicators only Outcome Indicators

Programme Output Indicators Outcome indicators

Kolarctic-Russia ― ―

Karelia-Russia ― ―

South East Finland-Russia ―

Estonia-Latvia-Russia Latvia-Lithuania-Belarus Lithuania-Poland-Russia ― ―

Poland-Belarus-Ukraine ― ―

Hungary-Slovakia-Romania-Ukraine ― Romania- Ukraine – Rep. of Moldova Italy-Tunisia ― ―

Baltic Sea Region Black Sea Basin ― Mediterranean Sea Basin ― ―

Opinion of the programme Programme data Integrated evaluation

ImpactThe impact criterion describes the degree to which a

programme’s specific objectives have promoted the achievement of the relevant policy objectives which

were outlined in the overall ENPI CBC policy/strategic framework specifically represented in the programme’s OP / JOP – Normally done Ex

Post But with sufficient information and a solid

evaluation framework (defined in a programmes OP / JOP) it is possible to assess probable/expected

impact.But this was not the case, so based on opinions

EU countries and Norway Partner countries and Russia Integrated evaluation

ImpactIn general and based on available information

BSB programme expects that programme outcomes should sufficiently impact the eligible programme area. In addition,

there appears to be no major deviations in the BSB CBC programme between participating partner countries and member states, with respect to this

finding.

SustainabilityThe Sustainability criterion introduces a time

dimension into the evaluation process. In general it measures the likely continuation of positive benefits (outcomes and impacts) after external

financial and technical support has ended. As far as actual achievement of outcomes at the

programme level (in OPs / JOPs) have not been reported by programmes, sustainability may only

be assessed according to the opinions of competent / mandated programme stakeholders

(PMS or Lead Partners at the project level).

Sustainability

The extent of sustainability of the outcomes/results expected by the BSB

programme can only be predicted based on available perception of PMS, although

with a high probability of overall sustainability, although in a 1-3 year (in

the medium term). No deviations registered between PMS and Partner

Countries

Coherence Coherence generally expresses the connection

of the programme to policies at the higher policy level. The assessment of individual

programmes based on the coherence criterion involves an evaluation of coherence

on a scale from conformity (coherent) to contradiction (incoherent) with regards to the

external policy context.

CoherenceThe only consistent outlier of all CBC programmes

was the Black Sea Basin Programme, which operates in a complicated geo-political

environment, with a significant number of post-soviet era countries at different stages of

transition coupled with an group of new MS (Romania and Bulgaria), MS (Greece), and

countries in the accession process (Turkey). This creates the potential for a significant variance in policy / strategic approach to CBC and regional

development in general, creating a perception of marginally incoherent policy linkage in the BSB.

Community Value AddedCommunity Value Added (CVA) implies the

extent to which interventions are complementary or overlapping / synergetic or

duplicating in every specific geographical region / country / area.

Community Value AddedIt is clear in the case of the BSB programme,

that the objectives of a project funded under ENPI CBC would most likely not be

fulfilled by another project funded by another EU or partner country

programme. In most cases, if the ENPI CBC funding was not available to

stakeholders, most projects would most likely have not happened.

Challenges experienced by both the BSB programme and most other ENPI-CBC Programmes

Common ChallengesIndicators

The quality of the indicators included within the individual Joint Operational Programmes

(OPs / JOPs) over the thirteen (13) programmes reviewed was an exceptional challenge. All OPs / JOPs have significant

weaknesses with respect to official indicators.Black Sea Basin was no exception......but the indicators were better than most programmes

Common ChallengesIndicators

Specifically, in many cases some programmes indicators represent lists of indicators, which are primarily ‘input’ in

nature for example - “the number of projects on”. These do not provide an adequate framework to identify impact and provides limited information on the

quality of projects critical to using indicators as a management tool.

Ref. No IndicatorAchieved value

Target (planned) value

Priority 1

OC I 1.1Number of project partnerships establishing permanent economic relations between the economic actors from different countries after the end of project activities

11 5

OC I 1.2Number of entrepreneurs adopting innovations and starting new production after involvement in projects

3 10

OC I 1.3Number of entrepreneurs / economic agents completing activities and achieving new skills and competencies

485 100

OC I 1.4 Number of new permanent joint products or partnerships in the area of tourism 13 5

OC I 1.5Number of local administrations and organizations activating new types of services or new ways of providing existing services

14 10

OC I 1.6 Number of cross border partnerships for local development projects created 24 10

OC I 1.7 Number of entrepreneurs / economic agents involved in project activities 1190 100

OC I 1.8 Number of training / innovation promotion initiatives for entrepreneurs initiated 29 10

OC I 1.9 Number of local administrations involved in initiatives for capacity building 173 100

OC I 1.10Number of new information, communication, transport and trade links researched and/or established

21 10

Ref. No IndicatorAchieved

value

Target (planned)

value

Priority 2

OC I 2.1

Number of partnerships contracts / agreements establishing permanent relations among institutions / agencies active in the environmental sector 4 5

OC I 2.2

Number of entrepreneurs / technicians / researchers completing activities and achieving new skills and competencies 140 100

OC I 2.3

Number of institutions active in environmental protection adopting innovations developed by projects 15 10

OC I 2.4 Number of environmental training and/or research initiatives carried out 8 20

OC I 2.5 Number of agencies / associations involved in project activities 12 100

OC I 2.6

Number of research / education institutions assisted / involved in project initiatives

11 50

OC I 2.7 Number of trainings initiatives begun in environmental protection 8 10

OC I 2.8 Number of inhabitants of natural areas participating in awareness events 660 100

Ref. No IndicatorAchieved

value

Target (planned)

value

Priority 3

OC I 3.1Number of permanent cultural and educational networks established after the implementation of projects 3 10

OC I 3.2Number of citizens completing cultural projects and achieving educational / cultural objectives 553 100

OC I 3.3Number of students completing an internship or training in partner countries

230 50

OC I 3.4 Number of partnerships created for cultural and educational initiatives 2 5

OC I 3.5 Number of media products produced and distributed by the projects 15 10

OC I 3.6Number of cultural agencies / associations participating in project activities

10 50

OC I 3.7 Number of education institutions assisted in project initiatives 11 20

OC I 3.8Number of citizens / students participating in events and activities implemented in the projects 2373 1000

Common ChallengesIndicators - Baselines

The lack of an effective baseline for all programmes, including the BSB is a significant challenge and therefore creates a lack of any

effective reference point for an evaluation at the programme or ENPI CBC levels).

Where possible use common statistics between countries / harmonise where possible......but this

needs time and effort to complete

Common ChallengesLack of a Logical Framework

The lack of effective logical frameworks across all programmes, which were either

incomplete or missing from the OPs / JOPs, and most importantly have never been

significantly updated (or progress assessed / reported on) in the intervening period (2007

to 2012).

Some Relevant Key Findings for the Black Sea Basin Programme

Project Selection - ChallengeKey challenge is the huge demand for

programme resources reflected in specific CfP

Create excessive workload for Programme Management Structures

Slow overall implementation of programme (Main Bottleneck)

And may significantly reduce operational efficiency

Opinion of the PMS Opinion of project beneficiaries (lead partners) ‘Integrated’ evaluation

Project Selection – SolutionsDefining selection criteria to limit the

number of proposals / applications;The utilisation of best practice experiences

between CBC Programmes – Finnish programmes and Baltic Programmes;

The utilisation of pre-selection processes, such as lists of approved applicants / pre-submission phase / concept notes;

Use of electronic submissions would be useful.

Enhanced Programme Management through Effective Indicators

A core function of any programme is ensuring the most effective and efficient implementation of resources to meet its requirements with the

highest possible sustainable impact.As a consequence, a programme’s ability to

adjust and react to changes and challenges is critical, especially for large and sometimes complex programmes such as the BSB

To meet these challenges, effective monitoring and evaluation systems are required as a management tool,

Programme Regulations and RulesWidely held perceptions of most stakeholders

collected during interviews and programme visits (qualitative responses) suggest that the current regulatory framework of ENPI-CBC

provides an un-necessary administrative burden. Most programmes have stated that

one of the major challenges is “understanding and implementing the various other

implementing rules relevant to ENPI CBC (PraG and national regulations – both

participating MS and PC).

Programme Regulations and RulesIt is clear that any opportunity to reduce the

regulatory burden and simplify the existing rules would be helpful and additionally extend

the overall efficiency of ENPI CBC.But......The Rules are the Rules

Changes to the rules require concrete recommendations from stakeholders which are

implementable. Therefore a comprehensive review of existing rules and regulations to attempt to rationalise and simplify where

possible.

Programme Regulations and Rules

But this involves all stakeholders – EU (PraG and implementing rules), MS national

regulations and Partner Countries regulations / enabling environment

...and there is room for improvement everywhere.

Enhanced Programme Management through Effective Indicators

The heart of any M&E system is the set of specific indicators which are designed to monitor progress towards achieving the

programmes specific objectives.

This is an essential management tool which should support effective operation of

individual programmes and discussed at monitoring committee meetings.

Enhanced Programme Management through Effective Indicators

Extensive Training and Technical Support to programmes during the planning process for the next ENI-CBC programming period, so

that programmes better understand Indicators and the linkage between Input-

Output-Outcome-Impact indicators.Development of effective and manageable

baselines to provide reference points for indicator assessment (especially at the

outcome level) is critical.

Enhanced Programme Management through Effective IndicatorsTherefore: i) review existing statistical fields which are

tracked in the participating countries in a programme;

ii) find common indicators (where adequate statistical data has been historically tracked);

iii) combine these data-sets to provide a useful reference point and significantly shorten the process of indicator / baseline elaboration.

Monitoring and Evaluation at the Programme Level

Plan and implement regular evaluations (including MTE) at the programme level, a task which is still possible during the current programming period.

For the future programmes, ensuring that routine / continuous monitoring actions are built into

individual programmes and that these occur is critical.

In addition, in the case that findings of monitoring / evaluations point to the need for remedial actions

that these are implemented in a timely and effective manner.

Inter-programme connectednessEnhancing inter-programme connectedness / exchanges on a more formalised and regular

basis would enhance the sharing of best practice examples. There is evidence that a

strong will of programmes to share experiences and assist other programmes exists where possible and where such a

sharing does not significantly interrupt or negatively impact own programme

operational efficiency. Some recommendations in this area include:

Inter-programme connectedness / Sharing experiences and best practiceCentralised database to share project

information with the public / other programmes / projects in a forum. Possibly establish a project forum in specific sub-priorities.

Examine the possibility for other best practice exchange options such as job exchanges and the mechanism for allowing for such best practice exchanges (for example how a job exchange could occur between member state civil servants / or between partner country civil servants and MS civil servants).

Final Draft OpinionIn general, based on available information,

the BSB programme will most likely achieve its results and objectives; the

delay in programme becoming operational may provide some risks but these are balanced against the relatively

small financial allocations, swiftly developing capability of the PMS and

significant demand of its stakeholders.