tool 4 - rural development · tool 4.1 examples of guiding questions for the appraisal of the...

TOOL 4.1 EXAMPLES OF GUIDING QUESTIONS FOR THE APPRAISAL OF THE PLANNED MONITORING, DATA COLLECTION AND IMPLEMENTATION ARRANGEMENTS OF THE CAP STRATEGIC PLAN THEMATIC WORKING GROUP NO 7 ‘PREPARING FOR THE EX-ANTE EVALUATION OF THE CAP STRATEGIC PLAN’ MARCH 2020

Disclaimer: This tool has been prepared by evaluation experts based on good practice available from the current programming period. The document has been consulted with a Sounding Board including Member States’ representatives in February 2020 and has been reviewed in line with the comments received. This document is non-binding and only intended to facilitate the work of evaluators and managing authorities in the context of preparing the ex-ante evaluation of the CAP Strategic Plans. The document is based on the legal proposal for the CAP Strategic Plan Regulation, COM/2018/392 final and does not anticipate any content of any legislative act. It has no interpretative value.

Copyright notice

© European Union, 2020

Reproduction is authorised provided the source is acknowledged.

Recommended citation:

EUROPEAN COMMISSION – Directorate-General for Agriculture and Rural Development – Unit C.4 (2020): Tool 4.1 Examples of guiding questions for the appraisal of the planned monitoring, data collection and implementation arrangements of the CAP Strategic Plan.

Disclaimer:

The information and views set out in this tool are those of the author(s) and do not necessarily reflect the official opinion of the Commission. The Commission does not guarantee the accuracy of the data included in this tool. Neither the Commission nor any person acting on the Commission’s behalf may be held responsible for the use which may be made of the information contained therein.

The Evaluation Helpdesk is responsible for the evaluation function within the European Network for Rural Development (ENRD) by providing guidance on the evaluation of RDPs and policies falling under the remit and guidance of DG AGRI’s Unit C.4 ‘Monitoring and Evaluation’ of the European Commission (EC). In order to improve the evaluation of EU rural development policy the Evaluation Helpdesk supports all evaluation stakeholders, in particular DG AGRI, national authorities, RDP managing authorities and evaluators, through the development and dissemination of appropriate methodologies and tools; the collection and exchange of good practices; capacity building, and communicating with network members on evaluation related topics.

Additional information about the activities of European Evaluation Helpdesk for Rural Development is available on the Internet through the Europa server (http://enrd.ec.europa.eu).

TOOL 4.1 EXAMPLES OF GUIDING QUESTIONS FOR THE APPRAISAL OF THE PLANNED MONITORING, DATA COLLECTION AND IMPLEMENTATION ARRANGEMENTS OF THE CAP STRATEGIC PLAN THEMATIC WORKING GROUP NO 7 ‘PREPARING FOR THE EX-ANTE EVALUATION OF THE CAP STRATEGIC PLAN’ MARCH 2020

TWG 7 Tool 4.1 (March 2020)

Thematic Working Group: Preparing for the ex-ante

evaluation of the CAP Strategic Plan

You can find all the tools for the Thematic Working Group, ‘Preparing for the ex-ante evaluation of the CAP Strategic Plan’, in the European Evaluation Helpdesk’s

eLibrary:

https://enrd.ec.europa.eu/evaluation/publications/_en

1. Working Package 1: Preparing the ex-ante evaluation; • Tool 1.1 - Indicative roadmap for the ex-ante evaluation and Strategic

Environmental Assessment of the CAP Strategic Plan; • Tool 1.2 - Indicative outline of terms of reference for the ex-ante

evaluation of the CAP Strategic Plan; • Tool 1.3 - Indicative outline of terms of reference for the Strategic

Environmental Assessment of the CAP Strategic Plan; • Tool 1.4 - Stakeholder mapping checklist.

2. Working Package 2: Appraisal of the SWOT and needs assessment; • Tool 2.1 – Examples of guiding questions supporting the appraisal of

the assessment of needs including the SWOT; • Tool 2.2 – Analytical tables supporting the appraisal of the assessment

of needs including the SWOT.

3. Working Package 3: Appraisal of the intervention strategy, targets and milestones;

• Tool 3.1 – Examples of guiding questions for the appraisal of the intervention strategy, targets and milestones;

• Tool 3.2 – Working steps for the appraisal of the intervention strategy of the CAP Strategic Plan;

• Tool 3.3 – Appraisal of quantified target values and milestones; • Tool 3.4 – Working steps for supporting the appraisal of the rationale

for the use of financial instruments.

4. Working Package 4: Appraisal of the planned monitoring, data collection and implementation arrangements of the CAP Strategic Plan.

• Tool 4.1 - Examples of guiding questions for the appraisal of the planned monitoring, data collection and implementation arrangements of the CAP Strategic Plan.

https://enrd.ec.europa.eu/evaluation/publications/_en?f%5B0%5D=im_field_enrd_publ_ehd_content_t%3A20709

https://enrd.ec.europa.eu/evaluation/publications/_en?f%5B0%5D=im_field_enrd_publ_ehd_content_t%3A20709


ACRONYMS

CAP Common Agricultural Policy

CPR Proposal for Common Provisions Regulation, COM/2018/375 final

EC European Commission

EIP European Innovation Partnership

EU European Union

FADN Farm Accountancy Data Network

FSS Farm Structure Survey

GHG Greenhouse Gas

GIS Geographic Information System

GQ Guiding Questions

HZR Horizontal Regulation

I&R Identification and Registration

IACS Integrated Administration and Control System

LAG Local Action Group

LDS Local Development Strategies

LEADER Liaison Entre Actions De Développement De L´Économie Rurale

LPIS Land and Parcel Identification System

MA Managing Authority

MS Member State

PA Paying Agency

PMEF Performance and Monitoring Evaluation Framework

RD Rural Development

SPR Proposal for CAP Strategic Plan Regulation, COM/2018/392 final

SWOT Strengths Weaknesses Opportunities Threats Analysis


CONTENT

Introduction ........................................................................................................................................... 1

Guiding questions ................................................................................................................................. 4 GQ 3.1 Is there an effective process in place for setting up the performance framework? .................... 4 GQ 3.2 Are the conceptual elements for the performance framework operationalised? ........................ 4 GQ 3.3 Are the requirements of the performance framework understood by the key actors? ............... 5 GQ 3.4 Have adaptations to the IT system started early enough to allow for its smooth operation from

the beginning of the implementation period? .................................................................................. 5 GQ 3.5 Is the planned process for collecting data on operations, areas and animals suitable? ............ 5 GQ 3.6 Is the planned process for the integration, analysis and clearance of data suitable for

monitoring and evaluation? ............................................................................................................. 6 GQ 3.7 Are the procedures for undertaking consultations suitable for the validation of the required

reports? ........................................................................................................................................... 6 GQ 3.8 Are the provisions adequate to deliver the required reports on time to the European

Commission (Summary assessment of all previous steps) ............................................................ 7 GQ 3.9 Are the planned follow up procedures suitable to deal with monitoring and evaluation

findings? .......................................................................................................................................... 7

Annex ..................................................................................................................................................... 8


1

INTRODUCTION

This document addresses the appraisal of the planned monitoring, data collection and implementation arrangements of the CAP Strategic Plan according to Articles 125(3)(f, g, i) and 125(4) of the proposal COM (2018) 392 final (SPR). The new performance monitoring and evaluation framework (PMEF, Article 115 SPR) and the transition to a performance-based delivery model is challenging, however, significant benefits can be gained through the involvement and input of an independent ex-ante evaluator(s).

The objectives of the performance framework (Article 116 SPR) are as follows:

• to monitor the progress made towards achieving the milestones and targets of the CAP Strategic Plans;

• to allow for the evaluation of the impact, effectiveness, efficiency, relevance, coherence and Union added value of the CAP;

• to support a common learning process related to monitoring and evaluation.

Each Member State will develop its own system to implement the requirements of the performance monitoring and evaluation framework in its own specific context. The ex-ante evaluation may accompany this process and assess the suitability of the performance monitoring and evaluation framework (PMEF) at the Member State level in order to facilitate reporting and monitoring and evaluation of the performance of the CAP Strategic Plan during its implementation.

Reporting activities related to the PMEF cover three areas for which there are different data collection and reporting requirements:

a) Performance Clearance, which is based on expenditures and output indicators and related to milestones and targets.

b) Performance Review, which is based on result indicators and related to milestones and targets.

c) CAP monitoring based on output indicators not used for the performance clearance, result indicators not used for the performance review and the aggregates of the output indicators reported for the performance clearance.

In addition to quantitative data, the annual performance report (APR) contains qualitative information (e.g. justifications in case of deviations from milestones and targets). The implementation of financial instruments is also part of the APR. The APR is accompanied by the annual financial clearance on expenditures and the management declaration on the governance system.

The certification bodies have a new and strong role in the performance audit as they need to verify and confirm the accurate and correct calculation of the result indicators and the functioning of the IT systems and the reliability of the performance reporting procedure (as part of the review and assessment of the governance system).

Many details of the performance framework are still under discussion and can, therefore, be addressed only at a later stage. To understand the requirements of the performance framework, the following internal working papers are important sources of information in addition to the draft CAP Strategic Plan Regulation (SPR) and the Horizontal Regulation (HZR):

• Commission non-paper on the content of the Annual Performance Report. • ‘Cover note’ on output and result indicators. • Presidency drafting suggestions on Indicator Annexes I-III. • Updated fiches on output and result indicators. • Context and impact indicator fiches. • Non-paper from the Commission services on certification bodies.


2

With a view to support the ex-ante evaluator in their evaluation tasks, it is proposed to examine the following examples of non-mandatory guiding questions (GQ) and judgement criteria related to the different phases when setting up the performance framework of the CAP Strategic Plans.

The guiding questions may be used by the evaluators in the ex-ante evaluation and where relevant in the ‘ongoing’ evaluation during the implementation period (acc. to Article 126 SPR). The applicability of the questions will depend on the available information and the implementation status of the performance framework at the time of the ex-ante evaluation. Evaluators can draw on the following information sources:

• Description of the monitoring and reporting structure in the CAP Strategic Plans (it is required to provide a brief description of the monitoring and reporting arrangements established to record, maintain, manage and report the information needed for assessing the performance of the CAP Strategic Plans).

• Lessons learned in terms of evaluation results and previous implementation experiences.

• Other relevant sources.

In the course of the ex-ante evaluation, the evaluator should check, as far as possible, whether the relevant pre-conditions are met and the planned procedures for the performance framework are adequate, while the ‘ongoing’ evaluation assesses the actual functioning of the system in practice. The guiding questions are formulated in a way to be potentially useful in both evaluation phases.


3

The following figure outlines the workflow to set-up and implement the PMEF (each number refers to its respective Guiding Question below).

Figure 1 Workflow and elements of the Performance Monitoring and Evaluation Framework1

Source: European Evaluation Helpdesk for Rural Development, March 2020

1 The timing and the elements of the workflow are based on the legal proposals which are still under discussion and may change.

Consider lessons learned from the

2014-2020 period

Submission by 15 February for previous financial year 16 Oct.

– 15 Oct. (2023 – 2030)

Management declaration by PA

Provision of sufficient information to all key actors on the performance framework and on data collection needs (e.g. manual, training)

Annual financial clearance Art. 51 HZR

(Expenditures)

Annual performance report Art.121 SPR by PA

Annual performance clearance Art. 52 HZR (Output indicators)

Annual performance review Art. 39 HZR + Art. 121 SPR (Result indicators)

Qualitative information (e.g. justification of deviation)

SFC templates

Consistency of data

Opinion by Certification Body

Summary for citizensTransfer of data

Consultation (January – February)

Opinion by Monitoring Committee

Integration,analysis and

clearance of data (June to December)

Collection of data on operations,

area, animals (ongoing)

Clarification of calculation methods

(at the beginning)

Awareness raising and information

(ongoing)

Follow-up Actions

(ongoing)

Unit costs for interventions (planned expenditure / planned outputs)

Calculation of result indicator values incl. yearly milestones and target

Sound link of interventions to specific objectives and result indicators

Approval / Action plan to take remedial action / Sanctions

Analysis of actual status and deviations by PA and MA

Clearance of double counting for output and result values for all interventions by PA

Elaboration of draft performance report by PA and MA including consistency check of data

Suitable data collection system for IACS interventions (full integration of new technologies and digital tools)

Suitable data collection system for non-IACS interventions

Verification of planned data with actual data for completed operations

Early involvement of Certification Body (e.g. as independent

observer)

Early start of the adaption of the IT-

systemSuitable time plan and resources for the adaptation of the IT system in parallel with the CAP Strategic Plan design

User friendly application forms

Integration of direct support and RD interventions data by PA

Supported by evaluation

Consultation of the regional actors (regional CAP Strategic Plan implementation)

Revision of Evaluation Plan Common learning process via CAP Network

Observation letter, Annual Review meeting

3

2

4

5

6

7

8

9

Setting-up phase1


4

GUIDING QUESTIONS

GQ 3.1 Is there an effective process in place for setting up the performance framework?

Explore if the following criteria are fulfilled:

a) A coordinating body or working group has been established at an early stage, which includes all relevant actors.

b) The overall timetable for setting up the system is designed in a way that the performance framework will be launched on time without generating delays.

c) The working results of the coordinating body or working group are well documented and are accessible for the ex-ante evaluator.

GQ 3.2 Are the conceptual elements for the performance framework operationalised?


a) A sound link between interventions to specific objectives and result indicators is established.

b) Clear procedures exist for encoding (linking) operations with a view to avoiding inconsistencies (e.g. assignment to type of intervention, specific objectives) and prevent double counting from the beginning.

c) The calculation of unit costs for all interventions is appropriate (planned expenditure/planned outputs).

d) The calculation of common result indicator values is appropriate2 and in line with the common indicator fiches including yearly milestones and overall targets.

e) Lessons learned for the calculation of (complementary) result indicators from the Rural Development Programmes in the 2014-2020 period have been taken into account (e.g. jobs created, GHG emissions and energy).

f) The certification body/bodies are involved at an early stage, possibly as an observer during the establishment of the IT system. This will allow these bodies to be familiarised with the necessary documentation, operationalisation of calculation methods and guidance, while at the same time respecting the independent role of the certification bodies.

g) The availability of relevant data sources is ensured (i.e. review of existing databases such as IACS, LPIS, FADN, FSS concerning their ability to provide the necessary data on the relevant population of beneficiaries and non-beneficiaries (for control groups)).

h) Consideration is given to the two-step procedure for identifying and updating the target values of result indicators in relation to LEADER Local Development Strategies (in accordance with the Cover Note of the output and result indicator fiches, Section 4).

2 The Annex provides a tool to check the technical feasibility of the indicators


5

GQ 3.3 Are the requirements of the performance framework understood by the key actors?


a) All relevant key actors are identified and involved in the planned monitoring, data collection and implementation arrangements, with particular attention given to the integration of Pillar I and II.

b) Sufficient information is provided to all key actors on the performance framework and on data collection needs.

c) Adequate information and manuals are prepared and disseminated to implementing actors.

d) Other awareness raising and training actions (e.g. workshops, events) are under preparation.

GQ 3.4 Have adaptations to the IT system started early enough to allow for its smooth operation from the beginning of the implementation period?


a) The time plan and resources are suitable for the development of the IT system in parallel with the CAP Strategic Plan’s design, allowing for room for adaption in line with the requirements of relevant regulations and working documents.

b) A risk-assessment has been carried out to identify any potential bottlenecks of the IT system (based on experiences from the past).

c) A back up plan is in place in case the adapted IT system is not ready for the start of the programme.

d) The potential to simplify the administration/implementation of the IT system has been explored.

e) Actors in charge of developing, maintaining and repairing the IT system have sufficient knowledge and/or are trained to acquire the relevant know-how.

f) In the case of CAP Strategic Plans with regionalised elements, adequate connections between the regional and national level IT systems have been installed.

GQ 3.5 Is the planned process for collecting data on operations, areas and animals suitable?


a) A suitable data collection system for IACS interventions is in place (e.g. definitions of geo-spatial application, area monitoring systems, system for the identification and registration (I&R) of animals, agricultural parcels, geographic information systems, claimless system).

b) A suitable data collection system for non-IACS interventions is in place.

c) There are clear requirements concerning data to be provided/collected (accurate, adequate/official sources, timing for delivery, aggregation issues, and individual data protection issues, etc.).

d) The planned application forms take into account a user-friendly design as well as the data needs for monitoring and evaluation of the CAP Strategic Plans.


6

e) Obligations for beneficiaries are introduced to provide data (in the funding contract, application form, etc.).

f) The verification process of planned data with actual data for completed operations to be undertaken by funding bodies is well prepared and feasible.

GQ 3.6 Is the planned process for the integration, analysis and clearance of data suitable for monitoring and evaluation?


a) Each year, adequate human resources and skills are made available in the Managing Authority and Paying Agency to carry out the following tasks:

- Integration of direct support and rural development intervention data by the Paying Agency (electronic databases and geographic information systems enabling the exchange and integration of data).

- Clearance of double counting for output and result values for all interventions ensured by the Paying Agency.

- Analysis of actual status and deviations by the Paying Agency/Managing Authority and supported by evaluation activities (link to evaluation during the implementation period).

- Tracing back and justifying deviations from the targets also in a regionalised context (e.g. if interventions in the national CAP Strategic Plan are implemented at the regional level).

- Adequate provisions to ensure that the Managing Authority can include LEADER contributions in the annual reporting and that at LAG level the contributions are accurately monitored.

- Preparation of the Draft Performance Report (incl. consistency check of data) by the Paying Agency and Managing Authority for further consultation.

GQ 3.7 Are the procedures for undertaking consultations suitable for the validation of the required reports?


a) Each year there is a commitment to have the Draft Performance Report ready for consultation by the relevant actors.

b) A suitable process is in place to allow for discussion and improvement of the Draft Performance Report before submission to the European Commission.

c) Adequate procedures are developed to enable meaningful consultation with the relevant regional actors (for those interventions in the CAP Strategic Plan with regional implementation).

d) Sufficient time is allocated to consult with the Monitoring Committee and the certification body.

e) The certification body has the necessary capacity and resources to verify and confirm the Draft Performance Report.


7

GQ 3.8 Are the provisions adequate to deliver the required reports on time to the European Commission (Summary assessment of all previous steps)


a) The procedures are adequate to deliver the following reports on time in adequate quality:

- Annual performance clearance Article 52 HZR (output indicators);

- Annual performance review Article 39 HZR and Article 121 SPR (result indicators);

- CAP monitoring information (output/result indicators not used in the annual performance clearance and annual performance review, and aggregates);

- Provision of qualitative information (e.g. justification of deviations);

- Annual financial clearance Article 51 HZR (expenditures);

- Summary of annual performance report for the citizens (Article121 of SPR);

- Management Declaration.

GQ 3.9 Are the planned follow up procedures suitable to deal with monitoring and evaluation findings?


a) Procedures are in place to receive, analyse and deal with possible comments of observation letters and annual review meetings (e.g. on deviations or detected weaknesses).

- Procedures are in place for the development of an action plan to take remedial actions (e.g. set up of working groups, evaluations and studies).

- A common learning process is supported (e.g. by involving the CAP Network) to avoid repeating the problem and to draw lessons for the future.

- Procedures are in place for the revision of the evaluation plan and/or adjustment of the evaluation activities to analyse underperforming areas.


8

ANNEX

Checking the feasibility of result indicators

The ex-ante evaluation is intended to check whether the result indicators relevant to the CAP Strategic Plans can be technically implemented and reported on in a given Member State. For this purpose, the indicators which may be difficult to implement should be identified in order to take remedial action. Additionally, the ex-ante evaluation may explore whether there is a plan for the collection of information to facilitate a robust evaluation. The following example shows, how the ex-ante evaluator could check this aspect.

Example of matrix to assess the technical feasibility of result indicators in a specific Member State

1 2 3 4 5 6 7 8

Indicator name, unit of measurement

Intervention concerned

Clarity and feasibility of the calculation method in line with fiches

Timing of data collection

Data base, availability

Need to clean up the data after collection

Availability of a control group if relevant (for evaluation purpose)

Summary assessment

R1: Enhancing performance through knowledge and innovation

Unit of measure-ment: Number of persons benefitting from support for advice, training, knowledge exchange, or participating in EIP operational groups

Education and advice for farmers (Code x)

EIP operational groups (Code x)

Required is the number of beneficiaries, counted cumulatively from the first payment.

Whatever the length of the training/advice action, the beneficiary is counted in the same way.

In order to increase the relevance of the indicator, the type of action and the target group reached should be observed (for evaluation purpose)

Operations for which a first payment (excluding advance payment) to the beneficiary was made in the financial year concerned

Data could already be collected continuously in the past without any problems

Avoiding double counting of persons benefitting from different trainings is too burdensome and will be neglected

No control group of non-beneficiaries is collected for the indicator

The indicator is technically feasible in this form

…

…


9

tool 4 - rural development · tool 4.1 examples of guiding questions for the appraisal of the...

Documents