development of a system of common indicators for european ... · confirmed as an output indicator...
TRANSCRIPT
-
Framework Contract
Conceptual framework and analysis of indicators
1
Andrea Gramillano (t33) – SWECO Consortium
Evaluation network meeting
Brussels, 5th March
Development of a system of common indicators for European Regional Development Fund and Cohesion Fund interventions after 2020
-
Outline of the presentation
I. Methodological approach
II. What can be used from the existing list?
III. Elements for discussion: examples of proposed indicators
IV. Conclusions
2
-
• For output and direct result indicators
1. Identification of the most frequent intervention fields at TO level
2. Analysis of the link between the existing common output indicators and the intervention fields
3. Proposal of additional indicators (if there is a gap) tightly linked with the intervention field and IPs based on:
Programme experience (scale-up)
Literature review
3
I. Methodological approach (1/5)
-
• Criteria of good indicators – Better regulation toolbox
RACER criteria
1. Relevant (thematic coverage and policy responsiveness)
2. Accepted (understood by those in charge of data collection)
3. Credible (unambiguous)
4. Easy to monitor (feasible in terms of costs)
5. Robust (clearly defined and with a sound methodology)
4
I. Methodological approach (2/5)
-
• Other criteria
Time-bound (when is the right timing to measure them? – for direct result indicators)
Clear (indicate unequivocal direction of change, in particular for direct result indicators)
Debatable (can be used for benchmarking and comparison)
• FOCUS on “aggregation rules”
from project to programme
from programme to EC
5
I. Methodological approach (3/5)
-
• Gap analysis underpinning the proposed list of common indicators (example TO 1)
PROCESS - CO01 “Enterprises receiving support” is the main candidate process indicator for TO 1. An additional process indicator might be included to measure the supported research institutions
OUTPUT - CO26 “Enterprises cooperating with research institutions” can be confirmed as an output indicator (proxy of technological transfer) and completed with other indicators measuring the value of purchased equipment / services and with an indicator on built / renovated research infrastructure (surface of intervention, etc.)
6
I. Methodological approach (4/5)
-
• Gap analysis underpinning the proposed list of common indicators (example TO 1)
DIRECT RESULT
Indicators on the private investments matching public support (CO06, CO07, CO27), employment increase (CO08) and new researchers (CO24) can be confirmed and used as direct result indicators, even if CO08 and CO24 are more difficult to be measured (as highlighted by the consultation).
Indicators CO28 and CO29 measuring the number of supported enterprises to introduce innovation can be refined with other indicators measuring the (product / process) innovations resulting from the projects.
Other indicators on research activities (prototypes, articles submitted for publication) and patenting can be added.
7
I. Methodological approach (5/5)
-
8
I. Methodological approach
II. What can be used from the existing list?
III. Elements for discussion: examples of proposed indicators
IV. Conclusions
-
II. What can be used from the existing list? (1/7)
• Most frequently used indicators in TO 1 and TO 3 are either process or direct result indicators
9
Common output
indicator IP1a IP1b IP3a IP3b IP3c IP3d
CO01 10 173 139 63 77 136
CO02 2 108 90 52 69 87
CO03 49 63 12 44 89
CO04 1 48 68 19 11 63
CO05 2 40 104 8 13 40
CO06 2 46 36 33 40 43
CO07 13 40 5 23 56
CO08 1 52 102 30 50 86
CO24 50 50 100
CO25 119 11 130
CO26 32 161 1
CO27 33 102 2 2
CO28 86 5 10 22 6
CO29 2 82 10 7 35 11
Looks like a process indicator Looks like an output indicator Looks like a direct result indicator
-
II. What can be used from the existing list? (2/7)
• TO 4 is well covered (ERDF better than CF) but with differences across the IPs
10
Share common / total
IP CF ERDF CF and ERDF
4a 70%
4b 75%
4c 69%
4d 64%
4e 23%
4f 63%
4g 52%
4i 57%
4ii 50%
4iii 48%
4iv 67%
4v 22%
4vi 29%
Grand Total 45% 58% 58%
Larger use in IPs 4a(4i) and 4b (4ii) than IPs 4e (4v)
CF programmes use less common output indicators and a part of those used by ERDF programmes
-
II. What can be used from the existing list? (3/7)
• Limited used of common output indicators in TO 5
11
5a
(ERDF)
5b
(ERDF)
5i
(CF)
5ii
(CF)
(a) Common output 21 55 6 5
(b) Programme-specific 23 102 16 24
Common output share
[a /(a+b)] 48% 35% 27% 17%
-
II. What can be used from the existing list? (4/7)
• In most cases in TO 6, common output indicators measure either a direct result (e.g. CO09 on expected number of visits) or a real output (land rehabilitation - CO22)
12
IP CO 09
CO 17
CO 18
CO 19
CO 20
CO 22
CO 23
CO 29
CO 34
CO 37
CO 38
CO 39
CO 40
Total
6a 31 1 32
6b 39 46 85
6c 96 1 1 1 2 101
6d 7 75 1 83
6e 1 31 1 2 17 41 22 4 119
6f 4 1 15
6g 5 2 18
6i 10 2 12
6ii 13 13 1 27
6iii 8 8
6iv 5 1 6
ERDF
96 40 39 47 1 38 77 1 4 18 43 24 4 453
CF 10 13 13 8 8 1
Tot 96 50 52 60 1 46 85 1 5 18 43 24 4 506
-
13
Type COI 2014-2020 (examples)
Process indicators CO01 Number of enterprises receiving support
CO05 Number of new enterprises supported
CO02 Number of enterprises receiving grants
CO03 Number of enterprises receiving financial support other than grants
CO04 Number of enterprises receiving non-financial support
II. What can be used from the existing list? (5/7) - Examples
-
14
Type COI 2014-2020 (examples)
Output indicators
CO11 Total length of new railway line (km) CO12 Total length of reconstructed or upgraded railway line (km) CO14 Total length of reconstructed or upgraded roads (km) CO15 Total length of new or improved tram and metro lines (km)
II. What can be used from the existing list? (6/7) - Examples
-
15
Type COI 2014-2020 (examples)
Direct result indicators
CO06 Private investment matching public support to enterprises (grants) CO07 Private investment matching public support to enterprises (non-grants) CO24 Number of new researchers in supported entities CO27 Private investment matching public support in innovation or R&D projects CO08 Employment increase in supported enterprises CO20 Population benefiting from flood protection measures CO21 Population benefiting from forest fire protection measures
II. What can be used from the existing list? (7/7) - Examples
-
16
I. Methodological approach
II. What can be used from the existing list?
III. Elements for discussion: examples of proposed indicators
IV. Conclusions
-
• IP 1a
Name: Number of articles submitted to peer-review due to the supported operations (number)
Type: Direct result indicator
Source: programme-specific output indicators, EEA and Norway Grants 2014-2021
17
III. Elements for discussion: examples of proposed indicators – TO 1
Thematic coverage: IP 1a
Harmonisation: Reference to the “peer-reviewed publications”, without
requiring the approval and high-impact factors (similar to Horizon 2020).
Capitalisation: programme-specific output indicators
-
• IP 1a / 1b
Name: Built / renewed research infrastructure (square meters / number)
Type: Output indicator
Intervention field: 58, 59, 60, 61
Source: programme-specific output indicators
Harmonisation: Definition of research infrastructure – Horizon 2020
18
Thematic coverage: IP 1a / 1b
Harmonisation: other EC programmes
Capitalisation: programme-specific output indicators
III. Elements for discussion: examples of proposed indicators – TO 1
-
• IP 3b / 3d
Name: Share of supported SMEs with increased exports (%)
Type: Direct result indicator
Source: programme-specific output indicators
Rule of aggregation at EU level = (1) Number of SMEs receiving support (process indicator),
(2) Supported SMEs with increased exports (supporting indicator of the direct result indicator calculated at programme level)
(3) = (2)/(1): Share of supported SMEs with increased export (%) (direct result indicator).
19
III. Elements for discussion: examples of proposed indicators – TO 3
Harmonisation: Aggregation rule at EU level (Rural programmes)
Capitalisation: programme-specific output indicators
-
• IP 4e / 4v
Name: Public recharging points for electric vehicles installed (number)
Type: Output indicator
Intervention field: 043
Source: programme-specific output indicators
20
III. Elements for discussion: examples of proposed indicators – TO 4
Thematic coverage: IP 4e / 4v
Harmonisation: EC Directive 2014/94 (definition of recharging points)
Capitalisation: programme-specific output indicators
-
21
III. Elements for discussion: examples of proposed indicators – TO 4
• IP 4a, 4c, 4d / 4i, 4iii, 4iv
Name: Modernised / renovated / extended heating / thermal network (km)
Type: Output indicator
Intervention field: 015
Source: programme-specific output indicators
Thematic coverage: TO 4
Capitalisation: programme-specific output indicators
-
22
III. Elements for discussion: examples of proposed indicators – TO 5
• IP 5a, 5b/ 5i, 5ii
Name: Average response time to emergency situations
Type: Direct result indicator
Intervention field: 087, 088
Source: programme-specific output indicators
Thematic coverage: The indicator measure the result not in terms of additional population but in terms of “performance” of the interventions in
emergency situations
Capitalisation: programme-specific output indicators
-
23
III. Elements for discussion: examples of proposed indicators – TO 5
• IP 5a, 5b/ 5i, 5ii
Name: Local plans for risks protection adopted / revised
Type: Output indicators
Intervention field: 087, 088
Source: programme-specific output indicators and Covenant of Mayors
Thematic coverage: TO 5 Harmonisation: Covenant of Mayors
Capitalisation: programme-specific output indicators
-
24
I. Methodological approach
II. What can be used from the existing list?
III. Elements for discussion: examples of proposed indicators
IV. Conclusions
-
IV. Conclusions
A. Continuity - Maintain / refine as much as possible the existing indicators if they have been used successfully and discard the others
B. Feasibility and harmonisation - Introduce indicators which either come from the literature review (other DGs, EU directives, practices, etc.) or from programme-specific indicators
C. Timeliness – Direct result indicators can be measured either at the project completion (e.g. private investment matching public support) or some time later
D. Simplification – Clear guidance and thematic coverage
25
-
Thank you for your attention
26
Andrea Gramillano
-
ANNEX – other indicators
27
-
• IP 4e / 4v
Name: Low-emission public transport vehicles purchased or refitted (number)
Type: Output indicator
Intervention field: 043, 083
Source: programme-specific output indicators
Link with programme-specific output indicators: other programme-specific output indicators can be added to specify the fuel and the type of vehicle.
28
III. Elements for discussion: examples of proposed indicators – TO 4
-
29
III. Elements for discussion: examples of proposed indicators – TO 6
• IP 6a, 6b, 6f, 6g, 6i, 6ii
Name: Additional population served by supported recycling facilities (Population)
Type: Direct result indicator
Intervention field: 017, 018, 019
Source: programme-specific output indicators, based on the existing CO17, CO18