the evaluation partnership evaluation of the commission’s impact assessment system presented in...

The Evaluation Partnership

Evaluation of the Commission’s Impact Assessment System

Presented in Brussels on June 28th 2007 by:John P. WatsonJohannes M. WolffMartin H. Kuehnemund

www.evaluationpartnership.com

2TEP-PresentationTEP-Presentation

AGENDA

1. Evaluation Overview & Methodology

2. Achievement of Objectives

3. Key Problem Areas & Options for Change


Evaluation Scope Assess the IA system from 2002 to 2006 with an emphasis on June 2005 onwards.

Review the set-up, its outputs (completed IAs), and the impact of IAs on the policy process.

Examine all IAs completed by Commission Services since 2003, for both legislative and non-

legislative proposals, covering all policy fields.

Evaluation Questions The Terms of Reference defined 31 evaluation questions grouped under 3 headings.

Setup / Inputs• 7 questions on General Approach (applied appropriately, timing, etc.)• 4 questions on Support Structures (SG, training, etc.)• 4 questions on Central overview of the System Implementation (co-ordination, etc.)

Outputs• 11 questions (compliance, duration, quality, etc.)

Impacts• 5 questions (IA and legislative work, quality of EC and MS IAs, etc.)

EVALUATION OVERVIEW


Methodology Overview

EVALUATION OVERVIEW

IA Report Screening(150+ IAs)

Quality Assessments(20 IAs)

CaseStudies(6 IAs)

DocumentReview

Interviews

On-line StakeholderConsultation

Data on individual IAsData on the EC IA system

Interviews

With competentEC, EP andCouncil Staff

Doc. Review

Policy documentsGuidelines

External reports

Evaluation of the Commission’s impact assessment systemComparison with other

IA Systems

EC Survey

Of staff in operational units

DG Info Request

Specific DG measures


IA Screening

Screened 158 IAs available on the europa website by September 2006.

35 criteria related to content and format.

EVALUATION OVERVIEW


IA Quality Assessments

20 IAs selected by type of proposal,

lead DG, and year adopted.

In depth analysis of IA reports,

roadmaps, proposals and 20

interviews with EC officials in charge.

EVALUATION OVERVIEW

DG IA Year

AGRI Review of the Sugar regime 2003

DEV Communication on interconnecting Africa 2006

EAC Decision establishing the programme "Citizens for Europe" 2005

EMPL Review of the European Employment Strategy 2003

ENTR Directive laying down rules on nominal quantities for pre-packed products

2004

Regulation on motor vehicles emissions 2005

ENV Directive establishing an infrastructure for spatial information in the Community (INSPIRE)

2004

Environment & Health Action Plan 2004

Thematic Strategy on Air Pollution 2005

FISH Council Regulation establishing financial measures for the implementation of the CFP and in the area of the Law of the Sea

2005

INFSO Communication on "i2010 - a European Information Society for growth and employment"

2005

JLS Framework Decision on procedural rights in criminal proceedings 2004

Regulation on the applicable law and jurisdiction in divorce matters

2006

General Programme Solidarity and Management of Migration Flows

2005

MARKT Directive on Postal Services 2006

Action plan for the implementation of the legal framework for electronic public procurement

2004

SANCO Regulation on plant protection products 2006

TAXUD Communication on the Community programmes Customs 2013 and Fiscalis 2013

2005

TREN Common rules and standards for ship inspection and survey organisations

2005

Electricity from renewable sources 2005

IA Case Studies

Sample of 6 Case studies selected.

Interviews undertaken to engage those

involved in all phases of the IA work

and subsequent policy process.

IA Screening

Screened 158 IAs available on the

europa website by September 2006.


EC Official’s Survey

Survey of officials in operational units to collect views and opinions about the IA system.

It was available via IPM for a 5 week period and received 89 responses.

PROGRESS TO DATE

Stakeholder Consultation

Stakeholder consultation surveyed via IPM to gather general information about IA system.

Available for an 8 week period and received 118 responses.


Information Request

Information Request was sent to 21 DGs.

Provided quantitative and qualitative information

about specific approaches, structures, provisions

and arrangements in different DGs.

Final Interviews

Held final round of interviews to validate findings

and to discuss the feasibility and usefulness of

options for change.

15 interviews with Commission senior managers,

representatives of the European Parliament and

Council.

EVALUATION OVERVIEW

- PART I -

GENERAL IA INFORMATION AND DOCUMENTATION REQUEST

A) Which unit do you work for? Which DG do you work for? (Please mark on of the following) Unit: __________ P O L I C I E S E X T E R N A L R E L A T I O N S

O Agriculture and Rural Development O Development

O Competition O Enlargement

O Economic and Financial Affairs O EuropeAid - Co-operation Office

O Education and Culture O External Relations

O Employment, Social Affairs and Equal Opportunities O Humanitarian Aid

O Enterprise and Industry O Trade

O Environment G E N E R A L S E R V I C E S

O Fisheries and Maritime Affairs O Communication

O Health and Consumer Protection O European Anti-Fraud Office

O Information Society and Media O Eurostat

O Internal Market and Services O Publications Office

O Joint Research Centre O Secretariat General

O Justice, Freedom and Security I N T E R N A L S E R V I C E S

O Regional Policy O Budget

O Research O Bureau of European Policy Advisers

O Taxation and Customs Union O Informatics

O Transport and Energy O Internal Audit Service

O Legal Service

O Personnel and Administration

O Translation

B) Impact Assessment Guidelines 1) Has your DG developed its own Impact Assessment Guidelines/Manual (such as DG SANCO’s ‘Scoping Paper’ or DG MARKT’s ‘Practical Guide for Impact Assessments’)? O Yes O No 2) Has your DG developed a definition of ‘proportionality’ (i.e. the principle of ‘proportionate analysis’) that is different from / goes beyond the one in the Commission’s Impact Assessment Guidelines? O Yes O No


Evaluation in Figures

Overview of the number of persons contacted during the evaluation.

EVALUATON OVERVIEW

Stakeholder Tools for Contact Number

Interviews

Commission Officials in Operational UnitsIA Quality Assessments 20

IA Case Studies 18

Commission Officials in IA Units

EU Interview Programme I 13

EU Interview Programme II 4

IA Case Studies 4

Commission Senior Managers and members of Cabinets


External experts / consultants IA Case Studies 6

European Parliament and Council Representatives


IA Case Studies 23

External Stakeholders (Civil Society & MS) IA Case Studies 18

Total Interviews 119

Surveys

Commission officials in Operational Units EC Officials’ Survey 89

Commission Officials in IA Units Information Request 21

External stakeholders Stakeholder Consultation 118

Total Survey responses 227


AGENDA





ACHIEVEMENT OF OBJECTIVES

1. Improve the quality of Commission proposals, in particular by Facilitating a more systematic, coherent, analytical, open, and evidence-based approach to

policy design; Providing a thorough, balanced and comprehensive analysis of likely social, economic and

environmental impacts.

2. Provide an effective aid to decision-making, in particular by Providing policy makers with relevant and comprehensive information on the rationale behind

proposed interventions, and their likely impacts; Enabling policy makers to assess trade-offs and compare different scenarios when deciding

on a specific course of action.

3. Serve as a valuable communication tool, in particular by Fostering internal communication and ensuring early and effective co-ordination within the

Commission; Enhancing external communication by making the policy development process more open

and transparent to external stakeholders.

Ultimately, the achievement of these key objectives should contribute to a better, simpler and more consistent regulatory environment that helps the EU to meet the objectives of the Lisbon and Sustainable Development Strategies.

Objectives

The IA system’s degree of success depends on how much emphasis is placed on each of

its key objectives.



Objective I: Improve the Quality of Commission Proposals

Results varied: 13 out of the 20 IAs showed some degree of influence on the respective

proposals, but the extent of this influence varied considerably.

• Half of the 20 IAs examined, identified, analysed and compared the most

relevant policy options. The remainder analysed only the preferred option.

• IAs were mostly balanced, but the analysis of economic impacts is often more

developed.

There were significant differences in quality depending on type of proposal (IAs on

legislative proposals were more thorough than non-legislative proposals).

Many IAs suffer from an inappropriate approach, i.e. are launched late in the process,

principle of proportionate analysis not clear, policy decisions already made ex-ante.

Sometimes insufficient tools, expertise, time and resources are available to conduct an IA

capable of significantly improving the quality of the proposal it accompanies.



Objective II: Provide an Effective Aid to Decision Making

A significant part of Commission staff in Services and at Cabinet level saw shortcomings in

relation to the usefulness of IAs, as they did not feel they were sufficiently detailed or were

too much oriented towards the preferred option.

For Members and staff of the European Parliament and Council the vast majority of IAs

failed to meet their expectations, or met with lack of trust in the “objectivity” of Commission

IAs]

Out of six IA case studies, three played no role in the decision-making process. The

remaining (all on legislative or ‘action plan’ type proposals) played some role, but only one

was genuinely useful as the basis for a better informed decision.

The working culture and capacity inside the institutions is not yet conducive to making full

use of IAs.

• Lack of understanding of IA purpose and role.

• MEPs and members of the Council WPs sometimes lack time and expertise.



Objective III: Serve as a valuable Communication Tool

Communication with external stakeholders is undertaken in accordance with the

Commission’s minimum standards on consultation.

Survey results showed that the majority of stakeholders agreed that the IA process gives

interested parties adequate opportunities to provide relevant opinions and information.

Nonetheless, the degree to which stakeholders are satisfied with how consultations are

managed, and how their views were taken into account, varies considerably.

ISSGs for the majority of recent IAs and the development of Roadmaps represent progress

in terms of ensuring early and effective co-ordination and communication.

Limits to the success of IAs as a communication tool are their quality and late timing.



Stakeholder Consultations

In many IA processes, consultation has been carried out to the satisfaction of stakeholders,

and provided relevant and useful input for the Commission.

Some others, however, have been organised in a manner and timeframe that left

stakeholders frustrated, and was unlikely to have had much of an effect on the IA.

Stakeholder opinions on consultations

0%10%

20%30%

40%50%

60%70%

80%90%

100%

Input w as sought atappropriate points

of time during the IAprocess.

There w as enoughtime and

opportunities toprovide input.

The input w assought through

appropriatechannels.

It w as clear w hatinput w as being

sought.

Contributions w ereacknow ledged and

a summarypresented in the IA

report.

Don't know

Strongly Disagree

Disagree

Neutral

Agree

Strongly Agree



Stakeholder Consultations

Most frequently expressed criticism by stakeholders:

• More time: Sufficient time should be provided for responses (eight weeks as an

absolute minimum).

• Earlier consultation: Stakeholders should be consulted as early as possible in

the IA process in order to ensure that balanced, comprehensive and realistic IAs.

• Limit the use of online consultations and closed questions: Many stakeholders

feel that internet surveys tend to be oversimplified.

• Balance and representativeness: More attention to the weighting of the

responses (e.g. from representative organisations vs. individuals).

• Transparency: The exact purpose and scope of the consultation and the

requested data should be made more explicit.

• Feedback and acknowledgment: Stakeholders often feel that they are not

adequately informed about what happens with their input.


AGENDA





KEY PROBLEM 1

Scope of Application and Proportionate Analysis

IA currently applied to a large number of very diverse initiatives on the CLWP.

Blanket coverage of all CLWP items raises difficulties:

• Important legislative and non-legislative proposals may not be included in the

CLWP, and may thus be exempt from the requirement to undergo an IA.

IA Board con prompt IAs on non-CLWP items.

Several DGs have begun to apply IAs voluntarily to non-CLWP items, but

the current system does not guarantee consistent coverage.

• A significant number of items on the CLWP are not well suited to an IA in its

present form:

Result: some extremely short IA reports that add little value, neither

examine different policy options nor properly assess any impacts.

Seen as a bureaucratic hurdle rather than a tool for better policy-making.

May undermine trust in the value and rigour of Commission IAs overall.


KEY PROBLEM 1

Scope of Application and Proportionate Analysis

The principle of ‘proportionate analysis’ provides an element of flexibility and pragmatism.

This is appreciated, but the principle leaves many open questions, and does not of itself

give drafting officials a clear enough indication of what is expected of individual IAs:

• Perceived to only refer to the depth and scope of the analysis of impacts.

• Not to other aspects that require some flexibility, such as: Which steps should be

emphasised? What types of options and impacts should be examined?

Sco

pe o

f applic

atio

n

Level of differentiation

A

B

C

Wide range of initiative are covered, but no clear

guidance as to which elements are required of

different IAs!

The IA system’s scope of application needs to be

brought in line with the level of differentiation,

understood as flexibility – within well established

parameters – to focus on different elements.


OPTIONS FOR CHANGE

Limit scope of application to proposals with most significant impacts

Reduce the number of IAs, by limiting the scope of application to those proposals that are

likely to have the most significant impacts:

• All legislative initiatives in the CLWP.

• Non-legislative CWLP and other non-CLWP proposals (incl. comitology measures) identified by IA Board.

Differentiate between different types of IAs and approaches

Maintain broad scope of application, but differentiate more between different types of IAs,

and provide clear guidance on what is expected of each type.

Operationalise the principle of proportionate analysis by defining broad categories of IAs,

and providing guidance on areas for differentiation and customisation:

Far-reaching legislative proposals Communications on well-defined future action

Legislative proposals with a narrow scope Broad policy-defining Communications

Expenditure programmes Communications of a purely reporting nature


KEY PROBLEM 2

Timing and Approach

Many IAs are perceived as not fully objective, in the sense that they are focused primarily

on justifying a pre-determined policy choice.

Two main factors have limited the ability of IAs to take a disinterested approach:

• Set mandates or political pressure from the Commission hierarchy and / or from the other EU Institutions.

• Late start of IA work, after key decisions on the nature of the proposed intervention (incl. the preferred option) have already been taken.

IAs that do not start from ‘tabula rasa’ can still fulfil important functions.

However, to maximise their value, IA work should start as early as possible, and

accompany the first phases of the policy development process.

Currently, the IA process is not sufficiently built into the early stages of policy design, when

the problems and objectives are framed and options for their achievement considered.

Risk: IAs that start too late become an artificial exercise with a predetermined result.

Late start to IAs also diminishes the usefulness of (internal and external) consultation.


OPTIONS FOR CHANGE

Better exploit the potential of Roadmaps and early consultation

Remind DGs to launch IAs sufficiently early, especially on far-reaching proposals.

Develop stricter minimum standards and quality control mechanisms for Roadmaps .

Ensure early circulation of Roadmaps to other DGs, and explore possibilities for engaging

external stakeholders and the other EU institutions.

Formalise early IA steps by means of a ‘scoping paper’

Require parts of IAs to be produced from the very beginning of the policy process.

Introduce an obligation to produce a document (scoping paper / initial IA) as soon as the

policy idea is generated, outlining the rationale, policy alternatives and their likely impacts.

De-couple the IA process from the drafting of the proposal

Do not allow proposals to be drafted until the IA has been completed:

• Produce IA before the initiative is included in the CLWP.

• Externalise the entire IA process to an expert body?


KEY PROBLEM 3

Quality Control Mechanisms

IA quality control system operates on three levels:

• Lead DG’s operational units and IA support functions, with input from relevant units in other DGs via the ISSGs and the ISC.

• Secretariat-General (SG): ‘guardian’ of the IA Guidelines.

• IA Board: provides guidance and plays a final quality control role.

IAs don’t always meet the desired and required quality standards in terms of their content,

presentation and procedural issues.

While the Commission has made great strides towards developing its IA capacity, not all

DGs have established dedicated IA support functions capable of providing effective initial

as well as ongoing support and quality control.

The SG can provide assistance on procedural issues, but not on policy specific issues.

More informal mechanisms (incl. ISSGs and stakeholders input from consultations) have

only partly been able to mitigate the lack of systematic quality control.


OPTIONS FOR CHANGE

Encourage DGs to further develop IA capacities

All DGs should assess and, where necessary, enhance their capacity to provide operational staff with support in developing high-quality IAs.

Require and encourage top-level sign-off / buy-in of IAs by Commissioners.

Centralise quality control at SG and IA Board

SG takes on the responsibilities of DG IA support functions, becomes central service providing ongoing quality control before passing IAs on to IA Board for final quality check.

Externalise (parts of) the quality control of IAs

Expand the IA Board to include external experts with full voting rights.

Consult external stakeholders on draft final IA reports.

Involve the European Court of Auditors in ex-post IA quality control.

Create a new independent agency for quality control.


KEY PROBLEM 4

Support and Guidance

Provide more / more relevant / more tailored IA training.

Enhance practical usefulness of the IA Guidelines.

Continue to improve methodologies:

• Lack of appropriate methodologies to assess certain types of impacts,

particularly social impacts.

• Integrate and co-ordinate the efforts of different DGs and Member States;

develop guidance documents for assessing impacts in particular fields.

Address problems with availability of data:

• Comprehensive, reliable and comparable data is often not easily available.

Efforts to collect it are resource-intensive and do not always produce the desired

results.

• Develop a more systematic, integrated approach to data collection, involving

DGs, Member States, stakeholder and expert networks.

the evaluation partnership evaluation of the commission’s impact assessment system presented in...

Documents

evaluation questions

questions ia

ia system

teppresentation ia screening

ia work

impact of ias

ias available

central overview