How to evaluate climate policy: Case of an evaluation-based audit of Finnish climate and energy strategy

2012 EEEN Forum 9-10 February, Leuven, Belgium

Paula Kivimaaa & Vivi Niemenmaab

a Senior Researcher, Finnish Environment Instituteb Principal Performance Auditor, National Audit Office of Finland

Pau

la K

ivim

aa, S

YK

E

2

● Challenges for evaluating climate policy

● Performance audit in comparison to policy evaluation

● Evaluation-based performance audit of Finnish climate and energy strategy

● Conclusions on challenges

10.2

.201

2

Contents

Pau

la K

ivim

aa, S

YK

E

3

● Uncertain and long-term cause-effect chains• Emergence of new scientific knowledge• Final impact measurement not possible

● Multiple other causes and policies influencing in desired and opposite directions

• E.g. economic fluctuations, policies supporting employee mobility, municipal interests

● Horizontal involvement of several administrative sectors and policy domains an important feature of climate policy

• How do we define climate policy?• Impacts on effectivess and evaluation

● Slowly changing practices• Despite awareness, it is difficult to change daily practices,

administrative processes, etc.• Marginal change or no change? Cause for change?

17.1

0.20

11

Challenges for evaluating climate policy

Pau

la K

ivim

aa, S

YK

E

4

● There are not enough systematic evaluations of climate policy to support evaluation-based policy making (Haug et al. 2010)

● Existence of both technical-analytical and reflexive evaluations carried out by a number of actors (Huitema et al. 2011)

● “Several long-standing measures reported as climate policies were initially designed as responses to other problems” and “climate policy evaluations do not, in general, challenge dominant framings of the policy problem as one of market and state failure with regard to greenhouse gas emissions” (Haug et al. 2010)

● Evaluations should (1) address the complexity of climate policy making, (2) be reflexive in challenging policy goals and means, and (3) be participatory (Huitema et al. 2011)

17.1

0.20

11

Findings from a climate policy meta-evaluation

Pau

la K

ivim

aa, S

YK

E

5

● ”the independent examination of the efficiency and effectivenss of government organizations, operations, or policies, with due regard to the economy’”(INTOSAI, 2010:11)

● Starts from the rationalistic side in that audits are carried out from the viewpoint of what states can do and should do better and relates to performance-based management

• Not so much concerned with e.g. rational social choice or future paths

● Often have a narrower focus than evaluation as a whole • focused on ex-post and have to be justified by direct or indirect

links to government economy

● Based on principles of good governance and decisions and goals set up in legislation (van der Knaap, 2011)

● Carried out by supreme audit institutions

17.1

0.20

11

Performance audits

Pau

la K

ivim

aa, S

YK

E

6

● Carried out between August 2010 and November 2011 (circa 14 effective working months)

● Focus on the quality of policy preparation & the consistency, effectiveness and cost-efficiency of implementation

● Addressed complexity through multiple methods and data sources & addressed participation through a stakeholder questionnaire, interviews, and a focus group

• Was not reflexive in a sense that the main goals were taken as given from the EU level and did not address the framing of goals and measures

17.1

0.20

11

Case of evaluation-based audit of Finnish climate and energy strategy

Pau

la K

ivim

aa, S

YK

E

7

The effectiveness model (source: Performance Audit Manual of the NAOF)

17.1

0.20

11

Social needs and

objectives

Impacts

Outputs and outcomes

Managing and steering

activities

Organisation and

resources

Main focus of climate and energy strategy audit

Feedback to objective setting

Pau

la K

ivim

aa, S

YK

E

8

17.1

0.20

11

● Was the preparation of the strategy of good quality?○ Was the preparation organised effectively, crossing administrative

sectors, transparently and involving different stakeholders?○ Was the information base versatile and sufficient, and was the

government economy comprehensively acknowledged?○ How were policy aims, instruments and their interrelations considered?○ How were the implementation and responsibilities planned?

● Has the implementation of the strategy strived for consistency, effectiveness and cost-efficiency from the perspective of climate change mitigation?

○ How does climate change mitigation shows in the preparation of government budgets?

○ Is implementation managed effectively and with sufficient resources?○ Has implementation been in line with the strategy statements?○ Has monitoring and evaluation been organised appropriately?

○ And more tentative evaluation of impacts and effectiveness

17.1

0.20

11P

aula

Kiv

imaa

, SY

KE

8

Audit questions

Pau

la K

ivim

aa, S

YK

E

9

17.1

0.20

11

Audit criteria (more tentatively effectiveness)Emission reduction & other legislation

based requirements (Kioto, EU)

Versatility and reliability of

information base

Long-term cost-

efficiency

Horisontality of

governance

Good governance

(transparency, participation)

Pau

la K

ivim

aa, S

YK

E

10

Methods and material17

.10.

2011

Pre-study (3 months)•Key public policy documents•Focus group discussion with policy scientists•Interviews in key ministeries (3/5)• Audit questions and criteria

Main study (9 months)•Electronic stakeholder questionnaire (67 responses, 50% response rate)•Preparatory material (meeting minutes [43], background studies)•Sectoral strategies and background material•Interviews of government officials and experts (23/40) •Government budget proposals from 2008 to 2011•Statistics Finland

Ending the process•Internal quality control•Comment round in government administration•Final meeting of audit organisation•Publication of report (December 2011)

11

ener

gy se

curity

ener

gy ef

ficien

cy

ener

gy sa

ving

rene

wable

ener

gy

econ

omic

instru

ments

build

ing an

d con

struc

tion

trans

port

commun

ity st

ructu

re

waste

manag

emen

t

agric

ultur

e

fores

try

fores

ts as

sink

s

rese

arch

, tech

nolog

y & in

nova

tion

educ

ation

, adv

ice &

commun

icatio

n

strate

gy as

who

le0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

3.38

3.042.82

3.53 3.51 3.51

3.04

2.622.86

2.54

3.11 3.033.23

3.00 3.09

Kaikkien vastausten keskiarvot

Preparation

Implementation

Effectiveness

General view of implementation in different sectors, questionnaire (5=excellent, 1=poor)

Pau

la K

ivim

aa, S

YK

E

12

17.1

0.20

11

Example from the transport sectorNeed for transport

Transport mode

Vehicle efficiency

Fuel emissions

GHG-emission

s

Adapted from Monni & Raes (2008)

Policies (resources, measures)

Sectoral objective setting

(options, information

base)

13

Preparation

Implementation

Effectiveness

Measures influencing transport emissionsArea Supporting and hindering measuresPolicy process - Sector specific target in climate and energy strategy (+)

- Sectoral climate programme, high level network, monitoring (+)- Distribution of responsibility to agencies (+)

Amount of transport (reduction of need)

-Increase in fuel taxation (+) (M. of Finance, fiscal policy)-Joint planning of land use and transport (+) (horizontal, vertical)-Applications for intelligent transport (+)-Other communications policy (+/-)-Tax reductions of commuting costs (-) (M. of Finance)

Choice of transport mode

- Subsidies for public transport (+)- Development programme of public transport (+)- Strategy for walking and cycling (+)- Employee transport ticket (+)- Targetting of transport route investments (+/-)- Tax free parking benefit offered by employer (-) (M. of Finance)

Vehicle fuel efficiency

- Emission limits for vehicles (+)- CO2-based vehicle taxation (+) (M. of Finance)- Energy labelling system for vehicles (+)- Energy efficiency agreement for public and freight transport (+)-Training for economic driving(+)-Energy-efficiency criteria in public procurement(+)

Fuel emissions - Biofuels distribution obligation (+/-) (M. of Employment and Econ.)- Biofuel development programme (+) (MEE)- Subsidies for piloting and demonstration of transport biofuel production plants (+) (MEE)

14

Preparation

Implementation

Effectiveness

Effectiveness and emissions tentativelyArea Main factors increasing and decreasing emissionsOverall - The growth in transport has ’used up’ emissions reduction from

improved vehicle efficiency and biofuels

Amount of transport (reduction of need)

- Tax reduction for commuting costs to support employee mobility (-)- Economic downturn (+)- New communication technology (+)

Choice of transport mode

-Budget funding for public and light transport exists (+)- Resources do not match the estimated need (-)

Vehicle fuel efficiency

- Emission limits to and labeling of new vehicles (+)- CO2-based vehicle taxation (+)- Lower than anticipated renewal of vehicle base (-)- Import of old vehicles (-)

Fuel emissions -- Distribution obligation for biofuels (+)

15

Preparation

Implementation

Effectiveness

Vehicle efficiency: kilometres per tonne of CO2

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 20093000

3200

3400

3600

3800

4000

4200

4400

4600

4800

5000

5200

5400

Km/CO2 t

km

16

Preparation

Implementation

Effectiveness

Summary of implementation

• Significant new measures in renewable energy, transport and energy efficiency of buildings

• Major flaws: • Agriculture, community structure and land use (that affect possibilities

to reduce traffic and its emissions)• No real and innovative openings to promote micro- or consumer-

generation of renewable energy• Budget funds for the transport sector are small

• Monitoring and follow-up

• The official monitoring of strategy is weak, but fulfilled partly through monitoring of government programmes and to reporting to the UNFCCC and EU, which in turn are carried out well.

17

Valmistelu

Toimeenpano

Vaikuttavuus

Example from climate-based budgeting185 % increase in renewable and energy-efficiency, increases in transport and agriculture are small

2008 2009 2010 20110

50

100

150

200

250

300

R&D funding renewable energy energy (other)

energy-efficiency of buildings public transportation agri-environmental measures

climate policy in general

18

Preparation

Implementation

Effectiveness

Summary of effectiveness

• Renewable energy• Achieving 2020 goals seems challenging and dependent on forest

industry as well as the effectiveness of the feed-in-tariff system

• Agriculture’s N2O emissions• In present situation planned emission reduction appear unrealistic and

depend much on EU agricultural policy

• Transport CO2 emissions• So far growth in transport has exceeded the benefits from biofuels

and technological development.

• Community structure• Preconditions for effectiveness are missing outside the MoE:s

administrative sector

Requires further measures in many sectors and in energy efficiency

19

Recommendations of the audit

• Further increasing coordination between administrative sectors

• Improving participation and transparency

• Developing the evaluation of the cost-efficiency of measures and the evaluation of the impacts of the measures on government economy

• Drafting a climate budget

• Identifying subsidy elements that have contradictory effects

• Improving implementation in the weakest areas

• Clarifying the responsibilities of strategy monitoring and development of climate policy evaluation and reporting

Pau

la K

ivim

aa, S

YK

E

20

● It is relatively straight forward to evaluate the progress of policy processes and measures after target setting, the methodological challenges lie evaluating both cost-efficiency and effectiveness

● Evaluation of climate policy (even within sectoral emissions) complicated by

• Multiple policy domains, organisations, levels of governance, scope of emissions analysis (national reported vs. global) and lack of scientific precision regarding impacts

● The effectiveness of climate policies could be tentatively evaluated based on their potential for change

• Incremental vs. system-level change• Intervention logic and whether it is likely to work• Actor perceptions

17.1

0.20

11

Conclusions

Pau

la K

ivim

aa, S

YK

E

21

Division of climate-based human resources in ministeries

17.1

0.20

11

37 %

24 %

5 %

21 %

1 %

1 %

1 %

11 %

employment and economyagriculture and forestrytranport and communicationsenvironmentfinanceprime ministers officeforeign affairs