Policy influence of indicators: The Point project Framework for analysis and lessons

from the UK energy sectorfrom the UK energy sector

Markku LehtonenSussex Energy Group

SRPU University of SussexSRPU, University of Sussexm.lehtonen@sussex.ac.uk

P INT19.6.2009

1

OutlineOutline

FP7 POINT project (Policy Influence of• FP7 POINT project (Policy Influence of Indicators)

• Analytical framework for energy indicator case studyy

• Preliminary findings, hypotheses, problems

P INT19.6.2009

2

Starting pointStarting point

“A substantial literature on knowledge utilizationA substantial literature on knowledge utilization documents how little, on the whole, formal analysis and information influence decisions”analysis and information influence decisions

“…when information is most influential, it is also most invisible. That is, it influences most when itmost invisible. That is, it influences most when it is part of policy participants’ assumptions and their problem definitions, which they rarely p y yexamine”

(J.E. Innes 1998)

P INT19.6.2009

3

Do indicators influence policy?Do indicators influence policy?

“ Indicators do not drive policy People are…Indicators do not drive policy. People are not suddenly converted because they are

f t d ith d t tt h tlconfronted with data, no matter how expertly or how collaboratively designed. Compendia of indicators are not used by policy makers as aids to decision.”

Innes and Booher (2000) ( )

P INT19.6.2009

4

Project objectivesProject objectives

• Find better ways of using indicators for policy making focussing on policies forpolicy-making - focussing on policies for sustainable development and environmental i t ti i t li iintegration in sector policies

• Foster two-way communication between yresearchers and policymakers/users/stakeholderspo cy a e s/use s/s a e o de s

P INT19.6.2009

5

How to get thereHow to get there

• Design a coherent framework of analysis and generate hypotheses on the influence of indicators in policy

Draw on ”knowledge use literature”– Draw on knowledge use literature• Test the framework in a number of policy areas related

to sustainable development and environmental policy p p yintegration– direct and indirect, intended and unintended influence of

indicators on policiesindicators on policies– identify factors that condition the types of indicator influence

• Recommend ways to enhance the role of indicators in policy

P INT19.6.2009

6

Policy areas and indicator typesPolicy areas and indicator types

• Sector integration• Agricultureg• Transport• Energy

• Sustainable development• monitoring of strategies, national and EU level• ex ante impact assessment

• Aggregated indicators, indices• EU-level indices, media usage

P INT19.6.2009

7

Project conceptProject concept

Expertphase

WP2: Framework for use and influence of indicators in policy

men

tWorkshop: “Testing the framework of indicator use and influence”

atio

n

Expertphase

WP2: Framework for use and influence of indicators in policy

men

tWorkshop: “Testing the framework of indicator use and influence”

atio

n

Stakeholder WP3: t

Use and influence of indicators

WP4:I di tR

ole

of

olde

rs”

nd m

anag

em

WP5:C it dn

diss

emin

Stakeholder WP3: t

Use and influence of indicators

WP4:I di tR

ole

of

olde

rs”

nd m

anag

em

WP5:C it dn

diss

emin

phase sector policy indicators

Indicators for policy instrumentsW

P6: “

Rst

akeh

o

rdin

atio

n anComposite

indicators

unic

atio

nadphase sector

policy indicators

Indicators for policy instrumentsW

P6: “

Rst

akeh

o

rdin

atio

n anComposite

indicators

unic

atio

nad

Experts andStakeholders

Workshop: “discussion of the research outcome”

WP7: Synthesis and recommendations

WP

1: C

oo

WP7

: Com

mu

Experts andStakeholders

Workshop: “discussion of the research outcome”

WP7: Synthesis and recommendations

WP

1: C

oo

WP7

: Com

mu

International workshop to disseminate the results WInternational workshop to disseminate the results W

P INT19.6.2009

8

PartnersPartners• National Environmental Research Institute, University of Aarhus (NERI), y ( )• (co-ordinator)• Finnish Environment Institute (SYKE)• Technical University of Denmark, Department of Transport, (DTU-T)Technical University of Denmark, Department of Transport, (DTU T)• Université Libre de Bruxelles, Institute of Environmental Management

and Land-use Planning (ULB)• University of Reading, School of Human and Environmental Science y g

(UoR)• Bayswater Institute (BI)• Science and Technology Policy Research, University of Sussex (SPRU) • University of Malta, International Environment Institute (UoM)• Slovakian Academy of Science, Institute of Landscape Ecology (ILE-

SAS)

P INT19.6.2009

9

Framework

1. Types & pathways of influence1. Types & pathways of influence2. Factors shaping influence

P INT19.6.2009

10

Types of indicator influenceTypes of indicator influenceIndividualROLE OF KNOWLEDGE/ Individual behaviour

Instrumental/

CONCEPTION OF POLICYMAKING

Process IndicatorsArgumentation &

dialoguePersuasion, Legitimisation

Critique Defence

Rational-positivist

Conceptual/ Critique, Defence

INFLUENCE

pDiscursive-interpretative

Intendedpolicy Non-policy

i t

Decisions and actionsNew shared understandings

Political/Strategic

p y

Otherpolicies

impactsg

(learning, framing, agenda-set.)NetworkingLegitimacy (internal, external)

P INT19.6.2009

11

What is specific to indicators?What is specific to indicators?

Use impact influence• Use, impact, influence…– Can there be influence without use? (cf.

l ti lit t ) ‘ ’evaluation literature) – e.g. ‘process use’• Separating indicator influence from the

processes they are embedded in– e.g. evaluations, assessments, working groupsg , , g g p

P INT19.6.2009

12

Types of influence and explanatory f tfactors

Producer factors Policy factors User factorsProducer factorsRepertoiresBeliefsInterests

Policy factorsLong-term frameworkShort-term politicsIndic purpose, pol. stageIssue characteristics

User factorsRepertoiresBeliefsInterests

Dynamics ofindicatordesign

IndicatorfactorsValidityR li bilit

Dynamics ofindicator useArgumentationDi l

SALIEN TI

MAC

Y

designArgumentationDialogue

ReliabilityTimelinessRelevance

Dialogue

NC

E

CREDIBILITY

LEG

IT

LEGITIMACY

Other policies Intended policy Non-policy impacts

Desicions & New shared Networking Legitimacy

I N F L U E N C E

actions(policy change)

understandings(Learning, framing,agenda-setting)

(internal & external)

P INT19.6.2009

13

UK Energy Sector IndicatorsUK Energy Sector Indicators

• White Paper 2003• White Paper 2003• Designed to monitor progress towards the

achievement of UK energy policy goalsachievement of UK energy policy goals– Climate

R li bilit ( it )– Reliability (energy security)– Competitiveness– Fuel poverty

• 4 headline, 28 supporting, and 100+ background indicators

P INT19.6.2009

14

UK energy policy since WP 2003UK energy policy since WP 2003

• Shift in substance• Shift in substance– Rise of energy security (reliability goal)

• JESS working group 2001 -g g p– Revival of nuclear

• Shift in policy style– From consultation, deliberation & participation back to

authoritarianism

What role for indicators in the policy shift?Focus on energy security indicators (?)Focus on energy security indicators (?)

P INT19.6.2009

15

Types of indicator influence:l fi di h thearly findings, hypotheses

• Process of indicator design: once an indicator becomesProcess of indicator design: once an indicator becomes mainstream, it is no longer needed?

• New shared understandings and professional networks– DTI engineers/policy pragmatists vs. Ofgem economists (paradigms!)– Impact from indicators or JESS collaboration as a whole?

• Legitimacyg y- Use of indicators to support & legitimise policy objectives (e.g. nuclear)?- Design of indicators to allow their use as legitimising tools?

• Non policy impacts• Non-policy impacts- Security indicators and performance management? (trust, blame

shifting, etc.)Manipulative use of indicators and the quality of democracy?- Manipulative use of indicators and the quality of democracy?

P INT19.6.2009

16

Factors shaping influence:l fi di h thearly findings, hypotheses

• Indicator frameworks rather than individual• Indicator frameworks, rather than individual indicators

• Forward-looking indicators more influential?Forward looking indicators more influential?• Causality: do indicators shape or are shaped by

policy priorities?p y p• UK policy context (analytic, adversarial,

managerial) enhances strategic and political role of indicators

• ‘Indicator revolution’ not (yet?) arrived in the UK t ?energy sector?

P INT19.6.2009

17

ChallengesChallenges

‘Isolating’ indicator influence• Isolating indicator influence• Concepts and frameworks: discursive-

interpretative vs. ‘rationalistic’ approaches– ‘process impacts’p p– indicator frameworks, not single indicators

• Access to information: value of participant• Access to information: value of participant observation, ‘action research’

P INT19.6.2009

18

Purpose of indicatorsPurpose of indicators

• “What do you mean by indicators?”• What do you mean by indicators?• “We produce indicators, because we have to”• “I don’t have a clear idea of who might use our• I don t have a clear idea of who might use our

indicators”• “Indicators are only a minor part of the evidence-base we• Indicators are only a minor part of the evidence-base we

produce to support policymaking”• “Indicators are not produced to influence policy. WeIndicators are not produced to influence policy. We

produce evidence to inform policy, not to influence it“(DECC indicator producer)

P INT19.6.2009

19

Belief in rational decision-makingBelief in rational decision making

• “in the end all decisions boil down to the• in the end, all decisions boil down to the allocation of your efforts: where does a unit of additional effort bring the greatest marginaladditional effort bring the greatest marginal benefit”

• “I wouldn’t like to imagine a time when• I wouldn t like to imagine a time when policymaking was not evidence-based; what a terrifying idea!”a terrifying idea!

• “The PM framework covers each and every task undertaken by the Department”task undertaken by the Department

P INT19.6.2009

20

Evidence-based policy vs. li b d idpolicy-based evidence

“It is more common that policies are indeedIt is more common that policies are indeed modified when the evidence does not support th li h i th th id bthe policy choices than the evidence base being modified (manipulated) in order to provide support for the already chosen policy.” (DECC indicator producer)

• But: no concrete examples…

P INT19.6.2009

21

Political & strategic role of i di tindicators

“Indicators often play a negative role in• Indicators often play a negative role in policy, as they are used to prevent rather th f t h t d l bthan foster change towards a low-carbon economy “ (energy policy researcher)

• e.g. competitiveness indicator – exported to EU

P INT19.6.2009

22

MethodsMethods

• Document analysis (e g consultation responses)• Document analysis (e.g. consultation responses)• Interviews

– BERR/DECC Defra CCC CSD PIUBERR/DECC, Defra, CCC, CSD, PIU– Regional (and local?) authorities– Industry– NGOs– Consultants

P liti i– Politicians– Academics

• Stakeholder workshop• Stakeholder workshop

P INT19.6.2009

23

Origin of theUK E S t I di tUK Energy Sector Indicators

“We will need appropriate indicators to monitor progressWe will need appropriate indicators to monitor progress. Government already publishes an extensive range of energy indicators and these will continue to be publishedenergy indicators, and these will continue to be published annually. But we need to focus on a smaller set of indicators to give a broad overview of whether overallindicators to give a broad overview of whether overall energy policy objectives are being delivered. Therefore, as a supplement to the white paper, we will be seeking pp p p gviews on the most appropriate indicators to focus upon.” (Energy White Paper 2003, p. 113)

P INT19.6.2009

24

The EndThe End

• “What do you mean by indicators?”• What do you mean by indicators?• Forward vs. backward-looking indicators &

evidence (reliability indicator vs CO ;evidence (reliability indicator vs. CO2; scenarios vs. stats/indicators)Wid di t h t ‘ d’• Wide disagreement on what are ‘good’ indicators (e.g. fuel poverty indicator)

• EBP and target-setting not a revolution in the energy sector?

• How to ‘buy’ civil servants’ time?

P INT19.6.2009

25

Long-term framework conditionsLong term framework conditions

• Policy style • Democracy (vs• Policy style• Politico-administrative

structures

• Democracy (vs. authoritarianism)

• Environmental policystructures• Position of change

agents

Environmental policy integration

• Independence of theagents• ‘Evaluation culture’• Discussion culture

Independence of the media

• Status of NGOs & civil Discussion culture• Level of development of

administrative

society• etc.

structures

P INT19.6.2009

26

Stage of policy process (Blumer 1971) & l f i di t1971) & role of indicators

POLICY STAGE INDICATOR ROLE1. The emergence of the problem

2 Legitimisation

Discursive & rational? Enlightenment?

C t l l /2. Legitimisation

3. Mobilisation of the public for action

Conceptual role/discursive-interpretative

Consensus Instrumental/4. Formation of an official plan of action

5. Implementation of the plan

Consensus

Controversy

Instrumental/Rational-positivist

Strategic, political,

6. Monitoring/evaluation/assessment/appraisal??

‘Potemkin village’

P INT19.6.2009

27

Research questions: t f i fltypes of influence

• What role have the UK Energy Sector Indicators played• What role have the UK Energy Sector Indicators played in supporting the follow-up of the four energy policy goals?– rational policymaking: monitoring, evaluations, assessment– policy learning: cross-sector integration, definition of shared

understandings, framingsg , g– forums for discursive battles over the definition of energy policy

priorities (e.g. legitimisation, persuasion, critique, defence)• How have the shifts in policy style and priorities been• How have the shifts in policy style and priorities been

reflected in the Energy Sector Indicators?– Do indicators drive policy shifts or are indicators designed

according to new policy priorities?

P INT19.6.2009

28