ilri process and partnership for pro-poor policy change methods for assessing policy impact process...

ILRI Process and Partnership forPro-Poor Policy Change

Methods for Assessing Policy Impact

Process and Partnerships for Pro-Poor Policy Change,

Project Initiation Workshop 1

ILRI, 21st February 2005


Workshop Outline• Introduction to the RAPID Framework and

ILRI/ODI Project• Case Study Approach• Episode Study Approach• Outcome Mapping Approach• RAPID Outcome Assessment (ROA)

ApproachLunch• Practical Sessions


Workshop Purpose & Objectives

ObjectivesBy the end of the workshop, participants will:• understand the Process and Partnership for Pro-Poor

Policy Change project’s purpose and general approach• have the opportunity to contribute their own suggestions to

improve the project;• understand, and have had the chance to try out the three

key methods which will be used in the project;• assess the usefulness of the approaches in their own work.

Purpose: To familiarise the participants with the general approach and specific methods to be used in the SDP case study

An introduction to the RAPID Framework and ILRI/ODI Project


Definitions• Research: “any systematic effort to increase the

stock of knowledge”

• Policy: a “purposive course of action followed by an actor or set of actors”

– Agendas / policy horizons

– Official statements documents

– Patterns of spending

– Implementation processes

– Activities on the ground


Policy Processes

- Identify a policy problem

- Commission research

- Assess the results

- Select the best policy

- Establish the policy framework

- Implement the policy

- The problem is solved


Reality…• “The whole life of policy is a chaos of purposes

and accidents. It is not at all a matter of the rational implementation of the so-called decisions through selected strategies 1”

• “Most policy research on African agriculture is irrelevant to agricultural and overall economic policy in Africa2”

1 - Clay & Schaffer (1984), Room for Manoeuvre; An Exploration of Public Policy in Agricultural and Rural Development, Heineman Educational Books, London2 – Omamo (2003), Policy Research on African Agriculture: Trends, Gaps, and Challenges, International Service for National Agricultural Research (ISNAR) Research Report No 21


Existing theory1. Linear model2. Percolation model, Weiss3. Tipping point model, Gladwell4. ‘Context, evidence, links’ framework, ODI5. Policy narratives, Roe6. Systems model (NSI)7. External forces, Lindquist8. ‘Room for manoeuvre’, Clay & Schaffer9. ‘Street level bureaucrats’, Lipsky10. Policy as social experiments, Rondinelli11. Policy Streams & Windows, Kingdon12. Disjointed incrementalism, Lindquist13. The ‘tipping point’, Gladwell14. Crisis model, Kuhn15. ‘Framework of possible thought’,

Chomsky16. Variables for Credibility, Beach17. The source is as important as content,

Gladwell

18. Linear model of communication, Shannon19. Interactive model, 20. Simple and surprising stories,

Communication Theory21. Provide solutions, Marketing Theory I22. Find the right packaging, Marketing II23. Elicit a response, Kottler24. Translation of technology, Volkow25. Epistemic communities26. Policy communities27. Advocacy coalitions etc, Pross28. Negotiation through networks, Sebattier29. Shadow networks, Klickert30. Chains of accountability, Fine31. Communication for social change,

Rockefeller32. Wheels and webs, Chapman & Fisher

www.odi.org.uk/rapid/lessons/theory


Existing theory – a short list• Policy narratives, Roe• Systems of Innovation Model, (NSI)• ‘Room for manoeuvre’, Clay & Schaffer• ‘Street level bureaucrats’, Lipsky• Policy as social experiments, Rondene• Policy streams and policy windows, Kingdon• Disjointed Incrementalism, Lindblom• Social Epidemics, Gladwell

• The RAPID Framework


An Analytical Framework

The political context – political and economic structures and processes, culture, institutional pressures, incremental vs radical change etc.

The evidence – credibility, the degree it challenges received wisdom, research approaches and methodology, simplicity of the message, how it is packaged etc

External Influences Socio-economic and cultural influences, donor policies etc

The links between policyand research communities – networks, relationships, power, competing discourses, trust, knowledge etc.


Case Studies• Sustainable Livelihoods: The

Evolution of DFID Policy

• The PRSP Initiative: Research in Multilateral Policy Change

• The adoption of Ethical Principles in Humanitarian Aid post Rwanda

• Animal Health Care in Kenya: Evidence fails to influence Policy


A Practical Framework

External Influences political context

evidencelinks

Politics and Policymaking

Media, Advocacy, Networking Research,

learning & thinking

Scientific information exchange & validation

Policy analysis, & research

Campaigning, Lobbying


What you need to know• The external environment: Who are the key actors?

What is their agenda? How do they influence the political context?

• The political context: Is there political interest in change? Is there room for manoeuvre? How do they perceive the problem?

• The evidence: Is it there? Is it relevant? Is it practically useful? Are the concepts familiar or new? Does it need re-packaging?

• Links: Who are the key individuals? Are there existing networks to use? How best to transfer the information? The media? Campaigns?


What researchers need to do

What researchers need to know

What researchers need to do

How to do it

Political Context:

Evidence

Links

• Who are the policymakers?• Is there demand for ideas?• What is the policy process?

• What is the current theory?• What are the narratives?• How divergent is it?

• Who are the stakeholders?• What networks exist?• Who are the connectors,

mavens and salesmen?

• Get to know the policymakers.• Identify friends and foes.• Prepare for policy

opportunities. • Look out for policy windows.

• Work with them – seek commissions

• Strategic opportunism – prepare for known events + resources for others

• Establish credibility• Provide practical solutions• Establish legitimacy.• Present clear options• Use familiar narratives.

• Build a reputation• Action-research• Pilot projects to generate

legitimacy• Good communication

• Get to know the others• Work through existing

networks.• Build coalitions.• Build new policy networks.

• Build partnerships.• Identify key networkers,

mavens and salesmen.• Use informal contacts


Policy entrepreneurs

Storytellers

Engineers

Networkers

Fixers


Practical ToolsOverarching Tools

- The RAPID Framework - Using the Framework - The Entrepreneurship

Questionnaire

Context Assessment Tools- Stakeholder Analysis - Forcefield Analysis - Writeshops - Policy Mapping - Political Context Mapping Communication Tools

- Communications Strategy- SWOT analysis - Message Design - Making use of the media Research Tools

- Case Studies - Episode Studies - Surveys - Bibliometric Analysis- Focus Group Discussion

Policy Influence Tools- Influence Mapping & Power Mapping - Lobbying and Advocacy - Campaigning: A Simple Guide - Competency self-assessment


Practical Application• Within ODI• Workshops for researchers, policy makers

and activists.• Advice to a DFID forest/ground water

research project in India:– Less research– More communication– Developing champions in regional and national

government – Local, Regional & National advocacy campaign


Further Information / Resources

• ODI Working Papers

• Bridging Research and Policy Book

• Meeting series Monograph

• Tools for Policy Impact

• RAPID Briefing Paper

• www.odi.org.uk/rapid


Yes, but:• It this its role?• “Global Public Good” Research vs Policy Advocacy• Probably needs to do both:

How?• Understand the political context• Get the evidence & package it well• Strategic networking / lobbying / campaigning• Collaboration….

Can ILRI do it?

ILRI Process and Partnership forPro-Poor Policy ChangeILRI

International Livestock Research Institute

The New DfID funded Project

Process and partnership for pro-poor policy change



• Project Leaders: ODI / ILRI

• Key collaborators: ECAPAPA

• Case study collaborators in Kenya: – MoLFD / KARI– Range of NGOs & other SDP partners

ILRI Process and Partnership forPro-Poor Policy Change Why would I be interested?ILRI


• Not all research is expected or intended to lead to policy change, but there may be;– Specific cases where research is expected to;

• provide evidence for policy change• identify potential policies (or impact of)• influence the policy making process

(advocacy)– Cases where speculative research becomes

relevant because of changes in circumstance

ILRI Process and Partnership forPro-Poor Policy Change The project …ILRI


• Ideas for methods and approaches

• Lessons learnt from earlier activities

• Identification of appropriate communication tools

ILRI Process and Partnership forPro-Poor Policy ChangeWhat will we be doing?ILRI


• Three case studies in three DIFFERENT countries

– A project considered to have influenced policy change

– A stream of research addressing a particular policy area

– A clear policy change;• New policy statement• New law• Irrefutable change in way something is done

ILRI Process and Partnership forPro-Poor Policy Change What will we be doing?ILRI


• Three case studies– SDP and impact on changed view of

informal milk trade– ????– ????



• Discussion:– Can ILRI hope to influence pro-poor policy

through research?– Any good case studies?

Case Study Approach


What is a Case Study?Definition:

" A systematic inquiry into an event or a set of related events which aims to describe and explain the phenomenon of interest" Bromley (1990)


Why is it useful?

• An ideal methodology when a holistic, in-depth investigation is needed

• Designed to bring out the details from the viewpoint of the participants by using multiple sources of data

Goal :to describe as accurately as possible the fullest, most complete description of the case.


Types of Case StudyTypes of case studies: • Exploratory, • Explanatory, • Descriptive (Yin, 1993)Stake (1995) included three others: • Intrinsic - when the researcher has an interest in

the case; • Instrumental - when the case is used to

understand more than what is obvious to the observer;

• Collective - when a group of cases is studied.


Issues• The unit of analysis is a critical factor• Typically a system of action rather than an

individual or group of individuals • Tend to be selective, focusing on one or two

issues that are fundamental to understanding the system being examined

• Case studies are multi-perspectival analyses • The researcher considers not just the voice and

perspective of the actors, but also of the relevant groups of actors and the interaction between them

• They give a voice to the powerless and voiceless.


Triangulation• Data source triangulation, when the researcher looks

for the data to remain the same in different contexts; • Investigator triangulation, when several investigators

examine the same phenomenon; • Theory triangulation, when investigators with

different view points interpret the same results; and • Methodological triangulation, when one approach is

followed by another, to increase confidence in the interpretation.


Applications

• To explain complex causal links between research and policy

• To describe the real-life context in which policy has been influenced by research

• To describe the policy influencing process itself

• To explore those situations in which the policy intervention being evaluated has no clear set of outcomes.


Process1. Design the case study protocol:

– determine the required skills – develop and review the protocol

2. Conduct the case study: – prepare for data collection – distribute questionnaire – conduct interviews

3. Analyze case study evidence: – analytic strategy

4. Develop conclusions, recommendations, and implications based on the evidence

Episode Study

Approach


What is an Episode Study“A study that focuses on a clear policy change and tracks back to assess what impact research had among the variety of issues that led to the policy change”.


• an excellent way of investigating the influence of research on policy

• Can focus on a single episode or comparative episodes.

What is the purpose?

Tracking backwards from policy change to any particular research which influence policy


• The process of working backwards in time gives a more realistic view of the broad range of factors – other than research – that influence policy

• Tracking forward probably overemphasizes

the importance of research

Advantage


• Policy processes are complex, multi-layered and change over time

• Often difficult to isolate the impact of research from other factors

• Actors may ‘re-write history’• Important to seek the views of a wide range of

informed stakeholders • The process of preparing an episode study is

iterative• Key facts and / or inconsistencies need to be

cross-checked with key informants

Issues


1. Identify a clear policy change.

2. Identify key Research Questions (draw on RAPID framework)

3. Explore how and why those policy decisions and practices took place

4. Assess the relative role of research in that process by drawing on the framework.

Process


Apply the RAPID Framework

External Influences political context

evidencelinks

Politics and Policymaking

Media, Advocacy, Networking Research,

learning & thinking

Scientific information exchange & validation

Policy analysis, & research

Campaigning, Lobbying


Key Questions• The external environment: Who are the key actors?

What is their agenda? How do they influence the political context?

• The political context: Is there political interest in change? Is there room for manoeuvre? How do they perceive the problem?

• The evidence: Is it there? Is it relevant? Is it practically useful? Are the concepts familiar or new? Does it need re-packaging?

• Links: Who are the key individuals? Are there existing networks to use? How best to transfer the information? The media? Campaigns?


• review of the literature;

• interviews with key actors;

• capturing the authors’ own experience; and

• discussions at workshops.

Methods

Steps 3 and 4 can be done through a variety of methods:

Episode Study Examples


Professionalisation of Public Services.

Structural Adjustment → Collapse

Paravet projects emerge.

ITDG projects.

Privatisation.

ITDG Paravet network.

Rapid spread in North.

KVB letter (January 1998).

Multistakeholder WSs → new policies.

Still not approved / passed!

1970s

1980s

1990s

2000s

Professionalisation of Public Services.

Structural Adjustment

Privatisation

ITDG Paravet network and change of DVS.

KVB letter (January 1998).

Multistakeholder WSs → new policies.

ITDG projects – collaborative research.

The Hubl StudyDr Kajume

Paravets in KenyaInternational Research


PRSPs – Political Context• Widespread awareness of a “problem” with

international development policy in late 90s• Failure of SAPs (and Asian financial crisis)• Mounting public pressure for debt relief• Stagnation of Comprehensive Development

Framework idea• Diverging agendas (UK – Poverty, US –

Governance)• WB/IMF Annual General Meeting, Sept 1999


PRSPs – Evidence• Long-term academic research informing new

focus on poverty, participation, ownership, aid effectiveness etc

• Applied policy research:– ESAF reviews– HIPC review– SPA Working Groups– NGO research on debt

• Uganda’s PEAP


PRSPs – Links

• WB, IMF, SPA, Bilaterals, NGOs all involved

• Formal an informal networks

• “None of the players was more than two handshakes away from any of the others”

Outcome Mapping


What is it?• an integrated PM&E tool• a system to think holistically & strategically about

how we intend to achieve result• an approach that focuses on changes in the

behaviour, relationships or actions of partners (as outcomes)

• a methodology that characterizes and assesses the program’s contributions to the achievement of outcomes

• an approach for designing in relation to the broader development context but assessing within your sphere of influence


Focus: On Behavioural Change


• For a program to tell its performance story in outcome terms by:– articulating its goals and designing its activities– designing a monitoring system for assessing

internal performance and outcomes of partners– setting a use-oriented evaluation plan

How can it be used?


• Focussing on changes in partners’ behaviour, relationships, or actions allows a program to:– measure results within its sphere of influence– obtain feedback about its efforts in order to

improve its performance– take credit for its contributions to the achievement

of outcomes– show progress towards outcomes

Why use it?


• Outcomes: changes in behaviours, relationships, activities and/or actions of the people, groups and organisations with whom we work

• Vision: the broad human, social and environmental betterment we desire

• Mission: how we intend to contribute towards the achievement of the vision

• Boundary partners: individuals, groups and organisations with whom we interact directly to effect change

• Outcome challenges: changes behaviours of the boundary partners as identified by the vision

Terminology


The Three Stages


Vision Statement Outcome Challenges Progress Markers

Why ? What ?

Boundary Partners

Who ?

MissionStrategy MapsOrganizationalPractices

How ?

Intentional Study Design


• Provides a framework for a continuous monitoring of the initiative as a tool to achieve its outcomes.

• The program uses progress markers, a set of graduated indicators of behavioural change, identified in the intentional design stage to clarify direction with its primary partners and to monitor outcomes

Performance Monitoring


• Helps identify the evaluation priorities assessing the strategy at greater depth than the performance monitoring stage

Evaluation Planning


Outcome Mapping: Main Elements

Boundary Partners

Vision

Mission

O.C. OutcomeChallenge

O.C.

Progress Markers

Strategies

Main Elements

RAPID Outcome Assessment


What is it?• A Visual Tool• Combines the outcome mapping concept within a

case study & episode study approach• Systematic approach to collecting information about

changes in behaviour of key project partners that contributed to the policy change

• Assessment of the contribution of the project (programme, strategy, etc.) to observed changes in behaviour –and ultimately to the policy change


Approach1. Describe policy environment at end2. Describe policy environment at the beginning3. Identify the key policy actors4. Identify key boundary partners5. Describe boundary partner behaviour at end6. Describe boundary partner behaviour at beginning7. Describe changes in BP behaviour8. Describe changes in project (strategic/opportunistic)9. Describe external influences10. Determine level of impact of changes in project11. Determine level of impact of external influences12. Check through external interviews13. Write report


Sources & Outputs1. Literature review

- Project background, progress, (published) achievements

2. Participatory workshop with staff (and BP)– Gather detailed information– Identify issues for further investigation

3. Interviews with key informants to:– Triangulate the result of the workshop,– Fill the gaps of information– Clarify causality

4. Report Writing– Visual and Narrative


ROA Terms and Definitions

• Boundary partners: individuals, groups and organisations with whom we interact directly to effect change.

• Outcomes: changes in behaviours, relationships, activities and/or actions of the people, groups and organisations with whom we work.

• Behaviours: the way we or our boundary partners do or think about things.


The key steps of the ROA framework

1. Describe the policy environment at the end of the project

BP 0 1 2 3 4 5 6 7 8 9 10

Project

EE

Po

lic

y E

nv

iro

nm

en

tP

olic

y C

ha

ng

e



2. Describe the policy environment at the beginning of the project

BP 0 1 2 3 4 5 6 7 8 9 10

Project

EE

Po

licy

En

vir

on

men

tP

olicy C

ha

ng

e



Before TodayBP 0 1 2 3 4 5 6 7 8 9 10

BP1

BP2

BP3

BP4

BP5

BP6

BP7

Project

EE

Po

lic

y E

nv

iro

nm

en

t

year/month

Po

licy

Ch

an

ge

3/4. Identify key policy actors and boundary partners (that were influential at end)



5. Describe the behaviours of the boundary partners that contributed to the change in the policy environment or policy Before Today

BP 0 1 2 3 4 5 6 7 8 9 10

BP1 5

BP2 4

BP3 8

BP4 7

BP5 4

BP6 9

BP7 3

Project

EE

Po

lic

y E

nv

iro

nm

en

tP

olic

y C

ha

ng

e

year/month


The key steps of the ROA framework6. Describe the behaviours of the boundary partners

at the beginning of the project

Before TodayBP 0 1 2 3 4 5 6 7 8 9 10

BP1 0 5

BP2 0 4

BP3 0 8

BP4 0 7

BP5 0 4

BP6 0 9

BP7 0 3

Project

EE

Po

lic

y E

nv

iro

nm

en

t

year/month

Po

licy

Ch

an

ge


The key steps of the ROA framework7. Map the key changes in behaviour for each

boundary partner from the start of the project

Before TodayBP 0 1 2 3 4 5 6 7 8 9 10

BP1 0 1 2 3,4 5

BP2 0 1 2,3 4

BP3 0 1 2 3 4,5,6 7 8

BP4 0 1 2 3 4 5 6 7

BP5 0 1 2 3 4

BP6 0 1,2 3,4,5 6,7,8 9

BP7 0 1, 2 3

Project

EE

Po

lic

y E

nv

iro

nm

en

t

year/month

Po

licy

Ch

an

ge


The key steps of the ROA framework8. Map the key changes in the project including

organisational changes, outputs and changes in behaviour during the same period.

Before TodayBP 0 1 2 3 4 5 6 7 8 9 10

BP1 0 1 2 3,4 5

BP2 0 1 2,3 4

BP3 0 1 2 3 4,5,6 7 8

BP4 0 1 2 3 4 5 6 7

BP5 0 1 2 3 4

BP6 0 1,2 3,4,5 6,7,8 9

BP7 0 1, 2 3

Project 0 1 2,3 4,5 6 7 8 9 10

EE

Po

lic

y E

nv

iro

nm

en

t

year/month

Po

licy

Ch

an

ge


The key steps of the ROA framework9. Map the external influences including the actions f

strategic partners and other exogenous factors during the same period

Before TodayBP 0 1 2 3 4 5 6 7 8 9 10

BP1 0 1 2 3,4 5

BP2 0 1 2,3 4

BP3 0 1 2 3 4,5,6 7 8

BP4 0 1 2 3 4 5 6 7

BP5 0 1 2 3 4

BP6 0 1,2 3,4,5 6,7,8 9

BP7 0 1, 2 3

Project 0 1 2,3 4,5 6 7 8 9 10

EE 0 1 2 3,4 5 6 7 8 9 10 11

Po

lic

y E

nv

iro

nm

en

t

year/month

Po

licy

Ch

an

ge


The key steps of the ROA framework10. Determine the level of impact/influence of the

project on the changes in behaviour of the boundary partners

Before TodayBP 0 1 2 3 4 5 6 7 8 9 10

BP1 0 1 2 3,4 5

BP2 0 1 2,3 4

BP3 0 1 2 3 4,5,6 7 8

BP4 0 1 2 3 4 5 6 7

BP5 0 1 2 3 4

BP6 0 1,2 3,4,5 6,7,8 9

BP7 0 1, 2 3

Project 0 1 2,3 4,5 6 7 8 9 10

EE 0 1 2 3,4 5 6 7 8 9 10 11

Direct influence

Indirect influence

External influence

Po

lic

y E

nv

iro

nm

en

t

year/month

Po

licy

Ch

an

ge


The key steps of the ROA framework11. Determine the level of impact/influence of external

influences on the changes in behaviour of the boundary partners and the project

Before TodayBP 0 1 2 3 4 5 6 7 8 9 10

BP1 0 1 2 3,4 5

BP2 0 1 2,3 4

BP3 0 1 2 3 4,5,6 7 8

BP4 0 1 2 3 4 5 6 7

BP5 0 1 2 3 4

BP6 0 1,2 3,4,5 6,7,8 9

BP7 0 1, 2 3

Project 0 1 2,3 4,5 6 7 8 9 10

EE 0 1 2 3,4 5 6 7 8 9 10 11

Direct influence

Indirect influence

External influence

Po

lic

y E

nv

iro

nm

en

tP

olic

y C

ha

ng

e

year/month


The key steps of the ROA framework12. Refine conclusions with in-depth interviews and

assess the real contribution of the project on the policy change

Before TodayBP 0 1 2 3 4 5 6 7 8 9 10

BP1 0 1 2 3,4 5

BP2 0 1 2,3 4

BP3 0 1 2 3 4,5,6 7 8

BP4 0 1 2 3 4 5 6 7

BP5 0 1 2 3 4

BP6 0 1,2 3,4,5 6,7,8 9

BP7 0 1, 2 3

Project 0 1 2,3 4,5 6 7 8 9 10 11

EE 0 1 2 3,4 5 6 7 8 9 10 11

Direct influence

Indirect influence

External influence

Po

lic

y E

nv

iro

nm

en

t

year/month

Po

licy

Ch

an

ge

ilri process and partnership for pro-poor policy change methods for assessing policy impact process...

Documents

policy slide

policy research

policy framework

policy communities

lipsky policy

policy windows

economic policy

multilateral policy