The Separate but Related Cultures of Evaluation &

Policy Analysis

AES 2011 ConferenceDavid Turner

Overview

This paper started with the observation that evaluation and policy practitioners read different journals, cite different authors, used different terminology but cover common ground

I acknowledge that policy and evaluation have wide and diverse literatures; more than one person will usually follow closely

Common Ground

Evaluation and policy analysis have a common focus on how policies are developed and implemented, and a general goal of public interest

Evaluation practitioners have to consider policy needs and stakeholders for evaluation work

Different Outlooks

Evaluation practitioners may treat policy goals and designs as fixed

Evaluators may not follow debates within policy community on policy & implementation

Separate Literatures

Analysis of journal publications shows differences in sources

Reviewed 2010 publications in Journal of Policy Analysis and Management (JPAM) and American Journal of Evaluation

Looked at content of the articles and the sources cited

JPAM Articles

50 research articles in 4 issues, on range of policy areas such as education, tax policies, health, employment, and housing

Focus often on evaluative questions concerning policy implementation and impacts

Public administration, economic journals cited (rarely evaluation)

AJE Articles

18 research articles in 4 issues, most often focused on evaluation methods or approaches (less focus on policy substance)

A few references to public administration or management journals, or journals in policy areas such as criminal justice or education

Book Buying Patterns

Looked for reading patterns among policy and evaluation books by using Amazon data Identified lists of evaluation and policy

texts Looked for links under “customers who

bought this book also bought…” Mapped linkages between books

(pointing from one to another bought by same people)

Evaluation Books

Patton, Utilization-Focused Evaluation Morris, Evaluation Ethics for Best Practice Chelimsky and Shadish, Evaluation for the 21st

Century Rossi, Lipsey & Freeman, Evaluation Scriven, Evaluation Thesaurus Posavac, Program Evaluation Stufflebeam, Evaluation Theory, Models, &

Applications Weiss, Evaluation

More Evaluation Books

Shadish, Foundations of Program Evaluation Mathison, Encyclopedia of Evaluation Wholey, Handbook of Practical Program

Evaluation Owen, Program Evaluation Davidson, Evaluation Methodology Basics Yarbrough, Program Evaluation Standards Fitzpatrick, Program Evaluation: Alternative

Approaches Bamberger, Real World Evaluation

Policy Books

Bardach, A Practical Guide for Policy Analysis Weimer & Vining, Policy Analysis Birkland, An Introduction to the Policy Process Stone, Policy Paradox: The Art of Political Decision

Making Kraft & Furlong, Public Policy Sabatier, Theories of the Policy Process Kingdon, Agendas, Alternatives, and Public

Policies

More Policy Books

Stokey & Zeckhauser, A Primer for Policy Analysis Pressman & Wildavsky, Implementation Hill & Hupe, Implementing Public Policy Smith, Writing Public Policy Dye, Understanding Public Policy Wilson, Bureaucracy

E8

E2

E7

E1

E4

E5

E6 E3

E10

E15

E9

E16

E11

E12 E13 E1

4P13

P6

P1

P7

P4

P3

P2

P5

P11P1

2

P10

P9

P8

Book Buying Networks (all links)

E8

E2

E7

E1

E4

E5

E6 E3

E10

E15

E9

E16

E11

E12 E13

E14

P13

P6

P1

P7

P4

P3

P2

P5

P11P1

2

P10

P9

P8

Book Buying Networks (reciprocal)

Learning from Policy

Evaluation practice could draw on developments in policy and public management How the policy process works What makes for successful policy

implementation

Policy Process Models

Descriptions of how the policy process works have advanced from the “stages heuristic” over time Programs emerge over time, decisions

are part of a continuous process Different models of policy provide a

range of insights

Policy Model: Example 1

Richard Matland described policy implementation as a function of ambiguity and conflict A policy can be ambiguous in either

means or ends A policy is subject to conflict when

parties with a stake in an issue have different views over its means or ends

Ambiguity-Conflict ModelConflict

Am

big

ui

ty

Low High

Low

Hig

h

Administrative implementationIssue: resourcesExample: smallpox eradication

wide range of policy areas such as education, tax policies, health, employment, and housing

Political implementationIssue: powerExample: busing

Experimental implementationIssue: ContextExample: Headstart for children

Symbolic implementationIssue: coalition strengthExample: community action agencies

Implications for Evaluation

The model describes a policy in ways that suggest what evaluation can achieve Administrative: focus on technical

questions Political: consider whose values and

needs are being met by evaluation Experimental: scope for contribution to

new ideas and approaches Symbolic: people may need to be seen

doing something, may not expect impacts!

Another Policy Model

Inside Government Agencies In the “Real World”

Hypothesis Action Note anomalies

Ask questions

Revised hypothesis and next action

Collaborative learning in and between networks of government and non-government actors

All available technical, practitioner, and user information about what is known and unknown informs policy hypothesis

Source: Institute of Policy Studies Working Paper 11/04, “Experimentation and Learning in Policy Implementation”, Eppel, Turner, Wolf

Implications for Evaluation

Program theory: need to adapt to emerging issues, anomalies; may not have a stable theory of action

Outcome evaluation: question of when (if ever) program goals stabilise for measurable results

Grounding Formative Evaluation

Implementation has been widely studied since Pressman & Wildavsky, but may not be considered in formative evaluation Wide range of endogenous and

exogenous factors, significance of key actors

Substantial and varied literature

Learning from Evaluation

It’s not just evaluators who should look outside their own boundaries; policy analysts can learn from evaluation: Alternative approaches for gaining

insight (not just “gold standard”) Need to define goals, indicators, logic

more precisely where possible

Evidence of Change

Recent attention to issues of complexity reflects awareness of need to adapt evaluation approaches Developmental evaluation one example

of a response to complex policy environments and emerging policy solutions

Attention also given to need for knowledge exchange across boundaries

Discussion

What do you think? Is there a cultural divide with policy analysis, and if so how significant is it?