VIII KONFERENCJA EWALUACYJNA

METHODOLOGICAL DEVELOPMENTS AND CHALLENGES IN UK POLICY EVALUATION DANIEL FUJIWARA

VIII Evaluation Conference‘Methodological Developments and Challenges in UK Policy Evaluation’

Daniel FujiwaraSenior Economist Cabinet Office & London School of Economics

Warsaw, November 2012.

Policy evaluation in the UK

Analysts in government have a long history and tradition of policy evaluation.

The Magenta Book

Assessing whether the policy had a causal impact on the outcomes of interest.

The Green Book

Cost-benefit analysis is the key tool for policy assessment. CBA enters at the Appraisal stage

Appraise policy interventions in terms of their social costs and benefits. Further supported through the Social Value Act (2012)

Challenges to policy evaluation

● Some constraints on use of impact evaluation findings:

i. Are the impact and valuation estimates robust?

ii. Have we acknowledged and included all possible impacts?

iii. Are there some things that are considered separately – eg, merit goods, distribution (of income, health etc).

Challenges to policy evaluation: Impact measurement (1)

Identifying and measuring thecounterfactual

Many public sector organisations moving to the Maryland Evaluation scale.

‘What Works’ institutes like Washington State Institute for Public Policy.

Scale ranks howwell counterfactuals have been measured in the analysis.

Level Design Statistical method

5 Randomised trials Evaluations with well implemented random assignment of treatment to subjects in treatment and control groups.

4 Quasi-Experiments

Evaluations that use a naturally occurring event (that makes the treatment assignment as good as random)

3 Matchingtechniques;Regressionanalysis

Non-experimental evaluations where treatment and comparison groups are matched on observable characteristics

2 Simplecomparisons

Studies with a treated and comparison group, but with no attempt made to control for differences among the groups.

1 Pre- and postanalysis

Studies where no comparison group is used. Outcomes are measured pre and post-treatment.

Challenges to policy evaluation: Impact measurement (2)

Level 5 study designs are being increasingly used for public policy interventions.

RCTs used in tax policy, environment policy and labour market interventions.

Challenges to policy evaluation: Valuation

Supplementary Green Book guidance provides guidelines on how to value social impacts

Preference methods or Wellbeing Valuation ?

Different methods are increasingly allowing us to value and include a broad range of impacts

Challenges to policy evaluation: Summary

These developments and new guidelines for policy evaluation help us to understand the impacts of a policy better and to continuously improve the success of policy interventions.

Policy interventions are enhanced during this process through the use of evidence from behavioural science.