evaluating the quality and use of impact assessments the role and approach of the nao

HELPING THE NATION SPEND WISELY

Evaluating the quality and use of Impact Assessments

The role and approach of the NAO


To cover

The role of the National Audit Office (NAO)

The potential benefits of Impact Assessments

Our examinations of Impact Assessments

Lessons learned


The role of the NAO

The NAO scrutinises public spending on behalf of Parliament through:

Financial audit of the accounts of all central government departments and agencies

Value for money audit of the economy, efficiency and effectiveness with which government spends public resources

Hold Government to account for spending taxpayers’ money

Independent analysis of the evidence


The Regulation teamWe cover:

The economic regulators – the design of regulators and risk-based approaches

Competition bodies – the introduction of competition and prevention of anti-competitive behaviour

Consumer issues – the impact or regulation on consumers and adequacy of consumer protection

The regulatory reform agenda – the effectiveness of the system and outcomes

5/6 reports each year.


Value for money

Hold regulators and departments to account but also seek improvements in delivery

Focus on effectiveness – low spend but big impact

Do not question the merits of policy decisions

Parliamentary scrutiny


Our Regulatory Reform work

Impact Assessments – since 2001

Reducing the burdens of regulation on business – since 2007

Reviewing regulators’ approach to inspecting and enforcing regulations – since 2008


Our objectives – regulatory reform

To hold the BRE and departments to account for delivering the regulatory reform agenda, and to help improve working practices

– Are ‘tools’ being used effectively?

– Are initiatives delivering their intended objectives?


The benefits of Impact Assessments

Increase transparency and strengthen accountability

Help encourage evidence-based policy formation

WHICH MEANS:

The need for regulation is challenged

A better chance of achieving good policy outcomes


Some challenges

Embedding IAs in policy development

Process and governance


Impact Assessments – NAO work

We began our work in 2001 and have published an annual report since 2003-04

Originally, we examined the quality of IAs

From 2006, we examined quality but also the use of IAs:• A review of departmental processes• Considered how depts have used IAs to inform policy development• Post-implementation review

2008 – our broadest review of quality


NAO criteria for assessment

Criteria NAO tests

A clear statement of the conclusion

Did the IA make comparisons against the preferred option?

Did the IA state the recommendation clearly?

Consideration of appropriate options

Did the IA consider a range of options, including the do-nothing option?

The use of impact tests and consultation

Did the IA consider social, environmental small business and competition tests?

Was consultation undertaken in accordance with guidance?

Standard of cost benefit analysis

Were costs and benefits of different options quantified? If not, was there a qualitative discussion?

What economic methods / expertise were used? Was the data source referenced?

The existence of and quality of implementation and enforcement plans

Is compliance considered? Does the IA detail who will enforce / monitor? Are there estimates of enforcement costs? Is there a plan?

Is there a statement on post-implementation review?


Results of NAO reviews

Generally ‘competent’ – few areas of significant weakness – policy officials understand the requirements


2007 Report

Scope and Purpose Consultation Costs and BenefitsCompliance and

Enforcement

Implementation, Monitoring and

Evaluation

Competition Assessment

GREEN 15 16 5 4 8 12AMBER 4 1 10 12 10 5RED 0 2 4 0 1 0N/A 0 0 0 3 0 2


2008 Report

Tests

Trend Overall assessment

Conclusion Improvement The strongest element of both samples – with a small improvement in 2008

Implementation and enforcement

Improvement The weakest area in 2006 but a considerable improvement in 2008

Analysis of costs and benefits

Improvement A weak area in 2006 but a number of improvements in 2008.

Impact tests and consultation

No change Scored well in both samples – although no improvement

The consideration of options Deterioration In 2008 fewer IAs presented an analysis of a range of options.


Results of NAO reviews - quality

COST-BENEFIT ANALYSIS

Improving but still persistent weaknesses

– greater incidence of quantification• 67% estimated costs (up from 56%)• 60% estimated benefits (up from 40%)

– type of analysis• 24% used sensitivity analysis (up from 13%)• 22% used specialists (up from 14%)



But wide variation between the best and worst IAs

– Superficial evidence base– Basic analysis– Lack of justification for an absence of monetisation

A NEED TO FOCUS ON THE EVIDENCE BASE

A need to consider a range of options, including non-regulatory



IMPLEMENTATION AND ENFORCEMENT

Practical enforcement considerations often missing:

– 80% did not include an implementation plan– 66% did not estimate enforcement costs

Impact tests need to be applied with more vigour – need to consider the full range of potential impacts

CONSIDER THE PRACTICAL IMPLCATIONS OF NEW REGULATIONS


Results of NAO reviews - governance

Departments are strengthening scrutiny arrangements

– increased frequency of review; – greater use of economists in review

IAs more embedded in the policy formation process

Challenge still lacking in places

A need to make scrutiny bite – inc external scrutiny


Lessons learned

Improve content of IAs

Establish robust processes to integrate into policy formation

Provide support for policy officials

Culture? Embed principles into a way of thinking


Lessons learned

A need to make IAs an integral part of the policy making process

How to create the right incentives?

– Senior level commitment– Parliamentary use in the passage of legislation– Scrutiny / review – Government / SAI / Parliament– Show benefits to policy officials / make easier to use


Contact:

[email protected]@nao.gsi.gov.uk

evaluating the quality and use of impact assessments the role and approach of the nao

Documents