Five principles and guidelines for an evidence-informed approach to policymaking in government

Louise Shaxson, Mapula Tshangela, Narnia Bohler-Muller, Ajoy Datta, Nikki Funke, Linda Godfrey, Bongani Matomela, Gary Pienaar, Nedson Pophiwa and Wilma Strydom December 2015

Supporting evidence in decision making

2

This document is an output from a project funded by UK Aid from the UK Department for International Development (DFID) to support South Africa’s Department of Environmental Affairs (DEA) in its implementation of the environment sector Research, Development and Evidence framework. The views expressed and contained in it are not necessarily those of or endorsed by DFID and DEA, which can accept no responsibility for such views or information or for any reliance placed on them.

3

Table of contents

Table of contents .................................................................................................... 3

Acknowledgements ................................................................................................ 4

1. Introduction ...................................................................................................... 5

2. Evidence, and evidence-informed policy-making .......................................... 6

3. Context and desired outcomes ....................................................................... 8

4. Five principles and guidelines ........................................................................ 9

Principle 1: Using a broad definition of ‘robust evidence’ ................................................... 9

Principle 2: Linking evidence needs to policy priorities ...................................................... 9

Principle 3: Linking an evidence-informed approach with business planning, reporting and

budgeting ........................................................................................................................ 10

Principle 4: Inclusive and participatory policy processes .................................................. 10

Principle 5: Co-production of evidence and policy ........................................................... 11

5. Implementing the principles and guidelines .................................................12

Promoting the adoption of the principles & guidelines ...................................................... 12

Ensuring accountability for implementing the principles and guidelines ........................... 12

Monitoring their effectiveness .......................................................................................... 13

Annex A: National principles include: ..................................................................14

4

Acknowledgements This document was developed as part of the VakaYiko consortium project supporting Department of Environmental Affairs (DEA) in South Africa. Teams from HSRC, CSIR (in South Africa) and ODI (in the UK), Department for Planning, Monitoring and Evaluation (DPME) and Department of Science and Technology (DST) have been working with DEA teams to help it understand what may need to be done to strengthen DEA’s approach to evidence-informed policymaking. It has been a highly collaborative effort and we are very grateful to all individuals who have consented to be interviewed, attended meetings, read and commented on draft reports, and offered their insights, opinions and advice. We are also grateful to those people who offered their time and expertise during interviews for the five diagnosis studies. For reasons of confidentiality they are not listed. A steering group participation consisting of officials from DEA, DPME, DST and the University of Cape Town (UCT) has guided us throughout. Particular thanks are due to the following: DEA: Wadzi Mandivenyi and Kiruben Naicker (Biodiversity & Conservation), Kgauta

Mokoena and Anben Pillay (Waste Management), Anna Mampye (Sustainability Reporting), Nhlanhla Sithole, Gertrude Matsebe and Bongani Maluka, (Environmental Programming), Keleabetswe Tlouane and Samukelisiwe Mncube (Outcome 10), and Brian Mantlana (Climate Change & Air Quality)

DPME: Ian Goldman (chair of the steering group) and Harsha Dayal DST: Shanna Nienaber and Henry Roman UCT: Alan Hirsch DEA, Provinces and Entities teams whose support is acknowledged: Alf Wills, Ester Makau, Tlou Ramaru, Limpho Makotoko, Peter Lukey, Jacob Kutu, Thandeka Mandigora, Obed Baloyi, Mamosa Africa, Mpho Tshitangoni, Khashiwe Masinga, Sharon Mogomotsi, Bonginkosi Dlamini, Leanne Richards, Reitumetse Molotsane, Tsepang Makholela, Pravin Pillay, Edward Moeketsi, Timothy Fasheun, Carmel Mbizvo, John Donaldson, Kristal Maze, Mandy Driver, Tammy Smith, Anthea Stephens, Karen Shippey, Frances van der Merwe, Magdel Boshof, Geoff Cowan, Wilma Lutsch, Humbu Mafumo, Stanley Tshitwamulomoni, Basani, Tharina Boshof, Nthabeleng Montsho, Godfrey Muneri, Zama Nkuna, Monana Dhliwayo and Thembinkosi Twalo

5

1. Introduction This document has been written to prompt discussion within South Africa’s Department of Environmental Affairs (DEA), and possibly government-wide. It proposes five principles that underpin an evidence-informed approach to policymaking. It derives from work done in DEA in 2014 and 2015 and builds on many good practices that have been identified across the department. Three principle-based documents that help guide policymaking in DEA. Chapter 5 of the National Development Plan (NDP) sets out fourteen principles to guide South Africa through the transition to an environmentally sustainable low carbon economy, moving from policy, to process, to action. The National Environmental Management Act of 1998 describes another set of principles. First, decisions must take into account all forms of knowledge, including traditional and ordinary knowledge. Second, participation of all interested and affected parties in environmental governance must be promoted. Third, all people must have the opportunity to develop the understanding. Fourth, development must be socially, environmentally and economically sustainable. The National Strategy for Sustainable Development (NSSD) states a further set: fundamental principles relating to human rights, substantive principles that underscore a systems approach to sustainable development, and process principles on implementation of sustainable development. All three sets of principles are set out in Annex A. This document draws from them to propose a single set of five principles and associated guidelines that are specifically directed at enhancing DEA’s use of evidence. They are:

1. Using a broad definition of ‘robust’ evidence 2. Linking evidence needs to policy priorities 3. Linking an evidence-informed approach with planning, budgeting and reporting 4. Participatory and inclusive policy processes 5. Co-production of evidence and policy

The next section 2 describes what we mean by evidence in the policy context and the four key processes that are underpinned by these principles. Section 3 describes the context within which the principles can be applied. Section 4 sets out the principles themselves. Each principle is accompanied by guidelines on how they could be applied. Section 5 discusses how these principles can be implemented in practice, how to ensure accountability for them and how to monitor their effectiveness. The principles have been developed specifically for DEA. However, we hope other government departments will find them helpful.

6

2. Evidence, and evidence-informed policy-making Policy work is very varied and different policy themes in DEA work with evidence in different ways. Trying to prescribe a single approach to evidence will be too limiting, as teams need to be flexible to cope with the range of issues they have to deal with. A set of principles will help each theme determine what is most appropriate at any given time, whilst ensuring that there is consistency across all the policy themes. Before describing the principles, we describe what we mean by evidence for policy. DEA’s Research, Development & Evidence (RD&E) Framework1 recognises that the evidence that policy teams require is not just research evidence. While research is a core component of the evidence base, it is also important to recognise the wealth of statistical and administrative data that DEA possesses, evidence from monitoring and evaluation, and evidence from citizens, DEA’s wide range of stakeholders and other role players. If these are considered as different but related categories of evidence, it will help DEA work out how to prioritise its spending on each of them so that—with limited budgets—it is best able to address all its policy goals (formulation, implementation, reporting and monitoring and evaluation).

The following definitions of the four categories are proposed, these are2:

Statistical and administrative data, whose purpose is to paint a picture of where we are now. It might include trend data on greenhouse gas emissions, the performance of landfill sites, information on regional water quality, or the distribution of endangered species;

Analytical (research) evidence, whose purpose is to explain causal relationships, enrich our understanding of complex issues or challenge received wisdom. This primarily includes evidence from engineering, natural science and social science research;

Evidence from citizens, stakeholders and role players, whose purpose is to inform policymakers of what different groups of people value and what they consider to be legitimate. This type of evidence may be collected using research methods, but inclusive and participatory processes of engagement are equally important;

Evidence from evaluations, whose purpose is to tell us what has worked in the past, for whom, how and why. This includes evidence from detailed evaluations that can be conducted of a specific policy or programme.

Together, all these forms of evidence make up the ‘evidence base’ that help DEA develop, implement and report on its policy goals. Some of this evidence may be needed in the short-term, and some in the long-term. Deciding the balance of what is required will be a matter for individual policy themes depending on the priority issue they are facing. Ensuring that the evidence is of high quality will be important. Advice is available within DEA and from external organisations, on how to devise appropriate processes for collecting evidence so that it is of the highest possible quality.

1 Available from https://www.environment.gov.za/sites/default/files/docs/environmental_research_framework.pdf

2 Legal expertise may be needed to help develop or amend regulations, and to formulate legislation.

We do not see this as a separate form of evidence, but as knowledge of how to use the evidence

that already exists and define what other evidence may be required.

7

The RD&E Framework also recognises that there are four internally driven processes that could support an evidence-informed approach to policymaking. These are:

Jointly scoping the question: working with the full range of stakeholders to set the agenda for key policy issues, and to identify and define the policy question that need answering;

Assembling existing evidence: reviewing what is already known to reduce the danger of reinventing the wheel. This is particularly appropriate where time is limited;

Procuring new evidence: where time is available, collecting new evidence through individually commissioned pieces of work;

Jointly interpreting the evidence to inform decisions: working across the evidence-policy interface to ensure that both sides share an understanding of what the evidence means within the current policy context.

An evidence-informed approach therefore pays equal attention to the quality of the evidence base and the quality of the processes that make use of it.

8

3. Context and desired outcomes Evidence is one of the inputs into government decision-making. Government policymakers have to exercise considerable judgement about what evidence to seek out, when, from whom, and how to ensure that it informs decisions effectively and in a timely manner. This means managing their devolved budgets for evidence and ensuring that funds are spent as wisely as possible. In the past, DEA policy themes have not systematically anticipated their likely future evidence needs over the next 1-3-5 years and construct detailed plans for how to acquire them in an inclusive and participatory way. Some themes have done this, but others are just beginning. It can be challenging to do this, but an intentional approach to managing the evidence base can help themes in several ways. First, it clearly aligns needs for evidence with policy priorities, ensuring that a broad range of evidence is collected and used. Second, it helps prioritise spending so that as many evidence needs as possible are met within the existing budget. Third, it develops a more consistent approach to sourcing and using evidence across the department, but one that is flexible to individual themes’ ways of working. Fourth, it sends signals to external organisations about what evidence each theme is likely to require in future, helping them plan more effectively. Finally, it allows themes to see whether there may be opportunities for sharing costs or processes with other themes or other organisations.

9

4. Five principles and guidelines Building from the context above, five proposed principles are set out below, with a short set of guidelines about how they could be put into practice. It will be up to DEA theme managers or other interested Government departments to work out how best to interpret and apply them to their work. All five principles are equally important.

Principle 1: Using a broad definition of ‘robust evidence’ DEA’s evidence base needs to be broad enough to address all its policy goals for the environment, society and the economy. Effective policy decisions will be based on a broad definition of evidence which recognises not only formal knowledge from a wide range of disciplines, but also information which may be practical or context-specific, as outlined earlier. Taking an evidence-informed approach does not mean simply searching for evidence to confirm predetermined decisions. Evidence can also explain complex relationships, enrich our understanding of an issue, challenge received wisdom and scope opportunities for change. Guideline, this can be done by:

Explicitly recognising the four types of evidence outlined earlier: o statistical and administrative evidence o evidence from research o evidence from stakeholders and citizens o evidence from evaluations

Categorising further for planning and prioritisation purpose, the evidence as either responding to statutory obligations, strategic evidence needs or applied evidence needs

Ensuring that appropriate training is given in how to appraise the quality of each type of evidence, supported by appropriate training and guidance

It may be helpful to set up one or more advisory committees that can oversee any quality issues. These should build on existing governance structures. They may be most effective when they appropriately comprise both senior policy officials and people from evidence-providing organisations and stakeholders. This could be combined with advice on the strategic direction of the evidence base (see principle 2 & 5).

Principle 2: Linking evidence needs to policy priorities It is policy themes’ responsibility to lead discussions about what evidence they will need to make their decisions. Policy themes should lead on setting the key questions for the sector, because they are ultimately responsible for answering them. This ensures that the priorities for spending on evidence reflect the department’s policy priorities, not the other way round. While DEA policy themes should lead this process, it must be done jointly with organisations and stakeholders that are responsible for providing the evidence (see principle 4 & 5). Guideline, this can be done by:

Setting out clearly each theme’s current and future policy goals and reporting priorities;

Specifying the key questions that need to be answered with evidence

Describing what evidence is needed to answer the key policy questions, making use of all four types of evidence

10

An Evidence Strategy is potentially a useful vehicle for ensuring that the evidence base is set up to respond to policy’s needs as effectively as possible. This can be done in such a way that it automatically informs departmental planning and budgeting processes (principle 3).

Principle 3: Linking an evidence-informed approach with business planning, reporting and budgeting While it is appropriate for each theme to manage their evidence base separately, at a departmental level it is important to have a consistent understanding of the overall prioritisation and pattern of spending on evidence. This will allow senior managers to monitor how much is being spent on the different types of evidence. It will also allow them to judge whether the department’s overall evidence budget is being spent as effectively as possible to help it meet its policy goals. Guideline, this can be done by:

Actively managing evidence as clearly defined line items in the departmental budget;

Developing detailed theme-level budgets for evidence that differentiate between the four types of evidence and possibly categories in terms of statutory obligations, strategic evidence needs or applied evidence needs;

Reporting on the planned and actual expenditure on evidence in the Departmental Annual Performance Plan.

The first step is to clarify how much is being spent on the different types of evidence (principle 1). Initially, this will involve a degree of judgement but over time the definitions will become clearer.

Principle 4: Inclusive and participatory policy processes Evidence is not simply a service provided to DEA policymakers. Different stakeholders have different views of what evidence is needed to inform policymaking; and different interpretations of what it means for developing policy positions, monitoring policy implementation and reporting on progress towards goals. An inclusive approach involves policy officials listening to as many of these voices as possible. A participatory approach ensures actively involving them throughout the processes of developing and submitting policies for approval, and reporting on progress. Evidence providers can contribute to both inclusive and participatory approaches by listening to and understanding the policy priorities, thus increasing the policy relevance of evidence they generate. Disagreement is an integral part of a process of inclusive and participatory development, not something to be ignored or swept under the carpet. Policymaking approaches that welcome a wide range of opinions do more than just help improve the quality of the evidence base. They also help strengthen the department’s legitimacy in the eyes of its stakeholders and of civil society. Opening up the ‘black box’ of DEA’s policymaking can be done by ensuring that the processes of joint scoping, assembling, procuring and joint interpreting the evidence are as inclusive as possible. There are time and resource costs, but this can help build trust, commitment and respect. An inclusive and participatory approach covers internal and external relationships. Internally, it means facilitating links between DEA staff that specialise in policy development, evaluation and policy implementation, and between DEA and other departments. Externally, it means strengthening relationships with a wide range of organisations and citizen groups—

11

from government entities to civil society organisations, private sector, non-government organisations, academia, think tanks, research councils, advocacy groups and others. Guideline, this can be done by:

Building in sufficient time for an inclusive and participatory approach. Where time is limited, it will be important to recognise whose voices are not being heard and how this will affect the outcomes of the discussions;

Ensuring that resources are explicitly allocated to developing and running inclusive and participatory evidence processes

o Internally, strengthening relationships between people with different expertise at key points in the policy and reporting cycle

o Externally, ensuring that resources for meaningful participation processes are built into procurement plans from the outset

Strengthening joint scoping and interpretation approaches and involving different expertise, both within government and with external stakeholders

The financial, human and time resource requirements for an inclusive and participatory approach will vary from issue to issue. In some cases they may be significant. It will be up to individual themes to decide what proportion of their resources, and how much time, to allocate to inclusive and participatory processes as guided by policy priorities. Specific guidance on how to do this may be helpful.

Principle 5: Co-production of evidence and policy Policymakers, researchers, citizens and non-state organisations are often thought of as occupying different worlds. An approach to evidence-informed policymaking that subscribes to this view will be limited by the established relationships that exist between the different groups. This is likely to result in a situation where the most powerful group defines the policy problem, the evidence that it needs to address the problem, and the most appropriate solution. In doing this it becomes both a player and a referee in the process. An alternative approach to evidence-informed policymaking subscribes to the view that it is only possible to understand what the real problems are, and to implement sustainable solutions, by working on local issues, and by working with all the different groups simultaneously. This approach focuses on enhancing mutual understanding, balancing priorities, learning from others, and thinking of innovative solutions that are not owned by a single group, but are co-owned. It encourages constant reflection on what progress is being made and co-benefits. The most dominant group does not impose its views on what evidence is needed to define the problem, develop the solution or demonstrate progress. Instead the evidence is co-produced by all the different groups throughout the process. Guideline, this can be done by:

Identifying and prioritising appropriate policy issues that could be approached through proactive and carefully managed processes of co-production and action research. These are likely to be local rather than national issues;

Jointly conceptualising evidence-informed policy making projects, to ensure their relevance and long-term sustainability

This principle is complex and may be difficult to implement within the existing processes. It may require pilot cases to establish the key lessons over time. However it is worth acknowledging its potential for improving an evidence-informed approach to policymaking and the science-policy interface. Similar to principle 4, the financial, human and time resource requirements for co-production of evidence and policy will vary from issue to issue.

12

5. Implementing the principles and guidelines Scaling the implementations of the principles and guidelines can help build confidence that DEA is taking an evidence-informed approach to all its policymaking and reporting. A strategy for implementation will show how the principles and guidelines are being adopted, set out who is accountable and describe how their implementation will be monitored to ensure they are having the intended effects. DEA is a complex department working in a complex environment. A one-size-fits-all template will be inappropriate, at least in the early years. Implementing the principles and guidelines will take time and a flexible approach. It may be necessary to make trade-offs between the speed and the depth at which the principles can be implemented. Individual themes within DEA will be responsible for many of the specific measures, but the intention is that these principles and guidelines promote co-operative working between themes within DEA and between DEA and external organisations.

Promoting the adoption of the principles & guidelines DEA is responsible for ensuring it uses its evidence base as effectively as possible to support achievement of its policy goals. This evidence responsibility is shared widely across the department and should not be delegated only to people who specialise in evidence. DEA could therefore:

Allocate specific resources to strengthening its evidence-informed approach;

Communicate the principles and guidelines internally and to external stakeholders, discussing their implications for current and future work;

Publicise case studies of good practice in the use of evidence in policymaking and reporting;

Ensure that appropriate professional development and training programmes for all types of evidence are available to all staff, reflecting the spirit of the guidelines. These should cover how evidence informs policy development, policy implementation and evaluation;

Ensuring accountability for implementing the principles and guidelines The principles and guidelines are closely linked to departmental planning, budgeting and reporting processes. The intention is that they become part and parcel of the on-going business of DEA and are used during policy formulation, implementation and M&E. This means that senior managers will be responsible for ensuring the principles and guidelines are implemented. DEA could therefore:

Designate senior managers responsible for the work to strengthen its evidence-informed approach; this could be through the work on Evidence Strategies and performance agreements

Account for the overall expenditure on evidence in annual budgeting processes at departmental and theme level—showing how the budget allocations reflect departmental policy priorities.

13

Monitoring their effectiveness Strengthening an evidence-informed approach will take time and resources. It needs to be planned and monitored to ensure the activities that underpin it remain on track and that they contribute towards continuous improvement. It will be important to develop a baseline from which progress can be monitored. DEA could:

Develop a strategy for change, based on these principles and guidelines. This should diagnose the main areas where improvement is needed, set out specific objectives within each area and list the activities to achieve those objectives. On-going and planned activities should all be included as long as they adhere to the principles and guidelines. This will form the baseline on which future progress can be assessed;

Consult widely on the change strategy, together with these principles, to ensure that the change strategy is widely shared;

Develop a set of indicators of change that can be monitored over time;

Share this process with DPME as the department responsible for performance management and with DST as the department responsible for research evidence.

14

Annex A: National principles include:

National Development Plan, 2012: page 200 National Environmental Management Act, Act 107 of 1998

National Strategy for Sustainable Development, 2011: page 9

Guiding principles for the transition

The following principles can guide the transition to an environmentally sustainable low carbon economy, moving from policy, to process, to action:

Just, ethical and sustainable. Recognise the

aspirations of South Africa as a developing country and remain mindful of its unique history.

Global solidarity. Justly balance national

interests with collective action in relation to environmental risks and existential threats.

Ecosystems protection. Acknowledge that

human wellbeing is dependent on the health of the planet.

Full cost accounting. Internalise both

environmental and social costs in planning and investment decisions, recognising that the need to secure environmental assets may be weighed against the social benefits accrued from their use.

Strategic planning. Follow a systematic

approach that is responsive to emerging risk and opportunity, and which identifies and manages trade-offs.

Transformative. Address the structural and

systemic flaws of the economy and society with strength of leadership, boldness, visionary thinking and innovative planning.

Managed transition. Build on existing processes

and capacities to enable society to change in a structured and phased manner.

Opportunity-focused. Look for synergies

between sustainability, growth, competitiveness and employment creation, for South Africa to attain equality and prosperity.

Effective participation of social partners. Be

aware of mutual responsibilities, engage on differences, seek consensus and expect compromise through social dialogue.

Balance evidence collection with immediate action. Recognise the basic tools needed for

informed action.

Sound policy-making. Develop coherent and

aligned policy that provides predictable signals, while being simple, feasible and effective.

Least regret. Invest early in low-carbon

technologies that are least-cost, to reduce emissions and position South Africa to compete in a carbon-constrained world.

A regional approach. Develop partnerships with

neighbours in the region to promote mutually beneficial collaboration on mitigation and adaptation.

Accountability and transparency. Lead and

manage, as well as monitor, verify and report on the transition.

Principle 3:

Development must be socially, environmentally and economically sustainable

Principle 4 (a):

Sustainable development requires the consideration of all relevant factors

Principle 4 (f):

Participation of all interested and affected parties in environmental governance must be promoted,

Principle 4 (f): All

people must have the opportunity to develop their understanding

Principle 4 (g):

Decisions must take into account all forms of knowledge, including traditional and ordinary knowledge

The NSSD vision is underpinned by a number of principles: Fundamental principles

The fundamental principles relate to the following fundamental human rights that are guaranteed in the Constitution of the Republic of South Africa: • Human dignity and social equity • Justice and fairness • Democratic governance • A healthy and safe environment Substantive principles

The substantive principles are based on the following sustainable development principles that are already enshrined in South African law and that underscore a systems approach to achieving sustainable development: • Natural resources must be used sustainably. • Socioeconomic systems are embedded in and are dependent on ecosystems. • Basic human needs must be met to ensure that the resources that are necessary for long-term survival are not destroyed for short-term gain. Process principles

The process principles apply to the implementation of the NFSD and the NSSD 1, and include the following: • Integration and innovation • Consultation and participation • Implementation in a phased manner

15

These evidence principles relate to the NDP, NEMA and NSSD principles as follows:

Evidence principles

NDP principles 1 2 3 4 5

Strategic planning. Follow a systematic approach that is responsive to emerging risk and opportunity, and which identifies and manages trade-offs.

Transformative. Address the structural and systemic flaws of the economy and society with strength of leadership, boldness, visionary thinking and innovative planning.

Managed transition. Build on existing processes and capacities to enable society to change in a structured and phased manner

Opportunity-focused. Look for synergies between sustainability, growth, competitiveness and employment creation, for South Africa to attain equality and prosperity.

Effective participation of social partners. Be aware of mutual responsibilities, engage on differences, seek consensus and expect compromise through social dialogue.

Balance evidence collection with immediate action. Recognise the basic tools needed for informed action.

Sound policy-making. Develop coherent and aligned policy that provides predictable signals, while being simple, feasible and effective.

Accountability and transparency. Lead and manage, as well as monitor, verify and report on the transition.

NEMA principles 1 2 3 4 5

Decisions must take into account all forms of knowledge, including traditional and ordinary knowledge

Participation of all interested and affected parties in environmental governance must be promoted,

All people must have the opportunity to develop their understanding

Development must be socially, environmentally and economically sustainable.

NSSD principles 1 2 3 4 5

Democratic governance

Socioeconomic systems are embedded in and are dependent on ecosystems

Consultation and participation