(c) F Cloete: GWM&ES as CAS, Cairo 2009 1

EVIDENCE-BASED POLICY ANALYSIS IN SOUTH AFRICA:

Critical assessment of the emerging GWM&ES

Fanie Cloete Department of Public Governance

(UJ)fcloete@uj.ac.za

(c) F Cloete: GWM&ES as CAS, Cairo 2009 2

Outline

• Evidence-based assessment• M & E as higher order management

function• The Emerging GWM&ES• Assessment of the GWM&ES• Complex adaptive systems (CAS)• Policy and GWM&ES as CAS• Conclusions

(c) F Cloete: GWM&ES as CAS, Cairo 2009 3

Evidence-based policy assessment 1• Evidence-based analysis largely theory before info

revolution• Info society enabled effective evidence-based

assessments• = approach that helps people make well informed

decisions about policies, programmes and projects by putting the best available evidence at the heart of policy development and implementation (Segone 2008: 27)

• = not opinion-based policy practice, which relies heavily on either the selective use of evidence (e.g. on single studies irrespective of quality) or on the untested views of individuals or groups, often inspired by ideological standpoints, prejudices, or speculative conjecture (Segone 2008:27).

(c) F Cloete: GWM&ES as CAS, Cairo 2009 4

M&E as higher order management function• M&E = not an isolated activity• M&E = integral part of good policy

management process• M&E = an applied research and planning

function needed to ensure goals are achieved.

• It is a higher order management function that compiles evidence of progress towards goal achievement and interprets the data to determine the extent of change

(c) F Cloete: GWM&ES as CAS, Cairo 2009 5

Process/ ongoing

evaluation Summative outputevaluation

Formative evaluation

Summative outcomesevaluation

Summative impactevaluation

Start date: Current base line data & goal to improve

Projected future end date: End or culmination data (feasibility/viability)

End data: Culmination data & concrete outputs/ services or products (Programmes/projects/instruments/ like quantity & quality of jobs, houses, water, police, roads)

Evaluation as gap assessmentEvaluation as gap assessment

Mid-term review: Activity progress data(Resource conversion processes: Efficiency, effectiveness, productivity)

Intangible short & medium term sectoral outcomes/results (mission)(Improved literacy, health, safety, affluence, communications, social security, environment)

Intangible long term multi-sectoral impacts/consequences (vision): (Poverty alleviation, quality of life, growth, empowerment, equity, accountability, democracy, sustainability)

(c) F Cloete: GWM&ES as CAS, Cairo 2009 6

Emerging GWM&ES in SA 1• SA behind many developed states but ahead of

many developing states• Until 2005 no coordinated M&E function in govt:

Isolated line function M&E by PSC, Treasury, DEAT & other depts

• Then:– Mil goals: 2015– WSSD: 2002– Presidency: POA– Donors

• Resulted in GWM&ES: 2005

(c) F Cloete: GWM&ES as CAS, Cairo 2009 7

Emerging GWM&ES in SA 2• Presidency coordinates GWM&ES• Emerging framework intended to provide

min structure and min uniform procedures (eg indicators, Treasury performance reporting framework, SASQAF)

• Implemented in decentralised way, maintaining existing systems & procedures asap

• Establish reporting system from community & eventually NGOs through prov, national depts to Presidency.

(c) F Cloete: GWM&ES as CAS, Cairo 2009 8

Main GWM&ES stakeholders• Presidency: GWM&ES Policy Framework, PoA,

NII, Mid-term dev indicators, 10 & 15 Yr Reviews• PSC: Guidelines & evaluations• Treasury: Perf reporting framework• StatsSA: SASQAF• DPSA: PERSAL, BAS data• DEAT: SoE, NFSD• DPLG: Prov & Loc Govt Key perform indicators• PALAMA: M&E Training• Line function depts: Nat, prov, distr, local• Businesses, NGOs, CBOs,

(c) F Cloete: GWM&ES as CAS, Cairo 2009 9

Structural components of the GWM&ES

(c) F Cloete: GWM&ES as CAS, Cairo 2009 10

(c) F Cloete: GWM&ES as CAS, Cairo 2009 11

(c) F Cloete: GWM&ES as CAS, Cairo 2009 12

Treasury performance information model

(c) F Cloete: GWM&ES as CAS, Cairo 2009 1336

Shared understanding of Local government indicator Shared understanding of Local government indicator development across all three spheres Phase 2development across all three spheres Phase 2

GWM&E System

National

Provincial

District

Local

GPOA

PGDS

IDP

IDPIndicators

(c) F Cloete: GWM&ES as CAS, Cairo 2009 14

Pres Mid-term dev indicators• Economic growth and transformation• Employment• Poverty and inequality• Household and community assets• Health• Education• Social cohesion• Safety and security• International Relations• Good Governance• One environ indicator only (greenhouse gas emissions)

(c) F Cloete: GWM&ES as CAS, Cairo 2009 1513

(c) F Cloete: GWM&ES as CAS, Cairo 2009 16

Complex adaptive systems 1• Simple, complicated, complex• Cooksey 2001: complexity = “…a non-linear

systems-oriented perspective that attempts to conceptualize, understand and intervene in organizational systems at multiple levels…in full recognition of the dynamic linkages and influences that operate within and between aspects of those system levels through time and space” & “…guiding instead of prescribing, adapting instead of formalizing, learning instead of defending, complexifying instead of simplifying and including instead of excluding”

(c) F Cloete: GWM&ES as CAS, Cairo 2009 17

Complex adaptive systems 2Van Buuren & Gerrits (2008: 382) …(e)very decision can be regarded as a temporarily stable state of equilibrium in which streams of negotiation, deliberation and fact-finding are connected stepping stones in an ongoing policy process which at the same time is influenced by parallel policy processes competing for the same resources (attention, money, legitimacy, support). Decisions that are made persist until a sufficient amount of system pressure (internal or internal) destabilizes the policy system towards a new state of equilibrium…

(c) F Cloete: GWM&ES as CAS, Cairo 2009 18

Complex adaptive systems 3Teisman & Klijn, 2008: …(w)e already know that managers are not the rational beings presented in many managerial handbooks and that they try to avoid choices or act according to the circumstances. The complexity theory gives us a different image of the manager as someone who is trying to survive in the ‘fitness landscape’ that he is creating jointly with other agents, by slightly bending and changing the conditions and using the moments and possibilities perceived. This will, almost certainly, also provide us with different prescriptions for these managers..(2008:297).

(c) F Cloete: GWM&ES as CAS, Cairo 2009 19

GWM&ES assessment 1• GWM&ES not integrated, coherent system• Regulated by collectivity of different sectoral

policy docs from different stakeholders not coherently integrated

• Many details still fuzzy or absent• Still emerging system: no time frames• SASQAF problems with stats quality• Not enforced • Rollout to provs & lower problematic• No vision: what to M&E: outputs, outcomes, etc?• Focus mainly on how:

(c) F Cloete: GWM&ES as CAS, Cairo 2009 20

GWM&ES assessment 2

• No link to African or other eval guidelines• Mid-term Dev Indicators very crude,

incomplete & unsystematic• Turf battles detrimentally affect system

implementation (eg PSC & DEAT).• Implementation capacity limited• Massive training effort needed by

PALAMA

(c) F Cloete: GWM&ES as CAS, Cairo 2009 21

GWM&ES assessment 3• A policy system = a complex adaptive system• Evidence-based policy assessment relies on

integrated higher order M&E function in govt • GWM&ES = emerging, complex adaptive

system:– many simple variables/components – of an open system subject to external influences, – interacting with each other in a dynamic, rich

historically determined and non-linear manner, – defying full understanding, – operating far from equilibrium (hovering on the

edge of chaos), – but surviving and expanding in a self-learning and

self-regulatory manner

(c) F Cloete: GWM&ES as CAS, Cairo 2009 22

GWM&ES assessment 4• GWM&ES does not have a formal hierarchical structure• operates as loose network of autonomous agencies

periodically interacting with one another. • No clear line of authority • frequent turf battles cause confusion and conflict

among stakeholders. • System still evolving, as new rules of the game are

being formulated or clarified, changing power and authority relationships among the main stakeholders.

• System has an inherent survival capability• Learning lessons in a self-regulatory manner illustrates

complex nature of system• The complex nature of the system justifies the

decentralised implementation approach • Optimal balance between centrifugal and centripetal

forces in the system is essential for success.

(c) F Cloete: GWM&ES as CAS, Cairo 2009 23

GWM&ES assessment 5• GWM&ES improvement strategies:

– Need for coherent and feasible, integrated and holistic national vision to guide M&E activities;

– Environmental and sustainability indicators to be integrated into Mid-term dev indicators, distinguishing output from outcome and impact indicators.

– Fast-track roll-out of system to prov & local govt levels– Improve capacity of M&E Coordinating Unit in the

Presidency – Improve inter-govt communication and marketing for

GWM&ES. – Reduce internal turf battles and overlapping M&E

mandates among main stakeholders – Build organisational culture of network co-operation

rather than hierarchical competition