Slide 1

Presentation to Uganda Evaluation Week Nokuthula Zuma and Antonio Hercules19-23 May 2014

The Presidency Department of Performance Monitoring and Evaluation South Africas National Evaluation System

1OutlineEstablishment of DPME Why evaluation?NEPF and NEP Timeline for developing the systemStage we are at with evaluations?Current status with the evaluation systemUse of information by ParliamentConclusions2The Presidency: Department of Performance Monitoring and EvaluationTimeline around DPME2005Government-wide M&E system document2007Framework for Programme Performance Information (Treasury)2008System for data quality (StatsSA)

32009New administration, emphasis on M&EMinister of Performance M&E createdWork starts on developing priority outcomesApril 2010DPME created in Presidency, as delivery unit12 outcomes agreed, Ministers performance agreements, delivery agreements, quarterly reports Systems for Management Performance Assessment (MPAT) created with assessment of 103/155 national and provincial departments, monitoring of front-line services developed. June/July Study tour to Mexico/Colombia/USAugustDraft National Evaluation Policy Framework.OctoberFirst evaluation starts as pilot for the systemNovember National Evaluation Policy Framework approved by Cabinet

The Presidency: Department of Performance Monitoring and EvaluationMonitoring but evaluation4The Presidency: Department of Performance Monitoring and EvaluationWhy evaluate?5Improving policy or programme performance (evaluation for continuous improvement): this aims to provide feedback to programme managers.

Improving decision-making: Should the intervention be continued? Should how it is implemented be changed? Should increased budget be allocated?

Evaluation for improving accountability: where is public spending going? Is this spending making a difference?

Evaluation for generating knowledge (for learning): increasing knowledge about what works and what does not with regards to a public policy, programme, function or organization.The Presidency: Department of Performance Monitoring and EvaluationNote Minister clear he wants all 45Scope of the Policy Framework approved Nov 2011Outlines the approach for the National Evaluation SystemObligatory only for evaluations in the national evaluation plan (15 per year in 2013/14), then widenGovernment wide focus on departmental programmes not public entitiesFocus on policies, plans, implementation programmes, projects (not organisations at this stage as MPAT dealing with this)Partnership between departments and DPMEGradually developing provincial (2) and departmental evaluation plans (3) as evaluation starts to gets adopted widely across governmentFirst metro has developed a plan (Tshwane)

6The Presidency: Department of Performance Monitoring and EvaluationWhy a National Evaluation PlanRather than tackling the whole system, focus initially on strategic prioritiesAllows the system to emerge, being tried and tested in practiceLater when we are all clear it is working well, make system wide7The Presidency: Department of Performance Monitoring and EvaluationProgress with National Evaluation Plan evaluations82012/13 National Evaluation Plan approved June 2012, 2013/14 NEP in November 2012, 2014/15 November 20132012/13: 7 evaluations (NSNP moved to 2014/15)2013/14: 15 evaluations (1 agreed by Cabinet to be dropped)2014/15: 15 evaluationsECD evaluation completed June last year and on DPME website, 4 others have final reports and gone to Cabinet been in Parliament in April18 other evaluations underway from 2012/13 and 2013/14 inc 1 not in NEP 3 completing in a few weeks, 15 underway15 from 2014/15 TORs mostly developed, procurement started with some aim for most to be underway by April 2014 cycle now much earlier (we were at this stage only in May or so in 2013, and September in 2012)The Presidency: Department of Performance Monitoring and EvaluationPriority interventions to evaluateLarge (eg over R500 million) or covering a large proportion of the population, and have not had a major evaluation for 5 years. This figure can diminish with time;Linked to 12-14 outcomes (particularly top 5)/NDPOf strategic importance, and for which it is important that they succeed. Innovative, from which learnings are needed in which case an implementation evaluation should be conducted;Of significant public interest eg key front-line services.9The Presidency: Department of Performance Monitoring and EvaluationImplication of evaluation being in National Evaluation PlanApproved by Cabinet and reports will go to Cabinet (with Improvement Plans)Political support from Cabinet and DPME, including to resolve problems emergingCo-funding available from DPME (or if necessary DPME will assist with sourcing donor funding)Have to follow national evaluation system - guidelines, standards, steering committees, training to supportAll evaluations are partnerships with DPME who will sit on Steering Committee, provide technical support and quality assurance, and be involved in improvement plan.All evaluations published on DPME (and dept?) website unless security concerns

10The Presidency: Department of Performance Monitoring and EvaluationApproach - ensuring evaluations are usedKey challenge internationally that where evaluations are done, often not used - waste of moneyKey issues to ensure use:Departments must own the evaluation concept and the process and so they must request evaluation (not be imposed on them)There must be a learning focus rather than punitive otherwise departments will just game the system so punish people not because they make mistakes, but if they dont learn from their mistakesBroad government ownership so selection by cross-government Evaluation Technical Working Group based on importance (either by scale or because strategic or innovative)Evaluations must be believed - seen as credibleThere must be follow-up (so improvement plans)

11The Presidency: Department of Performance Monitoring and EvaluationApproach credibility and transparency To ensure credibility:Ensure independence:Independent external service providers undertake the evaluation, reporting to the Steering CommitteeEvaluations implemented as partnership between department(s) and DPMESteering Committee makes decisions on evaluation not departmentEnsure quality:Design clinic with top national and international evaluators (giving time free)Peer reviewers (normally 2) per evaluationDPME evaluation director part of whole processHave to follow system - evaluation panel, standards, guidelines, training etcQuality assessment once completed must score >3/5. (actuals so far 4.14, 4.45, 3.67, 4.1 3.71)To ensure transparency:All evaluation reports go to Cabinet Then evaluations made public unless security concerns media briefing, DPME website, Parliament, publication, communicationWhen complete quality assess and go into Evaluation Repository

12The Presidency: Department of Performance Monitoring and EvaluationTimeline around evaluations2012/13 Plan2012JanuaryDevelop system for National Evaluation Plan .FebruaryCall goes out for evaluations for 2012/13 JuneFirst National Evaluation Plan 2012/13 approved by Cabinet with 8 evaluationsJulyWork starts on TORs for 2012/13 evaluationsOctoberFirst evaluation from NEP 2012/13 startsOther start soon after2013MayFirst evaluations complete132013/14 Plan2012MayCall goes out for evaluations for 2013/14July15 evaluations approvedAugTraining of depts and work starts on TORsNov Second NEP for 2013/14 approved with 16 evaluations2013MarchTORs for 15 evaluations for 2013/14 being developedJuneMost underway2014JanFirst evaluation complete2014/15 PlanCall out

SelectionNEP approvedTORsStartThe Presidency: Department of Performance Monitoring and EvaluationEvaluation process 2014/1514Call for evaluations for 2014/15 1 April 2013Depts submit concepts for evals 30 June 2013Work starts on refining conceptAug/Sept 2013Selection by Eval Tech Working GroupJuly 2013Plan submitted into Cluster/Cab systemSept 2013Cabinet approves PlanNov/Dec 2013Finalising TORs, procurementJan-May 2014Evaluation commissionedFeb-May 2014Evaluation completedOct 2014 to March 2015Results to Cluster and Cabinet 1-2 months afterReport public to Parliament and WebsiteImmediateManagement Response/Quality Assessment1 month after completionImprovement Plan drafted12 Guidelines and templates - ranging from TORs to Improvement Plans plus 6 draft ones being finalised FebruaryVery significant ones on Planning Implementation Programmes and Design Evaluation major focus on improving programme designStandards for evaluations and competences, and standards have guided the quality assessment tool4 courses developed, over 600 government staff trained so far 1 more courses being developed and piloted by MarchIncludes course for DGs/DDGs in use of evidenceStudy tours organised for SCOA to Canada/US, Kenya/Uganda, unfortunately SCOA Chair not able to come to South-South RoundtableEvaluation panel developed with 42 organisations which simplifies procurement - major focus on ensuring universities bid. W Cape now using the panel may become Government-wide PanelCreation of Evaluation Repository - 70 evaluations quality assessed and on the Evaluation Repository on DPME website.

26The Presidency: Department of Performance Monitoring and EvaluationProgress with the system (2)Gauteng, W Cape provinces have developed provincial evaluation plans. DPME working with other provinces Limpopo, NW, Free StateDepartmental evaluation plans for dti, DST, DRDLRMunicipal evaluation plans Tshwane developed but not focus at present

27The Presidency: Department of Performance Monitoring and EvaluationConclusionsIn two years the whole system is now established and 38 evaluations are completed, underway, or about to startInterest is growing more departments getting involved, more provinces, first metro, and more types of evaluationWork on programme planning and design evaluation will potentially have very big impact will build capacity in departments to undertakeChallenges emerging as the evaluation reports start being finalised and the focus shifts to improvement plans Some gaming by departments as they see critical findingsNeed close monitoring of development and implementation of improvement plans to ensure that departments do implement the recommendationsImportance of Parliaments oversight role committees could request departments to present the evaluation results to them, request departments to present improvement plans to them, and request departments to present progress reports against the improvement plans to themImportant for Committees to consider requesting evaluations for 2015/16 cycle start discussing now28The Presidency: Department of Performance Monitoring and EvaluationThank you

Outcomes Manager: OME, DPMENokuthulaz@po-dpme.gov.zaDirector: ERU,DPMEAntonio.Hercules@po-dpme.gov.zawww.thepresidency-dpme.gov.za 29The Presidency: Department of Performance Monitoring and Evaluation