Building a Results-Based

Monitoring and Evaluation

System for the

Provincial Government Western

Cape, South Africa

CONTEXT OF THE PRESENTATION

1. Background and rationale

2. Designing of the results-based M&E system

3. Phases of the provincial-wide M&E System

4. Type of data and information produced

BACKGROUND AND RATIONALE

• The primary aim of monitoring and evaluation (M&E) in the Provincial Government

of the Western Cape (PGWC) is to provide information for decision-making focussing

on the policy context.

• Data collected for M&E purposes is often wide-ranging; fragmented ; multiple

reporting requirements

• Make sense of the complexity/ diversity of M&E data collected against indicators,

turn data into useful strategic management information, integrate the data into a

system.

• The Provincial-wide M&E system developed for the PGWC - indicator and data

management system - a Results-based M&E system

• Moving towards an integrated system -with other governance systems- and fully

automated

• Provide strategic management information based on data that meets the criteria of

international best practice.

PROVINCIAL MANDATE• The PWMES for the PGWC focuses on the following three key principles:

– ensuring that the Executive is supported in the monitoring of provincial

performance

– monitoring and reporting the outcomes of the PGWC’s policies, strategies and

more specifically the 12 Provincial Strategic Objectives [PSOs] programmes and

projects

– ensuring that there is provincial alignment with National Government-wide M&E

System ; Performance Monitoring and Evaluation System; Framework for

Managing Programme Performance Information; National Statistical System

(NSS)

– Facilitate M&E for 13 provincial government departments

• The core and supporting processes of Provincial-wide M&E for PGWC is presented in

following diagram.

Provincial- wide Monitoring and Evaluation

Ensure the executive is supported in the monitoring of provincial performance in respect of national and provincial policy, strategy, programme and project implementationMonitor and report on the outcomes of the PGWC policies, strategies and programmes and project as well as monitor and report on the progress of provincial programme and project implementationEnsure provincial alignment to Government Wide M&E System and Performance M&E System

Support Processes

• Determine data and information requirements for the outcome indicators; policy areas and national statistical themes

• Facilitate the analysis of the causal relationships between outcomes and PGWC-specific policies, strategies, programmes and/or projects

Conducting Provincial-wide M&E

Methodologies; Standards and Data

Governance

• Develop and maintain a strategic framework for provincial wide M&E

• Monitor the implementation of methodologies, standards and procedures

Maintaining/Sustaining the

Provincial-wide M&E System

• Information Architecture to support the PWMES and its integration with other governance systems

• Departmental M&E Systems

• National Statistical System

• PGWC M&E forum• Co-ordination of data

producers and data users • Network of M&E

practitioners and special interest groups

• Data Quality Assessment Team

• Facilitate the implementation ond methodologies, standards and data governance for PWMES

• Facilitate workshops on building RBME systems and approaches

Policy context for Monitoring,

Evaluation (M&E)

Automated Provincial wide Monitoring and evaluation system

Monitoring and Evaluation Expertise

Capacity BuildingBusiness Processes

• Programme and Project Management methodologies

• Spatial Information• National Statistical

System

Core Processes

• Provincial alignment with the GWMES and Performance M&E System

• Provincial executive dashboard system meets the national and provincial requirements for Performance M&E

• Design and maintain indicator frameworks for core outcome indicators for provincial policies, strategies and programmes

• Develop and maintain a core directory of common data sources

• Develop and maintain a data management system and data warehouse

Core and supporting processes of Provincial-wide M&E

Data and Information on outcome indicators for MDGs; PSOs; Provincial Strategies; policy areas; statistical themes. Inform Provincial policy development and Provincial Strategic Planning is informed with strategic information

Outputs / Outcomes

The Provincial-wide Monitoring& Evaluation System(PWMES)• The Western Cape Model provides the platform for setting up RBM&E systems for

public sector M&E [13 provincial government departments] in the Province.

• The main aim regarding the use or application of the RBM&E system is to ensure

that it provides the data and information to measure against a core set of indicators

contained within the PWMES.

• It is based on methods of collecting data to measure outcomes and their links to

specific 12 Provincial Strategic Objectives.

• It is also based on linking outputs and their measurements, as identified by each

provincial department- Annual Performance Plans [APP] , and their contribution to

each outcome of the PSOs.

• Such information enables evidence-based decision-making in line with the provincial

government policies, strategies, programmes and projects.

2. Designing of the Results- based M&E System

Principles of RBM&E applied for the PGWC • The logical steps in building a RBM&E system and international results-

based M&E models – such as the Word Bank Model (Ten steps to Results-

based Monitoring and Evaluation), the IMA Model (six-step model) and the

Malaysian Governance System – informed the design and development of

the PWMES.

• To develop the RBM&E system for the Province, a specific sequence of

seven phases interdependent phases formulated

• Seven interdependent phases form the basis on which the PWMES seven-

phase model, also known as the Western Cape Model, is built.

Phases in designing the PWMES• Each of the seven phases constitutes a sub-system.

• These subsystems are interdependent and contained within the

overarching PWMES and the PGWC mandate .

• The subsystems provide the necessary components of the system

so that it can operate as a whole through effective indicator and

data management.

• The subsystems are reviewed annually to maintain an up-to-date

and comprehensive M&E system that will function effectively.

PWMES Provincial mandatePhases (subsystems) Components Core process Supporting processes

Phase 1Readiness assessment and stakeholder engagements

Provincial auditsInternal and stakeholder engagements

Building M&E capacityBuilding M&E expertise

Phase 2Overarching frameworks Conceptual framework

M&E strategyM&E framework

Policy context for M&E

Phase 3Indicator development Indicator frameworks:

PSONational statistical themesMillennium development goals (MDGs)

Maintaining and sustaining the Provincial-wide M&E System

Phase 4Monitoring and results framework

Monitoring and results frameworks:PSONational statistical themesMillennium development goals (MDGs)

Maintaining and sustaining the Provincial-wide M&E System

Phase 5Data management and data assessment

Core Directory of Common Data Sources (official data; mainstream data; provincial administrative data; spatial data)Indicator trends

Maintaining and sustaining the Provincial-wide M&E System

Data assessments on SASQAF application Methodologies, standards and data governance

Phase 6Information architecture Systems’ life cycle

Integration with governance systems Automating PWMES

Programme and project management Spatial informationNational Statistical System

Business processes

Phase 7Planning and sustaining the PWMES

M&E on results-based levelM&E on implementation level

Conducting provincial-wide M&E

The PWMES and its components aligned with the core processes and supporting processes of the provincial-wide M&E mandate

Phase 1Readiness assessment and stakeholder engagements

Phase 2Overarching frameworks for the PWMES

Phase 3Indicator development

Phase 4Monitoring and Results Frameworks

Phase 5Data management and data assessment

Phase 6Information architecture

Phase 7PWMES process: Planning to implement and sustain the PWMES

3. Phases of the PWMES Seven Phase Model

PHASE1:Readiness Assessment & Stakeholder Engagements• Readiness assessment involved conducting provincial audits

• Audits ascertained the capacity and readiness in each department:

– build a RBM&E system;

– be aligned to this provincial-wide system

– critical challenges faced in each department in relation to building RBM&E

systems

• Stakeholder engagement involved :

– identifying relevant stakeholders at local, provincial, national and international

levels;

– institutionalising stakeholder engagement through M&E forum

– external reference group

Policy Choices

Provincial Cabinet

Auditor General

Treasury

Provides Data onCross-Cutting Policy Areas

Compute Outcome &

Output Indicators

(PSO)

National Cabinet

Strategic Directives

Provincial – Wide Performance Management, M&E - Process

Produce:Indicator Assessment Reports

Bi-Annual Report Cards Programme of Action

Programmes/Projects (EPD) & Evidence

Submit Quarterly/Annual Reports

Provincial Strategic Objectives(aligned to National Outcomes)

Data

Provincial Administrative Data(Government Department)

Data

Official Data Sources

(STATS SA)

Data

External Data Sources(Research Institutions)

Compile Indicator Frameworks(Outcomes & Output)

Department Programme Officeand Monitoring & Evaluation

Projects Dashboard (systematic data collection &analysis as projects progress)

Departments

Spatial InformationProducts

Provincial Wide Monitoring& Evaluation System

(PWMES)

Provincial Programme

Office

Informs Cabinet

Report Quarterly on Project Performance Indicators

DotP Policy & Strategy

Annual Performance Plans

Premier’s Reports

Reports

Reports

PHASE 2: Developing Overarching Strategic Frameworks

• In this phase the policy objectives and outcomes for the PGWC were formulated as

agreed upon. The policy context of this PGWC for the period to 2014 is focused on

the provincial strategic objectives (PSOs) as approved by the Provincial Cabinet.

• The development of the strategic M&E frameworks for the PWMES:

– Provide conceptual and strategic understanding: provincial–wide M&E mandate,

RBM&E system and relationship to the policy context of the provincial

government.

– Sets out the strategic approach : how to implement RBM&E; assess how well

the provincial government is doing in meeting its strategic objectives.

– Integrates various nuances of M&E : programme and budget performance and

provincial-wide M&E (results-based M&E approach); the national context of

performance M&E -outcomes-based approach of the Presidency

PHASE 3: Indicator Development• This phase was the starting point in translating the Provincial Strategic Objectives

(PSOs) into broad outcome-based themes and then subdividing these themes into

aims linked to the desired outcomes, for the period 2010 to 2014.

• In this phase a compendium of indicators was selected to measure each aim ;

desired outcome for the PSOs taking into account the national statistical production

areas and global imperatives.

• Each indicator within the compendium was defined.

• The compendium of outcome/impact indicators; reference database

• Suggested measuring instrument to monitor and report on the outcomes of the

PSOs and national outcomes

Institutional Change

Change attributable to project funding /management

OUTPUT INDICATORS

Improvement or change in lives of

citizens

APP PERFORMANCE INDICATORS

OUTCOME INDICATORS

Results based M&E

Implementation M&E

APP: Programme and Budget Performance

Incorporating lessons learned into both

operational and strategic management

BASELINE DATA / EVIDENCE FOR POLICY ISSUE

IMPACT INDICATORS

ORIGINATION OF PROBLEM

STATEMENT

E.g. TEENAGE PREGNANCY, low

levels of Numeracy in Grade 6, ETC

Reduced TEENAGE PREGNANCY, Improved Numeracy levels in Grade 6, ETC

PROVINCIAL STRATEGIC OBJECTIVES

APP

TRANSVERSAL PROGRAMMES AND

PROJECTS

MANAGED THROUGH PROGRAMME AND

PROJECT MANAGEMENT METHODOLOGY

Indicators in the policy context

Outputs

Activities

Inputs

OutcomesResultsDirect

Intermediate

Impacts What we aim to change?

What we wish to achieve?

A long term developmental result at a societal level that is a

consequence of achieving specific outcomes

Medium-term results or consequences for society that are a logical consequence of achieving

specific outputs

The final products, or goods and services

produced by the institution for delivery to the

customer

The processes or actions that convert a range of inputs into the desired

outputs

The resources that contribute to production and

delivery of outputs

What we produce or

deliver?

What we do?

What we use to do the work?

Focu

s of th

e G

WM

&E sy

stem

g

en

era

lly

Focu

s of Pe

rform

an

ce

Info

rmatio

n

Focu

s o

f Pro

vin

cia

l Wid

e

M&

E

Imp

lem

en

tat

ion

M&

EDefinition

sFocus

Resu

lts-

based

M&

E

Ou

tcom

es b

ase

d a

pp

roach

PHASE 4:Developing Monitoring &Results Frameworks• In this phase all the attributes for each indicator selected identified in

order to build a monitoring system.

• The indicator frameworks developed in phase 3 were then extended to

accommodate all the attributes.

• The indicators and its attributes culminated into a monitoring and results

framework for each PSO.

• The monitoring system for each core indicator aims to drive all

subsequent data collection, analyses and data interpretation.

• It also provides the base for the selection of appropriate monitoring

methods, evaluation techniques and reporting requirements.

INDICATOR FRAMEWORK FOR PSOsSTRATEGIC OBJECTIVE 3: INCREASING ACCESS TO SAFE AND EFFICIENT TRANSPORT OUTCOME THEME 3: Improving transport safety

Aims/ Intended Outcomes

Indicator Definition /description

Baseline Frequency of data collection

Data source * Core directory

Data producer

Disaggregation

Reduce number of fatalities on Western Cape roads

Number of road fatalities in a given year in the Western Cape as a proportion of the total deaths

Number of road fatalities in a given year in the Western Cape as a proportion of the total deaths

1 623 (Percentage to be calculated)

Annual Causes of death (Road Traffic Accident Deaths)

Stats SA gender, age

Reduce number of accidents in Western Cape roads

Number of road accidents in the Western Cape in a given year as a percentage of the total number of road accidents in the country

Number of road accidents in the Western Cape in a given year as a percentage of the total number of road accidents in the country

New collection using administrative data

Provincial traffic; Municipal traffic

geographical area

Reduce number of accidents in Western Cape roads

Percentage of road accidents involving pedestrians in the Western Cape in a given year

Number of road accidents involving pedestrians in the Western Cape as a percentage of all road accidents in a given year

New collection using administrative data

Provincial traffic; Municipal traffic

Reduce fatalities in public transport in the Western Cape

Number of deaths in a given year occuring in rail transport as a percentage of total passenger trips in the Western Cape

Number of deaths in a given year occuring in rail transport as a percentage of total passenger trips in the Western Cape

New collection using administrative data

Metrorail

PHASE 5: Data Management and Data Assessment• This phase relates to data collection on the outcome indicators; assess if targets are

being met, analysis and reporting of results, the assessment of the quality of the

data

• The use of the most relevant data source ; the compilation of core directory of

common data sources

• In each data source, identified for the compendium of indicators, profiled and

electronically stored.

• This culminates in a composite system, draws its data from contributory systems

and provides official, mainstream and administrative data sources for the PSO

indicators

• Data quality standards for provincial administrative data records - South African

Statistical Quality Assessment Framework (SASQAF)

Thematic Area: Employment and economic activity. A. Labour Force Matters

Data Source: QLFS Profiling

Purpose:The QLFS is a household-based sample survey conducted by Stats SA. It collects data on the labour market activities of individuals aged 15 years and above who live in S.A.

Type of Data:Labour Force characteristics by :•Employment•Unemployment•Inactivity •Employed by:

• Industry plus sex• Industry and province; etc

•Key labour market indicators•Section 4: Not economically active [NEA}•Section 5: Social demographic characteristics•Section 6: Involvement in non-market activities and labour market status by province

Year:2nd quarter 2010

Tenure:Quarterly

Example 1 of Mainstream Data Profiling

Data Source: QLFS Profiling

Sample size:30 000 households per quarter

Statistical Production Area:•Employment, job creation and decent work•Economic growth and transformation

PSO Link:•Increasing opportunities for growth and jobs•Reducing and alleviating poverty

Limitation:•This survey only covers labour market activities of persons aged 15 to 65 years

Strength:•Quarterly data useful, unlike data being available bi-annually which reflects labour market activities

Source: Stats SA. Statistical release PO211: Quarterly Labour Force Survey. Q2: 2010

Continued…

Data Source: DOE: Exam Results Profiling

Purpose:To manage the calculation and publication of all results for the National Senior Certificate exams

Type of Data:National Senior Certificate exam results

Year:2003 - current

Tenure:Annually

Sample Size:[Results per learning area/subject]

Statistical production area•Education

Example 2 of Provincial Administrative Data Profiling

Data Source: DOE: Exam Results Profiling

PSO Link:Strategic Objective: Improving Education outcomesOutcome theme: Improved National Senior Certificate results

PTMS Data Theme:Skills Development[16.3.f Manage the calculation and publication of all exam results]

Limitations:External development of ‘system’ and maintenance thereof.

Strengths:None indicated

Source: Department of Education: Exam Results

Continued…

PHASE 6: Information Architecture• The information architecture is designed for supporting manual processes,

collecting PWMES information and for managing data collected to measure the

indicators.  

• Business Requirement Specifications

• PWMES broader computer-based relational data base.

• This data base contains data collected not only for the PWMES, but also for:

– Annual Performance Assessment System (APAS)

– Executive Provincial Dashboard (EPD).

• The PWMES, as an electronic system, draw its data by interfacing with other e-

platform systems.

• APAS, an electronic application, store output indicator results

• EPD, as an electronic application, receives and captures information on

departmental projects and tracks progress and budget utilisation.

Executive Projects Dashboard(Premier’s Provincial Dashboard)

http://www.capegateway.gov.za

Provincial-wide Monitoring and Evaluation System

PHASE 7: PWMES: Planning to implement

• In this phase one ensures that the PWMES delivers an effective indicator

and data-management system

• The annual review of the subsystems of the PWMES takes place in this

phase.

• This review is pivotal in maintaining an up-to-date comprehensive M&E

system

• Ensuring that the components in each phase adhere to the necessary

policy context, M&E elements and mechanisms for such a system.

4. Types of data and information

generated• Various reports have been produced annually through the effective

implementation of the PWMES

• Examples include:

– Six month report Card

– Annual Data Release

– MDG report

– Performance indicators for Gender, Youth and Disability

– National output indicators

– APRM provincial reports

Thank You

Project Team: Department of the PremierZeenat Ishmail zishmail@pgwc.gov.zaRowina Wynford rwynford@pgwc.gov.zaFaizel Noordien fnoordien@pgwc.gov.za

Quality Assurance Faldie Esau [South Africa Labour & Development Research Unit] faldiee@gmail.com Dr. Ros Hirshowitz rosh@mweb.com