AUDITING OF PREDETERMINED OBJECTIVES
24 November 2016
Reputation promise/mission
The Auditor-General of South Africa has a constitutional mandate and, as the Supreme Audit Institution (SAI) of South Africa, it exists to strengthen our country’s democracy by enabling oversight, accountability and governance in the public sector through auditing, thereby building public confidence.
Content
1. Introduction
2. Legislative requirements and framework for performance management and reporting
3. Audit strategy and approach
4. Findings for local government 2014-15
5. Recommendations and good practices
• Supreme audit institutions (SAIs) do not only conduct financial audits. They also provides assurance on the reporting of predetermined objectives produced by municipalities, government departments and entities
• Why is this necessary?
Public sector reforms Improving public reporting Providing better information on what government is accomplishing regarding service delivery
Need for reporting on predetermined objectives
Audits performed by AGSA
Annual mandatory audits
• Financial statements
• Report on predetermined objectives
• Compliance with laws and regulations
Discretionary audits
• Performance audits e.g. infrastructure
Difference between auditing of predetermined objectives (AOPO) and performance auditing
AoPO Audit of Predetermined Objectives
• Individual audits
• Focused on a specific government policy or management process
• Audit criteria are the 3E’s:
Economy
Efficiency
Effectiveness
• Done by subject matter experts
Annual audit of reported actual performance against predetermined objectives, indicators and targets as contained in the annual performance report.
Integral part of the annual regularity audit process, confirming:
compliance with related laws and regulations
usefulness of performance information
reliability of performance reporting
PA
Performance Auditing
An audit report must reflect an opinion or conclusion relating to the performance of the
auditee against predetermined objectives
Sections 20(2)(c) and 28(1)(c) of the Public Audit Act (PAA)
Applicable to all spheres of government
Legislative requirement – audit of predetermined objectives
AGSA strategy
2009-10 to 2010-11 2011-12 to 2015-16
2004-05 to 2008-09
Phased-in approach Factual audit findings
reported in both management and audit reports
No audit opinion in audit reports
Interaction with stakeholders (NT, Presidency, DPSA) to determine and test audit approach
Inputs to drafting of NT frameworks (FMPPI)
Completed phased audit approach
Audit to the extent necessary to express an audit conclusion
Audit conclusion in the management report for all high-capacity municipalities and their entities
Audit reports contain audit findings – not audit opinions
Completed phased audit approach
Audit to the extent necessary to express an audit conclusion
Audit conclusion in the management report for all municipalities and their entities
Audit reports contain audit findings – not audit opinions
This will continue until the environment shows a state of readiness to provide an audit conclusion in the audit report
Audit Approach
1
2
3
Obtaining an understanding of the entity's information system and related control
activities.
Assessing risks of material misstatement to form a basis for designing and
performing further audit procedures.
Obtaining sufficient appropriate evidence on which to base the conclusion on the validity, accuracy, and completeness of reported performance information.
• The audit approach for the audit of predetermined objectives is similar to that of the audit of the financial statements in the following ways:
• Auditees need to instil the relevant controls to ensure that the reported performance for each indicator is valid, accurate and complete in the same way controls are implemented to ensure line items in the financial statements are valid, accurate and complete
• Proper record-keeping measures should be in place to ensure that information is accessible and available to support performance reporting
Main criteria Sub-criteria
Presentation
Measurability
Relevance
Consistency
Validity
Accuracy
Completeness
Usefulness
Reliability
Audit criteria (derived from performance management and reporting framework)
Definition
Performance reporting using the relevant NT reporting principles
Indicators are well defined and verifiable Targets are specific, measurable and time bound
Indicators relate logically and directly to the entity mandate and realisation of strategic goals and objectives
Objectives, indicators and targets are consistent between planning and reporting documents.
Reported performance has occurred, have been recorded and reported correctly and completely.
Measurability and relevance
Measurability of indicators
Measurability of targets
Relevance of indicators
• Test well defined
• Test verifiability
• Specific
• Measurable
• Time-bound
• Direct and logical link of indicators and targets
• To mandate, strategic goals and objectives
What do auditors test when auditing measurability?
Measurability of indicators
• Test whether indicators are :
• well defined (Does the indicator have a clear definition and is there technical indicator descriptions for each indicator? )
• verifiable (Is it possible to verify the processes and systems that produce the indicator?)
Measurability of targets
• Test whether targets are:
• Specific (Is the nature and required level of performance clearly identified?)
• Measurable (Can the required performance be measured?)
• Time-bound (Is the time period/deadline for delivery specified?)
What is appropriate audit evidence?
Relevant
Reliable evidence
Appropriate evidence
Examples of relevant and reliable evidence
• Evidence is relevant to planned target and reported achievement e.g. Target is to build a dam but evidence provided is a report on a infrastructure plan to be developed
• The primary sources or origin of the actual performance is available to the auditor
Relevant
• Evidence is more reliable when
• In documentary form – e.g. paper, register, electronic or other media
• Generated through management and information systems with proper controls
• All source information exist and is complete – e.g. a list of houses built that can be verified against completion certificates
• Information is valid – e.g. every entry in register can be traced to a invoice/payment certificate or appropriate record
Reliable
Audit reporting – management report
An audit conclusion will be expressed for ALL municipalities and municipal entities in the management report, on –
USEFULNESS of reported performance information for the selected development priorities/ objectives
RELIABILITY of the reported performance for selected development priorities/objectives
Audit reporting – audit report
REPORT ON OTHER LEGAL AND REGULATORY REQUIREMENTS
Predetermined objectives
Usefulness of information Material audit findings focusing on consistency, relevance and measurability of reported performance information for selected development priorities and objectives
Reliability of information Material audit findings focusing on reliability of reported performance information for the selected development priorities and objectives
Compliance with laws and regulations
Non-compliance findings relevant to the performance management systems and processes
Findings for local government 2014-15
MFMA: Timing of submission and quality of APR improved over 5 years. Auditees continue to avoid material findings due to corrections during audit.
2014-15
MFMA
With no findings
With material findings (reported in audit report)
No APR/late submitted
Outstanding audits
Submitted APR with
material
misstatements
2% (6) 4% (11)
5% (13) 9% (25)
13% (36) 14% (40)
56% (154) 62% (172)
65% (181)
68% (189) 66% (183)
38% (107)
33% (93)
26% (72)
19% (53) 20% (55)
2014-15 2013-14 2012-13 2011-12 2010-11
79% (207)
78% (203)
80% (198)
85% (201)
94% (218)
278 municipalities
Findings for local government 2014-15
• There has been a significant improvement in the submission of APRs since 2010-11 when 14% of municipalities had either not prepared APRs or not submitted them on time for the audit.
• This contributed to significant improvement in the Free State, the Northern Cape and North West as a result of continued monitoring by provincial treasuries and the provincial departments of cooperative governance.
• There was also a further slight improvement in 2014-15 compared to the previous year.
• There has been an overall increase in the number of municipalities with no
material findings on the quality of their APRs since 2010-11 – the number has almost doubled.
• The main improvements during 2012-13 and 2013-14 occurred in the Eastern Cape, the Free State and KwaZulu-Natal.
Findings for local government 2014-15
• The improvement in these provinces was as a result of increased awareness of the requirement to report on performance and improved key controls.
• Our interactions with leadership and their willingness to implement our
recommendations including the implementation of performance information systems that are managed by competent personnel contributed to the improvement.
• Overall, 79 municipalities (29%) had no material findings in the current and
previous year, which means the controls and processes required to produce credible performance reports are in place to ensure sustainability .
• The figure also shows a reduction since 2010-11 in the number of
municipalities that submitted APRs that contained material misstatements. • 53 municipalities (21%) had no material findings in their 2014-15 audit report
ONLY because they corrected all the misstatements we had identified during the audit. This is an improvement compared to the previous year.
MFMA: Findings on usefulness and reliability 2014-15
MFMA
Findings on usefulness
Findings on reliability
54% (126)
58% (136)
61% (150)
57% (148)
56% (147)
71% (164)
73% (172)
68% (167)
57% (147)
47% (123)
2010-11(232 APRsprepared)
2011-12(236 APRsprepared)
2012-13(247 APRsprepared)
2013-14(259 APRsprepared)
2014-15(261 APRsprepared)
Findings on usefulness and reliability
• Significant improvement in the usefulness of the information in the APRs since 2010-11. The number of municipalities with findings on usefulness decreased by 10% compared to the previous year.
• The most common findings on usefulness in 2014-15 were: > Indicators not well defined (33%) > Indicators not verifiable (30%) > Reported information was not consistent with the objectives (29%) > Targets not measurable (23%) or > Targets not specific (23%) • The usefulness of the reported information improved as municipalities
corrected their performance indicators and targets as part of the annual planning and budget processes.
Findings on usefulness and reliability
• A slight regression in the reliability of APRs since 2010-11, with a slight improvement compared to the previous year.
• The processes and controls required to produce reliable information on performance have shown little improvement over the period.
• In the past five years there has been an improvement in municipalities’
reporting on delivery of services in accordance with planned targets.
• Progress made towards useful and reliable performance reporting is slow,
makes it difficult for provincial and national government to track progress towards the service delivery goals in the MTSF and national development plan
24
2014-15
MFMA
Findings on the usefulness and reliability of annual
performance reports remained stagnant
With no findings
With findings
• The annual performance reports of the auditees were useful and
reliable 63%
With no findings With findings
44%
62%
63%
56%
38%
37%
2012-13
2013-14
2014-15
Budget R59 billion (90%) Budget R6,4 billion (10%)
25
Annual performance report not materially misstated Annual performance report contained material misstatements
37% (25)
63% (43)
66% (45)
34% (23)
Outcome if
NOT corrected
Outcome
after corrections
[20 auditees (29%) [2013-
14: 15 auditees (22%)]
avoided findings due to
the correction of
material misstatements
during the audit
process]
38%
(26)
62%
(42) 60%
(41)
40%
(27)
Outcome if
NOT corrected
Outcome
after corrections
2014-15 MFMA
Quality of annual performance reports submitted for
auditing regressed – both submitted and after audit 2
01
4-1
5
68
au
dit
ee
s 2
01
3-1
4
68
au
dit
ee
s
Status of performance in KwaZulu-Natal
• The results on reported performance information show a steady improvement of 27% over the past five years as 24 auditees reported findings in 2014-15 compared to 40 auditees in 2010-11.
• However, many managed to avoid findings on reported performance in 2014-15 only because they corrected misstatements identified during the audit process.
• Auditees that were able to provide credible performance reports managed R59 billion (90%) of the total local government budget within the province.
• It is encouraging to note that the eThekwini Metropolitan Municipality has had no material findings on performance reporting for five years. Daily and monthly checks and balances, regular and accurate reporting as well effective oversight and risk management are essential for improved performance reporting.
• A slight regression in the reliability of APRs since 2010-11, with a slight
improvement compared to the previous year.
Root causes of defective service delivery reports
Inadequate formal planning
for PDO reporting
PDO reporting not included in performance agreements
Structures not capacitated
Organisational structures not
aligned
Inadequate internal audit
testing systems and data
Ineffective audit
committee oversight
Lack of monitoring by
leadership
Poorly defined roles and
responsibilities
Lack of accountability and consequences for actions / inaction
Do not meet regulatory requirements Not useful Not reliable
Recommendations and good practices
• Policies and procedures to report on performance information should be developed and implemented
• A set of performance indicators that are well defined and verifiable should be developed
• Performance targets that comply with the SMART criteria should be developed • Data definitions for all performance indicators should be defined
• CoGTA should support local government by developing customised indicators
and targets
• The methodologies and systems used for compiling budgets should be improved to strengthen the link between budgets and performance targets
• The correlation between planned and actual performance (performance implementation) in relation to the budgeted versus actual expenditure (financial implementation) should be monitored and evaluated
Recommendations and good practices • Predetermined objectives, financial reporting and compliance reporting should
be viewed in an integrated fashion as these are inseparable processes
• Formal processes and systems for the collection, collation, verification and storing of actual performance information should be developed, documented and approved by municipal manager.
• Auditees should ensure that there is an adequately resourced and functioning internal audit unit, audit committee and MPAC that provides assurance over the quality of the quarterly and annual performance reports
• Management should maintain portfolios of evidence to support reported targets, which are reviewed monthly and audited by the internal auditors.
• There needs to be sufficient oversight and monitoring of performance during the reporting cycle to ensure that performance targets are reported as planned
• Larger auditees should consider a dedicated strategic planning and/or monitoring and evaluation unit
Recommendations and good practices
• Auditees should prepare an audit action plan based on the audit findings of the previous financial year, including definite actions to address these audit findings
• Leadership must insist on credible quarterly performance reports that link to key
projects undertaken and progress to date • Effective human resource management and timely filling of key posts, effective
training programmes and performance appraisals, as well as succession planning • Councils and MPACs need to be more persuasive in their quest for assurance
from and accountability by audit committees and internal audit
• Induction and training programmes need to be further refined by CoGTA and Salga for these oversight structures to be effective.
Recommendations and good practices
• The coordinating departments (CoGTA and provincial treasury) can still amplify
their support to mobilise resources and efforts in the province. • Long-term proactive solutions should be found to address emerging risks
• Effective performance management is a fundamental ingredient for transparent
and credible reporting. • It is crucial that all data provided is carefully collated, reviewed and quality-
assured by management, internal audit units and audit committees before being reported on.
• Auditees need to ensure that they have adequate skills and resources to perform
proper strategic planning, performance monitoring and reporting in line with applicable requirements
32
Drivers of sustainability
THE POWER OF MEASURING RESULTS
• If you do not measure results, you
cannot tell success from failure
• If you cannot see success,
you cannot reward it
• If you cannot reward success, you
are probably rewarding failure
• If you cannot see success, you
cannot learn from it
• If you cannot recognise failure, you
cannot correct it
Adapted from Osborne and Gaebler, 1992, Reinventing Government
THANK YOU
34