Performance auditing, newpublic management and

performance improvement:

questions and answersFrans L. Leeuw

Utrecht University and Netherlands Court of Audit,

The Hague, The Netherlands

Introduction

This article discusses relationships between the new public management

(NPM) and performance auditing (PA). Usually, it is believed that public sector

performance audits as well as financial audits lead to a more efficient and

effective performance of this sector. However, in several West European

countries this assumption recently has been challenged. Adherents of NPM are

critical with regard to this assumption.

The article first describes characteristics of the new public management and

next answers the question why performance auditors should be interested in

this phenomenon. The major part of the paper discusses difficulties and

challenges performance auditors are confronted with when their aim is to

contribute to a better public sector performance. Most of the examples used and

studies referred to in this contribution refer to the situation in Western-

European countries like the UK and, in particular, The Netherlands.

New public management: some characteristics

In Western Europe the concept of new public management (NPM) is becoming

popular. It emphasizes economy, efficiency and effectiveness of governmental

organizations, policy instruments and policy programmes. NPM strives for a

greater quality of service delivery. Lesser attention is paid to compliance with

formally prescribed processes, rules and procedures. Other goals of NPM are

freeing up controls over and devolving greater responsibility to operating

managers, creating additional flexibility or autonomy for managers, making

public sector managers manage, stressing a greater focus on risk management

and measuring performance (PUMA, 1995, pp. 9-10).

The views expressed here are those of the author and no endorsement by The Netherlands Court

of Audit is intended or should be inferred. An earlier version of this paper was given at the OECD

symposium on Performance Auditing and Performance Improvement in Government, Paris,

6-7 June, 1995.

Organizational mechanisms through which these goals are achieved are the

following:

The establishment of quangos: quasi-autonomous non-governmental

organizations and similar executive agencies functioning at arms length

of (central) government.

The creation of quasi-markets: the government is paying for the services,

but it will no longer be providing them. Instead, welfare services will besupplied primarily by a variety of semi-independent agencies. Schools

will be competing for state-financed pupils and independent hospitals

will be competing with each other for patients (Bartlett and LeGrand,

1993).

Privatization.

Public-private sector partnerships.

Policy networks. Essential for these networks is that government acts

primarily as a conductor of different activities of different actors.

Together with pressure groups, independent experts and advisory

groups, government tries to define and realize policy goals[1].

In general it is believed that through these organizational mechanisms

government will get leaner, smarter, more efficient and more effective.

Why should performance auditors be interested in the new public

management?

Given the NPM focus on efficiency and effectiveness, it looks like there is hardly

a need for performance auditors to focus on reviewing NPM. Stressing the

importance of efficiency and effectiveness and contributing to their realization

has always been a central goal of PA. So why, it could be argued, bother NPM

officials with audits when the goals of both activities are so similar?

There are three reasons why performance auditors nevertheless are (and

should be) focusing on NPM and its possible consequences:

(1) NPM has goals that converge with the goals of PA. However, there is no

a priori evidence that these goals indeed will be realized. Striving for

performance improvement of governments is not equal to realizing

improvement. Studies on intended and unintended consequences of

policy making show that goals sometimes run the risk of being so

promising and challenging that politicians and bureaucrats, merely by

publicly stressing their importance, believe that they are already

realized. Sieber (1981) gives examples and calls this overcommitment.

Performance auditing makes it possible to distinguish ambitions and

intentions from realizations.

(2) Implementing NPM runs the risk of leading to unintended and

undesired consequences which are counterproductive to the goals

formulated. Auditing can unravel intended and unintended

consequences. In the literature examples of unintended consequences

are given. One is cream-skimming which is discrimination by either

purchasers or providers against the more expensive users: the

chronically ill patient, the incontinent, confused, elderly person, the

disruptive child If purchasers can choose for whom they will

purchase and providers can choose for whom they will provide, that is,

if they can skim off the cream, then welfare services may not reach those

who need them most and equity will not be achieved (Bartlett and LeGrand, 1993). This example stresses the importance of having clear

performance measures.

(3) Innovations like NPM run the risk of being mimicked. While the early

adopters of NMP may implement the philosophy in practice and indeed

create successes, later, other organizations start to imitate their

successful predecessors without, however, realizing the same effects

(isomorphism). Put somewhat differently: the latter organizations may

have the form right, but the substance wrong. Legitimation has become

more important than innovation. Performance auditing is able to

unravel these different aspects of the implementation and adoption of

NPM.

These three reasons play an important role for auditors at The Netherlands

Court of Audit to become involved in NPM. Before discussing difficulties and

challenges of auditing NPM, I will first give some background information on

The Netherlands Court of Audit.

The Netherlands Court of Audit: mandate for and practice of

performance auditing and programme evaluation

The Netherlands Court of Audit (in Dutch, Algemene Rekenkamer) is an

independent body and not subject to political supervision. Its primary focus is

the central government and related organizations. The Courts structure, duties

and competences are set out in specific acts of which the Budget and

Accounting Act (BAA) is central. The Court, which has a staff of 300 employees,

has a board of three persons (president and two members). The audit and

research work itself is done in three directorates, one with a special focus on

performance audits and evaluation, one with a focus on legality and one with a

focus on auditing quangos, government-sponsored enterprises and similar

public-private arrangements.

The Budget and Accounting Act (BAA) includes the following statements on

auditing efficiency and effectiveness (article 57): The Netherlands Court of

Audit pays attention to the effectiveness and efficiency of government

management, organisation and policy (BAA, 1991). The expression pays

attention to means that the audit office investigates efficiency and

effectiveness, in terms of procedures and regulations and through on-site

investigations and programme evaluations.

The BAA (article 57) specifies the goals of the performance audit work asfollows:

The auditing of effectiveness and efficiency includes:

(a) auditing the effectiveness and efficiency of management, which includes all possible

aspects of the internal management at (departments of) the public service;

95

(b) auditing the effectiveness and efficiency of the (departmental) organization; and(c) auditing the effectiveness and efficiency of policies (implemented) (BAA, 1991).

The BAA makes the distinction between goal achievement and effectiveness of

policies, programmes and organizations. This is important, because not every

goal achievement can be considered the result of the implemented policy or

programme.

The BAA also gives an answer to the question of which points in time the

Court of Audit is expected to audit the effectiveness and efficiency of policy,

management and organizational structures. The BAA says the following on

this subject:

The current situation is such that the effectiveness of policy is considered auditable as soon as

the government or a minister has taken the decision, whether or not after having consulted

with or after approval of Parliament. Before the policy itself has been implemented, an

analysis of objectives can then also be carried out, as well as an audit of the effectiveness of

policy-preparatory activities (BAA, 1991, p. 7).

In a study published by the Public Management Group of the OECD (PUMA,

1995, pp. 5-7), a distinction is made between substantive performance audits,

where on an ex post basis the functioning of a programme or government

activity is analysed by reference to certain standards or benchmarks and

systemic audits. This latter type of audit assesses the adequacy of performance

information and its use. Both substantive and systemic audits are carried out

by The Netherlands Court of Audit. With regard to the substantive studies the

focus not only is on programmes and activities but also on organizations as

such (how well are they organized and do they achieve the goals set?). With

regard to the systemic audits The Netherlands national audit office also

highlights the so called government-wide performance audits (GWAs), which

focus on policy instruments, programmes and activities relevant for all or most

of all ministries. These audits are comparative in nature. Since the late 1980s

they have formed an important part of the agenda of the Netherlands Court of

Audit. GWAs mirror the current state of the art with regard to different policy

instruments such as subsidies, grants, levies, loans and public information

campaigns (Leeuw, 1992a).

In The Netherlands, substantive and systemic audits make use of results

from programme evaluations. Sometimes these evaluations are carried out by

auditors themselves; more often they are carried out by social science research

institutes (universities, (not-for-profit) research firms). With regard to The

Netherlands Court of Audit with over 30 per cent of its professional staff having

a degree in the social, behavioural or economic sciences, not only reviews of

programme evaluations done by others (through meta-evaluations) are carried

out, but the organization also carries out substantive programme evaluations

(Netherlands Court of Audit, 1991). As a consequence, performance auditing

and programme evaluations converge.

Performance auditing and performance improvement: difficultiesPerformance auditors have to play a role in analysing and reviewing NPM and

its consequences. NPM needs auditors in order to verify accountability

information. This information can be used to compare the goals formulated

with the goals realized. As the goals of NPM and PA converge, it can be

hypothesized that NPM officials will welcome performance auditors. However,

in practice things are not that simple. It appears that the relationship between

PA and NPM is not without difficulties.

PUMA (1995, pp. 12-13) specifies some of the possible difficulties in the

relationship:

the focus of auditors on procedures may be too limited to be valuable for

performance improvement;

the fault finding approach of auditors may be too narrow-minded; and

the auditors assumed lack of formulating constructive suggestions may

make the relationship between NPM and PA also difficult.

Other difficulties are discussed next.

Ossification

Smith (1995, p. 299) illustrates that performance measures can inhibit

innovation and lead to ossification: organizational paralysis brought about by

an excessively rigid system of measurement.

Based on recent Dutch experiences, auditors are sometimes confronted with

the question of why they do not pay attention to the negative impact of their

(fault-finding) approach on the audited organizations human and social capital.

Sometimes commitment of people to their organization declines after a

performance audit is published, even when the findings of the report cover only

side-aspects of their organization. This in turn may reduce the motivation of

people. As public sector activities often are peoples businesses, it is wise not to

neglect this side-effect.

Auditors (and evaluators) like standards

Intosai, Eurosai, CPA-organizations and (learned) societies active in the field of

programme evaluation[2] seem to like standards. There is nothing wrong with

standards as such, but when viewed from the perspective of contributing to the

improvement of performance, there are challenges. PUMA (1995, pp. 13-14)

mentions the problem that stressing the importance of standards may lead to an

excessive attention on formal adherence to these standards.

What seems to underly this point is the interest auditors have for procedures

and procedural effectiveness. In most of the audit reports I know (Dutch,

German and Anglo-Saxon), auditors often are satisfied when on paper the

procedural and organizational prerequisites for an effective policy or an

effective public organization are met. However, these prerequisites are nothing

more than articulated assumptions that may be right or wrong. The relativelyfew empirical studies in which auditors go all the way (from studying the

preparation of a policy and its implementation to the assessment of the social

impact of the policy) are a signal of this.

Tunnel vision

Stressing procedural effectiveness has at least one other important side effect

tunnel vision:

Tunnel vision can be defined as an emphasis on phenomena that are quantified in the

performance measurement scheme at the expense of unquantified aspects of performance.

For example, maternity service managers in the UK National Heath Service are increasingly

being held to account by a single performance indicator: the perinatal mortality rate. There

is clear evidence that the emphasis on the quantifiable mortality rate is distorting the nature

of maternity services, to the detriment of the non-quantifiable objectives (Smith, 1995,

p. 284).

Smith (1995, pp. 290-91) also provides another example[3]. It deals with the

unintended consequences of adopting the Patients Charter in the UK as a

procedural benchmark. Charters are rapidly becoming a popular policy

instrument in countries like the UK and The Netherlands. The Patients charter

states that no patient should wait more than two years for surgery. The

consequences of implementing this benchmark have had the effect that health

authorities have almost universally eliminated waits in excess of two years.

However, there have also arisen some unintended consequences. First, there has

been an increase in the number of patients waiting more than one year: indeed,

there is evidence to suggest that the average waiting times have increased.

Further, early reports mentioned by Smith (1995, p. 291) suggest that the

patients who are benefitting from the waiting time initiative are those requiring

relatively minor procedures, at the expense of patients awaiting operations for

more serious conditions.

These descriptions are examples of unintended and negative consequences

of stressing procedural effectiveness and procedural standards. They also

reflect the general difficulties of developing good performance measures. What

is also at stake is that auditors may stress this type of information, because they

believe that the single, quantifiable indicator is an adequate proxy for effective

health programmes in general[4]. Sometimes, however, this is not true.

Other difficulties

In my opinion these difficulties are not related to the necessity of validating

accountability information as such. What are at stake are:

the ways in which this information is reviewed and validated;

what kind of information is reviewed;

to what extent the information in fact can improve performance; and to what extent the information can have (unintended) and possible

counterproductive effects on performance.

Frey and Serna (1990) have discussed the topic of counterproductive

consequences of auditing practices. Their study referred to national audit

offices in central European countries like Austria. They linked the (quasi)-

monopolistic nature of audit offices with the production of procedural and

formal information on performance. One of their arguments was that while this

information may be needed from a legality perspective, it is counterproductive

when viewed from the perspective of performance improvement.

These difficulties in the relationship between performance auditing and

performance improvement lead to the following question: what are important

challenges in the relationship between PA and NPM, given the goal of realizing

performance improvement of governments through PA?

Performance auditing and improving the performance of the

government: three challenges

The first challenge

The first challenge is how to develop and help implement performance

measurement schemes that do not run the above mentioned risks and to

identify inadequate performance measures. Maybe one of Smiths (1995)

recommendations is useful here. He suggests involving staff at all levels in the

development and implementation of performance measurement schemes

(p. 304). When one applies this idea to the world of auditing it may lead to a

greater and more open communication between auditors and auditees on the

criteria applied in PA and NPM.

The second challenge

This concerns an expansion of a point made by Moukheibir and Barzelay (1995)

about the (implicit) assumptions underlying different types of auditing. This

approach can be linked to earlier studies by Mason and Mitroff (1981) on

analysing strategic planning assumptions, work by Chen (1990) on theories-in-

action that underly programmes to be evaluated and to Leeuw (1992b) on

articulating and evaluating policy theories.

Moukheibir and Barzelay (1995, pp. 3-6) distinguish traditional (financial)

audits, performance audits and programme evaluations. Underlying the

traditional (financial) audit is the theory that the government resembles a

machine bureaucracy: this bureaucracy is thought to function well when

officials apply legal norms and technical standards to matters within their

assigned areas of authority and responsibility (p. 4). The performance audit is

based on the theory that government is an:

Adaptive organism: this image portrays managers as agents performing important functions,

including adapting organizations to shifts in their mandates and resources The concept of

performance audit is characterized primarily by the view that the public sector functions well

when managerial rationality is applied to the perennial task of adjusting means to ends and,

in particular, to accomplishing results with resources (p. 5). Programme evaluation is the third type. Here the underlying assumption is that

government functions well when the programmes (clinical interventions) are

subject to scientific scrutiny and more or less pass the examination. Govern-

ments are successful when the effectiveness of treatments (interventions)

increases (p. 4).

Auditors implicit feedback theory. To gain an insight into the difficulties in

the relationship between auditors and NPM, I have reconstructed the implicit

feedback theory which underlies most of the reporting in traditional and

performance auditing. Basically this feedback-theory can be stated as five

steps:

(1) feedback from the auditor to the auditee is needed when it is shown that

the standards or goals are not (or inefficiently) reached;

(2) the auditee will listen to the feedback;

(3) he/she subsequently will take follow-up action;

(4) which will lead to the realization of the formal goals; and

(5) this will not lead to unintended and undesired side-effects.

This theory assumes that the auditee refrains from strategic actions (merely to

make himself look good), elicited by the fact that he is familiar with the

standards or measurements auditors apply. The theory also assumes that the

likelihood of unintended and undesired side-effects is small to zero.

What can be said about the empirical content or validity of this audit

feedback theory? Meyer and OShaughnessy (1993) recently have presented

evidence that strategic behaviour does occur related to performance

measurement. In order to explain this they make use of the concept of

isomorphism and stress that not only managers run the risk of mimicking, but

also auditors/financial analysts and evaluators. Standards will increase

isomorphism and will therefore stimulate strategic response behaviour which is

not the same as improving real performance.

The performance paradox. Meyer and OShaughnessy (1993) have also

challenged the belief that unintended side-effects of measuring performance on

performance improvement are rare. They have coined the concept perfor-

mance paradox and have demonstrated the validity of this concept on the basis

of research in the private sector. I do not know of empirical evidence in the

public sector. Basically, the performance paradox is the simultaneous

proliferation and non-correlation of performance measures. An organization

which aims at safeguarding its performance through the application of

performance measurement instruments is not necessarily an effective

organization:

Even though organizational performance measures tend to be weakly correlated with one

another, performance measures have increased in number and sophistication over time and

staffs charged with monitoring performance have burgeoned correspondingly (p. 252).

This has contributed to the growth of second order performance assessment

by external auditors and financial analysts (p. 252).

Those who assume that (formal) standards and benchmarks reviewed by

auditors will lead to performance improvement, neglect this evidence. For the

coming years it is probably better to focus somewhat less on standardization of

measures and more on creative thinking and discourses between auditors and

auditees on topics like the performance paradox and other unintended

consequences. This is a second major challenge for PA and NPM.

The third challenge

Moukheibir and Barzelay (1995, p. 3) also stress that articulating and evaluating

ones own implicit underlying behavioural assumptions should have high

priority. In particular this is true with regard to the feedback theory underlying

audit work. Replacing this theory by a theory of organizational learning, like

the one developed by Argyris (1982), is a third challenge. It will probably

enhance the likelihood that audit work will contribute substantially to the

improvement of performance. Recent evidence from a comparative study on

organizational learning and evaluation research corroborates this assumption

(Leeuw et al., 1994). This study shows that programme evaluation contributes

to single and double loop learning. Argyris (1982) defines organizational

learning as a process of detecting and correcting error. It is a process in which

an organization continually attempts to become competent in taking action,

while at the same time reflecting on the action it takes to learn from its present

and past efforts. This conceptual approach differentiates learning as occurring

in either a single or double loop mode. Problem solving that enables an

organization to carry out better its present policies and achieve its current

objectives is defined as single-loop learning. A more comprehensive and

systemic learning process occurs when double-loop learning occurs. In this case

the assumptions underlying the policies and goals of a programme are

questioned, leading to the possibility of securing new and innovative

permanent solutions to problems[5].

Auditors should focus more on this (or on a similar) type of learning theory

as opposed to the naive and mechanistic feedback theory which was described

above[6].

Some final remarksOne of the crucial questions is whether or not there is an inherent conflict

between improved accountability and improved performance? My answer is

negative: there is no such thing as an inherent conflict because both

professional groups pursue the same goals. However, it cannot be denied that

there are conflicts between auditing performance on the one hand and

improving performance on the other hand. These conflicts deal with

imperfections or flaws in auditing: i.e. in its underlying (implicit) theories whichsometimes are naive, in the methodology applied, which sometimes causes

tunnel vision and other unintended side-effects and in the communication

processes between auditors and auditees. However, they also reflect flaws in the

performance measurement systems adopted under NPM. Apart from that,

isomorphism and unintended side-effects discovered by auditors are usually not

embraced by NPM officials or politicians because they are debunking and

critical.

A more open dialogue, focused less on procedures, rules and standards, may

be one of the solutions to minimize conflicts. Auditors and auditees need to

review (and discuss) their measurement schemes and measures. As suggested

earlier: commitment to the idea that auditors and NPM officials have their own

implicit pet theories that can be right but also wrong, will also reduce the

likelihood of unproductive conflicts between both groups of professionals.

Notes

1. That governmental agencies together with other corporate actors implement policies is of

course not new. What is new is that the formulation and development of goals of public

policy are carried out jointly with groups and organizations that themselves can

sometimes be seen as causing the problems government might try to solve.

2. For example, the Standards of the American and Canadian Evaluation Societies.

3. Smith calls this measure fixation (1995, p. 280).

4. This resembles what James Q. Wilson (1989, p. 161) calls a kind of Greshams Law: Work

that produces measurable outcomes tends to drive out work that produces unmeasurable

outcomes.

5. Based on Leeuw and Sonnichsen, Introduction: evaluations and organizational learning,

in Leeuw et al. (1994).

6. Policy and administrative scientists affiliated with Erasmus University Rotterdam

currently are engaged in a research project in which the impact of reports of the

Netherlands Court of Audit is described and explained by using a similar organizational

learning theory. During the First Conference of the European Evaluation Society (Den

Haag, 1-2 December 1994) the project director Frans B. van der Meer gave a paper on this

project.

References and further reading

Argyris, C. (1982), Reasoning, Learning and Action, Jossey-Bass, San Francisco, CA.

BAA (1991), Budget and Accounting Act (Fifth Amendment), Netherlands Central Government,

Government Printing Office, The Hague.

Bartlett, W. and Le Grand, J. (1993), Quasi-markets and Social Policy, Macmillan, Houndmills.

Chen, H. (1990), Theory-driven Evaluations, Sage, London.

Dekker, P.J. and Leeuw, F.L. (1989), Program evaluation and effectiveness auditing definitions,

models and practice, Impact Assessment Bulletin, Vol. 7 No. 1, 1989, pp. 113-35.

Frey, B. and Serna, A. (1990), Eine politisch-konomische Betrachtung des Rechnungshofs,

Finanzarchiv, Vol. 48, pp. 244-70.

Leeuw, F.L. (1992a), Government-wide audits in The Netherlands: evaluating central government

subsidies, in Mayne, J. et al. (Eds), Advancing Public Policy Evaluation: Learning from

International Experiences, North-Holland/Elsevier Science, Amsterdam, 1992, pp. 131-41.

Leeuw, F.L. (1992b), Policy theories, knowledge utilization, and evaluation, Knowledge and

Policy, Vol. 4 No. 3, pp. 73-92.Leeuw, F.L., Rist, R.C. and Sonnichsen, R.C. (Eds) (1994), Can Governments Learn?, Transaction

Books, Rutgers, New Brunswick, NJ.

Mason, R. and Mitroff, I.I. (1981), Challenging Strategic Planning Assumptions, Wiley, New York,

NY.

Meyer, K. and OShaughnessy, K. (1993), Organizational design and the performance paradox,

in Swedberg, R. (Ed.), Explorations in Economic Sociology, Russel Sage Foundation, New York,

NY, pp. 249-79.

Moukheibir, C. and Barzelay, M. (1995), Performance auditing: concept and controversies, paper,

PUMA/OECD symposium, Paris, June.

Netherlands Court of Audit (1991), Auditing at the Netherlands Court of Audit: Case Studies, The

Hague.

PUMA (1995), Background paper to the OECD Conference on Auditing, PUMA/OECD

symposium, Paris, June.

Shand, D. (1994), Performance auditing and performance improvement in government, paper

presented at the First Congress of the European Evaluation Society, The Hague, 1-2 December.

Sieber, S. (1981), Fatal Remedies: The Ironies of Social Intervention, Wiley, New York, NY.

Smith, P. (1995), On the unintended consequences of publishing performance data in the public

sector, International Journal of Public Administration, Vol. 18 No. 1, pp. 377-10.

Wilson, J.Q. (1989), Bureaucracy, Free Press, New York, NY.AAAJ

9,2

92

Accounting, Auditing &

Accountability Journal,

Vol. 9 No. 2, 1996, pp. 92-102.

MCB University Press, 0951-3574

PA, NPM and

performance

improvement

93

AAAJ

9,2

94

PA, NPM and

performance

improvement

AAAJ

9,2

96

PA, NPM and

performance

improvement

97

AAAJ

9,2

98

PA, NPM and

performance

improvement

99

AAAJ

9,2

100

PA, NPM and

performance

improvement

101

AAAJ

9,2

102