key performance indicator system for ipa institutions in...

NAO / NAO SUPPORT OFFICE

Key Performance Indicator System for IPA Institutions in Albania

DRAFT

March 2016

Republic of Albania Key Performance Indicators System for IPA Institutions in Albania NAO / NAO Support Office

Instrument for Pre-Accession Assistance IPA II Version 0.1 Date: March 2016 of 16

TABLE OF CONTENTS 1 Background and Initial Key Performance Indicator Workshop ....................................................... 3

1.1 Options for introducing KPIs in Albania .................................................................................. 4

1.2 Value Proposition of IPA Institutions in Albania ..................................................................... 5

1.3 Critical Success Factors of IPA Institutions in Albania ............................................................. 6

1.4 Approach to formulating KPIs for IPA institutions in Albania ................................................. 8

1.5 Draft Key Performance Indicators for IPA Institutions in Albania .......................................... 9

2 Steps to Make KPI System Work ................................................................................................... 10

2.1 Installation of iPerseus system ............................................................................................. 10

2.2 Agreeing and committing to the NAO SO reporting plan ..................................................... 10

2.3 Introduce check-lists for quality of reports .......................................................................... 10

2.4 Introduce check-lists for quality of tender documents ........................................................ 11

2.5 Agreeing and committing the CFCU procurement plan........................................................ 12

2.6 Cash flow forecasting ............................................................................................................ 12

2.7 Audit Authority involvement in KPI system .......................................................................... 13

3 System of Key Performance Indicators IPA Institutions in Albania .............................................. 13

3.1 Recording and reporting the KPIs ......................................................................................... 13

3.2 Reviewing the System of KPIs ............................................................................................... 14

4 Principles for Future Updates of the KPIs ..................................................................................... 15

4.1 Using ratios vs absolute figures for indicators ...................................................................... 15

4.2 Positive formulation of indicators......................................................................................... 15

4.3 Target vector of indicators .................................................................................................... 16



Key Performance Indicator System for IPA Institutions in Albania

1 BACKGROUND AND INITIAL KEY PERFORMANCE INDICATOR WORKSHOP An audit verification mission was carried out by DG NEAR from the 9 – 12 June 2015. During this

Mission a finding was put forward on establishment of Key Performance Indicators (KPIs) as follows:

Operational indicators (process, output, immediate outcome as intended in action programmes), strategic and

accession-related indicators are primarily the responsibility of the NIPAC. Nevertheless, a good mix of key budget

execution performance indicators (KPIs) shall be developed and used by the NAO (e.g. level of commitments and

payments; performance based on actual contracting vs. forecast; left-to-contract / left-to-pay; delays in

milestones; level of decommitments etc.) Using these KPIs is important in the context of standard NAO / NAO SO

monitoring and day-to-day performance follow-up

In January 2016 the TA project deployed a Non-Key Expert (NKE) to provide support in respect of KPIs.

This NKE carried out an initial 10-day mission in January 2016 and this mission was intended to conduct

the initial discussions with the NAO / NAO SO; CFCU; NIPAC and IPA Units and to collect information

for the initial design of suitable KPIs. In March 2016, the NKE returned for organisation and facilitation

of the workshop where the participants had to elaborate the KPIs for IPA institutions in Albania.

The present document has been prepared on the basis of the KPI Workshop participants’ contribution

that was held in Tirana, Albania on 7 March 2016. The participants of the workshop included:

No Name Institution

1 Ms. Jola Himci NAO Support Office, Ministry of Finance

2 Ms. Migena Kalthi NAO Support Office, Ministry of Finance

3 Ms. Lejda Frasheri NAO Support Office, Ministry of Finance

4 Ms. Alketa Vejseli NAO Support Office, Ministry of Finance

5 Ms. Veronika Korkaj National Fund, Ministry of Finance

6 Ms. Elona Mysliu National Fund, Ministry of Finance

10 Ms. Enkelejda Kokthi CFCU, Ministry of Finance

11 Mr. Neritan Totozani CFCU, Ministry of Finance

12 Ms. Lindita Shijeku CFCU, Ministry of Finance

13 Ms. Brikela Ajce CFCU, Ministry of Finance

14 Ms. Gentian Ramku CFCU, Ministry of Finance

15 Ms. Esi Fezolli CFCU, Ministry of Finance

16 Ms. Albana Gjinopulli Audit Authority, Ministry of Finance

17 Mr. Michael Leckey ICE TA key expert

18 Mr. Marko Rillo ICE TA non-key expert

At the workshop, the participants discussed and concluded that in addition to the requirements of

auditors, there are the following reasons to introduce KPIs for IPA institutions in Albania:

Observe current progress of the ongoing EU financed IPA projects in Albania;

Enable taking corrective actions, if necessary;

Understand and see better value for money in EU financed IPA projects in Albania;

See better linkages between inputs, outputs and outcomes of IPA projects in Albania

Make sure that the resources would be used more efficiently.



1.1 Options for introducing KPIs in Albania During the next phase of brainstorming, the participants analysed different options on how to

introduce KPIs in Albania and listed positive and negative aspects of those four scenarios.

Scenario Positive aspects Negative aspects

Plan A: Rely on the indicators identified in

high-level strategy documents (IPA country

strategy for Albania, high-level sector strategies)

There would be strong linkages between high-level strategy documents and actual indicators

The documents tend to be too general and the indicators tend to be missing.

Plan B: Copy the existing KPI system for IPA

projects from Macedonia

Simple approach to introduce KPIs in Albania

Well thought through and Comprehensive example from Macedonia

Too many indicators

Too detailed approach

Some of it is irrelevant in Albanian context

Plan C: Use existing system of EC

measurement that focuses on 1)

commitment and disbursement figures and

2) results of the evaluation of past IPA

programmes

Simple approach

Effective and proven set of indicators

Given that indicators are general enough, provides lots of flexibility

Too general

Requires that people working in IPA institutions would present ability to synthesize information and connect it to high-level indicators, which is unrealistic to expect

Indicators are mostly past oriented

Plan D: Develop ground-up the customised system

of KPIs for Albanian IPA institutions

Provides empowerment for people working in IPA institutions in Albania to come up with their own system

Provides flexibility to revise the system whenever necessary

Can be coherent if decided

Most relevant to Albanian context

Risky approach, because it requires good skills of strategy making and KPI definition

It cannot involve necessarily all stakeholders and therefore not fully comprehensive

May be too flexible if changed often

Thereafter the participants started working on defining the core strategy components of IPA

institutions working in Albania responding to the following questions:

OUR TAGLINE. 1-sentence summary of our unit

OUR MAIN CLIENTS. Who do we serve? What problems do we solve for them?

OUR VALUE PROPOSITION. What do we provide for them?

OUR CAPABILITY. What do our people need to know and do well to provide this value

proposition?

OUR SUSTAINABILITY. How do we ensure that we continue to have revenues and supporters

to do what we do?

The teams came up with the results.



1.2 Value Proposition of IPA Institutions in Albania Table 1 - Strategic Value Propositions of IPA Institutions in Albania

National Fund NAO Support Office

CFCU SPOs in line ministries

Tagline Effective and efficient financial management of EU funds as a central treasury entity with budgetary competence

Give assurance through internal control and monitoring, carry out coordination of EU projects

Tendering, contracting, payments and reporting of EU projects in compliance with PRAG and national legislation

Programming, implementation and monitoring of projects in their sector

Clients Implementing agencies and final beneficiaries

Minister of Finance, IPA bodies

Beneficiaries of EU projects in Albania SPOs in Line Ministries

Their own ministries for managing IPA projects CFCU and NIPAC

Value proposition We make funds available

Supporting NAO with 1) information and documentation and 2) administrative and on-the-spot verification

Providing the service of tendering, contracting and finance management to the beneficiaries

Provide ministry assurance for achieving project objectives. Provide CFCU and NIPAC for programming, technical expertise of project management and sustainability

Capability Competencies in financial management, financial control and accounting Values: integrity and ethics Knowledge of IPA funds

Good knowledge of the internal control systems applicable to IPA funds

Good knowledge of PRAG rules and procedures Capability of efficient management of EU projects

Knowledge of IPA rules Technical expertise relevant to their sector area

Sustainability preconditions

Good performance of staffing, disbursing funds properly leads to budget for NF

Ensuring information accuracy, timeliness and completeness

Putting the beneficiaries’ needs first

Leadership, expertise, accountability, results and network

In the second phase, the teams focused on the main elements of the Balanced Scorecard based

framework and added to this framework the Critical Success Factors of IPA bodies in Albania.



1.3 Critical Success Factors of IPA Institutions in Albania Table 2 - Five Dimensions of Balanced Scorecard for IPA Institutions in Albania

Purpose & Putting our community first

What are the things that IPA institutions do? What do IPA institutions provide Albanian

government, for its taxpayers? What is the wider impact that IPA institutions have?

Stakeholder first

What do IPA institutions do for our direct

stakeholders?

Funding the future

How do IPA institutions ensure their financing

for the future?

People matter

How do we make sure that people working for

IPA institutions are happy, productive and

innovative

Performance counts

How do we make sure that IPA institutions’

operations are efficient and timely

Respective summaries of the work for individual institutions are below

Table 3 - Balanced Scorecard Critical Success Factors (CSF) for the NAO Support Office

Purpose:

Give assurance to NAO, to NIPAC and to the Albanian Government on the functioning of IPA

systems in Albania through internal control and monitoring

Stakeholders:

Supporting the NAO, NIPAC and government by providing the appropriate information

Providing information about the IPA management to the European Commission

People:

Trained, skilled and competent people

Retention policy

Motivated people with proper pay and

promotion opportunities

Processes:

Timely and proper handling of information

regarding the IPA management in Albania

Well-functioning collaboration with other

IPA institutions in Albania

Clearly defined systems of reporting



Table 4 - Balanced Scorecard Critical Success Factors (CSF) for the National Fund

Purpose:

Proper channelling of EU funds in Albania, respecting the principles of Sound Financial

Management

Stakeholders:

Satisfaction of the NAO for sound financial

management

Satisfaction of the Implementing Agencies

for disbursement in time

Finances:

Proper bank account systems in place with

the accounts at the Bank of Albania and at

second level banks

Accounting system in place (Alpha)

Proper systems in place for co-financing

People:

Professional staff

Segregated duties

Processes:

Timely payments to Implementing

Agencies

Due management of cash flow

Requesting funds from the European

Commission in time

Table 5 - Balanced Scorecard Critical Success Factors (CSF) for the CFCU

Purpose:

Tendering, contracting, payments and reporting of EU projects in compliance with PRAG and

national legislation

Stakeholders:

Line ministries and beneficiaries for service

NIPAC, NAO and EU Delegation for

reporting

Finances:

Timely and appropriate cash flow

forecasting

Timely and appropriate requesting of funds

from the NF

People:

Experienced specialists, who are

motivated, know the rules and procedures,

have collaboration skills

Some specialist knowledge (e.g.

engineering, legal background)

Processes:

Timely procurement processes, leading to

high level of contracting

Timely reporting and control processes,

leading to assurance of the capacity and

performance of the stakeholders

Efficient and proper financial management

processes and due accounting, leading to

appropriate information

Quality assurance and checks.



Table 6 - Balanced Scorecard Critical Success Factors (CSF) for the SPOs at the Line Ministries

Purpose:

Provide on-time technical expertise on the Contract Notices and Terms of Reference during

the procurement phase

Ensure proper implementation of contracts

Provide information to NIPAC on the results of the projects

Stakeholders:

Continuously collaborate with the CFCU to

finalise tender documents in time and meet

procurement deadlines

Ensure the line ministry that all project

outputs and deliverables conform the

objectives of the ministry

Finances:

Prevention of delays in approval of reports

and deliveries to make sure that payments

will be processed in due time

People:

Continuous training of SPO and IPA unit

staff

Staff retention policy

Processes:

Ensure that tasks are delegated to

experienced staff to meet the deadlines

1.4 Approach to formulating KPIs for IPA institutions in Albania At the workshop the participants relied on the metaphor of a “Stool” with three

legs. This metaphor was put in place in order to describe that the Key

Performance Indicator system needs to be simple, prioritised and appropriate

for the context of the current strategy of Albanian IPA institutions. As such

three “legs” of the stool symbolize the need to have a few (ideally three)

chosen Critical Success Factors and associated KPIs in place.

Theory and best practices of due performance measurement suggest that a proper KPI system needs

to rely ideally on three KEY performance indicators that are observed by the management. We

therefore tried to keep the discussions also focused and generalise the indicators.

As of March 2016 the three main types of chosen KPIs would be:

- Quality of documentation - tender documents, financial documents and reports;

- Timeliness of work – meeting set deadlines in procurement plan, payments in time;

- Retention and professionalism of employees working for IPA institutions.

As such, those three “stool legs” or groups of KPIs reflect current stage of development at IPA

institutions in Albania. Chosen indicators are primarily process-focused. Given the situation where

activities of IPA institutions in Albania are at their inception this is a logical choice.

However, once the processes will start running more smoothly, it is worthwhile to consider updating

the chosen indicators to ensure that these would match the relevant strategic priorities of the time.



1.5 Draft Key Performance Indicators for IPA Institutions in Albania Finally, the participants of the workshop chose and formulated the KPIs for IPA Institutions in Albania,

that were linked to the most important Critical Success Factors that they prioritised.

Table 7 - Chosen KPIs for IPA Institutions in Albania

Indicator Description Frequency Responsibility

Global indicators

Commitments rate per IPA programme

% of the budget for a given IPA programme contracted

Monthly CFCU

Disbursements rate per IPA programme

% of the budget for a given IPA programme duly paid out to contractors

Monthly NF

NAOSO indicators

Timeliness and quality of reports to NAO

% of reports submitted in time that fulfil the requirements of reporting quality control check-list

Quarterly NAOSO head

National Fund indicators

% of funds disbursed

% of the committed funds disbursed to the Implementing Agencies on time.

Monthly NF head

Transactions are legal and regular

Based on measurement of Albanian Audit Authority reports, which checks financial statements and verifies whether the principles of Sound Financial Management are followed.

Annual NF head

CFCU indicators

Accurate procurement & contracting

Timely contracting rate as percentage of fulfilment of procurement plan

Monthly CFCU director

Percentage of timely and good quality documents submitted to the EC Delegation

This is measured by percentage of tender documents that arrive timely according to the procurement plan agreed with the CFCU, which fulfil the quality criteria specified in the checklists of the CFCU quality assurance sector and EC Delegation

Monthly CFCU director

% of filled staff positions

% of staff positions filled with competent staff Monthly Later: Quarterly

CFCU director

SPO / Line Ministry indicators

Percentage of timely and good quality documents submitted to the CFCU

This is measured by percentage of tender documents that arrive timely according to the procurement plan agreed with the CFCU, which fulfil the quality criteria specified in the checklists of the CFCU quality assurance sector and EC Delegation

Monthly SPO of the respective line ministry



2 STEPS TO MAKE KPI SYSTEM WORK Following the initial workshop, there are still a number of important steps that need to be taken to

make sure that the system of the KPIs would start functioning properly among the IPA institutions in

Albania.

2.1 Installation of iPerseus system The overall indicators “Commitment rate of EU funds in Albania” and “Disbursement rate of EU funds

in Albania” may be tracked using the regular Excel sheets and other accounting software. However, it

is recommended to discuss with the EC about the earliest convenience of installation of iPerseus

system, which would provide an accurate tool for proper tracking and follow-up of those indicators.

2.2 Agreeing and committing to the NAO SO reporting plan The NAO Support Office must prepare reporting plan for their KPI. This plan must include a timeline

of all the reports that they receive from the IPA institutions in Albania:

1. Create a draft annual plan of reports that are required according to the present system

2. Agree the deadlines with all IPA institutions that are in direct contact with NAO Support Office

– the NIPAC, the National Fund and the CFCU.

3. Follow up the detailed reporting approach with the institutions, which use adequate check-

lists that are mutually agreed and actively used.

2.3 Introduce check-lists for quality of reports The NAO Support Office needs to establish a simple checklist for its KPI “Timeliness and quality of

reports to NAO”, which will be measured as a percentage of reports submitted in time that fulfil the

requirements of reporting quality control check-list of the NAO SO. There are four main criteria that

could be used as the basis for assessing the incoming reports, as set out in the following table.

Criterion Description

Time Timeliness of reporting. Whether the particular report in question has been submitted in time to the NAO Support Office by the due deadline that was agreed in the original reporting plan.

Format Fulfilment of the reporting template. Whether the report has been prepared according to the reporting template of the NAO SO manual (e.g. monthly technical report template, procurement plan template, irregularity report template etc.).

Substance Substance of the contents of the report. Whether the substance of the report is optimal to provide full and adequate overview of the situation that the report is addressing. The substance of the report must be optimal for the purpose of the report. If the substance is too brief it may not provide enough information for decision-making. If the contents of the report are overly detailed, then it may also be too difficult to generalise and synthesise conclusions for decision making. While above indicators are easily quantifiable, measurement of substance is subjective and it is the matter of the NAO SO official to negotiate and reason the comments.

Learning Whether the authors of the regular reports have managed to learn from the feedback provided to the previous reports and have been able to improve their quality of writing and avoid repeated errors pointed out from the previous reports.



It is possible to use several different criteria in several different ways to come up with the composite

indicator to measure the quality of reports in a particular month.

Approach Criteria weighing Principle

Time Format Substance Learning

Absolute Yes/no Yes/no Yes/no Yes/no Simplest approach. If any of the four criteria have been assessed with “No” the report is deemed to not meet the quality criteria.

Absolute, scored

Yes / No ↓

25%

Yes / No ↓

25%

Yes / No ↓

25%

Yes / No ↓

25%

Alternatively, the absolute approach may also be scored depending on the Yes/no criteria met between 0-100%.

Weighted average

Yes / No ↓

10%

Yes / No ↓

10%

Content assess-ment: 50%

Yes / No ↓

20%

This approach is the most complex, because it requires first to set up adequate and reasoned weighing between the criteria and thereafter measure it duly.

Learning-focused

Yes/no Yes/no Content feedback

Yes/no If three yes/no criteria have been met, the report is deemed to meet the content criteria.

At the outset of the KPI system it is recommended to start using the last – learning-focused

measurement criteria. On the one hand it is reasonably simple approach and on the other hand it also

welcomes the authors of the reports to be actively engaged in working on improving the contents of

the reports for the future.

According to this approach - if three administrative criteria (timeliness, adhering to format and

capacity to demonstrate learning) have been achieved, the report is deemed to be “of good quality”.

However, the NAO SO employees will have to engage in discussions with the report authors regarding

the content of the report to make sure that the substantive elements of the report would improve for

the next period. If the content feedback brings about recurring errors during the following reporting

period, “Learning” may need to be assessed as “No”.

2.4 Introduce check-lists for quality of tender documents In a similar manner, the quality of tender documents prepared by the CFCU would need to be

assessed. At the outset it is suggested that the same criteria will be used for learning-focused

assessment of the tendering documents prepared by the SPOs and by various CFCU departments

during tendering process. Hence, the criteria for analysing the quality of the documents submitted

according to the procurement plan are the same as above:

- Time – whether the SPO submitted the tender documents in time?

- Format – whether the SPO has used the standard document formats?

- Substance – whether the SPO has written the tender documents in a substantially appropriate

manner?

- Learning – whether the SPO has managed to learn from the feedback provided to the previous

reports and have been able to improve their quality of writing?



2.5 Agreeing and committing the CFCU procurement plan The CFCU has formulated the procurement plan. Two of the KPIs identified are linked to measuring

performance according to the procurement plan.

Presently the CFCU updates the procurement plan on a monthly basis, revising almost all the due dates

of procurement documents referring to their manual of operations that has proposed the

procurement plan to be up to date.

While it is indeed important to make sure that the plans are up to date, they should not be treated as

moving targets, where the “planned documents” may be revised regularly by postponing the

implementation deadlines because no results were achieved during the previous month. Instead, the

planned submission dates for any documents, reports must be agreed upfront during mutual

discussions between the parties of the IPA institutions. After this agreement the procurement plan

dates and deadlines must be fixed and duly adhered to.

For example, every time an annual IPA programme is signed, the CFCU must dedicate time for

negotiations with the SPOs to clarify in due detail the situation at hand and set out a proper

implementation plan for each of the planned projects. As a result of those clarification meetings and

negotiations the CFCU together with the SPOs must formulate the deadlines of each of the project

stages that would be mutually agreed between them. At the later stage, the CFCU and the SPOs in the

line ministries must be in regular contact to ensure that the plan is still valid.

If and only when the project implementation is faced with some unforeseeable circumstances can and

should the procurement plan be revised. E.g. in case of preparation of works contracts it might turn

out during the preparatory land surveying that the original site for building turns out to be unsuitable

and the SPO / line ministry representatives would have to start negotiations to identify an alternative

plot of land for construction, which undeniably leads to delays in project implementation.

As presently no due negotiations have been carried out between the CFCU and the SPOs to formulate

mutually agreed, specified and realistic procurement plan to which all parties would be mutually

committed, the following has to be put in place as a temporary measure.

1. The CFCU and the SPOs must meet during the following 2 months and agree upon realistic and

specific procurement plan for implementation of all the projects.

2. Once the procurement plan has been agreed to, the deadlines specified for all the tendering

and contracting stages must be fully adhered to.

3. Resulting deadlines will formulate the basis for the procurement plan, which will function as

basis of two Performance Indicators (timeliness and quality tender documents prepared by

the SPOs and timeliness and quality of tender documents prepared by the CFCU). This KPI will

be measured on the percentage basis. The percentage of tender documents submitted in time

according to the agreed procurement plan, which fulfil the tender document quality criteria

set out by the EC Delegation.

4. It would be a good practice that the CFCU staff and the staff at the SPOs must proceed with

regular meetings to discuss the progress of preparation of tender documents. If necessary,

the SPOs and the CFCU staff would need to engage in teamwork and prepare the tender

documents together to speed up the process and increase mutual learning.

2.6 Cash flow forecasting As such, it is of utmost importance that the procurement plan is realistic, because it has direct impact

on due management of IPA finances in Albania.



Therefore, once the procurement plan has been properly established in such a way that the dates are

realistic and achievable, the CFCU needs to discuss and agree with the National Fund the appropriate

cash flow forecast that would satisfy the due implementation of IPA projects in Albania. Like with

procurement plan, the cash flow forecast has to be based on proper agreements that are based on

realistic background information. It must be made sure that the cash flow forecast thereafter is

continuously kept up to date to make sure that the CFCU and the National Fund can submit the

requests for funds at proper time.

It would be also a good practice to introduce regular meetings between the National Fund, the CFCU

financial department and the CFCU officials dealing with project implementation to discuss and update

the cash flow forecast regularly to make sure that it stays up to date.

2.7 Audit Authority involvement in KPI system To verify the adequacy of following the principles of Sound Financial Management, the Albanian Audit

Authority needs to check financial statements, verify the systems and performance of IPA programmes

in Albania. The Audit Authority has recently started its operations. Therefore, the NAO Support Office

needs to involve them into the process of managing the KPIs.

Furthermore, given that in 2017 / 18 the Audit Authority will start with performance audits of IPA

system in Albania, they need baseline information from performance indicators. Therefore, the proper

functioning of the IPA KPI system during that time would start providing valuable input for their

performance audits to provide a more comprehensive assessment of the functioning of IPA system in

Albania.

3 SYSTEM OF KEY PERFORMANCE INDICATORS IPA INSTITUTIONS IN ALBANIA

3.1 Recording and reporting the KPIs The overall responsibility for recording and reporting on the KPIs lies with the NAO Support Office who

has the overall responsibility for ensuring due management of IPA funds in Albania. However, the KPI

system is not the responsibility of the NAOSO alone. According to the KPI chart, there are institutions

who are responsible for providing inputs for measurement (the NF, the CFCU, the SPOs, the Audit

Authority, the NIPAC), but there are also institutions who have to make sure that as a result of their

work a positive change is going to be achieved in following those indicators.

For that reason, the NAO Support Office representatives will regularly interact with the other

institutions working in the IPA programming, implementation, monitoring, evaluation and auditing to

make sure that the KPI system is duly functioning and providing the management with information for

managerial decision making.

The NAO Support Office will handle the registry of overall KPIs in Albania, which is provisionally

formatted as an Excel table with indicator descriptions. At a later stage once the quantity of data

increases and the IPA organizations start using databases that allow for ODBC-based connections

between the data sources and the potential visual dashboards, it might be worth assessing whether

one of the visual KPI solutions might be worth the investment. There are dozens of companies that

provide the solutions in the field of “business intelligence” (e.g. smaller reporting-focused solutions

by Qlik, Spotfire, Tableau or more extensive analytics-focused solutions by Cognos, SAP, Business

Objects, Microsoft, Oracle, SAS or others).



3.2 Reviewing the System of KPIs It is suggested that the KPI system will have to be reviewed at least on an annual basis under the

leadership of the NAO Support Office. During review of the KPI system, the NAO Support Office will:

1. Review each of the KPIs that has been measured using the following quality control check-list:

a. Linked to our strategic goals? Is the KPI still linked to the important aspects for

managing IPA programmes in Albania. Is there still possibility to make active and

proper managerial decisions on the basis of this indicator?

b. Brings about positive change in people and formulated in a positive manner? Has

the KPI started creating overall positive change in managing IPA in Albania and has it

started supporting positive behaviours among people working at IPA institutions in

Albania?

c. Reliable? Does the KPI provide continuously reliable and valid information that can

be used for decision making?

d. Easy to understand? Do people working in IPA institutions in Albania understand what

the KPI means and what it is meant to achieve. Do they understand what do they need

to do in order to have an impact on the KPI?

e. Can people have impact on KPI? Have people working in the IPA institutions in

Albania had the chance to positively impact the KPI?

f. Cheap to measure? Is the workload for measuring for this particular KPI less than the

usefulness and importance of information that is received from reviewing this KPI on

a regular basis?

2. Review the whole KPI system for IPA systems in Albania using the following quality control

check-list:

a. Management decision making. Has the KPI system in its entirety given the

management team members of the IPA institutions (the NAO, NIPAC) tools and

information to make managerial decisions?

b. Focus on strategy and bottle-necks. Has the core strategy of IPA management

changed in the meantime? Have the main bottlenecks in the overall IPA system in

Albania been changed and do we therefore need to redefine some of the KPIs? E.g. in

March 2016 the main bottle-neck is the quality of tender documents and starting of

tendering – due and timely preparation of tender documents. However, in January

2017 the most crucial issue might be tender evaluations, in 2018 the KPIs might have

to be refocused on cash-flow management etc.

c. Positive change. Has the KPI system brought about positive change for IPA

management in Albania?

d. Easy to manage. Is the KPI system in Albania easy to handle and manage without too

much bureaucracy? Does the quality for decision making stemming from KPIs exceed

the cost of measurement?

e. Attention span. Is the KPI system in Albania simple enough so that people working in

IPA institutions have time to focus on the most important indicators. Do we need to

reduce the number of indicators in some core areas?

f. Relevance. Are all the KPIs still relevant from the point of view of overall picture of

managing IPA institutions in Albania?

As a result of this assessment, the NAO Support Office will have to renegotiate the updated Critical

Success Factors and Key Performance Indicators for Albanian institutions.



4 PRINCIPLES FOR FUTURE UPDATES OF THE KPIS

4.1 Using ratios vs absolute figures for indicators When updating the indicators there is always a core choice – whether to measure absolute figures

(e.g. number of tender documents passed the quality control) or ratios (e.g. percentage tender

documents that have been approved in good quality by the deadlines foreseen in the original

procurement plans). The choice between those two needs to be made in each particular instance

during the choice of the KPIs and it is worthwhile analysing which approach would be more useful.

At the outset it may seem to be more difficult to formulate the percentage figures of certain indicators.

However, the original KPI system has been designed primarily relying on percentage figures, because

percentage figures need more thinking about the baseline. While at the outset the definition of

baseline of the chosen KPIs might be more difficult because of no experience and lack of measurement

history, it is still recommended to spend some time thinking about the possible baseline.

The reason for suggesting so is that during discussions and analyses of potential baselines, the

participants of the process are forced to spend time together for due planning ahead. The result of

those planning discussions would also teach the participants of those discussions of the preparations

that they need to make and the potential future risks that they need to avoid.

4.2 Positive formulation of indicators Currently the EC Delegation in Albania uses an indicator “Rejection rate of tender documents” as a

percentage of documents received from the CFCU to assess the quality of work performed by the

CFCU and the SPOs in the line ministries.

As confirmed above, process quality and document quality indicators are important at this stage of

development for Albanian IPA institutions. Measuring quality of documents and processes aims to

induce learning among those people who are preparing those documents.

However, it is suggested to reformulate the quality indicators in a positive manner.

Research in the field of psychology of measurement has demonstrated that the way the indicators are

formulated has a strong impact on extrinsic motivation of people who are subject to measurement. If

the indicator is formulated in a negative manner – e.g. “number of documents submitted late”,

“rejection of documents” or “percentage of poor quality documents”, this indicator tends to generate

discussions that create negative feelings for everybody who are involved in the process. It generates

frustration among employees because results of their work have been labelled with “poor quality”.

This reduces motivation to learn and improve.

Instead the indicators must be formulated positively – e.g. “percentage of documents submitted in

time”, “percentage of good quality of documents”, “ratio of documents accepted with / without

comments” or alternatively “number of comments / suggestions / recommendations provided to draft

documents” it also has an impact. If the indicator has been formulated in a positive manner, team

discussions regarding the results of those indicators will also focus on positive. The conversations will

be carried out in the following manner: “The document was prepared well, but we also have some

comments …”. This increases much the openness towards listening to critical comments and will also

increase the motivation to learn and do better the next time.



4.3 Target vector of indicators The psychological impact of the vector of the target figure also has a considerable psychological

impact. If the vector of the indicators is targeted towards zero (e.g. zero mistakes, zero waste, zero

late submissions) then it creates a mental image of slowing down, which reduces the rate of

improvements. However, if the target vector is targeted towards maximum (e.g. 100% correct

documents, 100% quality, 100% in time), it creates mental image of speeding up, which in turn

increases the rate of improvements and creates additional extrinsic motivation among people to try

harder and achieve their maximum.

For that reason, the indicators in the Albanian KPI system are suggested to be formulated in a positive

manner, with increasing vector. We therefore also suggest the EC Delegation to review their quality

control indicators towards the CFCU.

key performance indicator system for ipa institutions in...

Documents