kpi monitoring

© 2005 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice

Monitoring KPI

Nick HechtHP Internal Audit – Corporate Finance

April 11, 2023 2

Monitoring MethodsThere are three monitoring methods used when monitoring KPI data

1. TREND: Look at trends over time. Look at the KPI graph and see if the trend is up or down over time. Any prolonged trend could be a sign that a process has changed or that significant assignable change is occurring in the system. A prolonged trend is seen as more than 3 consecutive + or _

2. CHANGE: Look at the percentage change period to period (month to month). If there is a significant change from one month to the next, then unnatural variation – or a significant change has occurred. Unnatural variation is seen as any data point greater than two standard deviations from the mean.

3. BENCHMARKING: Look at metrics of a system that is known to be stable and compare these metrics to another system.

April 11, 2023 3

1. Example of a trend− Consider that any pattern of data (upward movement, downward

movement, values above the mean, values below the mean) that continues for more than FOUR data points is likely due to assignable variation. This APL example constitutes a significant trend because more than four data points in a row have increased. The next step is to ask what is the explanation of this trend over time. Note: there may be valid explanations for trends and some trends may be a sign of good management, but the question to ask is why are the trends occurring, and is management aware of them?

APL Top 5

414

386366

341327318

388402 411 420

436

404

0

50

100

150

200

250

300

350

400

450

500

Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul

April 11, 2023 4

% change in transports

-600.0%

-400.0%

-200.0%

0.0%

200.0%

400.0%

600.0%

800.0%

1000.0%

1 2 3 4 5 6 7 8 9 10 11

2. Example of a change− It must first be assumed that the system is in control, or else any out of control data

will skew the guardrails. Any data point outside of two standard deviations is seen as assignable variation. The example below from D7 shows assignable variation in January for the % change in the number of transports. The next step would be to determine why January had a significantly higher change in the amount of transports.

− Note: Consider 2 standard deviations from the mean as the guardrails in KPI.

Outside the guardrail

April 11, 2023 5

3. Example of Benchmarking− D7 was used as the system to benchmark against, D7 is seen as being in

control. Determine the variation of a certain metric in D7 and apply the guardrails (2 standard deviations) of variation from D7 to the other systems.

− For the below example the percentage change in active users was benchmarked against D7. The guardrails are +/- 6% change. Using conditional formatting in excel, any value that is >6% or <-6% is highlighted orange. These values would be considered outside of D7 benchmark. Systems that are out of benchmark may not be using best practice processes or may be experiencing high change and hence are a higher risk to the company. Note: this analysis is in the excel file SPC Analysis.xcl

April 11, 2023 6

D7 Analysis

April 11, 2023 7

% change in all users

-3.0%

-2.0%

-1.0%

0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

Sep-04

Oct-04

Nov-04

Dec-04

Jan-05

Feb-05

Mar-05

Apr-05

May-05

Jun-05

Jul-05

D7 All users Using D7 as a benchmark I

determined that % change in all users should be between 3.6% and -2.4%. This is two standard deviations any change outside of these bounds would be seen as unnatural variation.

• Why have the total number of users increased for the past 4 months?

• Why is July’s % change out of control?

D7 All Users

1450

1500

1550

1600

1650

1700

1750

1 2 3 4 5 6 7 8 9 10 11 12

Series1

Note: The top graph can be obtained from KPI – it can be used for monitoring trends or change over time.


Note: I made the bottom graph from KPI data, it represents month to month change. This graph can be used to identify a significant change from one month to the next. This graph can also be used to identify an upward or downward trend – if more than 4 points are above or below 0% change in a row then there is assignable variation.


April 11, 2023 8

Active Users

% Change in Active Users

-8.0%

-6.0%

-4.0%

-2.0%

0.0%

2.0%

4.0%

6.0%

8.0%

Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul

• Using D7 as a benchmark I determined that 2 standard deviations of % change of active users is 6%. So 6% change can be seen as being “in control” anything greater would be out of control.

• January % change in active users is out of control. This is more than 6 months ago so may not be important for roll forward.

• There is a trend going upward.

# of Active Users

1150

1200

1250

1300

1350

1400

1450

1 2 3 4 5 6 7 8 9 10 11 12

April 11, 2023 9

Locked Users

% of Users Locked

0.0%0.2%0.4%0.6%0.8%1.0%1.2%1.4%1.6%1.8%


• % of locked users should be between 1.6% of all users and .5% of all users.

• Process is in control, however – why is the process oscillating?

% change in locked users

-80.0%

-60.0%

-40.0%

-20.0%

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun

April 11, 2023 10

Expired 1 Users

% of users expired

0.0%

5.0%

10.0%

15.0%

20.0%


• % of users expired should be between 17% and 9.6%.

• % change in expired users could also be used but is highly variable from system to system so, % of users expired is a better benchmark.

• Each system has their own method of expiring users so benchmarking on % change is not as effective. For example – AP Fusion expires users instead of locking them – which is still seen as an acceptable process but results in a greater variation of expired users period to period.

• January had an out of control % change in expired users.

# of expired 1 users

0

50

100

150

200

250

300


April 11, 2023 11

SAP All• SAP All – should not change dramatically – D7 does not move from 1 SAP ALL account. % of Active users that are SAP ALL should be no greater than

7% (2% + 2 standard deviations) of Active users.

• 1 SAP ALL user as in D7 is probably an unfair benchmark at this point – it is however best practice. For this benchmark I took the average percentage of active users that are SAP ALL for every system = 2%. The deviation was high due to system HPS as seen below.

% of active users that are SAP All

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

14.0%

16.0%


BW1

LH1

FI1

US1

HPS

PMG

MA2

GPG

APL

S20

A50

B50

D7C

R01

CPO

R00

PAP

IJ1

HPS

MA2

PMGPPC

April 11, 2023 12

Top 5 Basis• Top 5 transactions

should not change more than 30% (+ or -) from the mean.

• Why does April have a high % change? 0

5

10

15

20

25

30


D7 Top 5 % change

-40.0%

-30.0%

-20.0%

-10.0%

0.0%

10.0%

20.0%

30.0%

40.0%


Series1

April 11, 2023 13

VIP• VIP – should not be greater

than 27% increase. Decrease is seen as positive improvement. This seems high – D7 is probably not the best benchmark for this metric.

• Why is June out of control?

-30.0%

-20.0%

-10.0%

0.0%

10.0%

20.0%

30.0%

40.0%


D7 VIP

0

10

20

30

40

50

60

70

80


April 11, 2023 14

PORI• Purchase Orders, Goods

receipts, invoices. Should not go above 33% increase. This seems high, D7 is probably not the best benchmark for this metric.

• Why is June out of control?

-30.0%

-20.0%

-10.0%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%


D7 PORI

0

10

20

30

40

50

60

70


April 11, 2023 15

Transports• Transports – should not

increase more than 600% - seems very high. Something occurred in May that is skewing the data. This may be the consolidation of D7 and Colato.

• Why is May out of control?

D7 Transports

-1000

-500

0

500

1000

1500

1 2 3 4 5 6 7 8 9 10 11

% change in transports

-600.0%

-400.0%

-200.0%

0.0%

200.0%

400.0%

600.0%

800.0%

1000.0%


April 11, 2023 16

Emergency Transports• Emergency Transports –

this data is very skewed do to a spike in January. Using February through March data only the value should not change by more than 100% - as these months are stable.

• January is out of control – may not be important for roll forward.

-1500.0%

-1000.0%

-500.0%

0.0%

500.0%

1000.0%

1500.0%

2000.0%


D7 Emergency Transports

-40

-20

0

20

40

60

80

1 2 3 4 5 6 7 8 9 10 11

April 11, 2023 17

SE01• SE01 – no more than a

20% increase.• Notice gradual trend

downward for first half of fiscal year.

-30.0%

-25.0%

-20.0%

-15.0%

-10.0%

-5.0%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%


SE01

0

10

20

30

40

50

60

70


April 11, 2023 18

SE38• SE38 – no more

than 20% increase.

• Notice gradual trend upward.

• Why is there an upward trend?

• Why is June out of control for number of SE38?

-20.0%

-15.0%

-10.0%

-5.0%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%


SE38

0

10

20

30

40

50

60

70

80

90


April 11, 2023 19

SE80• SE80 – no more than

19% increase.

• Why is June out of control for % change in SE80?

-25.0%

-20.0%

-15.0%

-10.0%

-5.0%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%


SE80

0

20

40

60

80

100

120


April 11, 2023 20

Unplanned Downtime• Unplanned downtime should not be more than 4.5 hours. % change for unplanned

downtime is not measured because unplanned downtime should be consistently 0.

Unplanned Downtime

-3

-2

-1

0

1

2

3

4

5

6


April 11, 2023 21

Stability Analysis

April 11, 2023 22

Stability Analysis• Another way to benchmark systems is to determine which

systems are the most stable in given metrics.

− Taking the data from KPI, statistical analysis can be done using excel to determine the standard deviation of each metric by system.

− The standard deviations can then be sorted – the systems with the lower standard deviations in a given metric are the most stable in that metric.

• Following a risk based approach – the most unstable (or lowest ranked systems) are the systems that have the most change occurring and are therefore a higher risk and should be higher in auditing priority.

April 11, 2023 23

Stability Analysis

April 11, 2023 24

Stability Analysis

April 11, 2023 25

Stability Analysis

April 11, 2023 26

Regression Analysis

April 11, 2023 27

Regression AnalysisThere are several methods in doing regression analysis.

1. Compare a metric to time. This is good for metrics which are expected to change over time. ie – number of active users over time for a growing system or number of a certain transaction over time. Note: System data has a high r-squared value (high correlation) for this type of regression because a system metric usually follows a trend over time. System processes should be consistent and stable. System metrics do not vary based on market conditions like financial data. If system data does vary it should be explained by management – in fact a low r-squared value for this type of regression would mean that the system is undergoing a lot of change and is therefore a higher risk to the company.

2. Compare one metric of a system to a different metric for the same system. ie - % of users locked vs. % of users active. Note: There may be little advantage to compare one metric to another metric as the correlation is usually low – we also need 25 data points to get a more accurate reading on the correlation between metrics. Due to different system processes, correlation in one system for one metric to another metric may be high, but the same analysis may have low correlation in another system.

April 11, 2023 28

Regression AnalysisRegression analysis methods continued.

3. Compare one metric for all systems with another metric for all systems. ie - Used accounts for each system vs locked accounts for each system. Note: This type of regression has a low r-squared value (low correlation). This is due to the fact that every system uses different processes and is on a different stage in the management cycle. For example APL expires users instead of locking them – metrics from one system to another will not be as correlated. It is better to use benchmarking analysis to determine high risk when comparing system to system.

4. Compare one metric in one system with the same metric in another system. This would be useful to see if systems are changing at the same rate. Note: This type of analysis is more easily done by putting system data into percentages (so they are on the same scale) and then graphing the data for the metric and comparing the systems graphically.

April 11, 2023 29

Regression Analysis• If data is highly correlated (high r-squared value) then a best fit line

can be used and future data or trends can be determined by forecasting. Forecasting can be used to determine potential future risk.

• Different types of trend-lines can be used and depending on the trends of the data, a higher r-squared value will be obtained by using a different trend-line.

• Types of trend-lines are – linear, logarithmic, polynomial, power, exponential, and moving average. The trend-line with the highest r-squared value should be used.

April 11, 2023 30

Regression Analysis• Important points in Regression Analysis

− There should be at least 25 data points to get an accurate r squared value. If data is highly correlated then less than 25 points will still give a high r-squared value.

− Guardrails can be established by determining the equation for the trend-line and then adding (for the upper guardrail) or subtracting (for the lower guardrail) the standard error of the data to the y-intercept of the trend-line. Note: all of these variables are given when doing statistical analysis in excel.

− In some cases it may be good for a metric to be outside one of the guardrails. For instance if the system is being cleaned or consolidated then significant change would be acceptable – but it is always important that management knows why the system is trending a certain way and why a metric may be outside of trends.

− Guardrails can also be user defined based on the risk appetite of management for the metric. Example: management may not want any greater than 5% change in a metric – or management could feel a metric is not a high risk and guardrails could be set out further.

April 11, 2023 31

Regression Analysis

R01 Total Users over time

R2 = 0.967

0

500

1000

1500

2000

2500

3000

3500

0 5 10 15 20

Time (months)

Us

ers

Y

Predicted Y

Linear (Y)

•R01

April 11, 2023 32

Regression Analysis

APL # of active users

R2 = 0.888

2300

2350

2400

2450

2500

2550

2600

2650

2700

2750

0 5 10 15

Time (months)

# o

f ac

tive

use

rs

Y

Predicted Y

Poly. (Y)

•APL- Polynomial trend-line is the best fit for seasonal type data –

however, management should know why this data is seasonal.

April 11, 2023 33

Regression Analysis

D7 Expired Users over time

R2 = 0.1131

0

50

100

150

200

250

300

0 5 10 15

Time(months)

Exp

ired

Use

rs

Y

Predicted Y

Poly. (Y)

•D7- Polynomial trend-line is the best fit for seasonal type data –

however, management should know why this data is seasonal. – notice 2 outliers that make the data seasonal

April 11, 2023 34

Regression Analysis

% locked vs % active

R2 = 0.3177

-60.0%

-40.0%

-20.0%

0.0%

20.0%

40.0%

60.0%

-8.0% -6.0% -4.0% -2.0% 0.0% 2.0% 4.0% 6.0% 8.0%

% change active

% c

han

ge

lock

ed

• All systems r-squared = .31

April 11, 2023 35

Regression Analysis

Used compared to Top 5

R2 = 0.0196

0

2000

4000

6000

8000

10000

0 200 400 600

Top 5

Use

d Predicted Y

Linear (Y)

• All systems

April 11, 2023 36

Regression Analysis

Used vs VIP

R2 = 0.255

0

2000

4000

6000

8000

10000

0 200 400 600

# of VIP

Use

d Predicted Y

Linear (Y)

• All systems

April 11, 2023 37

Regression Analysis• All systems

# used vs # transports

R2 = 0.0014

0

200

400

600

800

0 5000 10000

Used

tra

ns

po

rts

Predicted Y

Linear (Y)

April 11, 2023 38

Regression Analysis

D7 All Users vs Used accounts

R2 = 0.1658

1540

1560

1580

1600

1620

1640

1660

1680

1700

1200 1250 1300 1350 1400

Used

All Predicted Y

Linear (Y)

•D7

April 11, 2023 39

Regression Analysis

% Top 5 vs % Active

R2 = 0.0829

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

-0.5 0 0.5

% change active

% c

ha

ng

e t

op

5

Predicted Y

Linear (Y)

•D7

April 11, 2023 40

Regression Analysis

Used vs Top 5

R2 = 0.0238

122012401260128013001320134013601380

0 10 20 30

# Top 5

# U

se

d

Predicted Y

Linear (Y)

•D7

April 11, 2023 41

Regression Analysis

System uptime vs # of transports

R2 = 0.001

99.2

99.3

99.4

99.5

99.6

99.7

99.8

99.9

100

100.1

0 500 1000 1500

# of transports

Sys

tem

up

tim

e

Predicted Y

Linear (Y)

•D7

April 11, 2023 42

Regression Analysis

Unplanned downtime vs transports

R2 = 0.0013

0

1

2

3

4

5

6

0 500 1000 1500

# of transports

Un

pla

nn

ed

Do

wn

tim

e

Predicted Y

Linear (Y)

•D7

April 11, 2023 43

Regression Analysis

Time vs Active Users D7

R2 = 0.0136

122012401260128013001320134013601380

0 5 10 15

Time(months)

Ac

tiv

e U

se

rs

Predicted Y

Linear (Y)

•D7

April 11, 2023 44

Regression Analysis

- Using logarithmic best fit gets a higher r-squared value for SE80

D7 SE80 over time

R2 = 0.4609

0

20

40

60

80

100

120

0 5 10 15

Time (months)

# o

f S

E80

Predicted Y

Poly. (Y)

•D7

April 11, 2023 45

Regression Analysis

- Polynomial trend-line is the best fit for seasonal type data – but why is this data seasonal?

D7 % locked over time

R2 = 0.2735

0.0%

0.2%

0.4%

0.6%

0.8%

1.0%

1.2%

1.4%

1.6%

0 5 10 15

Time (months)

% o

f u

se

rs lo

ck

ed

Y

Poly. (Y)

•D7

April 11, 2023 46

Other Monitoring ideas• If a metric increases over time then eventually it will go outside the

guardrails. Management will simply say that for example more users have been added because of increased support need due to system growth – the guardrails must then be redefined – regression analysis takes this type of trend into account and the guardrails follow the same slope or trend as the actual data accordingly over time.

• Guardrails on regression analysis will only show risk for a “significant change” – but what about risk from slow change over time? - This is why we must also look at trends in the data. A “significant change” may not have taken place, but slowly over time a risk could be building up – we can see this type of risk by looking at trends.

April 11, 2023 47

Other Monitoring ideas• Another way to catch this type of slow

rising risk would be to consider the % change in a metric over a long period of time. For example – you could take the average value of a metric last year (be sure there are no outliers to skew the average) and compare the % difference to the current metric reading. If there is a high % change then there may be risk in the system – an explanation should be provided by the system management.

• This could be done graphically by doing what is called a residual plot – the average is shown as a line across the graph at y=0 and the difference between the average and the data point is graphed. Guardrails can also be put on this graph. This is a good way to see which data points are outliers.

• A similar type graph could be done for all systems. The average value of a metric for all systems could be y=0 and then the difference between the average and each system would be graphed, easily showing which systems were the furthest from the average.

D7 Active Users Residual Plot

-100

-50

0

50

100

0 5 10 15

Time (months)

Dif

fere

nc

e f

rom

a

ve

rag

e

April 11, 2023 48

Business Benefits of KPI Monitoring• Traditional audit testing is costly and time consuming – both for the auditor

and auditee. Save time and money by quickly seeing where a system is at risk. Risk can be seen by looking at trends, changes, and benchmarking to other systems – all of which can be monitored.

• Monitoring helps stop risk in the early stages – since monitoring can be continual as opposed to a once a year audit. This can help save considerable time and money spent on remediation of risks.

• Seeing risk early helps to stop it early and hence potential loss is never realized that may have occurred had the risk continued.

• Quickly determine which systems are the higher risk systems and should be given audit priority – as shown by stability analysis and benchmarking.

• Benchmarking systems can help (if management is proactive) to determine and establish best practices – which make systems more efficient and manageable. Managers from different systems should work together to determine what the best practice process is. Example: if you see that D7 has a low percentage of expired users as compared to your system then you should inquire as to what the D7 process is.

kpi monitoring

Documents