interpret and use performance measures for improvement certification... · ©stacey barr step 7:...

©Stacey Barr www.pumpcertification.com

Step 7: Interpret and Use Performance Measures for Improvement

Transcript for December 1, 2011.

PuMP Performance Measurement Practitioner Certification Program Step 7: Interpret and Use Performance Measures for Improvement

Stacey Barr Pty Ltd the Performance Measure Specialist

www.staceybarr.com [email protected]

of 22


Copyright, etc…

No part of this Transcript may be reproduced, transmitted, re-posted or duplicated in any form or by any means without permission of the author and publisher, Stacey Barr.

Disclaimer: The information provided is for the purpose of expanding the awareness of the reader and the author accepts no responsibility for the subsequent use or misuse of this information.

© Stacey Barr, 1999-2011




of 22


Step 7: Interpret and Use Performance Measures for Improvement

Start of transcript.

Introduction

Well G‟day, Stacey here and we are up to Step 7 of the PuMP® Certification Program for 2011. Step 7 is interpreting and using measures for improvement, and it's all about your new measures finally doing the very thing they were created for, and that very thing that they were created for is improving performance fundamentally. You'll be learning how to facilitate the using measures workshop for your measures teams, helping them implement that Step 7 of the PuMP® Blueprint.

And, we have three specific objectives in Step 7 of the PuMP® Blueprint. The first one is to help your measures team to unlearn the bad habits of misinterpreting performance and other quantitative information. The second objective is to get them to adopt a simple, but sound approach to using performance measures in decision-making, an approach that focuses on performance improvement, rather than judgment -- so, improving, and rather than just judging performance. And, finally the third objective is to update the performance report that you will have had them create in Step 6, update those with XmR charts if you haven't got them using XmR charts already. XmR charts, as you'll recall from your Blueprint training are the charts that highlight the real signals in performance measures that you can track through time.

Now, we are just so close to the end it‟s not funny. We are at the final technical step in the PuMP® Blueprint using measures to improve performance. Now we do have one more step in the Blueprint that you will facilitate your measures teams, and that's the reflection step, but we will of course talk about that when we come to Step 8, but for now this is how far you‟ve come, up to Step 7.

Now for today the usual structure will apply. We‟ll start off looking at your PuMP® practitioner kit for the Blueprint Step Number 7. We'll also go a little bit more deeply into the process of facilitating your measures teams to implement Step 7. As we do that, of course, we'll take a closer look at most of those resources. Also, we‟re going to spend a little bit of time today talking about not only some of the challenges that can come up when you're helping people use their measures, and using XmR charts, but also some of the other things that become a little bit more relevant when you get to this end of the PuMP® Blueprint.

One of those things is setting performance targets. I don't know why it is but a lot of people will set targets in the business planning process before they've even decided how to measure something. And to me that's so counterintuitive, it's like trying to test-drive a car down the street before you put the wheels on it, it just doesn't make any sense. Setting targets, to me, is something that happens when your measures are designed, defined, and even when you started to collect some data for them, and then it becomes a more relevant thing to set targets. We'll look at the approach that I've taken for setting performance targets, and I've got a resource kit for you to download on how to set targets. It's going to have a few things in there, the framework for setting performance targets, it's going to have an example, also a template to follow, and a one sheet guideline for how to determine whether performance has reached target or not, some real objective, indisputable rules or guidelines for doing that.

We'll talk also, today, about integrating measurement with performance improvement, and what that really means. Like what role does performance measures play in any kind of performance improvement? And then we‟ll talk about one particular type of performance improvement that




of 22


happens, and that's business strategy. Really, that's what business strategy is, it is a type of performance improvement for the organization.

Your application guide, which we generally don't spend the time looking through, because it's pretty well structured all the time, but we may if we‟ve got the time, take a closer look at the application guide today because they are some facilitation tips in the back of there about dialog versus discussion that I'd really like to draw your attention to.

For now though, we‟ll take a quick tour through the PuMP® practitioner kit for Step 7, of course, so that you can just see it range of resources that are available for you. And I've got loads and loads of files open here; ready to show you as we go through today. So be patient with me as I try to land on the right one to show you. Now, again, I know this text or font is really quite small so if you happen to have your practitioner kit for Step 7 handy, refer to that so you can read the file names a little bit more easily, then you will be able to through this video.

The PuMP® Practitioner Kit for Step 7

Alright, so we have two articles for Step 7. And, they have a slightly different theme to each other, the first one is called Are You Listening to your Measures, and that article is about what a really good performance review meeting should be like. It is talking about what it means to listen to your performance measures as opposed to a lot of the things that we usually do with performance measures, which is blame something, or ignore them, or jump to the wrong conclusions, or a lot of those sorts of things. This is about really borrowing from the principles of the PuMP® using measures technique, and showing or discussing how they might look when they become manifest in a performance review discussion, a monthly performance meeting, for example.

The second article is called Are You Reacting to Trends That Aren’t Really There is what that article‟s called, and it is very much to do with interpreting signals in your performance measures. It goes through an example, which I'm pretty sure I discussed in the Blueprint Workshop, and it is a reliability measure for an energy company, like an electricity distribution company. And, it shows a before and after, and the before is a graph of how this organization was monitoring this reliability measure using a moving average. And the after is what I did with their data, with their permission, to show them the risks of using moving averages to draw conclusions from your data. And, it talks about how easy it is to completely misinterpret what your measures are saying if you've got no idea of the right way to present those measures visually.

There are two handouts for this session; one is our normal case study handout, the procurement department‟s case study. And that handout goes through a discussion, or the kind of process that they went through to interpret one of their performance measures. The second handout is the framework for using performance measures. So, again, it's just that one page, it's got this mind map in it -- filter out the noise, look for signals, find the causes, that model there.

Then we have -- oh I have these ordered by type rather than a name, I'm going to reorder them by name. Then you have got your usual agenda to send to your measures teams before you have the using measures meeting with them. You've got your e-mail templates again, the FAQs the frequently asked questions that facilitators have when they are implementing PuMP®, in particular, of course, implementing Step 7. You've got the presentation that you'll be using with your measures teams and the report document to do any documentation of what you create during that meeting. That won't used to document your XmR charts; you'll instead use the spreadsheet template for XmR charts. But, you've may create with your measures team an agenda for a regular performance review meeting, and that's something that you can document in that report. You've




of 22


got your session planner, and of course we'll be opening up shortly and using that to guide us how to facilitate this session.

There are two spreadsheet templates in the practitioner kit for Step 7. One of them, and it's not the first one in the list here, it's the second one in the list here, the template for the PuMP® XmR chart. You will already have played with that from your PuMP® Blueprint training, although I'm pretty sure this version I may have updated a little -- let me see, yes I have -- following a conversation I had with Stephen Few a couple weeks ago. And if you don't know who Stephen Few is, he's the author of some of my favourite books about visual analysis, Show me the Numbers, and Information Dashboard Design, and Now You See It. He's been a friend of mine for a while now; I think he's just a superb guy, and I think his work is superb as well. He and I were talking about XmR charts because I sent him a copy of Donald Wheeler‟s book, Understanding Variation. And he really enjoyed it, and loved it, and I think he's going to feature in the next book that he writes about XmR charts. Together over a Skype conversation we decided how we would redesign the way an XmR chart looks. So, this copy of the XmR spreadsheet template is going to be different to the one that you got at your Blueprint training. This one is Stephen Few approved, in a sense, where it's just visually a little bit more effective. So, I'll leave it up to you to contrast and look at the difference in formatting between the two, but effectively that's the way this template looks now. You can modify it; of course, there's no problem with doing that.

The other spreadsheet template is one that you may or may not use. I just had a few people over the last couple of years ask me questions about how to measure things that happened very rarely. One of my clients earlier this year is in the mining sector. And, the team I worked with is the exploration team. And, what they do is they go around to all different parts of the world and they use their geo-technical knowledge and instruments to try and find new places that have sufficiently rich amounts of a particular mineral to make it worthwhile to set up a new mine.

These things happen very rarely, they may find one a year. So, it's not something that you would measure the count of. “How many new discoveries did we make this year, One, yeah. Zero, ah no. Two, ah yeah… One". It's so infrequent that it doesn't make sense to put the count of discoveries on a chart. But, if you're still wanting to improve the rate at which make a new discovery, there's a different way that you can measure it. And, that's what a G chart is for, a G chart is a bit different to an XmR chart, but it does the same kind of thing. What you are measuring rather than the count of something over a regular period, like a monthly or quarterly or annually, instead what you are doing is measuring the days between discoveries, or the days between events.

For our mining sector exploration team, they might measure the number of days since their last discovery. And what they are trying to do is reduce that as time goes by. It's a slightly different way to measure things, but it can be very useful if you've got a situation where things happen quite rarely or infrequently.

I'll let you take a look at that chart, there are some instructions in there, and I'm pretty sure I've got some examples too. But, because that's unlikely to be a priority right now for you, we won‟t spend a lot of time on it. If at any stage you use that template, you want to understand it better, you just contact me directly when the time is right and we'll talk it through.

There's two tools also, in your practitioner kit for Step 7. One is tool for creating XmR charts. Now, this is several pages of fairly detailed instructions for how to create an XmR chart. Now in your Blueprint Workshop, or the online program, I stepped you through this, and you applied these steps to create your first XmR chart, so it's the same set of instructions, they're just there for you in a handy format, pretty much the same as what's in your Blueprint workbook, but it's in a handout. And, this way you can share this with your measures team members if they want to take a copy and go and start creating more XmR charts without you necessarily holding their hand.




of 22


The other tool is a checklist, or a guideline, I guess of differences that are real. So essentially it's a good thing to provide to your measures team members as well, because it gives them clear examples of the types of signals to look for in an XmR chart, in other words what are the signals to look for in their performance measures.

Finally, the last tool is another spreadsheet, and I showed you this, we went through it as a little experiment in the Blueprint training. I call it interpretation scenarios, but it was that tricky one where I showed you 10 different analyses and asked you to tell me in each case if performance was going up, going down, or staying the same. And, we were jumping all over the place, every analysis looked different, sometimes it looked like it was going up, sometimes going down, sometimes you couldn't tell, but effectively it was just 10 different analyses of the same set of data. So, you'll be taking your measures teams through that little experiment as well. It really helps to bring home this point that we can't just use any kind of analysis we feel like to display our performance measures. It‟s really laying that foundation for why we want to use XmR charts.

So that's the practitioner kit. Carolyn, I'm going to assume you don't have any questions right now, just going through the list of files because we're going to do them in more depth, but if you do just stick your hand up, or comments, I mean it doesn‟t have to be a question. I could just be something you might want to say.

The process of facilitating Step 7

I‟m going to go back to our slideshow here, and we'll open up the session planner now from your practitioner kit, and step through the process of preparing and facilitating and following up your measures team to implement this step. And most of that time we'll spend going through the PowerPoint, going through a simulation, really, of what you're going to do in that using measures meeting with your team.

Back to the practitioner kit, we're going to open up the session planner, just make it nice and big, I think for today, given that you've already been through six of these sessions, you'll be very familiar now with how the agenda works. You'll be very familiar with how the e-mail templates work, and those sorts of things. So, I may skip over some of these resources, just to not waste your time, we'll spend our time mostly on the key resources that are going to be quite different.

Preparing for Step 7

In reviewing the session, that's really your first step in preparing to run it. So, of course you‟ll do things like look at the agenda, you‟ll read through the PowerPoint presentation, I suggest you look over the handouts as well. Go through some of the resources that you‟ll use during the session, like the interpretation scenarios, read both articles too, because you may not want to give both articles to your team, one article may be more relevant than the other, just depending on where your team is at. For example, if you've already started introducing XmR charts with your team, then you don't need to give them that Reacting to Trends article, it would be much more relevant, probably, for them to have the Listening to Your Measures article.

And, conversely, if you haven't introduced XmR charts with your team, and they happen to be a team that already has a regular performance review meeting, maybe they have a team meeting every week where they talk about performance results and they discuss what may be causing the results they don't want and what to do about it, if they've already got that kind of continuous performance improvement approach to their meetings, you wouldn't necessarily need to get in that article Listening To Your Measures. Just give them other one, and get them warmed to bringing XmR charts into those meetings. So, it's up to you of course.




of 22


I would also suggest you go over again you tool for creating XmR charts, and also the tool being the instructions, and also the template and just remind yourself how to do it. I would say part of your preparation could even be take some of the data that your measures team already has for their measures, and put one of their measures into an XmR chart, and use that as an example to take to the workshop, to show people what it can look like.

The amount of preparation that you do personally to remind yourself of the content and to feel like it's really going to be a natural process for you to deliver it is entirely up to you. And, that amount of preparation will decrease over time, of course, as you get more familiar with it.

Your next part of the preparation is to get your measures team ready of course, and that's about updating the agenda with the appropriate details and sending it out to them. Deciding which of the articles, either/or or both, that you want to send to them, and then you'll be collating some resources. So, I would say certainly both handouts; print those. Print the summary page, which is the one page that shows the using measures framework and explains each step in that framework. Also print the procurement example too, particularly if you've been printing all the previous procurement case study handouts as well, this kind of becomes a complete set by the time you get to this stage in PuMP®, and they can see an end-to-end case study right there through the procurement handouts alone.

You may want to consider printing for them that tool, Differences That are Real, and that's a one-page document that shows the different signals an XmR chart can show you. I'm going to take you to that now, so you can see what I'm talking about. This is how it looks. I said it was one-page, but it's actually two. You can see it goes through each of the different signals that you can see in an XmR chart. There we‟ve got a special cause, there we‟ve got a run over the page, there will got an early warning run where we‟ve got three points closer to one of the limits of natural variation than to the mean line. You've got the trend where you've got seven consecutive points decreasing, or increasing, if the case be that. It‟s talking about those basic signals that you‟d expect see, and how to respond to them. It's a really good checklist for your measures team members to keep at hand until it becomes second nature for them in how to interpret these things.

Back to our session planner, I would suggest not printing some of the other resources. In particular, I'm thinking of the detailed instructions for how to do the XmR chart. Let me go there to show you those, just as a little reminder. This is the tool called Creating XmR Charts, now this is a five-page document, it‟s quite long; it's an excerpt from your Blueprint Training Workbook, that goes through, and you'll recognize it now, I hope, as you look at it, it goes through each of the steps to create an XmR chart using the spreadsheet. It's a useful tool for your measures team members to have for those measures team members that want to be actively involved in creating the XmR charts, but rather than print it for them, I would suggest e-mail it to them. If you print it for them it's quite overwhelming because they're not going to be able to sit there and read it during your using measures workshop or meeting anyway. And, yeah, just don't give them the feeling of overwhelm. They're going to appreciate the instructions a lot more after they've experienced with you the process of creating the XmR chart together. I would not print this document, this is what I want to reinforce for you; e-mail it later.

I always put in here this optional idea to print the presentation slides if you want to. My preference is not to do it. I'm not sure what you guys are doing, whether you're printing them for your measures teams or not. Do your usual preparation of arranging your data projector and laptop, and of course loading up your laptop with all the appropriate tools that you‟re going to need. Of course, you'll need the presentation, you'll definitely need the interpretation scenarios, you'll definitely need the template for XmR charts, you may or may not need that instruction that I just showed you for creating XmR charts. You may want to have a hard copy of that handy for your use, as a quick




of 22


reminder during the workshop, unless you've done a few of XmR charts in practice and it's really quite natural for you to do it. And, of course, your report document, which will help you document with your measures teams any particular changes they want to make to how they do performance review meetings, or changes to their performance report.

That's got you prepared then. Now we enter the stage of facilitating it. What we'll do is I'll open up the presentation file, and we'll go through that, and you can see how these other resources sort of come in, along the way as you're running the using measures workshop.

While that's opening up, Carolyn, I‟m going to un-mute you, just to check and see how you're going. How are you going so far, Carolyn, any comments, or thoughts, or questions for you?

Carolyn: Just one question, if you're working with a team who have shown no interest in the creation of charts or anything, how do you counter that sort of atmosphere? That they think its somebody else's job to do the charts and the analysis?

Stacey: Are they a team that still understand why charts are important?

Carolyn: I'm just thinking in terms of when you undertook the training for our group, and I think on the second day, which is when we were talking about the definition and the chart data. And, bearing in mind there were a lot of lawyers there as well, there was possibly -- I‟m not saying there is, but there‟s possibly a mindset of, "it's not my job, I‟m a lawyer. Surely our statistics team or the reporting team should be looking at this area.” I was wondering in terms of engaging them to make them realize that it's their data, not the reporting team‟s data.

Stacey: This is a little bit similar, Carolyn, to how managers often feel about the second half of the PuMP® Blueprint, that second day you are talking about in the two-day workshop, which really is the second half of the process. They often don't want to do it. You don't want to be involved in it. Now, usually what I'll do with an executive team is I don't take them through the workshops on Measure Definition or report design. And, when we come to this Step 7, so that‟s Steps 5 and Step 6, but when we come to Step 7, for them Step 7 is really about actually using their measures.

So, for an executive team, what I will have done is take them through that interpretation scenarios experience so they can understand the relevance for using something like an XmR chart. But, rather than teach them how to do an XmR chart, or expect them to create them themselves, instead I'll have created a couple of their performance measures in XmR charts, and show them a bit of a before and after. “This is how you traditionally have been looking at these sorts of measures, here's how an XmR chart would show it to you,” and then we'll have a discussion about why that's a better way of doing it.

So, really all I'm doing with the team like an executive team, is -- who really shouldn't be creating graphs quite honestly -- is just helping them understand why that's the chart to use and understand how to interpret the chart. So, I would suggest maybe something similar for the lawyers, or maybe in other organizations it's not just lawyers, it‟s people in a very specialist, professional role that just don't feel comfortable getting down into the weeds of the data. So, in that case I would just suggest you could run the using measures workshop in a similar way for them. Make it about establishing the relevance for an XmR chart, you need to do a little bit more preparation, you need to get some of the business analysts or the data people together to create XmR charts for the measures they‟ve come up with. But, then your using measures workshop with the




of 22


lawyers would be about just establishing the relevance for the XmR chart, making sure they understand how to interpret one. And, really it would be a shorter meeting, unless you wanted to turn it into a simulated performance review meeting, where you guide them through, “Here are the measures, here's how they‟re laid out, you now know how to interpret these particular kinds of charts. So, now we should be able to look at our measures and say are there any signals? What should we be doing?” So you treat it almost more like a real performance meeting.

How does that sound, Carolyn?

Carolyn: That sounds a lot better.

Stacey: It's a really important question that you‟ve asked, not just for Step 7, but for some of the other steps as well. Thinking about, you know, “Which steps do I take a particular team through and which steps shouldn‟t I take a team through? What I‟ll do again with an executive team, they'll do the Results Mapping, they‟ll do Measure Design, they will have a Measure Gallery, but then what we‟ll do is, at that stage, the executive team will know they need to establish some kind of implementation team to bring the measures to life. So, we put together a secondary measures team, and that measures team is usually made up of business analysts, strategy officers, and some people that know how to get the data out of systems. And, then they take responsibility for the Measure Definitions, the report design, and then we come back to the executive team again when the measures are ready to be used.

So you can do that, you don't have to take the one measures team all the way through, if it doesn't really make sense for that to happen.

Carolyn: Yeah, no, that sounds good.

Stacey: Anything else, Carolyn, before we continue?

Carolyn: No, that was it, thanks.

Stacey: Super. I'll pop you back on mute, just to get rid of some background noise, and I'll come back to you shortly.

Facilitating Step 7

We are now going to look at the presentation for the using measures workshop. Title slide is familiar to you by now, as would be the next slide, which is the purpose slide, what the purpose of the workshop, or meeting is about, which is of course about responding to measures in a valid way and focusing on performance improvement. Slide 3 is always a reminder of where your team is up to in the Blueprint, and clearly that is Step 7. Your agenda is similarly structured to all the other sessions, the background of using measures, a case study, what the framework is for interpreting and using measures, and then wrapping up with some action planning.

If you are going to adjust this session for the lawyers, Carolyn, or an executive team, then you‟d want to change that agenda, and I suggest what the agenda is essentially about is a bit of the background, certainly the case study, but then the doing it section really would be about tabling the team‟s performance report, which you already worked on, you‟ve already put XmR charts through, tabling that and facilitating them to use that performance report as though they were at a normal performance review meeting. You could keep the action planning in there because you would still want a reflection at the end and say, “Well, how did that go? Do you like that way of using performance measures? How did you feel about interpreting the charts? Did we come up with any




of 22


ideas for the performance improvement actions that we want to take?” You‟d still do that, but the doing it is the section that would change mostly. And, indeed, the second section, the case study, that could also change to be a brief look at the procurement case study, but then very much a focus on maybe one of the charts that you created out of their data, maybe a before and after… “Here's how we would traditionally look at a measure like this, but this is how we look at it through an XmR chart. Can you see how we interpret it differently?”

So there are the ways you could change the agenda if you're doing it with the lawyers.

Slide #5 is a focus on, you know, “Let's just go over some background here about using some measures, let‟s set some context.” That leads you to Slide 6; Slide 6 is very similar to what went through in your Blueprint training. Let's have a look at some performance measures. This is where you go and open up that interpretation scenarios spreadsheet, so let's do that. Into your practitioner kit, it‟s the file at the bottom, Tool --Interpretation Scenarios. Open that up, and you probably recognize it now.

It's all these different analyses and you're just going to do what I took you through, which is to say, "Is the trend going up, going down, or staying the same?" Now I will give you a tip, I made a mistake last week, or the week before, when I was running this session for the Performance Measure Blueprint Online Program, I made the mistake of asking the question of everybody live on that webinar, "Is performance going up, going down, or staying the same?" Now, that's a little bit more confusing because down is good with the examples I‟ve got here. For performance to go up, the trend has to go down, and that‟s too confusing. So, I would say always make sure you ask the question, "Is the trend going up, going down, or staying the same?"

Then you work your way through each of the analyses. That‟s Analysis #1: The moving average, then you‟ve got Analysis #2: the trend line, you‟ve also got Analysis #3 another trend line, but it's for the current year, ignoring any of the data beforehand. You've got a table; it's looking at percentage differences calculated this month to last month or the same month last year. You've got the stacked line, where you‟ve got year on year; bar chart, stacked bar chart, again, year on year. I‟ve added in a pie chart here, which you might not have seen from your Blueprint training, where each slice of the pie represents a particular year. You got your line charts, so now it's starting to get clearer. And, you've got your XmR chart, which is Analysis #10.

Now, if you remember how this was run during your training what I did was work fairly quickly through each of these analyses, and every time asking that same question, “In this graph is the trend going up, going down, or staying the same?" And, if people pause too long, if it takes them more than four or five seconds to answer, I go, " Taking too long,” and we go onto the next one, because really one of the points is that a graph should highlight in a split second what‟s going on, if you've got to stare at it for four or five or more seconds before you can figure out what's going on, the graph has failed. So, that's part of the learning as well.

And then at the end, you're going to make this point, “Well, okay, we kept interpreting different things. Sometimes we said the trend was going up, sometimes we said the trend is going down, sometimes we said there was no change. Sometimes we couldn't figure it out. What would you say if I were to tell you that all of those analyses were of exactly the same performance measure, exactly the same set of data?” And then you can have that discussion about, "Wow." A lot of people are quite startled and surprised they could be drawing such different conclusions, just because of the way the data is displayed is changing.

That then brings you to one particular point, which is we have to be a lot more deliberate and careful in how we interpret our performance measures and how we use them, so that what we end up with is performance improvement and not performance deterioration or ignoring performance.




of 22


Back to our presentation here, you are then going to do a bit of a summary of these are the reasons why performance measures don't improve performance. One of the problems is that we don't have that formal, regular review. It's fine to have charts, but if we don't even have a forum to even talk about the charts are saying, nothing is going to happen. We'll end up just randomly trying to improve business processes, and policies and whatever, just based on the wheels that are squeaking the loudest, the customers that are yelling the loudest, or the managers, or executives, or regulators that are barking at us the loudest. They are very rarely representative of all of the stakeholders, so if we constantly pander to their needs, what we're often doing is doing it to the detriment of the majority of everybody else. We need some more objectivity in the way we use performance information to help us decide what needs to be improved.

Knee-jerk reacting to data, that's what we just did in that experiment, seeing differences that weren‟t really there, and not seeing differences that really were there, mistaking excuses for reasons. So, this is about blame. Treating symptoms rather than causes is a very, very, very typical thing that people will do I think partly because they just haven't ever had training in cause analysis, or cause/effect thinking. It's much more satisfying to jump to the first solution that sounds like fun and go and implement it, whether or not you can be sure that it's the best solution to do. And, judging rather than learning is a habit that is fairly typical as well. It's using measures to say, "Well, how did we go, did we achieve our targets this year or didn‟t we?" As opposed to using them throughout the year to say, "Are we on track to achieve our targets, if not what do we need to do differently to get back on track?" The results at the end of the year should not come as a surprise; they should be well-planned and deliberately pursued. And, the feedback that your performance measures give you regularly, throughout the year, are key to you being able to do that.

So, this is why I'm really not an advocate of measuring things annually. You may need to measure some things annually, but that‟s because they are lag measures. You're interested in the long-term trends in things like that. If you're trying to measure something that you can influence on a week to week or month to month basis, and see some almost direct impact from the actions that you take, then you should be measuring that frequently to get that kind of feedback and know what you are doing.

With all of these slides of course there are notes underneath, and so I have got some notes there to prepare you to talk about each of those bullet points that are in that slide. Don't dwell too long on it, the point here is really to say measures are supposed to improve performance, but there‟s a whole bunch of reasons why they never get the chance to do that, and a lot of that has to do with our own behaviours and our own attitudes, and that‟s what that‟s summary of.

The next stage in your meeting is to look at the case study and that‟s the handout of the procurement department‟s use of their performance measures. Now, what you're going to do is you take through that case study just for refer to Slide 9, which is the process for using performance measures. Firstly filter the noise, secondly look for signals, thirdly go find the causes of those signals, then fix the causes; in other words choose strategies, initiatives, projects, improvement actions that are designed to remove or manage the causes, and then go back to your measures, and see after you've implemented as fixes, what kind of effect edit did they have? Did they have an effect? Was it the effect you wanted? Was it a big enough effect, given the investment that you put into those fixes? And, then keep going around the cycle, just keep continually improving, continually learning about what the best approaches or actions are to get the results that your team exists to produce.

So, introduce that framework and then you want to show them how it works with the case study. Back to your practitioner kit, so this is the handout called Procurement Using Measures. Now, you're going to have this other handout as well, which as I said, is a summary of that framework.




of 22


So, you can have that handy for them, but also you want to discuss this, because this procurement handout shows what kind of thing you're doing at each of those steps.

Filter the noise is to use the control chart, the XmR chart. Look for signals, they didn't see any signals in their particular performance measure, but in a sense, the absence of a signal can be a problem, particularly if you've got some kind of implementation of an improvement initiative going on, you want to know that it's having an effect. And if your performance measure is not showing any change at all, you've got to ask that question, “Well, why isn't it changing? Here we are trying to fix it, how come it's not being fixed?” So, find out why. In a sense here, we are trying to find the cause of a lack of change, of a lack of a signal, as opposed to what you often do, which is try and find the cause of a signal.

Now, the procurement team had come up with a bunch of strategies, you might remember that from your Blueprint training back in Step 1. The handout was four goals and six strategies. One of their six strategies was about managing inventory, I think. So they were measuring the inventory holdings, and what they were trying to do was reduce the inventory that they had. So, they had a closer look to see why the inventory management wasn‟t creating an improvement in their performance measure, which was operating expenditure. The inventory manager had been spending some time reducing inventory holdings, getting as much inventory as possible onto the auto-order system, and what that does is it takes a lot of the manual effort out of inventory management. I think he was measuring inventory turns, which is something quite good, but it wasn't achieving its target. So, that wasn‟t really increasing fast enough. If I remember correctly, the reason for that was he was trying to get all the big, expensive items onto the auto-order system. The items that, in an electricity company, cost multi-million dollars, like transformers and big things like that, and putting all his attention on doing that, but hadn't managed to get the bulk of the inventory that was taking up most of everybody's time and effort. When he realized that it was about reducing the time and effort he realized that just having one or two million dollar transformers on auto-order wasn't helping anybody. It was about getting all the small multitudes of inventory items managed a lot more smoothly. When he realized that, and started doing it, the performance started improving, and then they started to see the reduction in costs that they had expected. That's the look for signals again step, where they are able to see, “We found the reason why it wasn't working, and we've corrected that, and now we're seeing the performance measure improve.”

So, it can be a good example of that, it wasn't blaming the inventory manager either, it was about understanding what was going on. It gave the inventory manager a clearer idea of why he was trying to get the inventory onto the auto-order system, because he hadn't understood the reason to start with, because he hadn't been doing it the way it was expected. It's really just about learning and understanding and getting a better appreciation of what does have the effects on the results that we want. You can talk through that case study as a way of explaining how this framework operates. And, then what you're going to you is going to do is going into doing it.

Now if you're with a measures team that is going to create their own XmR charts, at least for the start, and then later on they may hand that over to a business analyst or reporting officer to do for them regularly, you want to do a couple of things with them now. One is to set up how they're going to go about reviewing performance on a regular basis, and also how to create an XmR chart.

So, I'm going to go to the report document right now. So, in your practitioner kit, Step 7, Using Measures Report, and just show you how that‟s structured. Now, it's up to you which way you go, you could start with XmR charts, if you want to, and then talk about how to design the performance review meeting, or the other way. It's what you feel comfortable doing, because the order doesn't really matter.




of 22


The first section here is to understand how performance is currently managed by your measures team. Now, that may be a simple answer of, "Oh, we don't do anything. We don't have a regular performance review meeting." Well, job done go on to the next step. But, if they do, make some quick notes about how it currently happens. Does it have a name? How frequently is it? Is it weekly, is a monthly, or quarterly, or is it ad hoc? What is its current purpose? What is its current intended outcomes? Who are the participants that go to it and what does the typical agenda look like? Now, just asking those questions will raise people's awareness so much more sharply about whether that really is a good way to review, monitor, and manage performance, or not. So, then hopefully they'll quickly want to get onto the next bit, which is to redesign how their performance review meeting could be. Now, I've got that in here as you simulating them through the application of those five steps in the using measures framework.

Step 1 is filter the noise. So, what you would do is firstly help them set up an XmR chart for one of their performance measures. Step 2, you‟d look at -- and you may have already done this, you may have baked this one earlier and you can pull it out of the oven and show it to them, but once you show it to them, whether you create it with them then and there or whether you already got it for them, is to say, “OK, here it is, do we see any signals?” You can show them or hand out the handout that shows -- give them the XmR chart interpretation guidelines handout that that shows all the different signals and what they mean, and they could look at that and look at their chart and see if they can pick one of those signals, and to talk about what it might mean.

Step 3, only for the measures that have signalled that attention is needed, in other words they are either moving away from target or they‟re not making satisfactory progress toward target, talk about what the causes might be. Now, there's only so much that you're going to be able to do in this meeting, like you won't be able to run off and grab some extra data and do some cause analysis, but what your measures team may be able to do then and there, is to come up with their theories about what might causing the signal, and from those theories formulate some questions that they can then hand to an analysis team to use data to verify. They may be looking at a performance measure, like before, like the procurement costs, and they see a signal and the signal for them is, “We're not improving, but we should be. So what do we think the causes might be? Well, something is going wrong with the way that we're managing inventory. The inventory costs might not be coming down.” Well, OK, there‟s a question, “What are inventory costs doing?” Another question might be, “Well if inventory costs aren‟t coming down then it could be that we've still got very low inventory turn rates. We're not able to turn our inventory over faster.” In other words, “We're not keeping lower inventory and having it easy to refill, instead what we're doing is holding more inventory than we need to. So, let's see what our inventory turn is doing.” So, there you got two questions that an analyst can go away and get the data for. One, “How much is inventory costing us?” “How much inventory do we have?” And two, “What's our inventory turnover rate?”

That's what‟s Step 3 is really about in the meeting. Sometimes you can answer the questions, sometimes you have to forward some hypothesis or theories about the causes might be, and then have a team go and do some analysis to come back to the next meeting.

If you have been able to identify a very likely cause, you might want to start thinking about what are some potential solutions to fix this? When you have this discussion here at Step 4 it shouldn‟t be, “Let's go with the first solution that comes to mind.” What it really should be are, “What are some potential solutions, and what are the merits and the disadvantages of each of those solutions?” Again, you may need to go away and do some more work to scope out costs or to flesh out details of some of these solutions. You may want to formulate a few pilot tests, to test maybe two or three of the solutions to find out which one works best.




of 22


I'll give you a really, really, really simple example of a pilot test. One of the performance measures for me is the open rate of my e-mail newsletter. And, what an open rate is is the percentage of people who subscribe to the newsletter that open that particular e-mail of a particular newsletter. I've got a newsletter going out next week, for example, the open rate for the newsletter would be, for the number of subscribers that received that e-mail, what percentage opened it up? Now, because the e-mail distribution software I use, it tracks that, it collects that data automatically for me. But, I might want to work out, “How can I get more people to open it up? They're missing out on great information, possibly because…” and here‟s a theory “…the title or the subject line I‟m giving that e-mail is not grabbing their attention, or it's not using language that they're comfortable with.”

So, I'm doing a split test on an e-mail that will probably be going out today, inviting everybody to come to a free webinar in December and what I've done is I've split tests -- I've done four pilot tests. The two questions I wanted to test is if I tell people it's a free webinar in the title, does that make it more or less interesting to them? And, what happens if I use the term „KPI‟ versus „performance measure‟? So, that creates four e-mail pilots that I can send out. Number one has got „free webinar‟ in the subject line, and it uses the term „KPI‟. The second one's got „free webinar‟ in the subject line, but it uses the term „performance measures‟. The third one doesn't have „free webinar‟ in the title, and it uses the term „KPI‟. And the fourth one doesn't have „free webinar‟ in the title, and it uses the term „performance measure‟. So I'm going to look at this, and do the open rates change depending on the combination of words that I use?

Now, the combination that gets highest open rate, that will become my new standard for how I write subject lines when I'm inviting somebody to a webinar. I don't know which one it will be yet because obviously I haven‟t the data, but that's what Step 4 is all about, is coming up with what are some potential solutions that we could test to find out what works?

Now, I've given you an example on a very small scale, but the theory, that concept of pilot testing, works on any level of scale, any type of project over any kind of time frame. But, they‟re critical to do, these pilot tests, sometimes they're called business experiments, because they teach you what works and what doesn't work very, very quickly. And, when you go to Step 5 to say, “Well, what‟s our performance measure based on the different pilot tests,” you can quickly work out what the best solution is, standardize on that, make that your new approach, and you should see your performance measure shifting up to a higher level, or a better level, or moving closer to target. That's what using a performance measures is all about. And, if you take your measures team through that process just for one of their measures they'll start getting a real sense of what it's all about. There's no blame involved in that, no blaming people, no blaming the economy, or the external environment, or the regulatory system. There‟s no sweeping things under the rug. It's just, "Yeah, let's be really curious and inquisitive and challenge ourselves, and see how well can we improve this."

Now after you've had that experience with them, you then have a discussion with them about, “What are the opportunities to improve how we go about using our performance measures?” You can do that by having a -- “Well, we already have a current performance review process in place,” a monthly performance review meeting for example. “What works well about that? What do we want to keep?” The next question is, “What doesn‟t work well about it? What do we want change?” And then the third bit is, “Well, what specifically will we do to make better use of performance measures in our performance review meeting?” Alternatively, you could just create a brand-new agenda from scratch; it's up to you how you want to do that.

But really the idea here, through the doing it part of this work shop, is to help them understand, objectively and realistically, how they‟re currently doing in managing performance, then to have the




of 22


experience of taking the PuMP® approach to using performance measures to review and manage performance. And, then after that, getting them to think about, “How will we go from this point forward?” “What can we improve about our current approach, or do we throw it away and come up with a new one based on the PuMP® method?”

That is essentially it then for the doing it part of this session. Then you‟re really just up to the action planning. I forgot to show you Slide 11, which is a summary of exactly what I just described to you. “How does our current decision process work?” “Let‟s experience the PuMP® approach.” And then, “How would we like our decision process to work?” Then Slide 12 is the action planning, so, “What are we going to change?” “What are we going to do differently” and reflecting really.

Slide 13 talks about the next session, which is the final session that you‟ll take your measures teams through. It‟s a bigger picture reflection on the entire measurement journey. What‟s been learned, how would we improve doing performance measurement into the future? Maybe starting to think about how you‟d integrate some of the PuMP® techniques into existing processes like planning, and reporting, and decision-making.

The where to next, again you will go back to your report document here and do some action planning. “Do we want to fine tune any of our measures?” “Do we want to fine tune our report?” “Do we want to change our performance review meeting schedule and agenda?” And finally, a reflection, “What worked well today, what could have worked better?”

And that's it, that's taking them through the whole session. Carolyn, I'm going to un-mute you and see how you're going, if you got questions or comments, you're un-muted now, Carolyn.

Carolyn: I've got no questions.

Stacey: Did that make sense?

Carolyn: Yep. I think the review of the performance meetings is a really good one to focus on, particularly, at our end, anyway.

Stacey: That's good. I often have clients that don't even have any kind of performance review meeting in place at all. So, this is a real wake up call for them, that, "Oh, maybe we should have." And, it's kind of nice because they get to design it from scratch and they don't have the struggles with unlearning the habits of what happens when they get together during those meetings. That's no great problem, and once you redesign and improve the way the meetings happen, you can facilitate a few of them until people get into that new routine of making sure that they‟re properly interpreting their measures, and doing cause analysis, and thinking of potential solutions before they lock something in.

OK, anything else, Carolyn, before we continue?

Carolyn: No, not from me.

Stacey: Back on mute, and back to our agenda.

Success Tips

FAQs

The next stage that I‟ll take you through is the success tips and common challenges. We'll have a quick look at some of the FAQs, ways to handle challenges that may come up as you're running your using measures session. And, then I'll get a bit of a heads up about the Setting Performance




of 22


Targets resource that I've got for you, and a bit of a chat about performance improvement as a process, and how measures fit into that.

To the practitioner kit and the FAQs document -- there's not a great deal for these, there are a few technical questions that can come up about XmR charts. So, I've got a few responses there. One of the general questions I, “Which performance measures can I use an XmR chart for?” Pretty much, it's any performance measure that you are calculating at regular intervals through time. For example, a monthly or a weekly value of something, whether it's staff turn over, or revenue, or new expenses, or hours of rework, or percentage of staff taking leave, whatever, it‟s something that you are tracking regularly through time. This also mentions the G charts one as well, where you've got rare events like, as much as I don't want to say it, deaths from workplace accidents, or like I said before mining discoveries, or even innovations, they're often quite rare too.

The funny numbers, 3.27 and 2.66 are constants that are used in the calculations in your XmR charts. Some people have asked where they come from. Well, basically, they‟re constants and you don't have to worry about where they come from. We would have to have a couple of lessons in statistical theory for you to understand why those numbers are there, but I think we trust the statisticians that have come up with them, and just use them.

I am just going to pick out some of these questions that are bit more hairy. “How do you convince managers to use XmR charts as opposed to what they're used to?” The thing here is that there used to what they're used to, that's comfortable for them. You need to establish a rationale for them to move away from it. But, not just a rationale, an emotional experience. When they have the want to use an XmR chart, then they're going to find it easy, no matter how technical they may be concerned that it is. The best way is to do that before and after thing. Show them a measure that they've currently got, the way they are displaying it now, and ask them to interpret it, and then show them the XmR chart version, and ask them to interpret it. And, what you're often find is that the interpretations are different, and the XmR chart gives them much more insight into what's really going on with performance. That article, Are You Reacting To Trends That Aren’t Really There is a perfect model for you to follow, to create that before and after example.

Another question that comes up a lot is that “Dashboard software doesn't have XmR charts as an option” and that's true. I don't know when that's going to change, or if that's going to change, but what I tend to do is do all my calculations in Excel, and then have the dashboard application import those Excel values, and create the chart as a normal chart that has multiple lines on it. But, the lines, as we know, have specific meaning in an XmR chart. You‟ve got your data series, which will be a normal data line, and then you'll have a line for the mean line and a line to the upper natural limit of variation, and the lower limit of variation. You kind of got to fiddle with it and do calculations and yourself in Excel and then import them.

“How often should you have performance review meeting?” It depends. I would say have them monthly at least, because if you have than quarterly too much time has gone by and you may very well may missed opportunities to make improvements to performance, or to learn from what your performance measures are telling you. Sometimes weekly can be really appropriate too because you may not look at every single measure every week, you may theme each week and say, “Well, this week we're looking at staff turnover. Let's see if we see some signals in there.” You can have a weekly thing, one week focus on one particular measure, and go through the whole using measures process for that one. Then the next week do a different measure, whatever. I think regularly is really important, so that you maintain that strong focus on managing performance, that discipline of working on the business, not just in the business.

“How can I stop people from comparing this month to last month, and those other sorts of habits?” I think using that tool, the interpretation scenarios is a great way to start. But, also, get




of 22


yourself a copy of Donald Wheeler‟s book, Understanding Variation, because there‟s lots of examples and ways of explaining these concepts in there that work really well. He wrote that book for management, so it's just got some really great ways of explaining these things that you could borrow from.

If you don't have enough time to go over all of the performance measures in a meeting, that's fine too. There's actually an article from the Harvard Business Review called Stop Making Plans, Start Making Decisions. It has this idea of you having those weekly or monthly themes where you do just focus on a subset of your performance measures, maybe just one result out of your Results Map and the two or three measures that go along with it. Again, it‟s about maintaining that momentum of improving things. You can‟t improve everything all at once. Exploit that.

To stop your measures teams from reverting to old habits, to help make sure that they continue to validly interpret their measures and do cause analysis, you‟re probably going to need to facilitate several of their meetings to get the new habits embedded and replacing the old ones, and then they‟ll be fine to go on their own after that. But, be patient, it can take a little while for people to get really comfortable with looking for real signals in a performance measure and talking about causes rather than blaming and testing potential solutions rather than just jumping to the first one that sounds really fun. Just stay with them for a while and facilitate maybe up to half a dozen of their performance review meetings, just to make sure that they‟re staying on track.

Sometimes, using a measure is the best way to work out whether it‟s useful or not, so often you can find at this stage people want to throw some of their measures away, or improve them, because the proof‟s in the pudding, you know? You start interpreting a measure and start deciding how to respond to it and you realise, “Hey, this really isn‟t that important.” Or,”This really isn‟t what this result is about,” and that‟s really brilliant, because then you‟ve got the opportunity to go back and make the measure better.

The last one is about data quality, where people are spending more time debating the problems with data integrity as opposed to using the measure. Something good to remember is that even though the data may have a certain amount of error in it, the trend over time is going to have less error. The trend over time is more reliable than any one particular data point is. You can still get good, practical information out of a performance measure, even when the data has a certain amount of error in it. No data is every going to be perfect. So, it‟s only in the cases where you really have got an incredible amount of variability in your data and you think that a lot of that is due to data entry errors, or not having a complete set of data. Certainly go and improve the data quality where you can, but if people are just obsessed with saying, “Aw, it‟s not perfect, I‟m not going to use it,” that‟s their problem, and that‟s something you‟ve got to be able to fix with them. You can‟t have perfect data for any sort of decision making. People in a technical role can be quite at risk of this, wanting things to be perfect before they use them. The point is it‟s about, “Is it giving us a general idea of what‟s going on, enough that we could take some action?” And, so long as you‟re looking at patterns and trends in the data, you‟re going to be fine. The individual points don‟t need to be perfect.

They‟re the most frequently asked questions that I get around this stage in the PuMP® process.

I‟m going to back to our slides here. I want to talk to you a little bit about target setting before we wrap up. This is the framework that I kind of use to do target setting. It‟s not about plucking a number out of thin air. It‟s about thinking very deliberately about targets as tools in improving performance, not as something that you just have to have because there‟s a target column in the business plan. This starts with thinking about what your current capability is. And, what that means, in plain English, is what is our performance measure say we‟re operating at now? Where is our mean line now in our XmR chart? That‟s your current capability.




of 22


It also helps to then think about, “Well, we know our current capability, what‟s our rationale for wanting to improve?” “Do we need to set a target for this measure because it needs to reach a certain level in order for another measure, further in, in the Results Map, we need to hit a target there.” Hitting that target for, let‟s say it‟s a green level measure, it depends on two or three of the blue level measures hitting their targets. That‟s what rationale is, it‟s about understanding is this target important because it‟s part of achieving a target for another measure?

The third part is thinking about the size and timeframe. Given you‟ve got that rationale worked out, given you know a higher level target that needs to be achieved, then what does that say about how big a target you need to set for your current performance measure, and over what time frame do you need to achieve that by? Sometimes this can mean setting several targets, staged through time. You might have a target to achieve in six months‟ time, and then a bigger target to achieve in twelve months‟ time, and then an even bigger one to achieve in two years‟ time, for example.

You also need to define, how do you know when you‟ve achieved that target? How do you know that your signal has met the target? And that‟s really, for example, saying, “OK, we‟ll know we‟ve hit the target where there‟s been cause to recalculate that mean line, and that recalculated mean line is now sitting at or above our targeted value.”

Finally, once you understand all of those aspects of what the target means, why it‟s important, how big it is, and how much of a gap you‟ve got to close between current capability and your target, that should help you understand, “Well, what kind of actions do we need to take to close that gap? And what sort of budget do we need for those actions? What‟s the initiative that we think is going to work, and how do we fund that appropriately?” I think this is really interesting, because that‟s not the approach that we take, by and large, in organizations and businesses today. Actually, the targets are set and they‟re not set in the context of what kind of funding we have available for performance improvement initiatives. We don‟t have any evidence what so ever, to know whether the initiatives we‟ve chosen are going to be able to achieve the target. And often we do all of this thinking before we even know where our current capability is. We don‟t even know how far we have to travel, and how hard it‟s going to be to achieve that target.

This is a different approach, you can appreciate this target setting process has got to happen in your planning cycle, but that also means that you‟re Measure Design and you‟re bringing to life some of your measures to get that benchmark data, to get those first few points on the graph, really has to happen before you choose your improvement initiatives. That‟s the most logical way to go, so that you‟re not wasting time funding things that aren‟t working.

Now, I know there‟s probably a whole lot of infrastructure in a lot of organizations that might prevent this from becoming the next approach that you take to your planning, but start the conversation going, and help people understand the implications of the current planning processes. Just see, are there incremental improvements that you can make to the way planning is done, so that you can get much more sensible target setting in there?

Now, I have got a kit for you, a resource, about target setting. So, I‟ll just quickly show that to you. It‟s a collection here that you‟ll be able to download from the Certification website. Now, I‟ve got that framework I just showed you on a one-pager explaining each of those aspects of the targets. I‟ve also got a guideline here for knowing when a target has been met. I just talked to you about the example of saying, “We‟ll know when we‟ve achieved the target when we have cause to recalculate the mean line and that recalculated mean line is at the target level,” well, these first two charts here shows you that. Here‟s our current level of capability for our performance measure in the first graph. We implement some improvement strategies and ideally what we want to see is that we have a run above that mean or an increasing trend such that we need to recalculate that mean




of 22


line, and when we recalculate the mean it‟s at the target value, on or before the date that we want to achieve the target by.

But, that‟s not the only way. You could set targets for upper and lower limits of normal variation. You can do that. You can also set targets for both so that you‟ve got a band of variation that you‟re aiming for. But, I‟ll let you go through that and see which way you want to go. I think the simplest is the first, just get people comfortable with mean line as the measure of capability. It‟s the mean line that you‟re setting your target for.

There‟s also a case study in that resource pack, which goes through an example of how to set a target for a measure called Chocoholics Anonymous Newsletter sign-ups, a little bit of a silly example, but it illustrates the points of what are you actually doing at each step in that framework as far as selecting a target. So, that‟s something for you to read too.

There‟s a template similar to the case study for you to use if you want to, to help guide you through selecting a target for a performance measure. And, finally, some target-setting tips, which is just a little bit more detail about each of those steps in the target-setting framework.

Yep, so that‟s your target-setting kit.

Now, Carolyn, I‟ll un-mute you. Carolyn, do you have any questions or comments about that target-setting stuff?

Carolyn: I just find it interesting that I know in our business plan there‟s targets that were just plucked out of thin air, which are then reported to the board quarterly, but on what basis they‟ve been set, other than someone‟s idea in their head. So, our planning cycle is coming around again next March. So, that will be worthwhile changing people‟s thinking before then.

Stacey: If you can build up a couple of examples for some of the performance measures that you‟re involved in developing now that will help them a lot -- not just the Chocoholics example I‟ve got there, but it will build your confidence and skill in how to create targets, but it will give them some real examples to look at too, so they can see how it can work, how much more sense it makes.

Anything else, Carolyn, before I put you on mute again?

Carolyn: No.

Some tips on xxxxxxxx

OK, thanks. Alright, just for wrapping up… there‟s not a lot of content in this that I‟m about to share with you, but more I think it‟s sort of stating the obvious, but while it‟s obvious, it‟s not really the approach that a lot people understand improvement to be. The improvement cycle starts with defining a problem, a performance problem, then identifying the process that problem lies in, and I‟m talking about the business process here, you know the way people go about doing their stuff. Then developing potential solutions, pilot-testing those selected solutions, selecting the best solution based on how it performed in the pilot test, and implementing it, and then monitoring the impact it has over time.

Now, that should sound familiar to you because that‟s how I described the PuMP® Using Measures Approach. Performance measures play a really important role not just in defining the problem, but throughout that improvement process as well. So, you‟d certainly use your performance measures to measure the problem that you‟re trying to fix and that way you‟ll know whether it‟s been fixed or not. But, you‟re also going to use performance measures when you‟re




of 22


identifying which process the problem is associated with and what the causes might be, because sometimes you can come up with potential causes that you want to measure as well. If you find specific causes to a problem and your solutions are aimed at fixing those causes, by measuring the causes you‟ll know whether your solutions has fixed them or not.

So, for example, your problem might be that costs are too high and what you may want to do is look at the particular business process step-by-step and examine why costs might be too high. And, something you might find as one of the steps might have a lot of rework, so you could measure that, “How much rework is there?” Rework could be measured in terms of number of hours spent doing rework, or number of items that required rework, or an item could be a document, or an invoice, or a customer interaction. Maybe there‟s also some non-approved spending going on and that‟s increasing costs, so you could measure what percentage of spending is happening by following the purchasing policy, as opposed to not following the purchasing policy. So, you could measure some of the causes, that could be quite useful.

Now, that doesn‟t necessarily make them long-term performance measures. These could be temporary performance measures that you‟ll only monitor until you‟ve got them under control. Your key performance measure is the measure that is the measure of the problem that triggers this whole improvement cycle.

The next place that performance measures are useful is in the pilot-testing because you‟re going to use those measures of your causes and of your problem to work out which solutions seem to be having the best impact.

Then, finally you‟ll use your measures again to monitor impacts too, and the measures you‟ll use mostly in monitoring the impacts is the measure of the problem, the same measure you‟re using there at the first step.

Understanding how measures play a role in improvement should help highlight to people that you can‟t just measure at the end of the year. You can‟t measure after you‟ve done everything, after you‟ve implemented your plan. You have to measure through your strategy execution. And, that‟s what strategy execution is, it is just an example of a performance improvement process. It‟s another form of continuous improvement, but the difference being that the problems don‟t just come up, the problems are defined already in business strategy. Your goals and objectives and business strategy are really problem statements, and each one of those represent something that you want to get better at. If you want to get better at it, it means you‟re not good enough yet, which means there are things holding you back, and therefore improvements is the process follow.

Like I mentioned before, we typically choose initiatives before we define the problems and before we define the measures, and before doing proper cause analysis. We‟ve got a delay our selection of initiatives just a little bit longer, and put a little bit more rigor into our planning processes to be able to do this kind of thing. The funny thing is we also lock in our budgets before we even test which initiatives works. It‟s absolutely find to throw away an initiative because it‟s not having the impact that you want, and reallocate its budget to a different initiative, either a new one or another one that seems to be working quite well. We need to be able to be that flexible, I‟m not sure that our systems are quite there yet.

There‟s one final resource that I have for you, and it does relate a little bit to improvement, Laws of Systems Thinking. This may be a little bit too much for you, I‟m not sure. But, I‟ll be absolutely fine if you choose to completely ignore it because you‟re just not there yet. But, I‟ve got to say that one of the biggest impacts on my whole thinking and approach to performance measurement and performance improvement has been what I‟ve learned from Peter Senge, who‟s the author of a book called The Fifth Discipline: The Art and Practice of the Learning Organization. And, in that




of 22


book he discusses eleven laws of systems thinking. Now, systems thinking is different to systematic thinking. Systematic thinking is linear, step by step thinking. Systems thinking is holistic thinking. It‟s thinking about all kinds of interactions and dynamics that happen between and among all the different parts in a system or organism, like an organization. I love these laws for systems thinking. I think they‟re great to talk about when people are rushing to solutions too quickly, without understanding the causes, without really knowing whether those solutions are going to work or not. You could just get a bit familiar with some of these laws and know which one to bring out and have a discussion about if you see people rushing to solutions too quickly.

I‟ll go through just a sample of them. Law #1: Today’s problems come from yesterday’s solutions. So, if someone‟s rushing to a solution too quickly you‟ll want to ask the question, “That sounds like a great solution, can you think of any long-term unintended consequences of that solution?” “Could that cause us some problems in the future that we‟re not really anticipating right now?”

Another law which I really love is Law #3: Behaviour grows better before it grows worse. Sometimes when you come up with a solution, you put it in place, everybody‟s on alert, “Yay, let‟s make this thing better, cool.” They‟ll be really optimistic and positive. But, once that solution really beds in, sometimes things can get a lot worse. People can get complacent, or the solution can just stop working the way you had at first thought it was. Or, it can require everyone‟s constant, high-level energy to work, and once everyone returns back to normal the solution can‟t sustain itself, so the performance can get worse. Look out for have you got a solution that requires a lot of energy and unnatural investment in order for it to be sustained.

Law #6: Faster is slower. Rushing into a solution too quickly can often mean that we‟ve got to fix more and more problems as time goes by, rather than just standing still for a minute, doing some proper planning, a bit of pilot testing, and then we‟ll know with a lot more confidence which solution to go with.

Another favourite of mine is Law #10: Dividing an elephant in half does not produce two small elephants. Splitting a budget in half so that you do two projects doesn‟t mean that those two projects are going to have as big an impact, or anywhere near the size of impact that one project, properly chosen and properly funded might have. That‟s a really great way to think about the return on investment, I suppose. Don‟t do two things half-arsed. Do one thing really, really properly.

I don‟t think I‟ve got too much discussion about each of those laws in there, but think about what they mean to you. Some of them are really quite obvious what they mean. Alternatively, grab a copy of The Fifth Discipline, or I‟m pretty sure Peter Senge‟s got a website. I just don‟t remember what it is off the top of my head, but if you search for Peter Senge and The Fifth Discipline, you‟ll probably find a whole bunch of resources, and possibly even some literature explaining each of these laws, if that becomes a relevant thing to do.

We‟ve gone over time by five minutes, which is fairly typical, so I‟m going to call it a day there. And, I‟ll un-mute you Carolyn, just for a bit, to see if you‟ve got any other questions. Comments or questions from you, Carolyn, before we wrap up?

Carolyn: Just a comment in terms of Peter Senge. I actually have the field book of The Fifth Discipline, which I call on quite frequently (and I can only recommend that to they other two) and when you were going through that circle, it made me think sort of like the action research group of that, defining the problem, and then looking at that. And, the problem tree/solution tree, sort of starting from that point, rather than starting at the solution. That was just comment.




of 22


But, it‟s really nice seeing Peter Senge coming into the conversation.

Stacey: Good, maybe I should think about bringing more of his stuff in there. Like, all of his templates for different system dynamics that can come up. Like, I think he‟s got one, escalating intervention -- I can‟t remember what it‟s called now, it‟s been a while since I looked at them. They‟re different names. But, he draws all these diagrams about --that‟s right, reinforcing and balancing feedback loops, and delays that happen, heaps of cool stuff. But, I‟m pretty sure that those things would be on his website. I‟ve got the field book too, and that‟s really great.

For Cath and for Mariko, if you want to check out those resources, The Fifth Discipline, and also The Fifth Discipline Field Book, you can pull out all sorts of really practical resources from that for all kinds of things, not just performance measurement, but for any kind of change you might want to make in a system.

Anything else, Carolyn?

Carolyn: No, no, that was just my comment.

Stacey: Awesome, thanks for sharing it, I‟m glad. I was a bit hesitant about sharing his stuff, but now you‟ve inspired me that there‟s people like you out there that really value it.

All right, we are done, nice to have you live today, Carolyn. And, Cath and Mariko, if you‟re listening to the replay and if you‟ve got any questions about it, of course, just email me and I‟ll look forward to having the next coaching calls that I have scheduled with each of you. Carolyn, I‟ll look out for some suggestions from your assistant about when you want to catch up.

Carolyn: Yeah.

OK, well, we are done, I‟ll catch up with you all next week in our Step 8 webinar, but until then, keep doing what you can with your implementation, email me anytime if you‟ve got questions or comments or thoughts. Other than that, enjoy the rest of your day, and I‟ll catch you later.

End of transcript.

interpret and use performance measures for improvement certification... · ©stacey barr step 7:...

Documents