Metrics MatterBy

Harold Kerzner, Ph.D.International Institute for Learning, Inc.

During the past decade, there has been a rapid growth in Agile Project Management practices, not just in IT, but in other types of projects as well. Most of the principles of Agile Project Management practices have provided benefi cial results when applied to non-IT projects. While all of this sounds good, there are also some challenges that accompany the growth.

There is an old adage used in project management, namely “You cannot manage what you cannot measure.” Therefore, to manage projects using agile techniques, you must establish metrics to confi rm that the benefi ts are being realized and agile practices are being executed correctly. Fortunately, accompanying the growth in agile practices has been a companion growth in metric measurement techniques whereby today we believe we can measure just about anything. There are good metrics for reporting performance.

Another aspect to be considered is part of the Agile Manifesto: Individuals and interactions over processes and tools; and working software (or product) over comprehensive documentation. Considering the Scrum framework, the team should provide artifact transparency; in other words, Scrum requires transparency to support decision making based on perceived state of the artifacts. This points to the direction that Agile principles would use just the absolutely necessary metrics, as long as they add value to deliver the items on the product backlog.

Metric Mania:“Metric mania” is the insatiable desire to create metrics for metrics’ sake alone rather than for what is really needed. There are disadvantages to having too many metrics and confusion in what metrics to choose.

The result of having too many metrics is that:

• We steal time from important work to measure and report these metrics.

• We provide too much data and the stakeholders and decision makers fi nd it diffi cult to determine what information is in fact important.

• We provide information that has little or no value.

• We end up wasting precious time doing the unimportant.

• Too many metrics can open the door for unnecessary questions from stakeholders and business owners, and eventually lead to a micromanagement environment.

In traditional project management, which uses waterfall charts, reporting has always been done around the metrics of time, cost, and scope. With the use of the Earned Value Measurement System, the number of metrics can increase to twelve to fi fteen. As companies become mature in using any new approach, the number of metrics reported is usually reduced.

Metric Management:

Having a good metric management program can minimize the damage of metric mania but cannot always eliminate it. There are four steps included in typical metric management programs:

• Metric identifi cation

• Metric selection

• Metric measurement

• Metric reporting

Metric identifi cation is the recognition of those metrics needed for fact-based or evidence-based decision making. Metric selection is when you decide how many and which of the identifi ed metrics are actually needed.

Metric selection is the fi rst step in resolving metric mania issues. Ground rules for metric selection might include the following:

• There is a cost involved to track, measure, and report metrics even if we use a dashboard reporting system rather than written reports.

• If the intent of a good project management approach such as Agile or Scrum is to reduce or eliminate waste, then the number of metrics selected should be minimized.

• Viewers of metrics should select the metrics they need, not the metrics they want. There is a difference!

• Asking for metrics that seem nice to have but provide no informational value, especially for decision making, is an invitation to create waste.

Paperwork is the greatest detriment to most project managers. The future of project management practices is to create a paperless project management environment. This does not mean that we are 100% paperless since some reporting is mandatory, but it does mean that we recognize that unnecessary paperwork is waste that needs to be eliminated. In doing so, we have gone to dashboard project performance reporting.

Dashboard reporting systems can force viewers to be selective in the metrics they wish to see on the dashboard. A typical dashboard screen has a limited amount of real estate, namely the space for usually only six to ten metrics that are aesthetically pleasing and can be easily read. Therefore, telling stakeholders or business owners that we wish to provide them with one and only one dashboard screen may force them to be selective in determining the metrics they actually need.

Graphical Displays of Metrics:Dashboard performance reporting systems have made it easy to report information. Typical metrics for Agile and Scrum include Stories committed x completed; Team Velocity and Acceleration; Sprint, Epics and Release Burndown rate; Net Promoter Score (NPS), just to name a few.

Below are several metrics that are common to traditional project management practices which might also be useful for Agile and Scrum.

1. Resource Management

Shows the amount of time people are committed to work on projects. Agile and Scrum recommend that the team be fully dedicated to the project. Resource utilization is critical. Without effective resource management, people may spend only 50% of their time doing productive work on projects. The remaining time could be devoted to rework or succumbing to time robbers such as unnecessary meetings, phone calls, multi-tasking, and other such activities.

2. Impediments, Defects and Scope ChangesAn impediment is anything that can block or slow down progress. Impediments require action items to resolve them and should be taken by the leadership of the project and/or team. If the impediments are not resolved in a timely manner, then the fault is usually with the leadership.

Exhibit 1 shows the impediments that occurred in each month of the project. The exhibit also shows how many impediments were discovered, how many were resolved and removed, and how many had to be escalated to higher levels of management.

One of the strengths of Agile and Scrum is the commitment to defect analysis and correction as the project progresses. In traditional project management, defect analysis is often performed at the end of the project where the cost to correct the defects becomes very expensive. Exhibit 2 shows how many defects were discovered each month, how many are still open, and how many escaped. Metrics such as Exhibit 2 give us some indication of our ability to meet time to market requirements.

Exhibit 1 Impediments

Exhibit 2 Defect Analysis per Month

Changes in scope are considered normal on Agile and Scrum projects and are dealt with at the end of each Sprint or Interaction. Some people believe that scope changes occur because of poorly defi ned requirements and faulty planning. While this argument may have merit, most scope changes occur because market conditions have changed or the business model requires reconfi guration. Care must be taken that the team can absorb all of the changes. The Exhibit 3 shows a process of scope changes being approved, denied, and pending.

Exhibit 3 Scope Change Options

3. Value Management:For more than fi fty years, we have defi ned project success as completing the project within the time, cost and scope constraints of a project. On the surface, this appears to be a good defi nition. But what this defi nition omits is the importance of “value.” Any company can complete a project within the constraints of time, cost, and scope and end up providing no business value. A better defi nition of project success is to create sustainable business value while meeting the competing constraints.

Value can come in many forms such as economic or business value, social value, political value, religious value, cultural value, health and safety value and aesthetic value. All of these are important, but we will focus on economic or business value. Exhibit 4 shows the quadrants that can be used to identify the various types of value.

In Exhibit 4, the quadrants can be represented as follows:• Internal values: projects designed to run the business such as the creation of new forms,

guidelines, templates, checks, software, etc.

• Customer-related values: projects designed to satisfy specifi c customer needs in hope of follow on work or improvements in product health and safety factors

• Strategic values: projects such as R&D or new product development where the revenue stream will exist sometime in the future

• Financial values: projects designed to support the existing revenue stream such as product modifi cations or enhancements

Exhibit 4 Categories of Value

The title of the quadrants can change for each company based upon their industry and the business model they use. Project portfolio management practices try to balance the number of projects in each quadrant as well as the amount of funding for each quadrant. This is shown in Exhibit 5 where, as an example, 40% of the projects in the portfolio are strategic and consume 25% of the cost of all projects.

Priorities are then established for the projects in each quadrant and the business owner should brief the project team on what the priority is and the reason for the priority. The priority could dictate the quality of resources needed for assignment to each project. The relationship between projects, business value, and the project portfolio is extremely important.

Exhibit 5 Categories of Value by Project Cost and Number of Projects

Another form of value can appear in improvements in customer satisfaction. If customer satisfaction continuously improves, the chances for additional work from customers and stakeholders can be expected to increase as well. Improvements in customer satisfaction can also be used as a team motivational factor. The argument holds true for providing business owners more value than they had expected. These two situations can be represented with the same type of metric as shown in Exhibit 6.

Improving customer satisfaction and/or giving the business owner added value may not happen all of the time. Project failure will happen. A good metric to use in this regard is to track the percent increase in project success rate or the decrease in project failure rate. This is shown in Exhibit 7. This metric is important to the senior levels of management because it could provide an indication as to how successful are the projects.

Exhibit 6 Increase in Customer Satisfaction or Business Value Delivered

Exhibit 7 Increase in Project Success Rate

The Love/Hate Relationship:Too many companies end up with a love/hate relationship with metrics, especially metrics related to agility. Metrics can be used to shine a light on the accomplishments of the team by tracking performance, reporting the creation of business value, and identifying ways to reduce waste. Metrics can also be used to identify “pain points” which are situations that bring displeasure to business owners, stakeholders, and cl ients. The team then looks for ways to reduce or eliminate the pain points.

The hate relationship occurs when metrics become a weapon used to enforce a certain behavior. While good metrics can drive the team to perform well, the same metrics can create a hate relationship if management uses the metrics to pit one team against another. Another hate relationship occurs when the metrics are used as part of an employee’s performancereview.

Reasons for this type of hate relationship are the result of the following:• Metrics are seen as the beginning of a pay-for-performance environment.

• Metrics are the results of more than one person’s contribution and it may be impossible to isolate individual contributions.

• Unfavorable metrics may be the result of circumstances beyond the employee’s control.

• The employee may fudge or manipulate the numbers in the metrics to look good duringperformance.

Conclusions and Recommendations:In this white paper, we have just scratched the surface in the identifi cation of metrics. Metrics are a necessity with any and all project management approaches, including Agile and Scrum. However, given the number of possible metrics that can be identifi ed, companies must establish some guidelines to avoid metric mania conditions and love/hate relationships. Possible recommendations might be the following, which is certainly not intended to be an exhaustive list, but rather a starting point.

• Select metrics that are needed rather than what people think they might want without any justifi cation

• Select metrics that may be useful to a multitude of stakeholders, clients and business owners

• Make sure the metrics provide evidence and facts that can be used for decision making

• Make sure the metrics are used, rather than just nice to have displayed

• Do not select metrics where data collection will be time-consuming and costly

• Do not select metrics that create waste

• Do not use metrics where the sole purpose is for performance reviews and comparing one team against another

• Make sure that the metrics selected will not demoralize the project teams

About International Institute for Learning, Inc. (IIL)IIL is a global leader in training, consulting, coaching and customized course development, and is proud to be the educational provider of choice for many top global companies. IIL’s core competen-cies include Project, Program and Portfolio Management; Business Analysis; Microsoft® Project and Project Server; Lean Six Sigma; PRINCE2®; ITIL® and IT Security; Agile and Scrum; Leadership and Interpersonal Skills; and Sustainability and Corporate Social Responsibility.

Contact us today to request a free consultation.Website: www.iil.comPhone: +1-212-758-0177Email: learning@iil.com