big apple scrum day 2015 - advanced scrum metrics presentation

© 2012, Asynchrony Solu2ons, Inc. All rights reserved.

ADVANCED SCRUM METRICS Big Apple Scrum Day – June 1, 2015

Jason Tice

@theagilefactor [email protected]


Alterna4ve Title: As many as 30+ metrics other than velocity

that may help your team improve


Hypotheses

Some scrum teams do not have easy access to sufficient data to assess the true impact of process changes and promote effec2ve self-‐management. Without sufficient measurements encompassing all delivery ac2vi2es, scrum teams can fall vic2m to improving one aspect of their process while other aspects of their process/product suffer. Lack of encompassing quan2ta2ve measurements reduces team effec2veness as decisions may be made based upon the beliefs, theories and/or egos of team members without scien2fic data to assess the impact and effec2veness of decisions made.


5 Types of Scrum Metrics

Process Health •  Assess day-‐to-‐day delivery team ac2vi2es & evaluate process changes

Release •  Direct focus to iden2fy impediments to con2nuous delivery

Product Development •  Align product features to user needs

Technical / Code •  Determine quality of implementa2on and architecture

People / Team •  Promote sustainable pace and team engagement

Recommenda4on •  Teams track several metrics from each type to have a well-‐rounded view

of their process & product development ac2vi2es


Metric: Velocity

Type -‐ Process Health

What •  Number of work items completed in a period of 2me Why •  Assess the impact of team process changes Goal •  Velocity is intended to promote effec2ve self-‐managing teams


Disclaimer

Teams should track sufficient metrics to guide & evaluate improvements •  The right metrics are determined by a team and its sponsors

Adop4on of metrics should be incremental •  Do not aVempt to start gathering all 30+ metrics at once

All metrics have costs to gather & review •  A team’s metrics should evolve based on observed

challenges, risks, and needs •  Teams should track enough just enough metrics to

understand how they work and how they can improve


Why Measure

•  Assess the impact / value of process improvements using scien2fic methods:

•  Problem Statement •  Hypothesis •  Ac2on Plan •  Gather Data •  Conclusion

•  Increase effec2veness of self-‐organiza2on – data doesn’t lie and creates a founda2on for team members to make cri2cal decisions

•  Collec2on and transparency of data promotes cross-‐team knowledge sharing – data enables teams to have engaging conversa2ons about impediments and mi2ga2ons


Best Prac4ces

•  Metrics are about understanding trends – not hi`ng magic numbers

•  Respect that desired trends and/or quan2ta2ve goals are unique to each team

•  Goals should be arrived at by team consensus – not mandated by management

•  Use metrics to encourage team/value contribu2on vs. establishing baselines across all teams

•  Metrics should be transparent and easily accessible to all teams – ideally posted as informa2on radiators / dashboards

•  Use metrics to direct the team’s focus to agreed upon challenges – enable team members to self-‐diagnosis scenarios


Metric: Cycle Time


What •  Amount of 2me needed to complete a work item Why •  Consistent cycle 2me increases the predictability of work Goal •  Rela2vely consistent cycle 2me for each work item size


Metric: Cycle Time


Metric: Escaped Defects

Type -‐ Release

What •  Count of defects that are discovered in produc2on Why •  Iden2fy root causes as to why defect not detected prior to

release Goal •  Agreement on acceptable level of escaped defects with

shared understanding of mi2ga2ons iden2fied to reduce future escaped defects


Metric: Business Value Burnup

Type -‐ Product Development

What •  The amount of business value provided by each completed

work item Why •  Allow customers & stakeholders to manage ROI Goal •  Focused on highest value work items first; consider moving on

as value per work item decreases


Metric: Test Coverage

Type -‐ Technical / Code

What •  Percentage of codebase exercised by various types of

automated tests Why •  Guide efforts / investments to improve test coverage to

sufficient levels Goal •  Examine trends & correla2ons between Defect Density and

Test Coverage


Metric: Test Coverage


Metric: Happiness Metric

Type -‐ Team / People

What •  Team member sa2sfac2on as a member of the team (5 point

scale) Why •  Create transparency regarding team member sa2sfac2on Goal •  Enable team to self-‐manage / improve team morale


Metric: Happiness Metric


Metric: Cumula4ve Flow


What •  Amount of work-‐in-‐progress at any state at any 2me Why •  Iden2fy boVlenecks that reduce flow and assess impact of

process changes Goal •  Team operates within capacity as incremental progress made

toward goal (narrow bands with consistent posi2ve slope)


Metric: Cumula4ve Flow


Metric: Escaped Defect Resolu4on Time

Type -‐ Release

What •  Amount of 2me required to resolve an escaped defect Why •  Understand the unplanned cost of resolving escaped defects Goal •  Achieve balance between proac2ve tes2ng and sustainable

Service Level Agreements (SLAs)


Metric: Risk Burndown


What •  Amount of known and unmi2gated risk shown across a period

of 2me Why •  Encourage self-‐management to reduce project risk Goal •  Product risk trends down to acceptable level over 2me;

balance investment between building product and reducing risk


Metric: Risk Burndown


Metric: Build Time


What •  Execu2on 2me to run build and tests to provide developer

feedback Why •  Guard against slow builds & test execu2on that reduce

frequency of feedback Goal •  Build provides feedback to team within an agreed upon

amount of 2me


Metric: Learning Log


What •  A lis2ng of items the team (or team members) have learned Why •  Direct focus to the importance of learning on scrum teams /

projects Goal •  Promote learning throughout a project; suppor2ve of self-‐

management and sustaining team morale & engagement


Metric: Percent Complete & Accurate


What •  Number of completed and acceptable work items Why •  Improve delivery by measuring comple2on and quality Goal •  High percentage of commiVed work is complete & accurate •  Promote clear defini2on and consensus on done criteria


Metric: Release Success Rate

Type -‐ Release

What •  Ra2o of accepted vs. rejected releases from the customer Why •  Encourage partnership between team & customer Goal •  High percentage of accepted releases due to effec2ve

process, consensus on done criteria & feedback loops


Metric: Push / Pull


What •  The ra2o / count of work items completed vs. work items

added Why •  Guard against team being overwhelmed with work that can

compromise promise Goal •  Push / Pull is balanced – new work is added when work-‐in-‐

progress is completed


Metric: Push / Pull


Metric: Defect Density


What •  Percentage of defects in each area of our system –

determined by func2onality or code architecture Why •  Iden2fy parts of the app/code where quality can be improved Goal •  Data can help to iden2fy hidden / unknown tech debt, and/or

unclear development steps / done criteria


Metric: Team Tenure


What •  How long has each team member been on the team Why •  Encourage ac2vi2es reflec2ve of tenure (mentoring for new

team members, job/knowledge sharing for long-‐standing team members)

Goal •  Promote whole team approach and ability to shik staff

between teams with less risk


Metric: Flow Efficiency


What •  Ra2o of 2me spent working on an item vs. 2me the item waits Why •  Calibrate Work-‐In-‐Progress (WIP) limits to minimize delay and

promote flow Goal •  Consistent wait / run ra2o for each work item, then increase

efficiency


Metric: Release Time

Type -‐ Release

What •  Amount of 2me required to release the product to a

produc2on-‐like environment (or produc2on itself) Why •  Establish consensus on sustainable cost/2me for a produc2on

release Goal •  Sufficient automa2on to reduce 2me/cost for desired level of

business agility


Metric: Product Forecast


What •  Future trend lines (best-‐case, worst-‐case) based on historical

performance of work item comple2on Why •  Predict when future work will be completed using work item

count Goal •  Stable performance to allow for sufficient forecas2ng


Metric: Product Forecast (shown as burn-‐up)


Metric: Product Forecast (shown as interval)


Metric: Code Churn


What •  Number of lines of code changed to complete a work item Why •  Assess if the amount of code changed is reflec2ve of the work

item addressed Goal •  Promote whole-‐team understanding of the code-‐base and

alignment to agreed upon design / code paVerns & standards


Metric: Code Churn


Metric: Phone-‐A-‐Friend Stats


What •  Number of 2mes a former team member needs to be

contacted for assistance Why •  Assess effec2veness of job sharing and knowledge transfer

ac2vi2es when team composi2on changes Goal •  Encourage whole-‐team approach and shared work


Metric: Time Blocked per work item


What •  Amount of 2me that a work item was blocked during its

comple2on Why •  Determine the cost of delay & propose proac2ve mi2ga2ons /

avoidances Goal •  Reduc2on in 2me blocked per story through avoidance and

faster mi2ga2on


Metric: Time Blocked per work item


Metric: Time Since Last Release

Type -‐ Release

What •  Amount of 2me since the team last released their product to

“real” users Why •  Encourage teams to integrate more user feedback into

development ac2vi2es Goal •  Team is able to get sufficient feedback from “real” users to

build the “right” product


Metric: Product Net Promoter Score


What •  Would you recommend this product to a colleague? Why •  Gather simple user feedback on if the product meets user

needs Goal •  Iden2fy successful elements of product design and

opportuni2es / ideas to increase user adop2on


Metric: Product Net Promoter Score


Metric: Code Ownership


What •  Frequency that team members change or commit to each

area of the code base Why •  Assess and promote collec2ve code ownership Goal •  No no2ceable areas of the code where single team members

make all the changes


Metric: Whole Team Contribu4on


What •  Percentage of team members that contribute to a work item

throughout its lifecycle Why •  Quan2ta2ve metric to assess / improve whole-‐team approach Goal •  Work items completed with at least the agreed upon

percentage of team members contribu2ng to them


Metric: Blocker Clustering


What •  Frequency and grouping of items that block work items Why •  Iden2fy largest sources of delay and propose common

mi2ga2ons Goal •  Reduc2on in occurrence of types of blockers via proac2ve

mi2ga2ons


Metric: Cost Per Release

Type -‐ Release

What •  The cost to complete a sokware release (planned and/or

unplanned) Why •  Enable considera2on of economic factors when deciding

when/if to release Goal •  Encourage investments to reduce release cost via automa2on

to an agreed upon level


Metric: User Analy4cs


What •  Iden2fy usage paVerns within the product Why •  Determine effec2veness of design; look for emergent usage

paVerns that warrant considera2on for future investment Goal •  Augment business analysis and product design ac2vi2es with

actual user behavior


Metric: Code Complexity


What •  Cycloma2c complexity score of product code base

determined by a tool Why •  Promote engineering prac2ces to create clean code using

quan2ta2ve data Goal •  Enable team to assess and understand upward/downward

trend of code complexity


Metric: Code Complexity


Metric: Release Net Promoter Score

Type -‐ Release

What •  Would you recommend this product based on the new

features included with this release? Why •  Determine if new features align to user needs Goal •  Integrate user feedback from prior releases into future

enhancements; ability to collect feedback to improve product / adop2on


Metric: Standards Adherence


What •  Assessment score of code alignment to architecture

standards Why •  Promote agreed upon coding standards to create clean code Goal •  Enable team learning to beVer align code to agreed upon

design paVerns; iden2fy code that can be improved that may not be evident by code review


Metric: Standards Adherence


Metric: Release adop4on / install rate

Type -‐ Release / Product Development

What •  Number of exis2ng users that have upgraded; Number of new

users gained from release Why •  Assess ROI on product development and validate business /

market assump2ons Goal •  ROI meets or exceeds business assump2ons; iden2fy users

that have not upgraded – determine why


Metric: Crash Rate


What •  Log incidents that cause our applica2on / product to crash Why •  Be able to perform root cause analysis to reduce crashes Goal •  Gain insights into product errors in released code that have

significant impacts on the user experience


Your Metric Many @mes the best measurements are defined by team members who have the most knowledge of a specific problem or challenge

Type -‐ ?

What •  ? Why •  ? Goal •  ?


Conclusion

Next Steps: •  Are you measuring enough to assess the impact of changes /

improvements made?

•  Have a team discussion and decide if any of these 30+ metrics (or new ones you think up) would promote beVer self-‐management.

Ques4ons & Contact: Jason Tice – www.asynchrony.com @theagilefactor – www.theagilefactor.com [email protected]

big apple scrum day 2015 - advanced scrum metrics presentation

Software

cycletime typeprocesshealth

velocity typeprocesshealth