it data collection, analysis and benchmarking: from ...€¦ · produce business value to the...
TRANSCRIPT
IT Data Collection, Analysis and Benchmarking: From Processes and
Benchmarks into Real Business Value
Dan Galorath
Copyright 2013 Galorath Incorporated
Key Points
Measurement needs to think like business
leaders… provide
information understandable to management
and other stakeholders
Measurement needs to produce value not cost in the eyes of management and other stakeholders
Measurement results should include confidence and risk so leaders can understand and include in decisions
Context
© 2011 Copyright Galorath Incorporated
3
•Cutter Consortium Software Project Survey:• 62% overran original schedule by more than 50%
• 64% more than 50% over budget
• 70% had critical product quality defects after release
• Standish Group CHAOS Report• 46% challenged
• 19% failed
• 35% successful
~$875 billion spent on IT~$300 billion spent on IT projects ~$57 billion wasted annually
ROI of better planning huge
IT Failures Are Pervasive: And Even Successful Projects May Not Be
Some suggest 80% of systems NEVER produce a positive ROI
Human Nature: Humans Are Optimists
HBR Article explains this Phenomenon:
• Humans seem hardwired to be optimists
• We routinely exaggerate benefits and discount costs
Delusions of Success: How Optimism Undermines Executives' Decisions (Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)
5
Solution - Temper with “outside view”:Past Measurement Results, traditional forecasting,
and statistical parametrics can help
Don’t remove optimism, but balance optimism and realism
Example of Tempering: German Mark V Tanks (Source: How To Measure Anything)
• Allies estimated production by analyzing serial numbers
• Used the captured tanks as a random sample and predicted probability of various production levels
Some of Dan’s Heroes
• Frederick Taylor: The Principals of Scientific Management 1901 “Let data and facts do the talking”
• W. Edwards Demming: “In God We Trust… All Others Bring Data”
• Frederick Brooks: “There is an incremental person when added to a software project that makes it take longer”
• Ed Yourdon: “Avoiding Death Marches in Software Projects”
• Steven Covey: “Sharpen the Saw” Focus on improvement
Measurement Manifesto For Software & IT
Measurement Perspective
• We measure to ultimately produce business value to the organization
• MEASURED Measurement should not ultimately be a cost
• The analysis of measurements produces decisions that produce business value
Management & Stakeholder Perspective
• Speak so I can understand
• Give me actionable items
• Don’t just give me problems… Give me solutions
• Help me make the best decisions so we can produce business value
© 2011 Copyright Galorath Incorporated 8
Measurement itself is N ENABLER….
Management decision making, better performance, quality improvements and better serving stakeholders is
Example: Project Cost Alone Is not The Cost of IT Failure (Source: HBR)
• Case Study: Levi Strauss
• $5m ERP deployment contracted
• Risks seemed small
• Difficulty interfacing with customers systems
• Had to shut down production
• Unable to fill orders for 3 weeks
• $192.5M charge against earnings on a $5m IT project failure
© 2011 Copyright Galorath Incorporated 9
“IT projects touch so many aspects of organization they pose a new singular risk”
http://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think/ar/1
Exuding Confidence or Concern With Measurement
© 2009 Copyright Galorath Incorporated 10
Frederick Brooks Classic Paper “No Silver Bullets”• “There is no single development, in either
technology or management technique, which by itself promises even one order-of magnitude improvement within a decade in productivity, in reliability, in simplicity.”
• -- Fred Brooks, 1986
• i.e. There is no magical cure for the “software crisis”
• Not software measurement
• Not better tools
• Not ---- (fill in the blank… the current great hope)
Size & Productivity
© 2011 Copyright Galorath Incorporated
12
Fundamental Metrics For Estimation, Planning & Control
• Size
• AKA Volume, Mass
• Units: Source Lines of Code (SLOC); Function Points (FP) Use Cases
• New versus rework
• COTS & Packages
• Technology
• AKA Productivity Potential, Efficiency
• Units: none
• Time
• AKA Duration, Schedule
• Units: Calendar Months, Calendar Weeks
• Effort
• AKA Work, Labor
• Units: Staff Months, Staff Hours
• Cost
• AKA Budget, Money
• Units: $, other currencies
• Staffing
• AKA Manpower Loading
• Units: FTE People
• Defects
• AKA Reliability, Quality
• Units: Defect Count
Many Size Measures Are Viable
• We know about:
• Lines of Code &
• COSMIC Both provide a method for Sizing Software
• Story Points (different since their values are team defined) provide the same capability for User Stories or Features or anything that can sized
• Many others
• Function Points
© 2012 Copyright Galorath Incorporated 14
A1-15
Size Is More Than Just Total Application
NewNew
Effective Effort Units
Lines, Functions, Objects, Use Cases, Etc.Lines, Functions, Objects, Use Cases, Etc.
Pre-existingPre-existing
Reuse: Watch Out For Low Cost Assumptions on “Heritage”
• Reuse or Heritage: applying existing software to a new mission (or additional innovation in its current mission)
• Effort to reuse software is routinely under estimated
Design
Test
ImplementationWhy should we care: Heritage is often underestimated and causes major schedule / cost overruns
Sample Effort Factors for Reuse
© 2013 Copyright Galorath Incorporated 17
Software Design For Reuse (Lower Cost Heritage)
• Designed for reuse• Reusability requirements during original development
• Significant extra coding and documentation effort during original development to insure reusability
• Minor to moderate work (mostly retest) required when reused IF NOT MODIFIED
• Not designed for reuse• Developed without extra effort to ensure reusability
• Sources include any legacy code not specifically designed for reuse
• Other projects & programs
• Prior builds of the current program
• Moderate to major rework required• Redesign
• Reimplementation
• Retest
Rework Causes Software Project Issues
• Rework: Doing the same work over again because it was incorrect the first time
• Not prototyping
• Not refactoring (tuning to make software better)
• Between 40% and 50% of the total effort on software projects is spent on rework.. Barry Boehm
• Initiatives to reduce rework can save significantly
Why should we care: Unplanned rework can drive cost up dramatically AND reducing rework can be a boon to projects
Customization
COTS or GOTS Hoped For Reuse
• Developmental Software:
• Functionality developed specifically for the project at hand
• May include customization of COTS
• “Glue” Code:• Code written to bind
COTS to developmental software
• Development effort must be captured
• COTS Software:• Purchased functionality
• Direct Cost component of COTS integration
• COTS Cognition: • Required functionality within the COTS
software that must be understood
• Effort component of COTS integration
DevelopmentalSoftware
COTS Software
“COTS Cognition”
Glue
Why should we care: COTS promises are often not accurate meaning additional development cost & Schedule
Packaged Applications Are Costly To Organizations
• “Commercial application program or collection of programs developed to meet the needs of a variety of users, rather than custom designed for a specific organization”
• Many are enterprise applications
• Often allows / requires customization
• Examples: SAP; Rational PPM, SEER for Software; Microsoft Excel, CA Clarity, Oracle Business Suite
Why should we care: Packages sometimes comprise solutions for parts of complicated systems and can be trouble
"One-third [of the budget] has to go to testing. Don’t ever short change testing. Everyone always underestimates it, and says it’s the last thing to worry about. Don’t do that!“
- Jim Larson, consultant for communications solutions provider
Software Productivity Laws
• LAW 1 – Smaller teams are more efficient: The smaller the team the higher the productivity of each individual person.
• LAW 2 – SOME schedule compression can be bought: Adding people to a project, to a point, decreases the time and increases the cost.
• LAW 3 – Every project has a minimum time: There is an incremental person that consumes more energy than he/she produces. Team size beyond this point decreases productivity and increases time. (aka Brooks’ Law--”Adding staff to a late software project makes it later.”)
• LAW 4 – Productivity is scalable: Projects of larger software size can use larger teams without violating LAW 3.
• LAW 5 – Complexity limits staffing: As complexity increases, the number of people that can effectively work on the project and the rate at which they can be added decreases.
Software Productivity Laws (continued)• LAW 6 – Staffing can be optimized: There exists an optimal
staffing function (shape) that is effectively modeled by the Rayleigh function. Flat (level load) staffing is rarely optimal.
• LAW 7 – Projects that get behind, stay behind: It is extremely difficult to bring a project that is behind schedule back on plan.
• LAW 8 – Work expands to fill the available volume: It is possible to allow too much time to complete a project (aka Parkinson’s Law).
• LAW 9 – Better technology yields higher productivity: More capable teams, better tools, and advanced, stable processes yield higher productivity.
• LAW 10 – No “silver bullets”: There is no methodology, tool, or process improvement strategy out there that yields revolutionary improvements in project efficiency.
The Bottom Line
• Effort for reuse and rework must be considered
• Standard definitions of productivity e.g.
• 18 hours per function point…
• Is necessary but not sufficient for detailed estimates that must be viable
• Use a range
• Identify key cost drivers that impact productivity
• Ensure you know what is IN and NOT IN the hours
© 2013 Copyright Galorath Incorporated 24
Relative Sizing Can Provide Quick ROM Numbers E.G. ISBSG
• War and Peace is 587,287 words
• How long will it take you to read it?
• Using a different reference try…
• Girl with the Dragoon Tattoo =144,000
• The Bourne Identity = 80,000
• Harry Potter and the Philosopher's Stone = 76,944
© 2012 Copyright Galorath Incorporated 25
Data Collection, Normalization, Usage, Caution
© 2013 Copyright Galorath Incorporated 26
Before You Start - Some Basic Rules
1. Know from where the data comes (total, effective, new, etc?)
2. Know what is included (Phases, People, Processes, …)
3. Know the different sizing schemas and how applied
4. Know what was intentionally left out (and why)
5. Consider the hindsight – if things happen they should be accounted for in the data collection
6. Data needs to be consistent or normalized
Galorath’s Data Review Process
Management is excluded from actual
Concept is excluded
from actual
CM / QA is excluded
from actual
…management from the estimate
…concept from the estimate
…CM / QA from the estimate
THE ACTUAL You should exclude...
Productivity Reasonable ?
Is Effort Accounting By Phase Probably
Complete?
Summary EffortMatch Accounting
By Phase?No No No
Adjust For Missing Phases
Is There Effort Accounting By
Phase?
Yes
YesYesYes
No
Do Not Adjust
Do Not Adjust
Do Not Adjust
Is“Project Activity
Scope” fieldComplete?
No
Do Not AdjustYes
Productivity Reasonable ?
Is Effort Accounting By Phase Probably
Complete?
Summary EffortMatch Accounting
By Phase?No No No
Adjust For Missing Phases
Is There Effort Accounting By
Phase?
Yes
YesYesYes
No
Do Not Adjust
Do Not Adjust
Do Not Adjust
Is“Project Activity
Scope” fieldComplete?
No
Do Not AdjustYes
Normalization
Mapping
29
Data Must Be Used With Caution
• Run sanity checks on data
• 10,000 Function Points can’t be developed in 3 months
• Ongoing issue between statisticians and engineers
• Some Statisticians claim.. “That is what the data says so it must be right”
• Sometimes even if it is obviously wrong
Data Doesn’t Have To Be Perfect To Be Useful: But Is Has To Be Viable
• 80 Calories per serving
• 2.5 Servings per can
• 4 Ounces, Condensed, 8 Ounces With Water
31
Data Improves Estimates For New Programs Source: John Vu, Boeing SEPG 1997
.
0 %
140%
-140%
..
..
.
..
..
.
..
.
. .
. . . .
.
. . .
. .
.
.
. . . .. . . . . . . ...
. . . .. .
.
. .
. ..
.
.
.
. .. .. ...... . .. . ... . ...
.. .
.
Without Historical Data With Historical Data
Variance between + 20% to - 145% Variance between - 20% to + 20%
(Efforts = Labor Hours)
(Mostly Level 1 & 2) (Level 3)
Ove
r/Und
er P
erce
ntag
e
.
(Based on 120 projects in
Boeing Information Systems)
.
. . .
.
.
..
..
.
. .
..
.
.
. .
.
.
.. .
. . .. . . . . .. . . . . .. .
..
. . .. . .. . . .
. . . .. . . . .
. . . . .
. . . . .. . . . . .
. . . . .. . . . . .
. . . . . .. . . . . . . .
. . .. . . . .
. . . . .. . . .
. . . . . .
. . . . . .
. . . . . .
John Vu, Boeing, keynote talk at SEPG ‘97,
“Software Process Improvement Journey (From Level 1 to Level 5)”
Just the Top Level Data Isn’t Sufficient For Estimation
Hours Versus UFPs
1
10
100
1000
10000
100000
1000000
1 10 100 1000 10000 100000UFPs
Hou
rs
Use Historical Data to evaluate your estimate!
The Error of Causal Analysis Is often encountered in data analysis!
NORMALIZE:There is no correlation that estimate of size ‘X’ is an
equivalent of historical projects of about ‘X’.
Code and Unit Test Only
System and SW Requirements
were done separately
This point does not include SW Testing Effort
Fallacy of Silent EvidenceWhat about what we don’t know?
How confident would you feel if the Silent Evidence was visible?
36
Software Has Additional Characteristics That Should Be Considered
Track defect discovery and removal rates
against expected rates
Increased defect reporting rate
shows a worsening trend
Heath and Status Indicator shows status and trends from
the previous snapshot
Thresholds are user definable
Watch Total System Requirements: IT Systems Total Ownership Costs; 60+% Can Be Infrastructure & Services
• Software Development
• Software Maintenance
• IT Infrastructure
• IT Services
• Other
Measurement Results must provide real analysis
Risk and Uncertainty
© 2013 Copyright Galorath Incorporated 38
39
Statistician Drowns in River with Average Depth of 3 Feet!
Software Is A Critical Component of Military & Aerospace Systems Failures Ariane 5
• Ariane 5 video
• Note: $500 Million payload
• Failure due to reused software from (Ariane 4)
• with incompatible assumptions for exception condition that was not required
ariane_5_explosion_video.mp4
“the culture within the Ariane program…” only addressed “random hardware failures… which can quite rationally be handled by a backup system”“the view had been taken that software should be considered correct until it is shown to be at fault”!
Why should we care: Software cost & schedule should be sufficient for successful missions
Minimum
Most Likely:
Maximum
If everything goes well (optimistic
estimate)
If everything goes badly
(pessimistic estimate)
Using Ranges To Address Uncertainty
pert beta Incremental Likelihood
Pert Beta
Adding Reality to Schedules Example (Source: SEI)
Process DurationsStep Expected1 302 503 804 505 906 257 358 459 7010 25
500
What would you forecast the schedule duration to be?
Adding Reality to Schedules – Example - 2
Process DurationsStep Best Expected Worst1 27 30 752 45 50 1253 72 80 2004 45 50 1255 81 90 2256 23 25 637 32 35 888 41 45 1139 63 70 17510 23 25 63
500
What would you forecast the schedule duration to be now?
Adding Reality to Schedules – Example - 3
With 90% confidence, the project will be under 817 days duration.
The project is almost guaranteed to miss the 500 days duration 100% of the time.
Adding Reality to Schedules – Example - 4
With only 50% confidence, the project will be under 731 days duration.
Typical Growth Range From Initial Sizing To Delivery
• Functional sizing for estimates should include growth range to mitigate risk
© 2011 Copyright Galorath Incorporated 46
Dealing With the “Problem of Assumptions”• Assumptions are essential but…
• Incorrect assumptions can drive an estimate to uselessness
• Use an assumption verification process
© 2012 Copyright Galorath Incorporated 47
Estimates must have assumptions defined but…Bad assumptions should not be justification for
bad estimatesASSUMPTIONS MAY BE RISKS
Business Value Is Part of Measurement & Estimation
© 2011 Copyright Galorath Incorporated
48
Software Projects Must Return Business Value“Economics is primarily a science of choice…
software economics should provide methods for analyzing the choices software projects must make.” Leon Levy
“Base choices on those providing the maximum business value to the organization” Eli Goldratt
Measurement and its uses such as estimating and defect analysis help this science of choice
• Some say business value is not our problem
• While others generally need to perform benefit analysis
• We need to build systems that optimize the business
• Make IT part of the solutionIf IT & measurement don’t generate sufficient profit money will go elsewhere
The Gap Between IT & Management Needs (Source: Fraunhofer)
© 2011 Copyright Galorath Incorporated 50
Financial
Customer Business Process
Innovation / Growth
Vision & Strategy
Goals
Data Collection
MeasurementMetricsAnswersQuestions
Goal Attainment
Software Development & Maintenance
6σ
GQM
PSM
CoBIT…
Business/OrganizationLevel
ITGovernance
ITGovernance
CustomerFocus
Software & IT Systems Are About Business Value
© 2009 Copyright Galorath Incorporated 51
Generating the Business Value Side of the Equation (Benefits)• The business owns benefit calculations
• IT should participate
• Exception: projects solely improving internal IT
• Beware of subjectivity translating soft benefits
• Use probability and risk
© 20011 Copyright Galorath Incorporated 52
Quantify Intangible Software Benefits & Costs When Possible … But Be Realistic; Examples
• Improved Customer Satisfaction: May yield Improved Customer Retention
• Higher Quality Product: May yield lower rework and higher customer satisfaction
• Improved Decision Making: May yield many things
• Improved Win Ratio: Yields additional revenue
• Time to Market: May improve profit/market share
© 2009 Copyright Galorath Incorporated 53
An ROI Analysis of A New System
© 2013 Copyright Galorath Incorporated
54
Cost of capital 8.0%
Initial Investment Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7Total
Ownership
Investment $100,000 $100,000
Increase/(dec.) in revenue ($40,000) $60,000 $110,000 $100,000 $100,000 $150,000 $150,000 $630,000
Increase/(dec.) in op. exp. $90,000 $70,000 $70,000 $22,000 $24,000 $27,000 $28,000 $331,000 Cash Flow ($100,000) ($130,000) ($10,000) $40,000 $78,000 $76,000 $123,000 $122,000 $199,000
PV of Cash Flow ($100,000) ($120,370) ($8,573) $31,753 $57,332 $51,724 $77,511 $71,186 $60,563
NPV 60,563 $60,563
IRR 13.5% 13.5%
ROI 121% 121.1%
The Rule of Measurement
When performance is measured Performance improves…
When performance is measured and reported the rate of improvement accelerates..
Thomas Monson
The work you do is a root cause of program success