estimation bias and mitigation with agile estimate ... · harvard business review explains this...
TRANSCRIPT
Estimation Bias and MitigationWith Agile Estimate Guidance
2017 Edition China November 2017
Dan Galorath
+1 310 414-3222 x614
© 2017 Copyright Galorath Incorporated
Key Points
Every forecast is subject to estimation
bias…a major cause of
program failure and corporate mis-spending
Agile is a good
thing.
Estimates are still
important
Experts Bias is probably costing
in cost, schedule, and
less than hoped for benefits
2
Human Nature: YOUR PEOPLE Are Optimism Biased
Harvard Business Review explains this Nobel Prize Winning Phenomenon:
• Humans seem hardwired to be optimists
• Routinely exaggerate benefits and discount costs
• Bias permeates opinions & decisions & causes waste & failure
Delusions of Success: How Optimism Undermines Executives' Decisions (Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)
3
Solution - Temper with “outside view”:
Past Measurement Results, traditional forecasting, risk
analysis and statistical parametrics can help
Don’t remove optimism, but balance optimism and
realism
Mitigate Bias With Reference Class Forecasting (Source SEER-SEM)
Anchoring Experiment: Anchoring Biases Estimates (Source: myweb.liu.edu/~uroy/eco23psy23/ppt/04-anchoring.pptx)
1. Subject witnesses the number that comes up when a wheel of fortune is spun
2. Is asked whether the number of African countries in the U.N. is greater than or less than the number on the wheel of fortune
3. Is asked to guess the number of African countries in the U.N.
Result: those who got higher numbers on the
wheel of fortune guessed bigger numbers
in Step 3If given a number that biases estimates
Flaw of Averages Case Studies (Source: HBR)
• Example: $2 billion property damage in North Dakota
• U.S. Weather Service forecast that North Dakota’s rising Red River would crest at 49 feet.
• Made flood management plans based on this average figure
• In fact, the river crested above 50 feet, breaching the dikes, and unleashing a flood that forced 50,000 people from their homes.
© 2016 Copyright Galorath Incorporated 6
Average productivity estimates or average velocity can be very inaccurate
Beware business decisions with such data
Agile: Business Leaders Need Estimates for Decision MakingAnd Bias Against Consistent Measurement Is Hurting Industry
© 2017 Copyright Galorath Incorporated7
Agile: Detailed Software Development Life Cycle Management (Scrum Example)
• Focus is on what features can be delivered per iteration
• Not fully defined what functionality will be delivered at the end?
• Iterations are often called “Sprints”
© 2016 Copyright Galorath Incorporated 8
Agile Is Not a Silver Bullet http://www.jamasoftware.com/blog/rethink-agile-manifesto-projects-still-fail/
• Projects still fail at roughly the same rate as 2001
• Dr. Dobbs: Agile is not a productivity revolution
© 2016 Copyright Galorath Incorporated 9
0%
20%
40%
60%
80%
Agile (72%) Traditional (64%)8% Improvementnot Revolution
Agile 8% More Success
Some think agile is more successful because they remove cost & schedule
goals from the evaluation
Agile Software Project Performance Still Disappointing (McKinsey)
-80%
-60%
-40%
-20%
0%
20%
40%
60%
45% Over Budget 56% less Functionality thanplanned
McKinsey Study: Projects over $15m
© 2016 Copyright Galorath Incorporated 10
So does skipping estimating solve this?Well: If commitments aren’t made
there can be no disappointment
Story Points: Good For Sprint Planning, Lacking for Project Decisions(Source I Brown)
Story Point Benefits
• Excellent for sprint planning works well for planning sprints
• Immediacy of feedback from previous sprint
• Encourages communication & expectation management
Challenges
• Generation of estimates to establish project budgets
• New development team with no history
• No consistent portfolio management metrics across organization
© 2016 Copyright Galorath Incorporated 11
Story Points Ineffective for Absolute Effort Estimation (Comyene-Abran)
© 2016 Copyright Galorath Incorporated 12
Relative Error% - (realEffort –Estimated Effort / RealEffort
Agilons: Standardized Sizing For Agile Projects (Source: SEER, I Brown)
• As a customer I would like to have the ability to search for and reserve a hotel room in order to spend the night in another city
• Using Agilon
• User story seems to have multiple Agilontypesthat need to be decomposed
• Hotel data (10 Agilons)
• Search for hotel room (4 Agilons)
• Reserve hotel room (4 Agilons)
• Total of 18 Agilons
• •If my team’s velocity is around 18 Agilons per sprint, we’re good to go…
© 2016 Copyright Galorath Incorporated 13
Agile Trying To Kill Estimates Without Considering Business Needs
• #noestimates
There Are Times When #Noestimate Is Appropriate
• A promise of 5 sprints with 4 people is good enough
• Projects are small and time is not concerning
Unsubstantiated Promise Good
Enough
•Willing to spend in whatever time
•No Business impact from software
•No concern with failure and contingency
•Resource optimization not concerning
•No system testing on top of development
Cost / Schedule not a
consideration
• You don’t need to consider total ownership costs
• If lack of documentation is ok for maintenance
No Maintenance COncern
© 2016 Copyright Galorath Incorporated 15
Manual Estimates: Human Reasons For Error EVEN IN AGILE SPRINTS (Goldratt)
• Manual Task estimates yield SIGNIFICANT error
• Desire for “credibility” motivates overestimate behavior (80% probability?)
• Then must spend all the time to be “reliable”
• Better approach: force 50% probability & have “buffer” for overruns
• Technical pride sometimes causes underestimates
16
Viable up-front estimates provide agile teams with fair terms
Agile Large Systems Back To Waterfall (Estimation & Planning Should Consider Hybrid)
• BACK TO WATERFALL or HYBRID
• UK’s Universal Credit Welfare System , “a complex IT project that involves switching off multiple benefits and reworking them into a new tax credit system” was, by most accounts, the most ambitious agile software development project in history. Suppliers include Accenture, Cap Gemini, HP, and IBM
• Quoting Computer Weekly “ DWP drops agile from flagship government software project”
• Cast Software research found that applications produced using traditional Agile or Waterfall methods alone have more security vulnerabilities, more reliability and performance issues, and a higher cost to maintain than those produced with a mixed method.
© 2016 Copyright Galorath Incorporated17
Hybrid Agile Development (Source IPPS-A)
Ag
ile T
errit
ory
Develop
Detailed Design
UnitTesting
Build
PDR -> CDR -> PLT
Requirements Analysis
Fit/Gap
Integration Testing
Operational Testing
Plan Analyze Deploy Run
Waterline
SRR -> SFR -> IBR TRR -> DIT -> GAT -> LUT -> OPT -> DEP
• Hybrid Agile - Establishes a waterline below for Agile development
• From Planning and Requirements Analysis down to Agile Territory for the Build and then back up to Testing for final Deployment
#noestimates Viable For Detailed Development -Should Not Abdicate In Substantial Developments
Business CaseEvaluation of alternatives
Agile or Hybrid Agile
Software Development
System Test (when
appropriate)
Maintenance & Support
© 2016 Copyright Galorath Incorporated 19
Agile development = root level software development management…
Story point estimating is short term productivity management
It is not a business decision making process
How Much? How long?Ownership
CostGo / no go
Hybrid Agile: Requirements
& Design
For substantial systems
Agile Developments Should Consider Risk and Probability
• For Estimation the Cone is the same – AND Loss Aversion may keep projects alive that should be killed
ROM Detailed Estimate
© 2016 Copyright Galorath Incorporated 20
with Data& SEER
EstimationBias
Strategic Mis-
estimation
When Making Management Decisions Remember: Software Often Less Than 10% Total Ownership Cost
SoftwareDevelopment
SoftwareMaintenance
IT Infrastructure
IT Services
© 2016 Copyright Galorath Incorporated 21
Development =Biggest Risk
Software Development is about 6-10% of total ownership cost…But much more of the risk
Assume $10m development could be over $100m total ownership
IT Services & Infrastructure Are Situational
but Generally 60% of TOC
Initial Backlog is Fraught with Uncertainties
Uncertainty Is Best Answered By Parametric Models
Known & Unknown Constraints
People SkillsProcess KnowledgeNew Technology
Customer AvailabilityMarket Uncertainty
Agile Can Only Help Mitigate This
© 2017 Copyright Galorath Incorporated
SEER-SEM Predictive Analytics Provide Outside View: Agile Estimates & Risk Analysis
Using Agile Artifacts: Story Point and Epoch Sizing
Estimate Using Project History:
• Development
• Defects
• Total Ownership Cost
• Estimate Backlog
• Estimate Maintenance Effort
• Single or parallel Agile teams
Agile Risk Still Needs Consideration
Least, likely, and mostinputs provide a range of
cost and schedule outcomes
Confidence (probability)can be set and displayedfor any estimated item
• SEER predictsoutcomes
• SEER uses inputs to develop probability distributions
• The result is a probabilistic estimate
• SEER will predict a likely range of outcomes
• Monte Carlo provides project-level assessments of risk
© 2016 Copyright Galorath Incorporated 24
Bia
s M
itig
ation:
Ris
k
SEER-SEM As Crosscheck to Supplier Quotes
• Example:
• 300 Functions 4GL
• Web, BI
• Team Size 3 FTE x 1 Month
• Reference is without Schedule Adjustment
• Current Estimate with Schedule Adjustment
Earned Value Management Process
INTEGRITY INNOVATION EXCELLENCE 26
How Galorath Applies Earned Value Management To Large Agile Systems
• As long as there is a plan and product, which can be measured EVM can be used
• Work planned and status rolled up at the EPIC Level (Often Control Account (CA))
• Features at the Work Package (WP) level breaks EPIC into functional packages
• Sprints statused by Stories and Story Points(statused in Agile Program Management Tool)
• Sprints are the process to the completing the Story(s) and Stories are Steps to completing the Feature, which rolls into the EPIC
• As Features are completed the percent complete is rolled up to the EPIC level
© 2017 Copyright Galorath Federal Incorporated 27
Agile Earned Value Management Building Blocks*
Sprints
Epic 1Feature 1
User Story 1
Epic 2Feature 2
User Story 2
Epic 2 Feature 3
User Story 3
Theme/Increment 1
Release 1 (made up of multiple Themes/Increments
Cost Estimating done at the Sprint Level
EVM work Packages
identified at Epic or
Theme level
* These “building blocks” are program specific and may be called by different names
Feature Point
values applied to each Feature
INTEGRITY INNOVATION EXCELLENCE 28
Back to this ProblemHow Much? vs. Show Me The Money
Product Owner
• Has a need and time to market expectations
• Can describe problem in large generalities
• May or may not be able to be proactive in development
• Needs to set budget for project approval
Development Staff
• Agile is perfect for everything mindset
• Tell us what you want
• Tell us what priority it is needed
• We’ll make a guess at how much it will cost
• Now show me the money
© 2017 Copyright Galorath Incorporated
Key Points
Every forecast is subject to estimation
bias…a major cause of
program failure and corporate mis-spending
Agile is a good
thing.
Estimates are still
important
Experts Bias is probably costing
in cost, schedule, and
less than hoped for benefits
30
Bibliography (Partial)
• FLYVBJERG, BENT, Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice, European Planning Studies Vol 16 No 1, Jan 2008,
• Kahneman, Daniel, and Amos Tversky. “Prospect Theory: An Analysis of Decision under Risk.” Econometrica 47, no. 2, March 1979.
• Johnson, H. Thomas. Relevance Regained. Free Press.
• Mislick, Gregory K.; Nussbaum, Daniel A.. Cost Estimation: Methods and Tools (Wiley Series in Operations Research and Management Science) (p. 143). Wiley.
• Rose, Todd. The End of Average: How We Succeed in a World That Values Sameness, HarperCollins.
• Taleb, Nassim Nicholas. Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets (Incerto). Random House Publishing Group.
• Glen, Paul; McManus, Maria. The Geek Leader's Handbook: Essential Leadership Insight for People with Technical Backgrounds. Leading Geeks Press.
• Kogon, Kory; Merrill, Adam; Rinne, Leena. The 5 Choices: The Path to Extraordinary Productivity. Simon & Schuster.
• Patterson. Crucial Conversations Tools for Talking When Stakes Are High, Second Edition. McGraw-Hill.
• Kahneman, Daniel. Thinking, Fast and Slow. Farrar, Straus and Giroux.
• Hubbard, Douglas W.. The Failure of Risk Management: Why It's Broken and How to Fix It . Wiley.
• Hubbard, Douglas W.. How to Measure Anything: Finding the Value of Intangibles in Business . Wiley.
© 2016 Copyright Galorath Incorporated 31
Backup Slides
© 2016 Copyright Galorath Incorporated32
IT Failure Can Impact Business Dramatically (Source: HBR)
• Case Study: Levi Strauss
• $5M ERP deployment contracted
• Risks seemed small
• Difficulty interfacing with customer’s systems
• Had to shut down production
• Unable to fill orders for 3 weeks
• $192.5M charge against earnings on a $5M IT project failure
“IT projects touch so many aspects of organization they pose a new singular risk”
http://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think/ar/1
© 2017 Copyright Galorath Incorporated 33
Cognitive Bias: Bias Guesses and Forecasts (Source BeingHuman.org)
Cognitive bias
• Tendency to make systematic decisions based on PERCEPTIONS rather than evidence
• “Perception has more to do with our desires—with how we want to view ourselves—than with reality." Behavioral economist Dan Ariely
Researchers theorize in the past, biases helped survival
• Our brains using shortcuts (heuristics) that sometimes provide irrational conclusions
Bias affects everything:
• from deciding how to handle our money
• to relating to other people
• to how we form memories
© 2016 Copyright Galorath Incorporated 34
Observations
• Agile is an excellent software development life cycle approach
• For substantial developments management decisions and commitments are still critical
• For some managers at least Agile keeps them from having to think about software (problem and sort of benefit)
© 2016 Copyright Galorath Incorporated 35
Working software maturing
COUNTERPOINT: #noestimate: No Problem?
Agile can sometimes remove software worries from the C level
Every Decision is a Forecast (L. Maccherone, AgileCraft)
• You are forecasting that your choice will have better outcomes than other alternatives
• So quality of decisions depends on
1. Alternatives considered
2. Processes and models used to forecast the outcome of these alternatives…
• Probabilistic models are superior
• http://www.slideshare.net/lmaccherone/you-want-it-when-probabilistic-forecasting-and-decision-making
© 2016 Copyright Galorath Incorporated 37
An Agile Approach to Planning (Source: Cohn)1. Projects rapidly generating useful new capabilities and
new knowledge
2. Flow of new capabilities & knowledge to guide work
3. Plan for what you want to learn – not what the product will be in the end
4. Traditional projects are like a 10K race – you know where the finish line is
• Get there as fast as possible
5. Agile projects are likea timed race
• See how far you can runin sixty minutes!
• Iterate until product owneris satisfied
© 2016 Copyright Galorath Incorporated38
Management Agile Manifesto #EstimatesSupportBusiness• We know you are technically capable… please help:
• Stop overpromising and under delivering
• Support management ROI, schedule needs and investment strategy
• Solve our problem
• Ensure it works, is usable, and secure
• Consider total cost to the business, not just initial costs
• Don’t waste too much resource
• “Base choices on those providing the maximum business value to the organization” Eli Goldratt
© 2016 Copyright Galorath Incorporated 39
Give Management The Whole Picture: Total Ownership Costs Includes On-Going Support
Sprint or ReleaseBacklog
Iteration to Build
CodeTest
Refactor
Delivered Functionality
Working System
n-Iterations per Release Warranty & MaintenancePlanning
User Stories, Use Cases, Business
Requirements, Etc.
Fixes, Enhancements, Sustainment
Defects & Unfinished Work
Collection of Functionality
Many programs need total ownership cost evaluation... Estimation is essential
Development + Maintenance
Identify Total Ownership Costs for the Software
Allows Independent Maintenance Team
Assumptions
Estimate Cost of: Corrective, Adaptive,
Perfective, and Enhancement support
The Planning Fallacy (Kahneman & Tversky, 1979)
•Manifesting bias rather than confusion
•Judgment errors made by experts and laypeople alike
•Errors continue when estimators aware of their nature
Judgment errors are systematic & predictable,
not random
•Underestimate costs, schedule, risks
•Overestimate benefits of the same actionsOptimistic due to overconfidence
ignoring uncertainty
•“inside view” focusing on components rather than outcomes of similar completed actions
•FACT: Typically past more similar assumed
•even ventures may appear entirely different
Root cause: Each new venture
viewed as unique
© 2016 Copyright Galorath Incorporated 41
Bias Mitigation Reference Class Forecasting
Attempt to force the outside view and
eliminate optimism and
misrepresentation
Choose relevant “reference class”
completed analogous projects
Compute probability distribution
Compare range of new projects to
completed projects
Provide an “outside view” focus on
outcomes of analogous projects
© 2016 Copyright Galorath Incorporated 42
Best predictor of performance is actual performanceof implemented comparable projects (Nobel Prize Economics 2002)
Predicts outcome of planned action basedon actual outcomes in a reference class:similar actions to those being forecast.