hubbard decision research the applied information economics company intro to quant. methods finding...
TRANSCRIPT
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Quantitative Methods Make A Difference
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Overview
• Quantitative methods (probabilistic analysis, operations research, etc.) are widely used in other industries, but mostly lacking in IT investment analysis
• Over the past 7 years, we have been focusing specifically on the application of more advanced quantitative methods to IT
• This presentation will review the key findings from the application of quantitative methods to over 30 projects
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Quantitative Methods Include:
• Computing uncertainties and risks
• Computing the economic value of information
• Measurement methods for many items usually considered intangible
• Optimizing solutions when there are huge combinations of options for:– Roll-out priorities of systems
– Selection of a portfolio of functions
– Any other problem where different alternatives about different aspects of the investment generate lots of possibilities
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
AppliedAppliedInformationInformationEconomicsEconomics
EconomicsEconomics
Decision/Game Decision/Game TheoryTheory
Empirical Empirical Decision TheoryDecision Theory
StatisticsStatistics
Information TheoryInformation TheorySoftwareSoftwareMetricsMetrics
InformationInformationEngineeringEngineering
Modern PortfolioModern PortfolioTheoryTheory
Operations Operations ResearchResearch
Method: AIEApplied Information Economics is the practical application of
scientific and mathematical methods to quantify the value of IT-enabled business investments
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Some HDR Clients• Booz-Allen & Hamilton• Accenture w/ the State of North Carolina• Giga Information Group • American Express • The Discovery Channel• U.S. Federal Government:
– Department of Treasury– Bureau of The Census– Department of Veterans Affairs– General Services Administration– Housing and Urban Development
• The Axa Group – 6 major companies• The Banking Industry Technology Secretariat • Blue Cross Blue Shield of Illinois
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
What Do the Critics Say?
• “Quantifying the risk and comparing its risk/return with other investments sets AIE apart from other methodologies. It can substantially assist in financially justifying a project -- especially projects that promise significant intangible benefits.” The Gartner Group
• “AIE represents a rigorous, quantitative approach to improving IT investment decision making…..this investment will return multiples by enabling much better decision making. Giga recommends that IT executives learn more about AIE and begin to adopt its tools and methodologies, especially for large IT projects.” Giga Information Group
• “AIE-like methods must become the standard way to make (IT) investment decisions.” Forrester Research, Inc.
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Five Key Revelations1. Quantifying risk radically changes IT investment
priorities2. Current measurement priorities are “upside-
down” when compared to priorities based on economic value of information
3. “Technology regret” is a significant and overlooked factor in the the value of IT investments
4. The true cost of “scope creep” is much higher than most would think
5. The value of quantitative analysis would make it the best investment in most IT portfolios
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Finding 1: Risk Analysis• When IT computes risk in the same way
that an actuary would, many IT investments will look completely different
• We define risk a “The probability of a quantified loss”
• Risk analysis of IT investments involves a probabilistic analysis of all uncertain variables
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Normal Distribution
Uniform Distribution
Lognormal Distribution
Hybrid
Threshold confidence 15% 85%
Ideal Values: Point
Real-world Meas.
Real-world Measurements vs. Ideal Values
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
When asked to provide a subjective 90% confidence interval, most managers providea range that only has about a 40%-50% chance of being right
When asked to provide a subjective 90% confidence interval, most managers providea range that only has about a 40%-50% chance of being right
Actual 90% Confidence Interval
Calibrated Estimates• Measuring your own
uncertainty about a quantity is a general skill that can be taught with a measurable improvement
• Studies show that most managers are statistically “overconfident” when assessing their own uncertainty
• Training can “calibrate” people so that when they provide a 90% confidence interval, it still has a 90% chance of being right (even though it is subjective)
Perceived 90% Confidence Interval
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Distribution-based ROI
Administrative Cost Reduction
Total Project Cost
% Improvement in Customer Retention
5% 10% 15%
10% 20% 30%
$2 million $4 million $6 million
ROI-50% 50% 100
%0%
Inputs
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Analyzing the Distribution
25% 50% 75% 100% 125%-25% 0%
Risk of Negative ROI
“Expected” ROIROI = 0%
Probability of Positive ROI
Return on Investment (ROI)
The “cancellation hump”
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Return
Risk
10%
20%
30%
40%
10% 20% 30% 40% 50% 60%
Pro
babi
lity
of le
ss th
an a
ri
sk-f
ree
retu
rn
A proposed IT investment with a 15% risk and 54% return
X
Plotting the Risk and Return
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Example of Risk Effects
50%
40%
30%
20%
10%
0%
0% 50% 100% 150% 200%
Expected IRR over 5 years
Cha
nce
of a
neg
ativ
e IR
R• These are real IT investments of $2M-$3M plotted against a client’s investment boundary• The 27% ROI investment is actually preferred to the 83% ROI investment
Region of Unacceptable Investments
Region of Acceptable Investments
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
10%
100%
1000%
20% 40% 60% 80% 100%
20%
30%
50%
200%
300%
500%
Size of the Project Relative to the Entire IT Portfolio(i.e. 50% = project makes up half the work in the entire portfolio)
Req
uire
d M
inim
um R
etur
n (I
RR
ov
er 5
yea
rs)
Most Risk
Avers
e
Approximate Median
Most Risk Tolerant
Range of Typical “Hurdle Rates”
Risk Increases Required ROI’s• Adjusting for risk causes some previously-acceptable projects to be
rejected• Also, some low return but low risk projects would now be acceptable• More projects with “intangible” benefits are now economically
justified• The net result: A completely reshuffled deck of IT project approvals
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Using Risk Analysis to Improve Decisions
If the Risk is significant (it usually is), consider doing the following:
• Reduce the size and functionality of the proposed system - focus on fewer high-return features
• Wait until specific uncertainties in the environment subside - e.g. major mergers, reengineering, etc.
• Wait to tackle big projects until proper skills are developed and methods are in place
• Consider purchased packages that aren’t a perfect fit but close enough - they may look more attractive now
• Invest more on a proper economic analysis of the largest IT investments - this should reduce uncertainty about critical quantities
• Include deferred benefits in any estimate of scope creep costs
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Finding 2: Measurements
• Information has a value that can be calculated with a formula known for 50 years
• Computing the value of additional information on uncertain variables causes some surprising changes in what gets measures
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
EVI p r V p r V p r V p r EVi j j ij
z
j j i l j j ij
z
j
z
i
k
( ) max ( | ), ( | ),... ( | ), *, , ,1
12
111
The Decision Theory Formula:
What it means: Information reduces uncertainty Reduced uncertainty improves decisions Improved decisions satisfy business objectives (by definition)
The Economic Value of Information
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
The IT Measurement Inversion
Typical A
ttention Eco
nom
ic R
elev
ance
Receives Most Attention
Least Relevant to Approval Decisions
Receives Least Attention
Most Relevant to Approval Decisions
• Costs– Initial Development Costs
– Ongoing Maintenance/Training Costs
• Benefits– A specific benefit (productivity,
sales, etc.)
– Utilization (when usage starts and how quickly usage grows)
• Chance of Cancellation
See my article “The IT Measurement Inversion” in CIO Magazine(its also on my website at www.hubbardresearch.com under the “articles” link)
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Finding 3: Technology Regret• Most business cases treat IT investments implicitly
as a “now or never” choice• Technology regret is an economic quantity
associated with committing to a technology after which, for whatever reason, becomes regrettable
• The equivalent of “buyers remorse”• Technology regret becomes and important
consideration in any environment where changing technology is a constant
• The issue becomes balancing technology regret (which tends to defer IT investments) vs. the opportunity loss of deferring the benefits of making the investment now
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
0
50
100
150
200
250
300
350
Year
Rel
ativ
e C
ompu
ting
Pow
erPe
r $
(198
0=1)
19951980 1985 20001990
32% Annual Growth Rate
Some Areas of Growth:• Processors & Memory
(Moore’s Law)
• Storage• Communications
(Payne’s Law)• Internet Users• Sensory devices
Competition makes capitalizing on new technology more critical to survival
Changing Technology
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Changing With Technology
A C
riti
cal Tech
nolo
gy
Measu
re
Time
How often should you change with technology? Avoiding “technology regret” is often a major driver in IT decisions.
Upgrade 1
Upgrade 2
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
“Real Option” Theory
Sin
gle
Per
iod
Opt
ion
Val
ue(V
alue
of
Wai
ting
one
per
iod)
Invest in this cycle, high priority
Net Value of the Investment0 +
Invest this cycle, low priority, may be deferred if resources are strained
-
Re-evaluate in the next decision cycle
Reject the investment
1. The option value tells us when an investment, even if it looks positive now, should be deferred until the opportunity is better
2. In the case of IT, waiting for the possibility of better technology around the corner should be considered
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
The Effects of Tech Regret
• Very long duration IT projects that commit to a proprietary solution tend to look much less favorable
• Short turnaround projects based on non-proprietary standards tend to look better
• Large scale commitments to the fastest improving technologies (like data storage, bandwidth) tend not to be favorable
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Finding 4: Scope Creep
• The cost of adding one additional function to an software development project is rarely addressed properly
• If anything, the only cost of new features considered is development cost
• Long term maintenance, increased risk of cancellation plus deferred benefits is much more significant
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
True Scope Creep Costs• 24%: Initial development
costs
• 24%: Future maintenance costs (computed over 5 years)
• 1%: Incremental contribution to probability of total project cancellation
• 51%: Deferred benefits of the other functions delayed by the proposed new function
24%
24%
1%
51%
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Finding 5: Value of Quant.• Organizations have successfully adopted more advanced
quantitative methods for evaluating IT investments• The cost of analysis routinely comes in below 1% and has always
been under 2% of the investment size - including initial training• This is still less than non-IT investments of similar size and risk• It is also sometimes less time-consuming than the previous non-
quantitative analysis techniques used by the firm (One of the reasons this analysis is efficient is we conduct a Value of Information Analysis - we only measure what is economically justified)
• Using the standard VIA calculation for the value of AIE analysis, AIE itself was the best investment of all the IT investments we analyzed - very conservative measures of payoffs put $20 to every $1 spent on AIE
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
Overview of RRA Analysis
Intangibles“Customer Satisfaction”“Strategic Alignment”“Technology Risk”“Information Quality” etc.
MeasurablesErrors in Decision XChange to Strategic Measure MProductivity in Activity YChance of cancellation, etc.
5% 10% 15%
10% 20% 30%
$2 mill $4 mill $6 mill
Measurements
$
$$$
$$
Value of Info.
Calculate Risk/Return Position"expected" ROI
50% 100% 150% 200% 250%-50% 0%
Probabilityof a negative ROI
Probability of a positive ROI
Organization's investment
limit
Acceptable region of investment
Return
Risk
Classification
HubbardDecision Research
The Applied Information Economics Company
Intro toQuant. Methods
Finding 1 The effect of Risk Analysis
Finding 4Scope Creep
Finding 2 What to Measure
Finding 3Tech. Regret
Finding 5AIE Value
In Conclusion…
• Quantitative methods like AIE cause major IT decisions to be very different – and better
• Advanced methods can and have been learned and adopted by IT organizations
• More quantitative analysis can be the highest return investment in your IT portfolio