the value of requirements uncertainty, louvain-la-neuve, october 2013
DESCRIPTION
TRANSCRIPT
![Page 1: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/1.jpg)
The Value of Requirements Uncertainty
Emmanuel Letier http://letier.cs.ucl.ac.uk
Joint work with David Stefan and Earl Barr
1 Louvain-la-Neuve, 4 October 2013
![Page 2: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/2.jpg)
Embrace Uncertainty!
2
![Page 3: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/3.jpg)
Software Design Decisions
Uncertainty is inevitable We must decide without knowing everything
3
What software to build? What
quality level? What to build
in next iteration?
What components and interfaces?
How to deploy them? When to
change the architecture?
![Page 4: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/4.jpg)
The Surfer’s Approach to Uncertainty
Instead of learning to
surf, conventional
organizations try to
control the waves. This
almost never works.
— Allen Ward
4
Mary Poppendieck “Learning to Surf”
industry keynote @ ICSE2013
![Page 5: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/5.jpg)
The Surfer’s Approach to Uncertainty
Instead of learning to
surf, conventional
organizations try to
control the waves. This
almost never works.
-- Allen Ward
5
![Page 6: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/6.jpg)
The Scientific Approach to Uncertainty
Decision Analysis, a discipline
for understanding, formalising,
analysing, and communicating
insights about situations in
which important decisions
must be made
6
Ron Howard, Stanford
![Page 7: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/7.jpg)
The Pseudo-Scientific Approach
Resembles the scientific approach, except that
• the decision criteria are numbers without verifiable meaning
• the decision models are not falsifiable
• no retrospective evaluation of decisions and outcomes
Most widely used example, the Analytical Hierarchy Process (AHP)
7
![Page 8: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/8.jpg)
What do we mean by uncertainty ?
8
![Page 9: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/9.jpg)
Uncertainty
Uncertainty is the lack of complete knowledge about a state or quantity. There is more than one possible value and the “true” value is not known. Measurement of uncertainty. A set of possible values with a probability assigned to each.
9
Will it snow at Christmas?
yes no
0.4 0.6
How cold will it be?
2oC 8oC -5oC
Pro
babi
lity
![Page 10: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/10.jpg)
Accuracy and Precision
10
For a measurement or prediction • Precision refers to how close the measured or predicted
values are to each other • Accuracy refers to how close the measured or predicted
values are to the true value
37oC 41oC 34oC Less precise, but more accurate
How hot will it be in Hyderabad, India on 1st June 2014?
30oC Precise: yes; Accurate: ?
![Page 11: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/11.jpg)
Key Insights
The more precise, the higher risk of being wrong (inaccurate)
The less you know, the harder it is to be both precise and
accurate; if you want to be accurate, you have to be less
precise
Reducing uncertainty has economic value because it leads to
better decisions that will, on average, increase profit
11
![Page 12: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/12.jpg)
12
Things Software Engineers Say ...
Clients don’t know what they want Requirements documents are always too vague, incomplete, inconsistent, out-of-date, etc. Requirements change is inevitable It’s not possible to discover the true requirements before building the system
![Page 13: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/13.jpg)
13
Things Academics Say ...
Requirements are inherently
unknowable!
Linda Northrop “Does Scale Really Matter? – Ultra-Large-Scale Systems Seven Years after the Study” plenary keynote @ ICSE2013
![Page 14: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/14.jpg)
What they really mean...
Requirements are uncertain
14
![Page 15: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/15.jpg)
“Requirements engineering is the branch of software engineering concerned with the real-world goals for, functions of, and constraints on software systems. It is also concerned with the relationship of these factors to precise specifications of software behavior, and to their evolution over time and across software families.”
15
Yet, we insist on requirements being precise
Pamela Zave, ACM Computing Surveys, 1997
![Page 16: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/16.jpg)
Why do we want precision?
16
![Page 17: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/17.jpg)
Boehm’s Cost-to-Fix Curve (1981)
17
200
50
20 10 5 1
Rela
tive
Cos
t of
Err
or C
orre
ctio
n
![Page 18: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/18.jpg)
An Hypothesis
18
Requirements engineering focuses on precision as a way to detect and fix defects as early as possible
when it is cheaper to do so
The cost-to-fix curve crystallised software engineering thinking around questions of costs (time and money)
and defects
We have lost sight of the end goal!
![Page 19: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/19.jpg)
What is the end goal of Software
Engineering?
19
![Page 20: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/20.jpg)
The end goal of Software Engineering is ...
A. To deliver software on time
B. To deliver software on budget
C. To deliver software with low number of bugs
D. All of the above
E. None of the above
20
![Page 21: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/21.jpg)
The end goal of Software Engineering is ...
A. To deliver software on time
B. To deliver software on budget
C. To deliver software with low number of bugs
D. All of the above
E. None of the above
21
E. None of the above
F. To deliver software that provides value for money (or no software at all if there are better ways to provide value)
![Page 22: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/22.jpg)
Beware of Treating Subgoals as End Goals
Delivering on time, on budget, with low defect rate doesn’t
necessarily provide value for money (e.g. £80 million mobile
technology for UK police)
22
Minimising requirements defects (ambiguity, incompleteness,
etc.) doesn’t necessarily yield a valuable system
![Page 23: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/23.jpg)
Approaches Focussing on Value for Money
23
Goal Modelling
![Page 24: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/24.jpg)
24
![Page 25: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/25.jpg)
25
code coverage for its own sake ...
bugs finding for its own sake ...
bugs fixing for its own sake ...
Precision for its own sake …
Goal-based & Value-based Software Engineering
![Page 26: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/26.jpg)
Goal-Based Decisions Under Uncertainty
26
A new perspective on software engineering
![Page 27: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/27.jpg)
Goal-Based Decisions Under Uncertainty
27
The Simplest Possible Example
![Page 28: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/28.jpg)
A Typical IT Project Business Case
28
Expected Cost €2m
Expected Benefit €10m
Expected Net Benefit
€8m
ROI 400%
![Page 29: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/29.jpg)
Cost-Benefit Analysis with Uncertainty
29
90% Confidence Interval Most Likely Cost [€2m , €5m] €3.5m
Benefit [€2m , €10m] €6m
Benefit
£6m £10m £2m
Pro
babi
lity
Cost
£5m £2m
Pro
babi
lity
![Page 30: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/30.jpg)
Where Do the Numbers Come From?
• Cost and benefit are functions of a set of uncertain variables (eg. development cost, operating cost, market size, ...)
• Uncertainty about each variable is elicited from experts and decision makers – using simple effective methods – having sound mathematical
foundations and significant empirical validation
30
![Page 31: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/31.jpg)
Cost-Benefit Analysis with Uncertainty
31
90% Confidence Interval Most Likely Cost [€2m , €5m] €3.5m
Benefit [€2m , £10m] €6m
Expected Net Benefit €2.5m
Loss Probability 16%
Average Loss Magnitude €1.3m
![Page 32: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/32.jpg)
The Expected Value of Perfect Information (EVPI)
EVPI(X) = the expected gain in net benefit from obtaining perfect information about X to inform decision
32
(Ronald Howard, 1966)
Highest expected net benefit among
all alternatives given current knowledge BK
Highest expected net benefit among all alternatives given
current knowledge BK and X = x
Expected gain (expectation
over X)
![Page 33: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/33.jpg)
The Expected Value of Information
33
• Information about benefit has high value and impact on risk – Current 90% confidence interval: €2m-€10m
• Information about cost has no value and impact on risk – Current 90% confidence interval: €2m-€5m
EVPI Remaining Loss Probability
Total Perfect Information €0.22m 0%
Info about Benefit €0.18m 3%
Info about Cost €0.001m 16%
Reminder: Expected Net Benefit = €2.5m; Loss Probability = 16%
![Page 34: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/34.jpg)
The Measurement Inversion Paradox
Lessons from applying decision analysis to 20 IT business cases, each having 40 to 80 variables
1. Most variables have zero information value
2. Variables with high information values were routinely
those the client never measured
3. Clients spent most of their effort measuring quantities
with low or even zero information value
34
(Douglas Hubbard, 1999)
![Page 35: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/35.jpg)
Application to Software Design Decisions
A mobile system for coordinating emergency rescue teams • Design space: 10 design
decisions, around 7,000 candidate architectures
• Objectives: Cost, Response Time, Reliability, Battery Life, ...
• Models given by design team: Utility score defined as weighted sum of objective satisfaction
• Lessons Learnt – Risks specific to requirements
and architecture decisions – Need to reason about model
uncertainty in addition to parameter uncertainty
– Decision models must be falsifiable
35
(with D. Stefan and E.T. Barr)
![Page 36: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/36.jpg)
Research Roadmap
36
Scientific Approach to Software Decisions
Parameter uncertainty
Model uncertainty: quantifying “good enough”
Incremental value delivery
Showing applicability
Showing cost-effectiveness Overcoming cultural barriers
???
Incremental evidence-based model tuning
![Page 37: The Value of Requirements Uncertainty, Louvain-la-Neuve, October 2013](https://reader033.vdocuments.us/reader033/viewer/2022052618/54c3638c4a7959a20f8b4576/html5/thumbnails/37.jpg)
A Call to Action
Who do you want to inform our IT projects decisions?
37
Uncertainty is at the heart of most major challenges for the 21st Century
The Surfers The Scientists The Pseudo-Scientists