saturation effect and the need for a new theory of software reliability
DESCRIPTION
Saturation effect and the need for a new theory of software reliability. Aditya P. Mathur Professor, Department of Computer Science, Associate Dean, Graduate Education and International Programs Purdue University. Department of Computer Science North Dakota State University, Fargo, ND - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/1.jpg)
1
Aditya P. Mathur
Professor, Department of Computer Science,Associate Dean, Graduate Education and International ProgramsPurdue University
Department of Computer ScienceNorth Dakota State University, Fargo, NDThursday April 19, 2007
Saturation effect and the need for a new theory of software reliability
![Page 2: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/2.jpg)
2
Dependability
Availability: Readiness for correct service
Reliability: Continuity of correct service
Safety: Absence of catastrophic consequences on the user(s) and the environment
Security: The concurrent existence of (a) availability for authorized users only, (b) confidentiality, and (c) integrity.
Source: Wikipedia.
Focus of this talk
The presence of software errors has the potential for negative impact on each aspect of Dependability.
![Page 3: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/3.jpg)
3
Reliability
Probability of failure free operation in a given environment over a given time.
Mean Time To Failure (MTTF)
Mean Time To Disruption (MTTD)
Mean Time To Restore (MTTR)
![Page 4: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/4.jpg)
4
Operational profile
Probability distribution of usage of features and/or scenarios.
Captures the usage pattern with respect to a class of customers.
![Page 5: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/5.jpg)
5
Reliability estimation-Early Work
Operationalprofile
Random or semi-random Test generation
Test execution Failure data collection
Reliability estimation
Decision process
[Shooman ‘72, Littlewood ‘73, Musa ‘75, Thayer ‘76, Goel et al. ‘78, Yamada et al. ‘83, Laprie ‘84, Malaiya et al. ‘92, Miller et al. ‘92, Singpurwalla ‘95]
![Page 6: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/6.jpg)
6
Reliability estimation: Correlation, Coverage, Architecture
Cheung ’80: Markovian modelOhba ’92. Piwowarski et al.‘93: Coverage basedChen et al. ’92: Coverage based
Garg ’95, Del Frate et al.’95 : Coverage/reliability model and correlation.
Littlewood ’79: architecture based
Malaiya et al. ’94: Coverage based
Xiaoguang et al. ‘03: architecture based
Krishnamurthy et al. ’97: architecture basedGokhale et al. ’98: architecture based
Chen et al. ’94, Musa ’94: Reliability/testing sensitivity
![Page 7: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/7.jpg)
7
Need for Ultrahigh Reliability
Medical devices
Automobile engine controllers
Aircraft controllers
Track/train control systems
No known escaped defects that might create unsafe situations and /or might lead to ineffective performance.
![Page 8: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/8.jpg)
8
A reliability estimation scenario (slightly unrealistic)
An integrated version of the software P for a cardiac pacemaker is available for system test.
Operational profile from an earlier version of the pacemaker is available.
P has never been used in any implanted pacemaker.
Tests are generated using the operational file and P tested.
Three distinct failures are foundand analyzed.
The management asks the development team to debug P and remove causes of all failures.
The updated P is retested using the same operational profile. No failures are observed. What is the reliability of the updated P?
Unrealistic
![Page 9: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/9.jpg)
9
Issues: Operational profile
Variable. Becomes known only after customers have access to the product. Is a stochastic process…a moving target!
Random test generation requires an oracle. Hence is generally limited to specific outcomes, e.g. crash, hang.
In some cases, however, random variation of input scenarios is useful and is done for embedded systems.
Human heart: Variability across humans and over time.
![Page 10: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/10.jpg)
10
Issues: Failure data
Should we analyze the failures?
If yes then after the cause is removed, the reliability estimate is invalid.
If the cause is not removed, because the failure is a “minor incident,” then the reliability estimate corresponds to irrelevant incidents.
![Page 11: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/11.jpg)
11
Issues: Model selection
Rarely does a model fit the failure data.
Model selection becomes a problem. ~200 models to choose from? New ones keep arriving!
![Page 12: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/12.jpg)
12
Issues: Markovian models
Markov models suffer from a lack of estimate of transition probabilities.
To compute these probabilities, you need to execute the application.
During execution you obtain failure data. Then why proceed further with the model?
C1
C3
C212
13 32
2112 + 13=1
![Page 13: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/13.jpg)
13
Issues: Assumptions
Software does not degrade over time; e.g. memory leak is not degradation and is not a random process; a new version is a different piece of software.
Reliability estimate varies with operational profile. Different customers see different reliability.
Can we have a reliability estimate independent of the operational profile?
Can we not advertise quality based on metric that are a true representation of reliability..not with respect to a subset of features but over the entire set of features?
![Page 14: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/14.jpg)
14
Sensitivity of Reliability to test adequacy
Coverage
low
low
high
high
Desirable
Suspect modelUndesirable
Risky
Rel
iabi
lity
Problem with existing approaches to reliability estimation.
![Page 15: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/15.jpg)
15
Basis for an alternate approach
Why not develop a theory based on coverage of testable items and test adequacy?Testable items: Variables, statements,conditions, loops, data flows, methods, classes, etc.
Pros: Errors hide in testable items.
Cons: Coverage of testable items is inadequate. Is it a good predictor of reliability?
Yes, but only when used carefully. Let us see what happens when coverage is not used or not used carefully.
Are we interested in reliability or in confidence?
![Page 16: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/16.jpg)
16
Saturation Effect
FUNCTIONAL, DECISION, DATAFLOWAND MUTATION TESTING PROVIDETEST ADEQUACY CRITERIA.
Reliability
Testing EffortTrue reliability (R)Estimated reliability (R’)Saturation region
Mutation
DataflowDecision
Functional
RmRdfRdRf
R’f R’d R’df R’m
tfs tfe tds tde tdfs tdfe tms tfe
![Page 17: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/17.jpg)
17
An experiment
Tests generated randomly exercise less code than those generated using a mix of black box and white box techniques. Application: TeX. Creator: Donald Knuth. [Leath ‘92]
![Page 18: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/18.jpg)
18
Modeling an application
OSComponent Component Component Interactions
Component Component Component Interactions
Component Component Component Interactions
……….
![Page 19: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/19.jpg)
19
Reliability of a component
R(f)= (covered/total), 0<<1.
Reliability, probability of correct operation, of function f based on a given finite set of testable items.
Issue: How to compute ?
Approach: Empirical studies provide estimate of and its variance for different sets of testable items.
![Page 20: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/20.jpg)
20
Reliability of a subsystem
R(C)= g(R(f1), R(f2), ..R(fn), R(I))
C={f1, f2,..fn} is a collection of components that collaborate with each other to provide services.
Issue 1: How to compute R(I), reliability of component interactions?
Issue 2: What is g ?
Issue 3: Theory of systems reliability creates problems when (a) components are in a loop and (b) are dependent on each other.
![Page 21: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/21.jpg)
21
Scalability
Is the component based approach scalable?
Powerful coverage measures lead to better reliability estimates whereas measurement of coverage becomes increasingly difficult as more powerful criteria are used.
Solution: Use component based, incremental, approach. Estimate reliability bottom-up. No need to measure coverage of components whose reliability is known.
![Page 22: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/22.jpg)
22
Next steps
Develop component based theory of reliability.
[Littlewood 79, Kubat 89, Krishnamurthy et al. 95, Hamlet et al. 01, Goseva-Popstojanova et al. 01, May 02]
Do experimentation with large systems to investigate the applicability and effectiveness in predicting and estimating various reliability (confidence) metrics.
Base the new theory on existing work in software testing and reliability.
![Page 23: Saturation effect and the need for a new theory of software reliability](https://reader036.vdocuments.us/reader036/viewer/2022081604/568167a9550346895ddcf66d/html5/thumbnails/23.jpg)
23
The Future
Apple Confidence: 0.999
Level 0: 1.0
Level 1: 0.9999
Level 2: 0.98
Boxed and embedded software with independently variableLevels of Confidence.
Mackie Confidence: 0.99
Level 0: 1.0
Level 1: 0.9999