software testing metrics & standards
TRANSCRIPT
By: Razieh RezaeiSupervisor: Dr.M.Kahani
What’s Software test?Any software product, no matter how
accomplished by today’s testing technologies, has bugs.
Software testing is the process of detecting and removing the bugs from the software product.
The goal of software testing is ensuring of software product quality according to the customer requirements.
2
Some typical software testing questions:• How long will it take to test software?• How much will it cost to test?• What kind of bugs were found?• How many of the bugs that were
found were fixed?• Will the product be ready on time?• How much is the risk of release?• …
3
Test MetricsMetrics are ways of answering questions.There are numerous metric for software
testing.In ideal a metric must have these
characteristic:simpleApplicableEasily collectableValidRobust
4
Software Metrics
5
Test MetricsTest metrics classifies in two category:
Product Metrics: Metrics that are related to test results or quality of software under test
Process Metrics: Metrics that used for evaluation of testing process effectiveness
6
TestabilityTestability can be considered the inverse of
fault tolerance. Code which tolerates faults will not, by definition, exhibit them.
Software can be temporarily rendered more testable by including assertions in it.
An approach to determining testability involving fault injection in three stages: propagation infection execution
7
Bug (Fault) DensityEach system has two characteristics:
the number of bugs and the number of non-commented lines of
source code (KLOC)
Bug Density can be used to predict release readiness
8
Weighted Fault DensityBug density can be calculated by bug
severity.Fault Density (Weighted) = W1 S / N + W2 A / N +
W3 M/ N
W1, W2, W3 , weights assign according to user viewpoint
N, number of faultsS, number of severe faultsA, number of average faultM, number of minor faults
9
SeveritySeverity is a fundamental measure of a bug
or a failure. Many ranking schemes exist for defining severity.
10
Bug Density per UnitBug densities should be monitored throughout the test effort.
The graph presented here can be used successfully to allocate test and development resources toward the end of the test effort so that the product could be released on time.
11
Bug Prediction KLOC EstimateThis model can be applied only when coding is
finished.
The Number of bugs per KLOC=
f1 = the number of bug reports required to change the design and code during the coding phase per KLOC
f2 = the average number of years experience of programming of all programmers involved
f3 = the number of pages of new and modified high- and low-level design documents used per KLOC
12
Zero-Failure Approach to Release Readiness EstimationIt is based on some period being defined during
which no further failures must be found if some level of reliability is to be achieved.
a = no. of test hours since the last problem was observed
f = expected number of field failuress = number of test failures discovered to far
13
Bug CostThe bug cost usually includes the
cost of bug analysis,cost to fix, cost in lost revenues, cost of the testers' salaries, Cost of equipment, systems, software, and
other tools.
14
Bug Find Rate This is a most useful derived metric both for measuring the
cost of testing and for assessing the stability of the system. It can give a good indication of the stability of the system
being tested.
15
CoverageThere are some coverage metric:
Code coverageBranch coveragePath coverageTest coverageData flow coverage
16
Mean Time to FailureThis is measured usually against a clock (e.g.
CPU cycles) and assumes:Constant execution loadMeasurement against some error threshold
17
AvailabilityAvailability is the probability that the system
will still be operating to requirements at a given time.
One way of determining a system’s availability is to model it as a Markov chain.
18
IEEE Std 829, IEEE Standard for Software Test DocumentationThe purpose of this standard is to
describe a set of basic software test documents.
This standard specifies the form and content of individual test documents. It does not specify the required set of test documents.
It is assumed that the required set of test documents will be specified when the standard is applied.
19
IEEE Std 829Benefits:
Facilitate communication by providing a common frame of reference
Provide a baseline for the evaluation of current test documentation practices.
The use of these documents significantly increases the manageability of testing.
Increase visibility of each phase of the testing process.
20
The documents outlined in this standard cover test planning, test specification, and test reporting.
Test Planning DocumentsThe test plan prescribes the scope, approach,
resources, and schedule of the testing activities.Test Specification Documents
A test design specification, A test case specification, A test procedure specification
Test ReportingA test item transmittal report, A test log, A test
incident report, A test summary report
21
22
ISO/IEC 9126(1991), Software Engineering –Product Quality-ISO/IEC 9126 (1991): Software product
evaluation - Quality characteristics and guidelines for their use, which was developed to support these needs.
ISO/IEC 9126 (1991) has been replaced by two related multipart standards: ISO/IEC 9126 (Software product quality) and ISO/IEC 14598 (Software product evaluation)
23
ISO/IEC 9126ISO/IEC 9126 consists of the following parts,
under the general title Software engineering — Product quality:Part 1: Quality modelPart 2: External metricsPart 3: Internal metricsPart 4: Quality in use metrics
24
First part of ISO/IEC 9126 describes a two-part model for software product quality:
a) internal quality and external quality b) quality in use.
25
Quality model for external and internal quality
26
Quality in UseThe capability of the software product to
enable specified users to achieve specified goals with effectiveness, productivity, safety and satisfaction in specified contexts of use.
27
Relationship between ISO/IEC 9126 and ISO/IEC 14598 standards
28
IEEE P1671, Standard for Automatic TestMarkup Language (ATML) for Exchanging Automatic Test Equipment and Test Information via XMLATML defines a standard exchange medium for
sharing information between components of automatic test systems.
This information includes test dataresource datadiagnostic datahistoric data.
29
ATML ObjectiveATML is intended to accomplish the following
objectives:Facilitate the communication, sharing, and reuse of
product design and test information for the purpose of testing the product.
Facilitate TPS portability and interoperability.Facilitate the development, integration, and use of test
software and test software development tools.Supports modular software architectures based upon a
framework that supports reusable software products.….
30
IEEE P1671As previously stated, the ATML
framework is defined in the form of ATML Components. The ATML Components define the domain-orientated information. ATML Components are segmented into:ATML Component Standards (in the form of
formal IEEE Standards).XML Schemas.Reference to ATML Instance Documents
31
32
ConclusionSoftware testing metrics
Product metricsProcess metrics
Software testing standardsIEEE 829: Software Test DocumentationISO/IEC 9126(1991), Software Engineering –
Product Quality-IEEE P1671, ATML for Exchanging Automatic
Test Equipment and Test Information via XML
33
References[Bre89] Ralph Brettschneider(Motorola). Is your software ready for release? IEEE Software, July 1989.
[Far08] Peter Farrell-Vinay, Manage Software Testing, Auerbach Publication, 2008.
[Hut03] Marnie L. Hutcheson, Software Testing Fundamentals: Methods and Metrics, John Wiley & Sons, 2003.
[IEEE 829] IEEE Std 829-1998 Software Test Documentation. Defines the content and format of documents that cover the testing process.
[IEEE P1671] IEEE Std P1671-2006 Archived Approved Draft - Draft Trial-Use Standard for Automatic Test Markup Language (ATML) for Exchanging Automatic Test Equipmqnt and Test Information via XML.
[IEEE90] IEEE (1990). IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. New York: IEEE. ISBN 1559370793.
[ISO/IEC 9126] ISO/IEC standard 9126 Software engineering product quality — Part 1: quality model. ISO Genève. 2001
34
References[ISO/IEC 9126] ISO/IEC standard 9126 Software engineering
product quality — Part 1: quality model. ISO Genève. 2001[Mic02] James B. Michael, Bernard J. Bossuyt, and Byron B.
Snyder. Metrics f or measuring the effectiveness of software-testing tools. Proceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE’02).
[Nag05] Nachiappan Nagappan. A Software Testing and Reliability Early Warning (STREW) Metric Suite. A dissertation submitted to the graduate faculty of North Carolina State University, 2005.
[Tak89] M. Takahashi and Y. Kamayachi. An empirical study of a model for program error prediction. IEEE Transactions on Software Engineering, January 1989.
35
Question?
36