software testing metrics & standards

36
By: Razieh Rezaei Supervisor: Dr.M.Kahani

Upload: javierarmando8076

Post on 29-Nov-2014

52 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Software Testing Metrics & Standards

By: Razieh RezaeiSupervisor: Dr.M.Kahani

Page 2: Software Testing Metrics & Standards

What’s Software test?Any software product, no matter how

accomplished by today’s testing technologies, has bugs.

Software testing is the process of detecting and removing the bugs from the software product.

The goal of software testing is ensuring of software product quality according to the customer requirements.

2

Page 3: Software Testing Metrics & Standards

Some typical software testing questions:• How long will it take to test software?• How much will it cost to test?• What kind of bugs were found?• How many of the bugs that were

found were fixed?• Will the product be ready on time?• How much is the risk of release?• …

3

Page 4: Software Testing Metrics & Standards

Test MetricsMetrics are ways of answering questions.There are numerous metric for software

testing.In ideal a metric must have these

characteristic:simpleApplicableEasily collectableValidRobust

4

Page 5: Software Testing Metrics & Standards

Software Metrics

5

Page 6: Software Testing Metrics & Standards

Test MetricsTest metrics classifies in two category:

Product Metrics: Metrics that are related to test results or quality of software under test

Process Metrics: Metrics that used for evaluation of testing process effectiveness

6

Page 7: Software Testing Metrics & Standards

TestabilityTestability can be considered the inverse of

fault tolerance. Code which tolerates faults will not, by definition, exhibit them.

Software can be temporarily rendered more testable by including assertions in it.

An approach to determining testability involving fault injection in three stages: propagation infection execution

7

Page 8: Software Testing Metrics & Standards

Bug (Fault) DensityEach system has two characteristics:

the number of bugs and the number of non-commented lines of

source code (KLOC)

Bug Density can be used to predict release readiness

8

Page 9: Software Testing Metrics & Standards

Weighted Fault DensityBug density can be calculated by bug

severity.Fault Density (Weighted) = W1 S / N + W2 A / N +

W3 M/ N

W1, W2, W3 , weights assign according to user viewpoint

N, number of faultsS, number of severe faultsA, number of average faultM, number of minor faults

9

Page 10: Software Testing Metrics & Standards

SeveritySeverity is a fundamental measure of a bug

or a failure. Many ranking schemes exist for defining severity.

10

Page 11: Software Testing Metrics & Standards

Bug Density per UnitBug densities should be monitored throughout the test effort.

The graph presented here can be used successfully to allocate test and development resources toward the end of the test effort so that the product could be released on time.

11

Page 12: Software Testing Metrics & Standards

Bug Prediction KLOC EstimateThis model can be applied only when coding is

finished.

The Number of bugs per KLOC=

f1 = the number of bug reports required to change the design and code during the coding phase per KLOC

f2 = the average number of years experience of programming of all programmers involved

f3 = the number of pages of new and modified high- and low-level design documents used per KLOC

12

Page 13: Software Testing Metrics & Standards

Zero-Failure Approach to Release Readiness EstimationIt is based on some period being defined during

which no further failures must be found if some level of reliability is to be achieved.

a = no. of test hours since the last problem was observed

f = expected number of field failuress = number of test failures discovered to far

13

Page 14: Software Testing Metrics & Standards

Bug CostThe bug cost usually includes the

cost of bug analysis,cost to fix, cost in lost revenues, cost of the testers' salaries, Cost of equipment, systems, software, and

other tools.

14

Page 15: Software Testing Metrics & Standards

Bug Find Rate This is a most useful derived metric both for measuring the

cost of testing and for assessing the stability of the system. It can give a good indication of the stability of the system

being tested.

15

Page 16: Software Testing Metrics & Standards

CoverageThere are some coverage metric:

Code coverageBranch coveragePath coverageTest coverageData flow coverage

16

Page 17: Software Testing Metrics & Standards

Mean Time to FailureThis is measured usually against a clock (e.g.

CPU cycles) and assumes:Constant execution loadMeasurement against some error threshold

17

Page 18: Software Testing Metrics & Standards

AvailabilityAvailability is the probability that the system

will still be operating to requirements at a given time.

One way of determining a system’s availability is to model it as a Markov chain.

18

Page 19: Software Testing Metrics & Standards

IEEE Std 829, IEEE Standard for Software Test DocumentationThe purpose of this standard is to

describe a set of basic software test documents.

This standard specifies the form and content of individual test documents. It does not specify the required set of test documents.

It is assumed that the required set of test documents will be specified when the standard is applied.

19

Page 20: Software Testing Metrics & Standards

IEEE Std 829Benefits:

Facilitate communication by providing a common frame of reference

Provide a baseline for the evaluation of current test documentation practices.

The use of these documents significantly increases the manageability of testing.

Increase visibility of each phase of the testing process.

20

Page 21: Software Testing Metrics & Standards

The documents outlined in this standard cover test planning, test specification, and test reporting.

Test Planning DocumentsThe test plan prescribes the scope, approach,

resources, and schedule of the testing activities.Test Specification Documents

A test design specification, A test case specification, A test procedure specification

Test ReportingA test item transmittal report, A test log, A test

incident report, A test summary report

21

Page 22: Software Testing Metrics & Standards

22

Page 23: Software Testing Metrics & Standards

ISO/IEC 9126(1991), Software Engineering –Product Quality-ISO/IEC 9126 (1991): Software product

evaluation - Quality characteristics and guidelines for their use, which was developed to support these needs.

ISO/IEC 9126 (1991) has been replaced by two related multipart standards: ISO/IEC 9126 (Software product quality) and ISO/IEC 14598 (Software product evaluation)

23

Page 24: Software Testing Metrics & Standards

ISO/IEC 9126ISO/IEC 9126 consists of the following parts,

under the general title Software engineering — Product quality:Part 1: Quality modelPart 2: External metricsPart 3: Internal metricsPart 4: Quality in use metrics

24

Page 25: Software Testing Metrics & Standards

First part of ISO/IEC 9126 describes a two-part model for software product quality:

a) internal quality and external quality b) quality in use.

25

Page 26: Software Testing Metrics & Standards

Quality model for external and internal quality

26

Page 27: Software Testing Metrics & Standards

Quality in UseThe capability of the software product to

enable specified users to achieve specified goals with effectiveness, productivity, safety and satisfaction in specified contexts of use.

27

Page 28: Software Testing Metrics & Standards

Relationship between ISO/IEC 9126 and ISO/IEC 14598 standards

28

Page 29: Software Testing Metrics & Standards

IEEE P1671, Standard for Automatic TestMarkup Language (ATML) for Exchanging Automatic Test Equipment and Test Information via XMLATML defines a standard exchange medium for

sharing information between components of automatic test systems.

This information includes test dataresource datadiagnostic datahistoric data.

29

Page 30: Software Testing Metrics & Standards

ATML ObjectiveATML is intended to accomplish the following

objectives:Facilitate the communication, sharing, and reuse of

product design and test information for the purpose of testing the product.

Facilitate TPS portability and interoperability.Facilitate the development, integration, and use of test

software and test software development tools.Supports modular software architectures based upon a

framework that supports reusable software products.….

30

Page 31: Software Testing Metrics & Standards

IEEE P1671As previously stated, the ATML

framework is defined in the form of ATML Components. The ATML Components define the domain-orientated information. ATML Components are segmented into:ATML Component Standards (in the form of

formal IEEE Standards).XML Schemas.Reference to ATML Instance Documents

31

Page 32: Software Testing Metrics & Standards

32

Page 33: Software Testing Metrics & Standards

ConclusionSoftware testing metrics

Product metricsProcess metrics

Software testing standardsIEEE 829: Software Test DocumentationISO/IEC 9126(1991), Software Engineering –

Product Quality-IEEE P1671, ATML for Exchanging Automatic

Test Equipment and Test Information via XML

33

Page 34: Software Testing Metrics & Standards

References[Bre89] Ralph Brettschneider(Motorola). Is your software ready for release? IEEE Software, July 1989.

[Far08] Peter Farrell-Vinay, Manage Software Testing, Auerbach Publication, 2008.

[Hut03] Marnie L. Hutcheson, Software Testing Fundamentals: Methods and Metrics, John Wiley & Sons, 2003.

[IEEE 829] IEEE Std 829-1998 Software Test Documentation. Defines the content and format of documents that cover the testing process.

[IEEE P1671] IEEE Std P1671-2006 Archived Approved Draft - Draft Trial-Use Standard for Automatic Test Markup Language (ATML) for Exchanging Automatic Test Equipmqnt and Test Information via XML.

[IEEE90] IEEE (1990). IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. New York: IEEE. ISBN 1559370793.

[ISO/IEC 9126] ISO/IEC standard 9126 Software engineering product quality — Part 1: quality model. ISO Genève. 2001

34

Page 35: Software Testing Metrics & Standards

References[ISO/IEC 9126] ISO/IEC standard 9126 Software engineering

product quality — Part 1: quality model. ISO Genève. 2001[Mic02] James B. Michael, Bernard J. Bossuyt, and Byron B.

Snyder. Metrics f or measuring the effectiveness of software-testing tools. Proceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE’02).

[Nag05] Nachiappan Nagappan. A Software Testing and Reliability Early Warning (STREW) Metric Suite. A dissertation submitted to the graduate faculty of North Carolina State University, 2005.

[Tak89] M. Takahashi and Y. Kamayachi. An empirical study of a model for program error prediction. IEEE Transactions on Software Engineering, January 1989.

35

Page 36: Software Testing Metrics & Standards

Question?

36