1 software testing see word file “software testing”
TRANSCRIPT
1
Software Testing
See Word file
“Software Testing”
2
8 historical definitions
• Establishing confidence that a program does what it is supposed to do (Hetzel, 1973).
• The process of executing a program or system with the intent of finding errors (Myers, 1979).
• Detecting specification errors and deviations from the specification.
• Any activity aimed at evaluating an attribute or capability of a program or system (Hetzel, 1983).
3
8 historical definitions
• The measurement of software quality (Hetzel, 1983).
• The process of evaluating a program or system.• Verifying that a system satisfies its specified
requirements or identifying differences between expected and actual results.
• Confirming that a program performs its intended functions correctly.
4
IEEE/ANSI definitions
• The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component (IEEE/ANSI, 1990 [Std 610.12 1990]).
• The process of analysing a software item to detect the difference between existing and required conditions (that is, bugs) and to evaluate the features of the software items (IEEE/ANSI, 1983 [Std 829‑1983]).
5
Verification: Validation
One kind of testing, verification, can be performed on any work product any intermediate product in the development cycle.
The other kind of testing, validation, can only be performed by executing the code.
6
Validation axioms
• Testing can be used to show the presence of errors, but never their absence.
• One of the most difficult problems in testing is knowing when to stop.
• Avoid unplanned, non-reusable, throw-away test cases unless the program is truly a throw-away program.
• A necessary part of a test case is a definition of the expected output or result. Always carefully compare the actual versus the expected results of each test.
7
Validation axioms
• Test cases must be written for invalid and unexpected, as well as valid and expected, input conditions. "Invalid" is defined as a condition that is outside the set of valid conditions and should be diagnosed as such by the program being tested.
• Test cases must be written to generate desired output conditions. Less experienced testers tend to think only from the input perspective. Experienced testers determine the inputs required to generate a pre-designed set of outputs.
8
Validation axioms
• With the exception of unit and integration testing a program should not be tested by the person or organisation that developed it. Practical cost considerations usually require developers do unit and integration testing.
• The number of undiscovered errors is directly proportional to the number of errors already discovered.
9
Validation Testing
• Testing has been the basic form of defect removal since the software industry began.
• For a majority of software projects, it is the only form of defect removal utilised.
• It is surprising that so little quantitative data has been published on this topic.
10
Testing (Jones)
• "the dynamic execution of software and the comparison of the results of that execution against a set of known, predetermined criteria.”
• i.e. validation
11
Categories
• The forms of testing are divided into three broad categories: general testing, specialised testing, testing that involves the users or clients
themselves.
12
Categories
• The general forms of testing are concerned with any kind of software and seek to eliminate common kinds of bugs such as branching errors, looping errors, incorrect outputs, and the like.
• The specialised forms of testing are more narrow in focus and seek specific kinds of errors such as problems that only occur under full load, or problems that might slow down performance.
• The forms of testing involving users are aimed primarily at usability problems an ensuring that all requirements have been in fact implemented.
Approximate Distribution of Testing Methods for U.S. Software Projects
Testing Stage Percentage of ProjectsUtilising Test Stage
General Forms of TestingSubroutine testing 100Unit testing 99System testing of full application 95New function testing 90Regression testing 70Integration testing 50Specialised Forms of TestingViral protection testing 45Stress or capacity testing 35Performance testing 30Security testing 15Platform testing 5Year 2000 testing 5Independent testing 3Forms of Testing Involving UsersCustomer acceptance testing 35Field (Beta) testing 30Usability testing 20Lab testing 1Clean-room statistical testing 1
14
Black & White
• Black box testing: the inner structure or control flow of the application is not known or viewed as irrelevant for constructing test cases.
• The application is tested against the external specifications and/or requirements in order to ensure that a specific set of input parameters will in fact yield the correct set of output values.
15
Black & White
• White box testing: the test-case developer is privy to the inner structure of the application and knows the control flow through the application, or at least knows the control flow if the software works correctly.
• This form is also sometimes known as "glass box" testing
16
Black & White
• Black box testing is useful for ensuring that the software more or less is in concordance with the written specifications and written requirements.
• White box or glass box testing is useful for ensuring that all or at least most paths through the application have been executed in the course of testing.
Black Box, White Box, and Mixed Testing Approaches
Subroutine testing White boxUnit testing White boxViral protection testing White boxStress or capacity testing White boxPerformance testing White boxSecurity testing White boxYear 2000 testing White box
System testing of full application Black boxNew function testing Black boxLab testing Black boxUsability testing Black boxCustomer acceptance testing Black boxField (Beta) testing Black boxClean-room statistical testing Black box
Independent testing MixedRegression testing MixedIntegration testing MixedPlatform testing Mixed
18
General: Subroutine
• Subroutine: a small collection of code that may constitute less than 10 statements or perhaps one-tenth of a function point.
• Testing consists of executing a just completed subroutine to see if it compiles properly and performs as expected.
• Subroutine testing is a "white box" form of testing.
19
General: Unit
• Unit testing is the execution of a complete module or small program that will normally range from perhaps 100 to 1,000 source code statements, or roughly from 1 to perhaps 10 function points.
• May be performed informally, it is also the stage at which actual test planning and test‑case construction begins.
20
General: Unit
• Usually performed by the programmers who write the module
• Unit testing is a "white box" form of testing.
• Unit testing is also often plagued by "bad test cases" which themselves contain errors.
21
General: New
• Often teamed with regression testing, and both forms are commonly found when existing applications are being updated or modified.
• Aimed at validating new features that are being added to a software package.
• For entirely new projects, as opposed to enhancements, this form of testing is also known as "component testing”.
22
General: New
• Normally supported by formal test plans, planned test cases, and occurs on software that is under full configuration control.
• Defect reporting for new function testing is both common and reasonably accurate.
• Both "white box" and "black box" forms of new function testing have been noted, although black box testing is more common.
23
General: Regression
• The opposite of new function testing.
• In the context of testing regression means accidentally damaging an existing feature as an unintended by-product of adding a new feature.
• Regression testing also checks to ensure that prior known bugs have not inadvertently stayed in the software after they should have been removed.
24
General: Regression
• After a few years of software evolution, regression testing becomes one of the most extensive forms of testing because the library of available test cases from prior releases tends to grow continuously.
• Regression testing involves the entire base code of the application which for major systems can exceed 10,000,000 lines of code or 100,000 function points.
25
General: Regression
• Both white box and black box forms of regression testing have been noted.
• Regression test libraries, though often extensive, are sometimes troublesome and have both redundant test cases and test cases which themselves contain errors.
26
General: Integration
• Testing on a number of modules or programs that have come together to comprise an integrated software package.
• Since integration testing may cover the work of dozens or even hundreds of programmers, it also deals with rather large numbers of test cases.
27
General: Integration
• Often occurs in “waves” as new builds of an evolving application are created.
• Applications undergoing integration testing are usually under formal configuration control.
• Integration testing normally make use of formal test plans, planned suites of test cases, and formal defect reporting procedures.
28
General: Integration
• Both black box and white box forms of integration testing have been noted.
• Integration testing can be performed by developers themselves, by professional test personnel, or by software quality assurance.
29
General: System
• System testing of full application is usually the last form of internal testing before customers get involved with field testing.
• For large systems, a formal system test can take many months and can involve large teams of test personnel. Also, the entire set of development programmers may be needed in order to fix bugs that are found during this critical test stage.
30
General: System
• System testing demands formal configuration control and also deserves formal defect tracking support.
• Normally based on black box principles, although sometimes the white box testing form is used.
• System testing can be performed by developers, professional test personnel, or by quality assurance personnel.
31
General: System
• For software that controls physical devices (such as telephone switching systems) the phrase "system test" may include concurrent testing of hardware components.
• System testing may sometimes overlap a specialised form of testing, termed “1ab testing," where special laboratories are used to house complex new hardware/software products that will be tested by clients.
32
Special: Capacity
• Aimed at judging the ability of an application to function when nearing the boundaries of its capabilities in terms of the volume of information used.
• Usually a black box form of testing.• Capacity testing may either be a separate test
stage, or performed as a subset of integration or system test (it cannot be performed earlier, since the full application is necessary.)
33
Special: Performance
• Aimed at judging whether or not an application can meet the performance goals set out for it.
• For many applications performance is only a minor issue, but for some kinds of applications it is critical.
• White box form of resting.
• Most performance testing is associated with integration and system testing.
34
Special: Viral
• This is rapidly moving from a specialised form of testing to a general one, although it still has been noted on less than half of our client's projects.
• Virus testing is a white box form of testing. • Although commercial virus protection software can be
run by anybody, major commercial developers of software also use special proprietary tools to ensure that master copies of software packages do not contain viruses.
35
Special: Security• Most common and most sophisticated for
military software, followed by software that deals with very confidential information such as bank records, medical records, tax records, and the like.
• Security testing is a white box form of testing usually performed by highly trained specialised personnel.
• Some military projects use "penetration teams" who attempt to break the security of applications by various covert means …
36
Special: Platform
• Found among companies whose software operates on different hardware platforms under different operating systems.
• While the features and functions of the application may be identical on every platform, the mechanics of getting the software to work on various platforms requires separate versions and separate test stages for each platform.
37
Special: Platform
• Usually a white box form of testing.
• Another aspect of platform testing is to ensure that the software package correctly interfaces with any other software packages that might be related to it. This is the stage where the interfaces between the two would be verified.
• Platform testing is also termed "compatibility testing" by some companies.
38
Special: Y2K
• The most recent form of specialised testing noted among SPR clients.
• By 1996 over 50 commercial tools for seeking out Year 2000 hits had entered the commercial market, and several hundred consulting and outsourcing companies had started Year 2000 services.
• Year 2000 testing was primarily a white box form of testing, although black box testing of inputs and outputs occurred too.
39
Special: Independent
• Very common for military software - it was required by Department of Defence standards.
• It can occur for commercial software, and indeed there are several commercial testing companies who do testing on a fee basis.
• Very rare for management information systems, civilian systems software projects, and outsource or contract software.
40
Special: Independent
• Performed by a separate company or at least a separate organisation from the one that built the application.
• Both white box and black box forms of independent testing are noted.
• A special form of independent testing may occur from time-to-time as part of litigation. The plaintiff or defendant, or both, may commission a third-party to test the software.
41
User: Usability
• Performed in usability laboratories.
• Usability testing involves actual clients who utilise the software under controlled and sometimes instrumented conditions so that their actions can be observed.
• Usability testing is common for commercial software produced by large companies such as IBM and Microsoft.
42
User: Usability
• Usability testing is a black box form of testing and usually occurs at about the same time as system test.
• Sometimes usability testing and Beta testing are concurrent, but it is more common for usability testing to precede Beta testing.
43
User: Beta
• Field (Beta) testing is a common testing technique for commercial software.
• An external test involving customers.
• Beta testing usually occurs after system testing, and is a black box form of testing.
• External Beta testing and internal usability testing may occur concurrently.
44
User: Beta
• Microsoft has become famous by conducting the most massive external Beta tests, with more than 10,000 customers participating.
• High-volume Beta testing with thousands of customers is very efficient in terms of defect removal efficiency levels and can exceed 85% removal efficiency if there are more than 1,000 Beta test participants.
• However, if Beta test participation comprises less than a dozen clients removal efficiency is around 35% to 50%.
45
User: Lab• Found primarily with hybrid products that
consist of complex physical devices that are controlled by software, such as telephone switching systems, weapons systems, and medical instruments.
• Normally a black box form of testing, although if certain kinds of problems are noted, such as capacity problems, white box testing can be part of the process, too.
46
User: Customer Acceptance
• Commonly found for contract software and often found for management information systems, systems software, and military software.
• The only form of software where acceptance testing is rare or does not occur is that of high-volume commercial “shrink wrapped” software.
• Usually a black box form of testing.
47
User: Clean-room• Found only in the context of clean-room development
methods. • The clean-room approach is unusual in that the developers
do not perform unit tests, and the test cases themselves are based on statistical assertions of usage patterns.
• Clean-room testing is inextricably joined with formal specification methods and proofs of correctness.
• Black box and always performed by testing specialists or quality assurance personnel rather than developers themselves.
Approximate Distribution of Testing Stages for U.S. Software Projects
Number ofTesting Stages
Percentage of ProjectsUtilising Test Stages
1 stage 22 stages 83 stages 124 stages 145 stages 166 stages 187 stages 58 stages 59 stages 710 stages 511 stages 312 stages 113 stages 114 stages 115 stages 116 stages 117 stages 018 stages 0
Total 100 %
49
Survey Results
• Majority of software projects in the USA (70%) use six or fewer discrete testing stages, and the most common pattern of testing observed includes the following:– Subroutine testing– Unit testing– New function testing– Regression testing– Integration testing– System testing
50
Survey Results
• Below 1,000 function points (and especially below 100) sometimes only three testing stages are found, assuming the project in question is new and not an enhancement:– Subroutine testing– Unit testing– New function testing
Defect Removal and Testing Stages Noted During Litigation for Poor Quality
Reliable Software Software Involved inLitigation for Poor Quality
Formal design inspections Used Not usedFormal code inspections Used Not usedSubroutine testing Used UsedUnit testing Used UsedNew function testing Used Rushed or omittedRegression testing Used Rushed or omittedIntegration testing Used UsedSystem testing Used Rushed or omittedPerformance testing Used Rushed or omittedCapacity testing Used Rushed or omitted