software testing testing in the software lifecycle

43
Software Testing Testing in the Software Lifecycle

Post on 21-Dec-2015

272 views

Category:

Documents


10 download

TRANSCRIPT

Page 1: Software Testing Testing in the Software Lifecycle

Software Testing

Testing in the Software Lifecycle

Page 2: Software Testing Testing in the Software Lifecycle

Component Test

• Explanation of term– Verifies whether each software component performs correctly

according to its specification– Checks internal aspects and behavior of a component– Also called module test, unit test, or class test depending on

programming language used

• Test objects– The software components are tested individually and isolated

from all other software components of the system• Isolation is to prevent external influences

• If a problem exists, it is definitely originating from the component

Page 3: Software Testing Testing in the Software Lifecycle

Component Test

• Test environment– Test objects coming directly from the developer’s desk– Close cooperation with development– Case study

• Testing a class method

• C++ function to calculate the total price

• In order to test the class method a test driver is necessary: a program that calls the component under test and then receives the test object’s reaction

• The test driver

– Information could be recorded in test drivers:• Test data and results

• Date and time of test

• A function that reads test cases from a file or database

Page 4: Software Testing Testing in the Software Lifecycle

Component Test

• Test objectives– Important task is to guarantee the particular test object executes

its entire functionality correctly and completely as required by the specification (functional testing)

• This means the behavior of input/output of test object

– The component is tested with a series of test cases, where each test case covers a particular input/output combination (partial functionality)

– What do the input/output combinations in test cases for the case study tell us?

• Test case 2 checks the partial functionality of “discount in the case of five or more special equipment items”. If test case 2 is executed the test object calculates an incorrect total price. Test case 2 produces a failure

Page 5: Software Testing Testing in the Software Lifecycle

Component Testing

• Typical defect found:– Wrong calculation– Missing or wrongly chosen program paths

• Eg special cases that were forgotten or misinterpreted• Robustness testing

– Testing to determine the robustness of the software product• The degree to which a component or system can function correctly in

the presence of invalid inputs or stressful environment conditions– Function calls and test data are used that are either known to be

wrong or at least are special case not mentioned in the specification

– The component’s reaction should be an appropriate exception handling

– If there is no such exception handling, wrong inputs can trigger domain faults such as like division by zero or access through null pointer

– Such faults could lead to a program crash

Page 6: Software Testing Testing in the Software Lifecycle

Component Testing

• Such test cases are also called negative tests

• Example for case study (negative test)

// testcase 20

price = calculate_price(-1000.00,0.00,0.00,0,0);

test_ok = test_ok && (ERR_CODE == INVALID_PRICE);

//testcase 30

price = calculate_price(“abc”,0.00,0.00,0,0);

test_ok = test_ok && (ERR_CODE == INVALID_ARGUMENT);

Page 7: Software Testing Testing in the Software Lifecycle

Component Testing

• Testing nonfunctional characteristics that cannot be tested at higher test level such as efficiency and maintainability

• Efficiency states how efficiently a component uses computer resources– Use of memory measured in kilobytes– Computing time measured in milliseconds– Disk or network access time measured in milliseconds– Time required to execute the component’s function and

algorithms

• Efficiency must be verified only in efficiency-critical paths or if efficiency requirements are stated in spec.– Eg: testing embedded software, realtime system

Page 8: Software Testing Testing in the Software Lifecycle

Component Testing

• Maintainability means all the characteristics of a program that have an influence on how easy or difficult it is to change the program or to continue developing it

• Maintainability testing requires:– Code structure

– Modularity

– Quality of comments

– Adherence to standards

– Understandability

– Currency of the documentation

• Can be done by analysis of the program text and spec– Static test

• What are your analysis on the method in the case study?

Page 9: Software Testing Testing in the Software Lifecycle

Component Testing

• Test strategy– Use white box testing

• Tester can design test cases using their knowledge about the component’s program structures, its functions, and variable.

– Use debugger• Observe program variables during test execution

• Can help checking for correct and incorrect behavior

Page 10: Software Testing Testing in the Software Lifecycle

Integration Test

• Explanation of terms– second test level after component test– Supposes that the test objects subjected to components have

been tested– Groups of these components are composed to form larger

structural units and subsystems (integration)– After assembling the components, it must be confirmed through

testing that all components collaborate correctly– The goal of integration testing

• To expose faults in the interfaces and in the interaction between integrated components

– Is necessary as a further test level to find collaboration and interoperability problems and isolate the causes

Page 11: Software Testing Testing in the Software Lifecycle

Integration Test

• Explanation of terms– Component integration test is also called integration test in the

small– System integration test is also called integration test in the large

• Testing interfaces to external systems

• Test objects– Assembled components– External system or component off-the-shelf (COTS)

Page 12: Software Testing Testing in the Software Lifecycle

Integration Test

• Test Environment– Also needs test drivers– Can reuse test drivers that were used earlier for component

testing– Additional tools, called monitors, are required that read and log

data traffic between components

Page 13: Software Testing Testing in the Software Lifecycle

Integration Test

• Test objectives– To reveal interface and cooperation problems, as well as

conflicts between integrated parts– Example of integration not working

• Interface format may not compatible– Some files are missing– Developers have split the system into completely different components

than were in original design

• Faults in data exchange– A component transmits syntactically wrong or no data– The communication works but the involved components interpret the

received data in a different way (contradict or misinterpret spec)– Data is transmitted correctly but is transmitted at the wrong time

Page 14: Software Testing Testing in the Software Lifecycle

Integration Test

• Is it possible to do without component testing and execute all test cases after integration is finished?

• It is possible but only at the risk of great disadvantages– Most of failures that will occur in a test design are caused by

functional faults within the individual components– Because there is no suitable access to the individual component

some failures cannot be provoked and many faults, cannot be found

– If a failure occurs in the test, it can be difficult or impossible tp locate its origin and to isolate its cause

Page 15: Software Testing Testing in the Software Lifecycle

Integration Test

• Integration strategies– In what order should the components be integrated in order to

execute the necessary testing as quickly and easily as possible?– Components may be completed at different times

• Ad hoc strategy is to integrate the components in order in which they are ready

• As a component has passed the component test, check if it fits with another already tested component, or if it fits into partially integrated subsystem

• If so, both parts are integrated and tested

• Need to write stubs to help in integration

• Stub is a skeletal or special-purpose implementation of a software component, used to develop or test a component that calls or is otherwise dependent on it. It replaces a called component

Page 16: Software Testing Testing in the Software Lifecycle

Integration Test : Basic strategies

• Top-down integration– The test starts with the top level component that calls other

components but not called itself. – Stubs replace all subordinate components– Successively, integration proceeds with lower level components– The higher level that has already been tested serves as test

drivers– Advantage

• Test drivers are not needed or only simple ones are required, because the higher-level components that have been tested serve as main part of the test environment

– Disadvantage• Lower level components not yet integrated must be replaced by

stubs. This can be costly

Page 17: Software Testing Testing in the Software Lifecycle

Integration Test : Basic strategies

• Bottom-up integration– The test starts with the elementary system components that do

not call further components, except for functions of operating system. Larger subsystems are assembled from the tested components and then these integrated parts are tested

– Advantage• No stubs are needed

– Disadvantage• Higher-level components must be simulated by test drivers

Page 18: Software Testing Testing in the Software Lifecycle

Integration Test : Basic strategies

• Ad hoc integration– The components are being integrated in the (casual) order in

which they are finished– Advantage

• Save time, because every component is integrated as early as possible into its environment

– Disadvantage• Stubs as well as test drivers are required

Page 19: Software Testing Testing in the Software Lifecycle

Integration Test : Basic strategies

• backbone integration strategy– A skeletal or backbone is built into which components are

gradually integrated– Advantage

• Components can be integrated in any order

– Disadvantage• Labour intensive skeletal or backbone is required

Page 20: Software Testing Testing in the Software Lifecycle

Integration Test : Basic strategies

• Top-down and bottom-up integration in their pure form can only be applied to program structured in a strictly hierarchical way

• A more or less individual mix of the integration strategies may be chosen

• Any non-incremental integration, also called big bang should be avoided– Disadvantages

• The time leading up to the big bang is lost time that could have been spent on testing

• All failures will occur at the same time. It will be difficult, if not impossible, to get the system to run at all. And it will be very difficult and time-consuming to localize and correct defects

Page 21: Software Testing Testing in the Software Lifecycle

System Test

• Explanation of terms– The next level of testing after integration test is complete– Checks if the integrated product meets the specified

requirements– Why still need system test?

• In the lower test levels, the testing was done against technical specifications which is from the technical perspective of software producer. The system test looks at the system from the perspective of the customer and the future user. The tester validate whether the requirements are completely and appropriately met

• Many functions and system characteristics result from the interaction of all system components, consequently, they are visible on the level of the entire system and can only be observed and tested there

Page 22: Software Testing Testing in the Software Lifecycle

System Test

• Test object and environment– Completely assembled software system– Tested in an environment as similar as possible to the intended

operational environment– Hardware and software products should be installed on the test

platform– Example: a system test environment

Page 23: Software Testing Testing in the Software Lifecycle

System Test

• Test object and environment– Example: a system test environment

mainframe

DB server file server application server

web server

workstation

Page 24: Software Testing Testing in the Software Lifecycle

System Test

• The system test requires a separate test environment not in customer’s operational environment– Failures are likely to occur during system test and will damage

customer’s operational environment such as system crashes and data loss in the production system

– The testers have limited or no control over parameter settings and configuration of operational environment. The test conditions can gradually change because of simultaneous running of other system with the test.

Page 25: Software Testing Testing in the Software Lifecycle

System Test

• Test objectives– The goal of system test is to validate whether the complete

system meets the specified functional and nonfunctional requirements and how well it does that

– Failures from incorrect, incomplete, or inconsistent implementation of requirements should be detected

– Requirements that are undocumented or have been forgotten should be identified

Page 26: Software Testing Testing in the Software Lifecycle

System Test

• Problems in system test– Unclear system requirements

• Use exploratory testing

– Missed decision• Different people have completely different views and idea

– Project may fail• If no requirements, then not likely to meet the implicit requirement of

the customer

Page 27: Software Testing Testing in the Software Lifecycle

Acceptance test

• Focuses on customer’s perspective and judgment

• Involves customer to test

• Can be executed within lower test levels or distributed over several test levels– COTS software product may be acceptance tested when it is

installed or integrated

– Acceptance test of a component’s usability may be done during component test

– Acceptance test of new functionality may come before system testing using a prototype

Page 28: Software Testing Testing in the Software Lifecycle

Acceptance test

• How much acceptance test?– Varies considerably depending upon application risk

• The risk is high and a full acceptance test required for customer specific software

• The risk is low and not full acceptance test if an acquisition of a standard product

– Acceptance test consists of installing the system and running a few representative use cases.

• If the system is expected to cooperate with other systems in a new way, at least the interoperation should be tested

Page 29: Software Testing Testing in the Software Lifecycle

Acceptance test: Typical forms

• Testing for acceptance according to contract– For customer specific software customer in cooperation with

vendor will perform the test according to the contract• Customer, based on the results of testing, considers whether the

ordered software system is free of deficiencies and whether the development contract or the service defined by the contract has been accomplished

– The test criteria are acceptance criteria determined in the development contract

• These criteria must be formulated clearly and explicitly

• Also regulation (government, legal, or safety) must be adhered to

• Acceptance test cases be designed by, or at least reviewed by customer

Page 30: Software Testing Testing in the Software Lifecycle

Acceptance test: Typical forms

• Testing for acceptance according to contract– Software producer checks these criteria within his own system

test and rerun the test cases relevant for acceptance

– System test takes place in producer’s environment, acceptance test takes place in the customer’s actual operational environment

• Be careful, a test case that works correctly during system test may suddenly fail.

Page 31: Software Testing Testing in the Software Lifecycle

Acceptance test: Typical forms

• Testing for user acceptance– Recommended if the customer and the user are different

individuals

– Different user groups usually have different expectations

– Need to get acceptance from all user groups

– Customer organizes tests, selects test cases based on business processes and typical use scenarios

– Allow a number of representatives from future users to examine prototypes at an early stage of the project to avoid acceptance problems

Page 32: Software Testing Testing in the Software Lifecycle

Acceptance test: Typical forms

• Operational (acceptance) testing– Assures the acceptance of the system by the system

administrators

– May include the testing of backup/restore cycle, disaster recovery, user management, maintenance tasks, and checks of security vulnerabilities

Page 33: Software Testing Testing in the Software Lifecycle

Acceptance test: Typical forms

• Field test (alpha and beta testing)– Choose to carry out field test if software is supposed to run in

many different operational environments

– To identify the influences from users’ environment that are not entirely known or that are not specified, and to eliminate them if necessary

– Producer delivers stable prerelease versions of the software to pre-selected customers

• Adequately represent the market

• Whose operational environment appropriately cover possible environments

Page 34: Software Testing Testing in the Software Lifecycle

Acceptance test: Typical forms

• Field test (alpha and beta testing)– The customer then either

• Run test scenarios prescribed by the producer

• Run the product on a trail basis under realistic conditions

– Customer gives feedback to the producer• about the problems encountered

• General comments and impression

– Alpha testing• Test carried out at the producer’s location

– Beta testing• Test carried out at the customer’s site

Page 35: Software Testing Testing in the Software Lifecycle

Testing New Product Versions

• Software maintenance– Typical problems found:

• The system is run under new operating conditions that were not predictable and were not planned

• The customers express new wishes

• Functions are necessary for rarely occurring special cases that were not anticipated

• Crashes that happen rarely or only after very long uptime are reported. These crashes are often caused by external influences

– After its deployment, every software requires certain correction and improvement

Page 36: Software Testing Testing in the Software Lifecycle

Testing New Product Versions

• Software maintenance– The strategy is:

• Anything new or changed should be tested

• The remainder of the system should be regression tested

– If only its environment is changed, maintenance testing is necessary

• Migration from one platform to another– testing should repeat the operational tests within the new environment

Page 37: Software Testing Testing in the Software Lifecycle

Testing New Product Versions

• Release development– Product is not finished with the release of the first version

– Changes and extensions that are intended from beginning• A planned change of a neighbor system

• Functionality that had been planned from the beginning, but could not be implemented as early as intended

• Extensions become necessary in the course of a planned market expansion

– The strategy is:• Anything new or changed should be tested

• The remainder of the system should be regression tested

Page 38: Software Testing Testing in the Software Lifecycle

Generic type of testing : Functional Testing

• Includes all kind of tests which verify a system’s input-output behavior

• Black box testing methods are used to design functional test cases

• Test bases are the functional requirements

• In requirement-based testing– The released requirements are used as the basis for testing

– For each requirement, at least one test case is designed and documented in the test specification (usually more than one)

– The test specification is then reviewed

– Used in system testing and acceptance testing

Page 39: Software Testing Testing in the Software Lifecycle

Generic type of testing : Nonfunctional Testing

• Testing attributes of the functional behavior, or the attributes of the system as a whole– “how well” or with what quality the system should carry out its

function

• Nonfunctional characteristics that should be considered in tests (usually in system testing)– Load test

• Measuring of system behavior for increasing system loads (e.g., the number of users that work simultaneously, number of transactions)

– Performance test• Measuring of the processing speed and response time for a

particular use cases, usually dependent on increasing load

Page 40: Software Testing Testing in the Software Lifecycle

Generic type of testing : Nonfunctional Testing

• Nonfunctional characteristics that should be considered in tests (usually in system testing)– Volume test

• Observation of the system behavior dependent on the amount of data (e.g., processing of very large files)

– Stress test• Observation of the system behavior when it is overloaded

– Testing of security against unauthorized access, denial of services etc

– Stability or reliability test during permanent operation (e.g. mean time between failure or failure rate with a given user profile)

Page 41: Software Testing Testing in the Software Lifecycle

Generic type of testing : Nonfunctional Testing

• Nonfunctional characteristics that should be considered in tests (usually in system testing)– Robustness test

• Measuring the system’s response to operating error, or wrong programming, or hardware failure etc, as well as examination of exception handling and recovery

– Testing of compatibility and data conversion• Examination of compatibility to given system, import/export of data

– Testing of different configuration of the system, e.g., different version of OS, user interface language, hardware platform etc. (back-to-back testing)

– Usability test• Examination of the ease of learning the system, ease and efficiency

of operation, understandability of the system output

Page 42: Software Testing Testing in the Software Lifecycle

Generic type of testing : Nonfunctional Testing

• Nonfunctional characteristics that should be considered in tests (usually in system testing)– Checking of the documentation for compliance with system

behavior (e.g., user manual and GUI

– Checking of maintainability• Assessing the understandability of the system documentation and

whether it is up to date; checking if the system has a modular structure etc.

Page 43: Software Testing Testing in the Software Lifecycle

Testing related to Changes and Regression Testing