testing concepts

Software Development

Life Cycle

Software Development Life-Cycle Models

� Four Essential Phases of any Software Development Process

� Requirements, Analysis, Specification.

� System Design.

� Program Implementation.

� Test.

Software Development Life-Cycle Models Cont..

� Each phase has an output

OutputPhase

Test Report, Change RequestsTest

CodeImplementation

Design Document, Design ClassesDesign

Software Requirements Specification (SRS), Use Cases

Requirements analysis


� Different projects may interpret these phases differently.

� Each particular style is called a “Software Life-Cycle Model”

� Life-Cycle Model

� Single-Version Models

� Incremental Models

� Single-Version with Prototyping

� Iterative Models


� Single-Version Models

� Big-Bang Model

� Waterfall Model

� Waterfall Model with “back flow”

� “V” model: Integrating testing


� Big-Bang Model

� Developer receives problem statement.

� Developer works in isolation for some extended time period.

� Developer delivers result.

� Developer hopes client is satisfied.


Waterfall Model Requirements

Design

Implementation

TestEach phase “pours over” into the next phase.


Why Not Waterfall ?

� Complete Requirements Not

Known at Project Start

Developer works in

isolation for some

extended time period.

0

10

20

30

40

50

60

10 100 1000 10000 100000

Project Size in Function Points

Cre

ep

ing

Re

q's

as

% o

f O

rig


Why Not Waterfall ? cont..

� The market changes constantly.

� The technology changes.

� The goals of the stakeholders change.

Function Point

is a unit of complexity used in software cost estimation.

Function points are based on

number of user interactions, files

to be read/written, etc

SLOC means number of source

lines of code, also a measure of program complexity


Why Not Waterfall ? Cont..

� The design may need to change during implementation The technology changes.

� Requirements are incomplete and changing.

� Too many variables, unknowns, and novelties.

� A complete specification must be as detailed as code itself.

� Software is very “hard”.


� A function point is a unit of complexity used in software cost estimation. Function points are based on number of user

interactions, files to be read/written, etc.

� SLOC means number of source lines of code, also a measure

of program complexity.


Waterfall Model With Back Flow

Requirements

Design

Implementation

Test

Adjustments made to immediately previous

phase based on issues

with successive phase

Data Gathering (business requirement)


� V Model

Requirements Analysis

System Design

Program Design

Implementation

Unit Test

Integration Test

Acceptance Test

Each phase has corresponding test or validation

counterpart

Incremental Vs Iterative

� Incremental

� Add to the product at each phase

� Start with a modest house, keep adding rooms and upgrades to it.

� Iterative

� Re-do the product at each phase

� On each iteration, the house is re-designed and built anew.

� Big Difference: One can live in the incremental house the entire time! One has to move to a new iterative house.

� Some of the models could be used either way

What is Software Testing

� Several Definitions..

� Testing is the process of establishing confidence that a program or system does what it is supposed to.”

- by Hetzel 1973

� Testing is the process of executing a program or system with

the intent of finding errors.”

- by Myers 1979

� Testing is any activity aimed at evaluating an attribute or

capability of a program or system and determining that it

meets its required results - by Hetzel 1983

Verification and Validation

� Verification

� Methods to ensure that the system complies with an

organizational standard or process

� Did we build the right system?

� Done by project team (Review & Test)

� Validation

� Ensure that the system operates according to a plan by

executing system functions

� Did we build the system right?

� Done by customer (including review and acceptance test).

Verification and Validation

Verification and Validation Cont…

� Verification strategies

1. Requirements Review.

2. Design Review.

3. Code Walkthrough.

4. Code Inspections.

� Validation strategies

1. Unit Testing.

2. Integration Testing.

3. System Testing.

4. Performance Testing.

5. Alpha Testing.

6. User Acceptance Testing (UAT).

7. Installation Testing.

8. Beta Testing.


Verification strategies in detail..

Software ready for

testing by the testing

team.

Formal analysis of the program source code to find

defects as defined by meeting system design

specification.

Code

Inspection

Software ready for initial

testing by the developer.

Informal analysis of the program source code to find

defects and verify coding techniques.

Code

Walkthrough

System Design

Document, Hardware

Design Document.

The study and discussion of the computer system

design to ensure it will support the system

requirements.

Design Review

Reviewed statement of

requirements.

The study and discussions of the computer system

requirements to ensure they meet stated user needs

and are feasible.

Requirements

Review

DeliverableExplanationVerification

Strategy


Validation strategies in detail..

Stable application performance.Testing of the application for the

performance at stipulated times and

stipulated number of users.

Performance

Testing

Tested computer system, based on

what was specified to be developed.

Testing of entire computer system.

This kind of testing can include

functional and structural testing.

System

Testing

Portions of the system ready for testing

with other portions of the system.

Testing of related programs, modules,

or units of code.

Integration

Testing

Software unit ready for testing with other

system component.

Testing of single program, modules,

or unit of code.

Unit Testing

DeliverableExplanationValidation

Strategy


Validation strategies in detail..

Successfully installed and

running application.

Testing of the application after the installation

at the client place.

Beta Testing

Successfully installed

application.

Testing of the Computer System during the

Installation at the user place.

Installation

Testing

Tested and accepted

system based on the user

needs.

Testing of computer system to make sure it will

work in the system regardless of what the

system requirements indicate.

User Acceptance

Testing (UAT)

Stable application. Testing of the whole computer system before

rolling out to the UAT.

Alpha Testing

DeliverableExplanationValidation

Strategy

When Testing Should Occur..

� Testing can and should occur throughout the phases of a project.

� Requirements Phase

� Determine the test strategy.

� Determine adequacy of requirements.

� Generate functional test conditions.

� Design Phase

� Determine consistency of design with requirements.

� Determine adequacy of design.

� Generate structural and functional test conditions.

When Testing Should Occur..

� Program (Build) Phase

� Determine consistency with design.

� Determine adequacy of implementation.

� Generate structural and functional test conditions for programs/units.

� Test Phase

� Determine adequacy of the test plan.

� Test application system.

� Installation Phase

� Place tested system into production.

� Maintenance Phase

� Modify and retest.

Work Products Of Test

� Input

� Software Requirement Specification (SRS)

� Design documents

� Programs (Modules)

� Output

� Test documents: Test plan, Test cases, Test script, Test data

� Defects in DMS

� Test results

Types Of Testing

Two types of testing

1. Functional or Black Box Testing.

2. Structural or White Box Testing.

Types Of Testing

Functional or Black Box Testing.

� Focuses on functional requirements of the software

� Derive sets of input conditions that will fully exercise all

functional requirements for a program

� Incorrect or mission functions

� Interface errors

� Errors in data structures or external db access

� Performance errors

Types Of Testing

Structural or White Box Testing.

� It guarantees that all independent paths within a module have been exercised at least once

� Exercise all logical decisions on their true an false sides

� Execute all loops t their boundaries and within their operational bounds

� Exercise internal data structures to assure their validity

Types Of Testing

Functional or Black Box Testing.

Error introduced into the

test.

Errors can be prevented or detected and

then corrected.

Error Handling

Unchanged system

segments function.

Verifies that anything unchanged still

performs correctly.

Regression

Prove system

requirements.

System performs as specified.Requirements

ExampleExplanationTechnique

Types Of Testing

Functional or Black Box Testing Cont...

File reconciliation

procedures work.

Controls reduce system risk to an

acceptable level.

Control

Old and new system can

reconcile.

Old systems and new system are run

and the results compared to detect

unplanned differences.

Parallel

Intersystem parameters

changed.

Data is correctly passed from system to

system.

Inter Systems

Manual procedures

developed.

The people-computer interaction works. Manual

Support


Types Of Testing

Structural or White Box Testing.

Evaluate adequacy of

backup data.

System can be returned to an operational

status after a failure.

Recovery

Transaction turnaround

time adequate.

System achieves desired level of

proficiency.

Execution

Sufficient disk space

allocated.

Determine system performance with

expected volumes.

Stress


Types Of Testing

Structural or White Box Testing cont...

Access denied. System is protected in accordance with

importance to organization.

Security

Standards follow. System is developed in accordance with

standards and procedures.

Compliance

Determine systems can

run using document.

System can be executed in a normal

operational status.

Operations


Testing Phases

Requirements

Review

Design

Review

Code

Walkthrough

Code

Inspection

Integration

Testing

Unit

Testing

System

Testing

Performance

Testing

Alpha

Testing

User Acceptance

Testing

Installation

Testing

Beta

Testing

Testing Phases

Formal Technical Review’s (FTR)

The focus of FTR is on a work product (e.g. Requirements document, Code etc.). After the work product is developed, the Project Leader calls for a Review. The work product is distributed to the personnel who involves in the review. The main audience for the review should be the Project Manager, Project Leader and the Producer of the work product.

Major reviews include the following:

� 1. Requirements Review.

� 2. Design Review.

� 3. Code Review.

Testing Phases

Unit Testing

Verifies that the component/module functions properly

� Done in a controlled environment

� Normally white box oriented

� What to test:� Data structure is examined

� Boundary conditions are tested to ensure that the module operates properly for input and output data

� All independent paths are exercised

� Error handling paths are tested

� Use of stubs (modules to be called) and drivers (modules that calls)

Testing PhasesIntegration Testing - Compare with Architectural Design Document.

� Check that modules integrate with one another correctly

� Problems may occur due to:

� Data lost across interface

� One module can have an adverse affect on another

� Sub functions may not produce the desired major functions

� Individual acceptable imprecision but unacceptable if magnified

� Global data structures presents problems

� Methods of Integration testing are followed:

1. Top-down Integration approach.

2. Bottom-up Integration approach.

Testing Phases

Top-down Integration Testing

� Top-down integration testing is an incremental approach to construction of program structure.

� Modules are integrated by moving downward through the control hierarchy, beginning with the main control module.

� Modules subordinate to the main control module are

incorporated into the structure in either a depth-first or breadth-

first manner.

Testing Phases

Top-down Integration Testing cont..

� The Integration process is performed in a series of five steps:

1. The main control module is used as a test driver and stubs are substituted for all components directly subordinate to the main control module.

2. Depending on the integration approach selected subordinate stubs are replaced one at a time with actual components.

3. Tests are conducted as each component is integrated.

4. On completion of each set of tests, another stub is replaced with the real component.

5. Regression testing may be conducted to ensure that new errors have not been introduced.

Testing Phases

Bottom-up Integration Testing cont..

� Button-up integration testing begins construction and testing

with atomic modules (i.e. components at the lowest levels in

the program structure).

� Because components are integrated from the button up,

processing required for components subordinate to a given level is always available and the need for stubs is eliminated.

Testing Phases

Bottom-up Integration Testing cont..

� A Bottom-up integration strategy may be implemented with the

following steps:

1. Low level components are combined into clusters that

perform a specific software sub function.

2. A driver is written to coordinate test case input and output.

3. The cluster is tested.

4. Drivers are removed and clusters are combined moving upward in the program structure.

Testing PhasesSystem Testing

� System testing is a series of different tests whose primary purpose is to fully exercise the computer based system.

� Sources of Software Faults

� Not all faults are seen, faults may occur in different phases

� Programmers cannot be expected to spot design faults

� Faults may be added when changes are made to correct other

faults

� maintenance and documentation faults

Testing Phases

System Testing

� The following tests can be categorized under System testing:

1. Recovery Testing.

2. Security Testing.

3. Stress Testing.

4. Performance Testing.

Testing Phases

Recovery Testing

� Recovery testing is a system test that focuses the software to fall in a variety of ways and verifies that recovery is properly

performed.

� If recovery is automatic, re-initialization, check-pointing

mechanisms, data recovery and restart are evaluated for

correctness. If recovery requires human intervention, the

mean-time-to-repair (MTTR) is evaluated to determine

whether it is within acceptable limits.

Testing Phases

Security Testing

� Security testing attempts to verify that protection mechanisms built into a system will, in fact, protect it from improper

penetration.

� During Security testing, password cracking, unauthorized

entry into the software, network security are all taken into

consideration.

Testing Phases

Stress Testing

� Stress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volume.

� The following types of tests may be conducted during stress testing:

� Special tests may be designed that generate ten interrupts per second, when one or two is the average rate.

� Input data rates may be increases by an order of magnitude to determine how input functions will respond.

� Test Case that require maximum memory or other resources.

� Test Case that may cause excessive hunting for disk-resident data.

� Test Case that my cause thrashing in a virtual operating system.

Testing Phases

Performance Testing

� Performance tests are coupled with stress testing and usually require both hardware and software instrumentation..

Regression Testing

� Regression testing is the re-execution of some subset of tests that have already been conducted to ensure that changes have

not propagated unintended side affects.

� Regression may be conducted manually, by re-executing a

subset of al test cases or using automated capture/playback

tools.

Testing Phases

Regression Testing cont..

� The Regression test suit contains three different classes of test cases:

� A representative sample of tests that will exercise all

software functions.

� Additional tests that are focused on software functions that

are likely to be affected by the change.

� Tests that focus on the software components that have

been changed.

Testing Phases

Regression Testing cont..

� Can be conducted at each of the test levels: unit, integration,system

Version XModifications Changed

Version

X+1

T1 T2 T3 Tn

… T1 T2 T3 Tn

…

Test cases

to

modify

• find affected test cases (red)• change affected test cases (red)• execute them• define new test cases, if necessary

Testing Phases

Alpha Testing

� The Alpha testing is conducted at the developer sites and in a controlled environment by the end-user of the software.

Beta Testing

� The Beta testing is conducted at one or more customer sites by the end-user of the software.

� The beta test is a live application of the software in an environment that cannot be controlled by the developer.

Testing Phases

User Acceptance Testing

� User Acceptance testing occurs just before the software is released to the customer.

� The end-users along with the developers perform the User Acceptance Testing with a certain set of test cases and

typical scenarios.

Testing Phases

Beta Testing

� The Beta testing is conducted at one or more customer sites by the end-user of the software.

� The beta test is a live application of the software in an environment that cannot be controlled by the developer.

Configuration Management

� Software Configuration management is an umbrella activity that is applied throughout the software process.

� SCM identifies controls, audits and reports modifications that

invariably occur while software is being developed and after it

has been released to a customer.

� All information produced as part of software engineering

becomes of software configuration.

� The configuration is organized in a manner that enables

orderly control of change.

Configuration Management

� The following is a sample list of Software Configuration Items:

� Management plans (Project Plan, Test Plan, etc.)

� Specifications (Requirements, Design, Test Case, etc.)

� Customer Documentation (Implementation Manuals, User Manuals, Operations Manuals, On-line help Files)

� Source Code (PL/1 Fortran, COBOL, Visual Basic, Visual C, etc.)

� Executable Code (Machine readable object code, exe's, etc.)

� Libraries (Runtime Libraries, #include Files, API's, DLL's, etc.)

� Databases (Data being Processed, Data a program requires, test data, Regression test data, etc.)

� Production Documentation

Quality Assurance

� All the planned and systematic activities implemented within the quality system that can be demonstrated to provide confidence that a product or service will fulfill requirements for quality.

� Set of support activities.

� Facilitation

� Training

� Measurements & Analysis

Quality Control

� The operational techniques and activities used to fulfill requirements for quality.

� Set of control methods.

� Software Testing

� Reviews

� Walkthroughs

� Inspections

What is a bug?

� In computer technology, a bug is a coding error in a computer program.

� Myers defined it by saying that “A software error is present

when the program does not do what its end user reasonably

expects it to do.” (Myers, 1976.).

Who can report a bug?

� Anyone who can figure out that the software isn’t working properly can report a bug.

Sales and marketing staff (especially when interacting

with customers).

End users

Beta sites

Technical Support

Developers

Testers / QA personnel

Bugs Rating

� To aid in assessing the state of the product and to prioritize bug fixes, bugs are rated.

� The easiest way to rate bugs is to assign each bug a severity

rating and a likelihood rating.

� This assignment is done by the bug reporter when the bug is

created.

� The bug’s rating is a combination of the severity and likelihood

ratings.

Bugs Rating

� Severity - The severity tells the reader of the bug how bad the problem is.

� The below list takes the guess work out of assigning a severity

to bugs.

5Enhancement

4Inconvenient

3Loss with a work around

2Loss without a work around

1Blue screen

ValueRating

Bugs Rating

� Likelihood – How likely is a user to encounter this bug?

5Never

4Rarely

3Sometimes

2Usually

1Always

ValueRating

Bugs Rating

� Severity * Likelihood = Rating

� Computing the rating of a bug is done by multiplying the

numeric value given to the severity and likelihood status’.

� The trick is to remember that the lower the number, the more severe the bug is.

� The highest rating is a 25 (5 X 5), the lowest is 1 (1 X 1).

The bug with a 1 rating should be fixed first while the bug with a 25 rating may never get fixed.

Bugs Rating

Sample screen of a typical bug tracking system.

Errors, Faults, Failures

� Error or Mistake–something a person thought or did he/she shouldn’t have (bad idea or action)

� Fault or Defect–something wrong within the design or

machine (bad state or flaw), due to an error during the design

or manufacturing process

� Failure–wrong behavior, malfunction of an artifact due to the

activation of a fault

� Incident or Accident–visible effect of a failure onto the

environment of the system, esp. on people

Errors, Faults, Failures Example..

� Failure: x = 3 means y =9 �Failure!

� This is a failure of the system since the

correct output would be 6

� Fault: The fault that causes the failure is

in line 5. The * operator is used instead of

+.

� Error: The error that conduces to this fault

may be:

� a typing error (the developer has

written * instead of +)

� a conceptual error (e.g., the developer doesn't know how to double a number)

/* program for sum two numbers */LOC Code1 program double ();

2 var x, y: integer;3 begin

4 read(x);

5 y := x * x;6 write (y)

7 end

Test, Test Cases and Test Suits

� Test –the execution of a test case

� Test Case –an entity identifying preconditions, inputs

and expected outputs or post-conditions for a particular SUT (Software Under Test) behavior

� Test Suite –set of test cases for a particular testing

objective (quality measure), usually with common points of observation and control (PCOs) in the SUT

� Test Design –the construction of test suites

Example Of Test Case

Test Input Description:

1. Login to <Abc page> as administrator

2. Go to Reports page

3. Click on the ‘Schedule reports' button

4. Add reports

5. Update

Expected Results:

The report schedule should get added to the report schedule Table

Software Maintenance

� Software maintenance has been defined as …

“the modification of a software product after delivery to correct faults, to improve performance or other attributes, or to adapt the product to a changed environment.” (ANSI/IEEE, 1983)

� Maintenance activities can be classified into four categories:� Perfective maintenance

� Adaptive maintenance

� Corrective maintenance

� Preventive maintenance

Software Maintenance Steps

Change

RequestProgram

Understanding

Change location

identification

Change

implementation

Ripple effectsTestin

g


� Forward engineering is the traditional process of moving from high-level abstractions to the physical implementation of a system.

� Reverse engineering is the process of taking something (a device, an electrical component, a car, a software, …) apart and analyzing its working in details, usually with the intentionto construct a new device or program that does the same thing.

Requirements Design Implementation

Requirements Design Implementation

“Abstract Code Representation” Code


� Re-engineering is the examination (reverse engineering) of a system to reconstitute it (forward engineering) in a new form.

� This process may include modifications with respect to new requirements not met by the original system (Semantics cannot be preserved).

Software Design

� An iterative process transforming requirements into a “blueprint” for constructing the software.

� Bridging the gap between requirements & Implementation

Requirements Analysis

Software Design

Implementation

TestingDeploymentEvolution

Systems Engineering

Design Model

� Data Design

� Transforms information

domain model into data

structures required to

implement software

� Architectural Design

� Defines relationship

among the major

structural elements of a

program

Procedural Design

Interface Design

Architectural Design

Data Design

The Design Model

Design Model

� Interface Design� Describes how the software communicates with itself, to

systems that interact with it and with humans.

� Procedural Design� Transforms structural elements of the architecture into a

procedural description of software construction

Software Design

Moving from problem to solution

Requirements Spec Design Spec

System attributes, anticipated

changes, design constraints

Use Cases

Contracts

Class Diagrams

Other models

Design goals

Architecture

• subsystem structure• subsystem interfaces

•etc.

Design Process

� Design must enable all requirements of the analysis

model and implicit needs of the customer to be met.

� Design must be readable and an understandable guide for developers, testers and maintainers

� The design should address the data, functional and

behavioral domains of implementation.

Design Process

� Decompose entire project into units / modules and prepare dataflow diagram and communication.

� CDD (Comprehensive Design Document)

= HCL + LLD

Design Process

HCD LLD

High Level DesignLow Level Design

Design Process

� High-Level Design (system Design)

� High-level design is the phase of the life cycle when a logical

view of the computer implementation of the solution to the

customer requirements is developed

� Low Level Design (Detailed Design)

� During the detailed design phase, the view of the application

developed during the high level design is broken down into

modules and programs. Logic design is done for every program.

testing concepts

Documents

software cost estimation

bigbang model developer

flow v model

previous phase

function pointis

complete requirements

number of source lines

number of user interactions