ceng 557 software quality assuranceceng557.cankaya.edu.tr/uploads/files/ceng 557.pdf · ceng 557...

1

CENG 557

Software Quality Assurance

&

Testing

Spring 2015-16

Dr. Sadık Eşmelioğlu

General Information

Dr. Sadık Eşmelioğlu CENG 557 2

E-Mail : [email protected]

Phone : +90 (312) 233 13 43

Hours

Class : Tuesday 18:00 – 20:50

Office : Tuesday 20:50 – 21:50

Room

Class : A202

Office : A202 – New Campus L-215

Text Book and References


Text Book:

“Software Quality Assurance: From Theory to

Implementation”, D. Galin, Addison Wesley, 2004, ISBN 978-0-

201-70945-2.

Reference Material:

Software Testing Techniques, B. Beizer, Van Nostrand Reinhold,

1992.

Software Engineering Body of Knowledge, Chapter 5 Software

Testing, 2004.

http://www.swebok.org/ironman/pdf/SWEBOK_Guide_2004.p

df

mailto:[email protected]



http://www.swebok.org/ironman/pdf/SWEBOK_Guide_2004.pdf

http://www.swebok.org/ironman/pdf/SWEBOK_Guide_2004.pdf

2

Grading Components


Midterm 25%

Will cover the concepts learned thus far.

Closed books, notes, etc.

Presentation 10%

15-Minute Presentation

Each student will pick a test technique as topic

Research using references

PC or Overhead Projector will be used, copies to be distributed to the class

Should include examples

50% of the grade will be given by the classmates (Anonymously)

Assignments 10%

4-5 assignments (Approx. every other week)

Project 20%

Pick a small piece of software

A list of Requirements for the software should be available

Write test plan, test design specs., and test cases

Final 35%

Will cover all the topics learned

Closed books, notes, etc.

Course Outline


SQA Concepts

Basic notions: Quality Assurance, Detection vs. Prevention, Verification & Validation, testing

Testing Concepts

Definition, Types and Levels of testing, Black vs. White Box testing

Test Techniques

White Box techniques

Black Box techniques

Test Planning

Test Plans

Test Design Specifications

Test Cases

Test Metrics

Pre-process metrics: Estimation

In-process metrics: Process Management

End-process metrics: Process Improvement

Test Management

Test planning, resource management, test reporting, tools

Test Automation

What and How to automate

Calendar


Wk Date Subject

1 09-Feb Introduction

2 16-Feb SQA Concepts

3 23-Feb Testing Concepts

4 01-Mar Test Techniques



7 22-Mar Test Planning

8 29-Mar Midterm

9 05-Apr Presentations

10 12-Apr Presentations

11 19-Apr Test Metrics

12 26-Apr Test Metrics

13 03-May Project Presentations

14 10-May Test Management and Automation

3

CMM Levels – Characteristics


Level Features

1 Initial Add hoc, no defined processes

Success by heroes

2 Repeatable Basic project management

Repeatable success (same team)

3 Defined Processes defined

Consistently used, applied

4 Managed Metrics gathered

Used in process management

5 Optimizing Metrics analyzed

Used in process improvement

CMM Levels – Processes


Level Processes

1 Initial Saving the day, act according to the situation

2 Repeatable

Requirements management

Software project planning

Software project monitoring

Subcontractor management

Software quality assurance

Software configuration management

3 Defined

Organizational process focus

Organizational process definition

Training Program

Integrated software management

Software product engineering

Coordination between groups

Peer reviews

4 Managed Quantitative project management

Software quality management

5 Optimizing

Defect prevention

Technology change management

Process change management

Software Development Process


Requirements

Design

Coding

Testing

Maintenance

Defect

SQA

4

Concepts – Testing in SQA


Process

and

Procedures

SQA

Change

Control … Standards

and

Templates

Verification

and

Validation

Inspection Review Audit Test

Software Dev – Waterfall Model


R

D

C

T

Software Dev – Incremental Model


R D C T

R D C T

R D C T

PH 1

PH 2

PH 3

5

Software Dev – Cyclic Model


R

D

C

T

R

D

C

T

SQA, V&V, and Testing


SQA is a set of activities and processes to ensure that an

end product is of desired quality.

V&V ensures that the product of each phase of software

development confirms to the requirements of that phase.

Testing is a process to detect faults in the end product.

V&V is a part of the QA process and Testing is a part of the

V&V process.

Detection vs. Prevention


QA focuses not only on detection but prevention of

faults (a.k.a. defects, failures, errors, etc.) as well.

Establish a suitable development environment

Define processes and procedures to be followed

Define how the effectiveness of the processes and

procedures are to be measured and improved

Etc.

V&V and Testing focus on detection only.

Inspections, Audits, Reviews, Walkthroughs (Static)

Unit, Integration, System, and Acceptance Testing (Mostly

Dynamic)

6

Cost of Fixing a Defect


12

5

10

20

0

2

4

6

8

10

12

14

16

18

20

Cost

Requirements

Design

Coding

Testing

Maintenance

Software Testing


Software testing consists of the dynamic verification of the

behavior of a program on a finite set of test cases, suitably

selected from the usually infinite executions domain, against

the expected (“specified and implied”) behavior.

(SWEBOK, 2004)

Dynamic : Code is executed

Finite : Limited # of Test Cases, not exhaustive

Selected : Appropriate set of Test Cases

Expected : Describing the desired outcome

(Pass/Fail Criteria – Test Oracle)

Scope of Testing


The purpose of testing is to find faults. If no faults are

found, then the testing cannot be considered successful!

Testing should cover not only the expected scenarios

(Sunny Day Scenarios), but unexpected scenarios (Rainy

Day Scenarios) as well.

Testing starts with requirements.

7

System Under Test (SUT)


Mod

A

Mod

B

Mod

D

Mod

C

Mod

E

Sys X

SUT

Sys Y

Input/Output

The function of each line is described in documents called

Specifications/Requirements

UI

XI

Levels of Testing wrt Target


Unit Testing

Testing a module wrt a low level design specification

Debuggers, Simulators, and Stubs are used for testing

Integration Testing

Testing several modules together wrt a high level design specification

Debuggers, Simulators, Drivers, and Stubs are used for testing

Top-down, Bottom-up, or Architecture driven approaches can be used

System Testing

Testing the SUT wrt a system specification

Simulators are used for stress and capacity testing

Levels of Testing wrt Objective


Installation (OA&M)

Cannot use the system if cannot install properly. Other typical administrative function are also tested, backup, restore, user administration, etc.

Alfa & Beta

Friendly users. No specific plan for testing. Users report the problems they encounter.

Functional

Heart of testing. Conformity to specifications

Performance

Users are not very patient. Testing response time under different load conditions

Stress

Break SUT. Increase the load and capacity until the system breaks.

Back-to-back

Just compare results. Same tests executed on two version of a product and results are compared.

8

Levels of Testing wrt Objective (Cont’d)


Recovery

System crashed. Can system be restarted without any major loss of data?

Configuration

Same SW different environments. Unix/Windows, ORACLE/SYBASE, etc.

Usability

User Friendly. On-line Help available, selecting instead of typing, comprehensible error messages, etc.

Acceptance

Before I pay. Customer needs to agree to what they are getting is what they asked for.

Reliability

This is not what I got before! System behaves in a consistent manner. When a test is executed twice under the same conditions, results should be the same.

Regression

New version old functionality. When new functionality is working, what used to work before should still work.

Test Techniques


Intuition-Based

Ad Hoc

Specification-Based

Equivalence partitioning

Boundary-value analysis

Decision table

Finite-state machine-based

Testing from formal specifications

Random testing

Code-Based

Reference models for code-based testing (flow graph, call graph)

Control flow-based criteria

Data flow-based criteria

Test Techniques (Cont’d)


Fault-based

Error guessing

Mutation testing

Usage-based

Operational profile

SRET

Based on nature of application Object-oriented testing

Component-based testing

Web-based testing

GUI testing

Testing of concurrent programs

Protocol conformance testing

Testing of distributed systems

Testing of real-time systems

Testing of scientific software

9

Black-Box Testing


Equivalence partitioning

Boundary-value analysis

Decision table

Finite-state machine-based

Testing from formal specifications

Error guessing

Random testing

Operational profile

SRET

White-Box Testing


Reference models for code-based testing (flow graph, call

graph)

Control flow-based criteria

Data flow-based criteria

Mutation testing

Black Box/White Box Testing


Black Box Testing

SUT is treated as a closed box

Does not pay attention to how the software is written

System Testing is mostly Black Box

White Box Testing

SUT is treated as an open box

Knowledge of code and design is needed

Unit and Integration testing is mostly White Box

10

An Example


Requirements for “Stock Purchase Wizard” (SPW)

1. Stock Selection(ss)

1. Screen will include a select box which lists all the stocks available for purchase

2. The user will be able to select only one stock

3. The selection will be done either by double clicking on a stock or single clicking and pressing “Ekle” button.

4. The selected stock symbol will be automatically populated in the “Hisse” text box.

2. Number of Stocks (ns)

1. The number of stocks to be purchased will be determined by two entries:

1. A box for entering the number “Miktar” and

2. Another box to indicate whether the number denotes lots or individual number of stocks.

3. Purchase Price (pp)

1. The purchase price will be selected from a box of valid numbers. The valid values will be automatically generated and can be “Serbest” or a number rounded to the nearest 0.01 between 10% below or above the opening price of the stock.

2. Once the purchase price is selected, the total purchase amount (# of stocks * purchase price) will be automatically populated into the “Tutar” box

An Example (Cont’d)


4. Number of Sessions (sn)

1. The user will be able to select from a pull down menu, the number of sessions the

order will be in the market.

2. The valid values in this menu are

1. “Bir” if the order is entered during the afternoon session

2. “Bir” and “Iki” if the order is entered during the morning session

3. The order will be canceled if the stock did not reach the purchase price indicated by

the user during the number of sessions selected by the user

5. Confirming the order (co)

1. Once the user fills all the boxes and clicks on “Devam”, the program will respond

with

1. A six-digit confirmation number if the order is valid

2. An error message if one of the fields is missing or invalid



11



TC Id Req. Id Priority Description

SPW-FN-ss-01 1.1 H Check stock list to make sure it contains all the stocks avail. for purchase

SPW-FN-ss-02 1.2 H Click one stock and check if highlighted

SPW-FN-ss-03 1.2 M Try to click on more than one stock with clk, shift-clk and cntl-clk

SPW-FN-ss-04 1.3, 1.4 H Click on one stock, press “Ekle” and check “Hisse” box

SPW-FN-ss-05 1.3, 1.4 H Dbl-clk on one stock and check “Hisse” box

SPW-FN-ns-01 2.1.1 L Enter a negative number in “Miktar”

SPW-FN-ns-02 2.1.1 L Enter zero in “Miktar”

SPW-FN-ns-03 2.1.1 M Enter a positive number in “Miktar”

SPW-FN-ns-04 2.1.2 M Check if selection for unit of stocks contain only “Lot” and “Adet”

SPW-FN-pp-01 3.1 M Check if “Emir Fiyati” selection contains only Serbest and valid values

SPW-FN-ns-02 3.2 M Select “Serbest” and check the “Tutar” box

SPW-FN-ns-03 3.2 M Select lowest value and check the “Tutar” box

SPW-FN-ns-04 3.2 M Select highest value and check the “Tutar” box

SPW-FN-ns-05 3.2 M Select a value in the middle and check the “Tutar” box

Test Scenarios for “Stock Purchase Wizard”



TC Id Req. Id Priority Description

SPW-FN-se-01 4.1 Check if there is a pull down menu to select the number of sessions

SPW-FN-se-02 4.2 Check if “Gecerli Seans” box contains only “Bir” before noon

SPW-FN-se-03 4.2 Check if “Gecerli Seans” box contains only “Bir” and “Iki” after noon

SPW-FN-se-04 4.3 Enter an order for one session with a price which is not reached, check if

order still exists during the second session

SPW-FN-se-05 4.3 Enter an order for two sessions with a price which is not reached, check if

order still exists during the second session

SPW-FN-se-06 4.3 Enter an order for two sessions with a price which is not reached, check if

order still exists after the second session

SPW-FN-co-01 5.1.1 Enter a valid order, click “Devam” and check the confirmation number

SPW-FN-co-02 5.1.2 Enter an invalid order, click “Devam” and check the error message

Test Scenarios for “Stock Purchase Wizard”

Another Example


Requirements for “Vending Machine”

1. The vending machine will accept only ¢5 and ¢10 coins

2. The cost of a can of soda is ¢25

3. When the coins inserted reach ¢25 or more, a can of soda will be dropped and excess coins will be returned.

4. After a soda can is dropped, the machine will be reset to initial state.

12

Another Example (Cont’d)


0

30

10

5

15

20 25

25

20

25 30

15

20

25 30

25

15

10

20

25 30

25

20

25 30

Decision Tree

13 TCs

Another Example (Cont’d)


State Diagram

TC #1 5 + 5 + 5 + 5 + 5 3 TCs

Each State and path must be

traversed at least once.

0 5 10 15 20 5/00

5/00 5/00

5/00

5/10

10/15

10/00 10/00

10/00 10/10

TC #2 5 + 10 + 10

TC #3 10 + 10 + 10

x/yz

x 5 – 10

y 0 no soda

1 soda

z 0 no coin

5 5 kurus back

Yet Another Example


Requirements for “Home Alarm System”

1. The system is activated and deactivated by entering the security code

2. The system has three sensors: One is connected to the main door, one to the living room window and the last one to the bedroom window.

3. If at least one of the sensors indicate an entry, the system will sound the alarm if it is in active mode.

4. If the system is not in the active mode, no alarm will be sounded.

13

Yet Another Example (Cont’d)


SA DR W1 W2 Alarm

N N N N Off

N N N Y Off

N N Y N Off

N N Y Y Off

N Y N N Off

N Y N Y Off

N Y Y N Off

N Y Y Y Off

Y N N N Off

Y N N Y On

Y N Y N On

Y N Y Y On

Y Y N N On

Y Y N Y On

Y Y Y N On

Y Y Y Y On

Test all scenarios: 16 TCs



SA DR W1 W2 Alarm

N N N N Off

N N N Y Off

N N Y N Off

N N Y Y Off

N Y N N Off

N Y N Y Off

N Y Y N Off

N Y Y Y Off

Y N N N Off

Y N N Y On

Y N Y N On

Y N Y Y On

Y Y N N On

Y Y N Y On

Y Y Y N On

Y Y Y Y On

Testing for single-mode

faults: 2 TCs

Each input value is tested

once. This testing

guarantees finding single-

mode faults.



SA DR W1 W2 Alarm

N N N N Off

N N N Y Off

N N Y N Off

N N Y Y Off

N Y N N Off

N Y N Y Off

N Y Y N Off

N Y Y Y Off

Y N N N Off

Y N N Y On

Y N Y N On

Y N Y Y On

Y Y N N On

Y Y N Y On

Y Y Y N On

Y Y Y Y On

Orthogonal Array Testing: 6 TCs

Orthogonal Array Testing

(a.k.a. Pairwise Testing)

means, each input value is

tested with each other input

value. This testing

guarantees finding double-

mode faults.

14

Another Decision Table Example


Requirements for “FlyHigh Airline Reservation System”

The SUT is an online reservation program for an airline, called FLYHIGH.

The requirements for this software are listed below:

FLYHIGH flies from Istanbul to only four destinations, Ankara, Izmir,

Antalya, and Paris.

Flights to domestic cities have only Business and Economy classes, but

international flights have First Class as well.

The customers can select an isle, middle or a window seat.

The customers can also request vegetarian or regular food.

Once the user selects the Destination, Class, Seat Type, and Food Type, the

software will either make the reservation if the selections are valid or

print an error message if the selections are not valid.

Total scenarios = 4x3x3x2 = 72

Another Decision Table Example (Cont’d)


Dest Class Seat Food Out

Pairwise scenarios = 12

ANK ANK

IZM

ANT

PAR

BUS

ECO

FST

BUS

ECO

FST

ANK

ANK

IZM

IZM

IZM

BUS

ECO

FST

ANT

ANT

BUS

ECO

FST

PAR

PAR

PAR

BUS

ECO

FST

ISL

MDL

WIN

ISL

MDL

WIN

ISL

MDL

WIN

ISL

MDL

WIN

ISL

MDL

WIN

REG

VEG

REG

VEG

VEG

REG

REG

VEG

VEG

REG

ANT

REG

VEG

REG

VEG

VAL

VAL

VAL

VAL

Err

VAL

Err

VAL

Err

VAL

VAL

VAL

DEST CLASS SEAT FOOD

One More Example


Requirements for “ATM – Withdrawing Money”

1. ATM allows cash to be withdrawn in TL20 increments.

2. Maximum cash limit for withdrawals in a day is TL600

3. If a valid amount is entered, the transaction is successful, if not an error message is displayed according to the error

1. E1: The amount entered should be in TL20 increments

2. E2: The amount entered exceeds daily limit

3. E3: Invalid number (All other errors)

15

One More Example (Cont’d)


TC ID Input Expected

Result

ATM-FN-mw-1

20 S

ATM-FN-mw-2 300 S

ATM-FN-mw-3 600 S

ATM-FN-mw-4 10 E1 (or E3?)

ATM-FN-mw-5 620 E2

ATM-FN-mw-6 250 E1

Equivalence Partitioning

1. Valid Values (20, 300, 600)

2. Invalid Values

1. <20 (10)

2. >600 (620)

3. Invalid Number (250)

An Additional Example


Requirements for “Calculate Commission”

1. Minimum commission is TL1

2. If Amount > TL500, then commission is 0.2%

3. If Amount > TL5K, then commission is 0.15%

4. If Amount > TL50K, then commission is 0.1%

Pseudo Code for “Calculate Commission”

1 If Amount > 50K

2 then Commission = Amount * 0.001

3 Else If Amount > 5K


5 Else If Amount > 0.5K


7 Else Commission = 1

8 Return Commission

An Additional Example (Cont’d)


1

>50K

3

>5K

7

<=.5K

5

>.5K

8

Return

2

4

6

n n n

y

y

y

Flow Graph Cyclomatic Complexity

V(G) = E – N + 2

= 10 – 8 + 2 = 4

V(G) = R = 4

E: Edges (Links)

N: Nodes

R: Regions

R1

R2

R3

R4

16




Result

CC-FN-fg-1

0.5K 1.0

CC-FN-fg-2

5K 10

CC-FN-fg-3

50K 75

CC-FN-fg-4 60K 60

Paths:

a) 1-3-5-7-8 (<=0.5K)

b) 1-3-5-6-8 (>0.5K)

c) 1-3-4-8 (>5K)

d) 1-2-8 (>50K)

TCs

Apply Mutation Testing technique to this

example.

M1: Change the first > to <

M2: Change the second > to <

M3: Change the third > to >=

M4: Change all the commission % to 0.002

Etc.




Result

CC-FN-bv-1 499 1.0000

CC-FN-bv-2 500 1.0000

CC-FN-bv-3 501 1.0020

CC-FN-bv-4 4,999 9.9980

CC-FN-bv-5 5,000 10.0000

CC-FN-bv-6 5.001 7.5015

CC-FN-bv-7 49,999 74.9985

CC-FN-bv-8 50,000 75.0000

CC-FN-bv-9 50,001 50.0010

If we do not know the code, we need to use a Black Box Technique. Best one is Boundary Value:

TCs

0.5K 5K 50K

1M .2% .15% .1%

Compare the two techniques.

If the developer used “>=“ instead of “>”

comparisons in the code, would both

techniques uncover this error?




Result

CC-FN-bv-1 495 1.00

CC-FN-bv-2 500 1.00

CC-FN-bv-3 505 1.01

CC-FN-bv-4 4,995 9.99

CC-FN-bv-5 5,000 10.00

CC-FN-bv-6 5.007 7.51

CC-FN-bv-7 49,993 74.99

CC-FN-bv-8 50,000 75.00

CC-FN-bv-9 50,010 50.01

If we do not know the code, we need to use a Black Box Technique. Best one is Boundary Value:

TCs

0.5K 5K 50K

1M .2% .15% .1%

Compare the two techniques.

If the developer used “>=“ instead of “>”

comparisons in the code, would both

techniques uncover this error?

17

Planning for the Test Phase


Identify Roles and Responsibilities

Become Familiar with Requirements and Design

Write a Test Plan

Write Test Design Specifications

Write Test Cases

Have Test Plan and Design Spec. Reviewed and Approved

Prepare the Test Environment

Prepare the Test Tools

Prepare the Test Data

Test Documentation


Project Plan Master

Test Plan Quality Plan

Test Plan 1 Test Plan K . . .

Test

Design 1

Test

Design L

Test Case 1 Test Case M

. . .

. . .

SRS

Design

Test Plan


1. Test plan identifier: Specifies the unique identifier assigned to the test plan.

2. Introduction: Summarizes the software items and software features to be tested; Defines what this test plan covers and provides references to the documents relevant for testing (overall project plan, quality assurance plan, configuration management plan, applicable standards…); Provides a list of terms and abbreviations along with their definitions.

3. Test items: Identifies the items to be tested, including their version/revision level; provides references to the relevant item documentation (requirements specification, design specification, user’s guide, operations guide, installation guide …); also identifies items which are specifically excluded from testing.

4. Features to be tested: Identifies all software features and their combinations to be tested, identifies the Test Design Specification (TDS) associated with each feature and each combination of features.

5. Features not to be tested: Identify all features and significant combinations of features which will not be tested, and the reasons for this.

6. Approach: Describes the overall approach to testing (the testing activities and techniques applied, the testing of non-functional requirements such as performance and security, the tools used in testing); specifies completion criteria (for example, error frequency or code coverage); identifies significant constraints such as testing-resource availability and strict deadlines; serves for estimating the testing efforts.

18

Test Plan (Cont’d)


7. Item pass/fail criteria: Specifies the criteria to be used to determine whether each test item has passed or failed testing.

8. Suspension criteria and resumption requirements: Specifies the criteria used to suspend all or portion of the testing activity on the test items (at the end of working day, due to hardware failure or other external exception …), specifies the testing activities which must be repeated when testing is resumed.

9. Test deliverables: Identifies the deliverable documents, typically test-design specifications, test-case specifications, test-procedure specifications, test-item transmittal reports, test logs, test-incident reports, description of test-input data and test-output data, description of test tools.

10. Testing tasks: Identifies the set of tasks necessary to prepare and perform testing (description of the main phases in the testing process, design of verification mechanisms, and plan for maintenance of the testing environment …).

11. Environmental needs: Specifies both the necessary and desired properties of the test environment (hardware, communications and systems software, software libraries, test support tools, level of security for the test facilities, drivers and stubs to be implemented, office or laboratory space …).

Test Plan (Cont’d)


12. Responsibilities: Identifies the groups of persons responsible for managing, designing, preparing, executing, witnessing, checking, and resolving the testing process; identifies the groups responsible for providing the test items (section 3) and the environmental needs (section 11).

13. Staffing and training needs: Specifies the number of testers by skill level, and identifies training options for providing necessary skills.

14. Schedule: Includes test milestones (those defined in the overall project plan as well as those identified as internal ones in the testing process), estimates the time required to do each testing task, identifies the temporal dependencies between testing tasks, and specifies the schedule over calendar time for each task and milestone.

15. Risks and contingencies: Identifies the high-risk assumptions of the test plan (lack of skilled personnel, possible technical problems …), specifies contingency plans for each risk (employment of additional testers, increase of night shift, exclusion of some tests of minor importance …).

16. References: Lists all the reference documents.

17. Approvals: Specifies the persons who must approve this plan.



1. Subfeatures to be tested

Provide a list of subfeatures to be tested

2. Subfeatures not to be tested

Provide a list of subfeatures not to be tested and why

3. Approach

List refinements or deviation from the Test Plan.

4. Item Pass/Fail Criteria

List refinements or deviation from the Test Plan.

5. Environmental Needs

Provide any special environmental needs, such as data, tools, etc.

6. Test Cases

TC_ID Req Scenario Description

OM.LG.fn.01 1.1.1 Login with a valid id and password

OM.LG.fn.02 1.1.1 Login with a valid id and an invalid password

OM.LG.fn.03 1.1.1 Login with an invalid id and a valid password

19

Test Cases


TC ID : MO.LG.fn.01

Purpose : Login with a valid id and password

Requirements : 1.1.1

Priority High (High, Medium, Low)

Category Acceptance (Acceptance, Regression, ...)

Est. Time : 5 Minutes (Estimated Time to run the test case)

Dependency : None (List any test cases that need to be run first, if any)

Setup : Bring up Netscape Web Browser (List the needs prior to executing this test case)

Procedure : 1. Go to http://online.metu.edu.tr web site

2. Enter the valid id and password into the corresponding fields

3. Click on the Login button

4. Metu-Online main screen should appear

Cleanup : Logoff (Describe what needs to be cleaned for next set of test cases)

Need for Measurement - Metrics


Why do we need metrics?

Pre-process Estimation

How much effort is needed?

How many defects do we expect to find?

In-process Monitoring/Managing

How much efforts is being spent?

Are we finding enough defects?

Post-Process Process Improvement

Is the quality of the material sufficient to release to the next phase?

What can be done to improve the next project?

Update benchmark data.

Types of Metrics – Pre-Process


Helps us estimate the size of testing effort

Size of SUT

# of Requirements

# of Lines of Code

Function Points

Complexity

Coding Effort in Staff-Months

Quality of SUT

Requirements/Code Inspected

# of Faults Found During

Requirements Inspection

Code Inspection

Unit Testing

Integration Testing

Results from Previous Projects

http://online.metu.edu.tr/

20

Metrics DB

Date Size

(KLOC)

Effort

(Staff-Months)

Defects

2005 120 18 240

2006 160 24 310

2007 200 29 450

2008 250 35 490

2009 300 38 550

2013 200 30 350

TOTAL 1030 144 2040

Dr. Sadık Eşmelioğlu 58 CENG 557

Types of Metrics – In-Process


Helps us track progress and determine quality of software

Number of Test Cases Scheduled Run

Passed Failed Inconclusive

Blocked

Number of Faults Found Severity 1/2/3/4

Faults on Fixes Fault Density

Time Spent on Test Execution CPU Hours Staff Hours

Testing Effectiveness Requirements Coverage/Traceability Fault Seeding

Mutation Score

Types of Metrics – End-Process


Helps us determine final quality of product and improve our process

Totals of In-Process Metrics

# and Severity of Faults Outstanding (not Fixed)

Faults Found in Test Cases

Defect Removal Efficiency (DRE)

FT

DRE = -------------- * 100

FT + FC

FT: Faults found during System Test

FC: Faults found by Customer

21

Reporting Faults Found


Severity 1: The whole system or a major functionality does not work and

there is no workaround.

Severity 2: A major functionality does not work, but there is a

workaround or a minor functionality does not work and there is no

workaround.

Severity 3: A minor functionality does not work, but there is a

workaround.

Severity 4: Spelling and cosmetic type of faults or enhancements that are

not explicitly stated in the requirements

These definitions need to be included in the Project Plan or Master

Test Plan and agreed to by all the parties involved.

Classification of Faults – What and When


Source

Requirements

Design

Code

Hardware

Testing

Phase Detected

Requirements

Design

Code

Unit Testing

Integration Testing

System Testing

Field

Defects by Phases – Case 1


Req Des Code Unit

TstInteg

TstSys

TestCust

Code

Req0

5

10

15

20

25

30

35

40

22

Defects by Phases – Case 2


Code

Req0

10

20

30

40

50

60

Re

q

Co

de

Inte

gT

st

Cu

st

Classification of Faults - Nature


Logic

Standards

Interface

Functionality

Performance

Data

Maintainability

Testability

Human Factors

Input/Output

Documentation

Duplicate

Not Fault

Tracking the Progress of Testing


0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10

Planned

Executed

Passed

23

Tracking the Quality– Faults Found by Week


0

20

40

60

80

100

120

0 1 2 3 4 5 6 7 8 9 10

Sev1

Sev2

Sev3

Sev4

Total

Tracking the Quality–Faults Found by CPU Hrs


0

20

40

60

80

100

120

0 1 2 3 4 5

Sev1

Sev2

Sev3

Sev4

Total

Tracking the Quality– Fault Density (Faults/KLOC)


0

0,5

1

1,5

2

2,5

1 2 3 4 5 6 7 8 9 10

Benchmark

Actual

24

Tracking the Quality – Faults Outstanding


0

5

10

15

20

25

30

35

40

45

1 2 3 4 5 6 7 8 9 10

Sev1

Sev2

Sev3

Sev4

Total

Deciding When to Stop – Exit Criteria


100% of the test cases are executed

95% of the test cases passed

All High Priority test cases passed

Sev 1 Faults = 0

Sev 2 Faults ≤ 2

Sev 3 Faults ≤ 5

Sev 4 Faults ≤ 10

These criteria need to be included in the Project Plan or Master Test Plan and agreed to by all the

parties involved.

Questions


1. When can we stop testing?

2. How many bugs can we expect?

3. Which test technique is more effective?

4. Are we testing hard or are we testing smart?

5. Do we have a strong program or a weak test suite?

25

Lessons Learned – Postmortem


Conducted with the SQA Engineers

Can be for only Testing or Project-wide

Present all the data

Brainstorm strengths and weaknesses

Praise the strengths

Identify areas of improvement

Use a voting technique to identify the importance of each area

Use a Pareto diagram to prioritize the results

Select 3 top priority areas

Establish a QIT for each area

Test Management


Test Plan


Test Cases

All the fields in the template and more

Traceability to Requirements

Schedule

When/What

Results

Run/Pass/Fail

Defects (Traceability to Test Cases)

History

Test Automation


Why Can be run multiple times

Can be run 7x24

Faster execution

When/What Stable feature set

Frequent regression testing

Automation Degree of Difficulty (Cost/Benefit)

How Scripts (Semi/Full Automation)

Tools (TSL, Capture/Replay)

26

Test Automation – Road Blocks


Management

Wants automation, but not willing to spend the money

(Tool, Training, Learning Curve)

Time

Effort is similar to Code development

Changes in Requirements need to be reflected

Constant Maintenance

Technical

Synchronization

Windows pop up in different locations

THE END


QUESTIONS

ceng 557 software quality assuranceceng557.cankaya.edu.tr/uploads/files/ceng 557.pdf · ceng 557...

Documents