introduction to software testing. types of software testing unit testing strategies – equivalence...

Introduction to Software Testing

• Introduction to Software Testing• Types of Software Testing• Unit Testing Strategies– Equivalence Class Testing– Boundary Value Testing– Output Testing

• System Testing• Usability Testing

Outline

2

What is Software Testing?

3

• software testing: Evaluating software by observing its execution

• Primary goal of testing is to find bugs!• During testing, should execute tests that find

bugs.• “Testing can only prove the presence of bugs,

not their absence.” - Dijkstra • When do you stop testing?

What is Software Testing?

4

• test case: A set of inputs, execution conditions, and a pass/fail criterion.– Example: height input is 5, width input is “Hello”, the

test passes if the program produces an error message.

• test or test execution: The activity of executing test cases and evaluating their results.

• test suite: A set of test cases.

Test Terminology

5

• defect: A mistake in the source code (static)– Synonyms: fault, bug

• failure: An observable incorrect program behavior (output)

• debugging: The process of finding and correcting a defect in a program given a failure

More Test Terminology

6

Software Testing is Hard! Why?

7

// Solve ax2 + bx + c = 0struct RootAnswer { float root_one; float root_two;}RootAnswer roots(float a, float b, float c);

• Which values for a, b, c should we test?

• Need to select test cases intelligently!

Testing Inputs

8

• Due to time constraints, only a small subset of actual inputs can be used to test.

• Need to choose a test suite such that…– Each “functionality” provided by a program is

tested by at least one test case.– Tests target common programming mistakes.– Tests exercise areas of code that are likely to be

“buggy”.

Test Case Selection

9

• Code Inspection: Looking at the source code directly for bugs.

• Tools– compiler– static analyzer– memory leak detector– profiler / performance monitors

Other Techniques for Finding Bugs

10



Outline

11

• Functional testing: Testing to determine if the software is working properly based on the specification (has the proper function).

• Non-functional testing: Testing other desirable properties of the software.– Examples: performance, usability– Testing is vastly different.

Functional vs. Non-Functional Testing

12

• Unit Testing: Testing of a “unit” in the program.– Unit: smallest part of the program – only assigned

to one developer.• Often a single function or class.

– Focuses on the functionality of that unit.– Can be automated using unit testing tools• Visual Studio• Junit• Google Testing Framework

Unit Testing

13

• Integration Testing: Testing the interactions of the different units.– Assumes unit testing is complete or will catch

errors within the unit.– Focuses on the interactions between units.– Can be tested manually or can be automated.

Integration Testing

14

• System Testing: Testing the entire system.– Requires the entire system to be present.– Used to test common usage scenarios.– Used to test non-functional requirements.– Best tested manually.• Testing approaches are vastly different for non-

functional requirements.

System Testing

15



Outline

16

• Derive tests from the specification of the unit.• Need to choose tests…– Intelligently– Systematically– Thoroughly

• Consider the unit under a test as a “function” with inputs and outputs– All unique and interesting inputs should be tested– All unique and interesting outputs should be produced– Focus on common errors

Unit Testing Strategies

17



Outline

18

• Equivalence Class Testing: Partition the input space into equivalence classes.

• The key is to divide the input space into equivalence classes such that all inputs within each equivalence class have the same functionality.– Very likely that each input within a equivalence class will

exercise the same code within the function.

• Need to have at least one test in each equivalence class.– More tests are better.

Equivalence Class Testing

19


Failure (valuable test case)Failures are sparse in the space of possible

inputs ...

... but dense in some parts of the space

If we systematically test some cases from each part, we will

include the dense parts

Functional testing is one way of drawing orange lines to isolate

regions with likely failures

The

spac

e of

pos

sibl

e in

put v

alue

s

No failure

20

• The division into equivalence classes is based on the desired / specified functionality of the subject under test.

• Equivalence classes may consist of:– one particular value– a range of values– a subset (possibly discontinuous) set of values

• The set of equivalence classes should include both valid and invalid inputs.


21

1. Volume discount if you buy more than 1000 units.

2. Function that returns the number of days in a month (1-12).

3. If the area code is 800 or 888, there is no charge for the call.

4. Shipping rates differ depending on what country you are in.

Equivalence Class Testing Examples

22

Identify equivalence classes to test for each for a function that returns a letter grade (A, B, C, D, or F) based on a numeric score from 0-100.

Class Problem: Equivalence Class Testing

23



Outline

24

• Boundary Value Testing: Test the values near and at the boundary of an ordered input.

• For ordered inputs, history suggests that values near the boundary are most error prone.– Programmers may forget to check for a boundary condition– Programmers may code the boundary incorrectly (use < instead

of <=)– Special case or different code may be used at the boundary

Boundary Value Testing

Possible test case

25

• For each boundary n, want to have three cases:n – 1, n, n +1

• For a range of values from x to y, have at least six cases:x – 1, x, x + 1, y – 1, y , y + 1

• Boundary value testing can be combined with equivalence class testing:– Boundaries mark the endpoints of equivalence classes.– Need to test the boundaries of each equivalence class.– Still desirable to test one or more values in the middle

of the range.


26

• Some values are outside of the range:– May also be part of another equivalence class

(worry about redundancy later)– May be invalid – should check to make sure it’s

invalid• Boundary analysis can be applied to more

situations than handling numeric data:– string length– size of arrays (or other data structures)– data type limitations (size of integer)


27

Class Problem: Boundary Value Testing

28

Identify boundary values to test for each for a function that returns a letter grade (A, B, C, D, or F) based on a numeric score from 0-100.



Outline

29

• Equivalence class and boundary value testing can also be applied to the output of a function.

• Outputs can be divided into equivalence classes.– At least one test should be executed for each output

equivalence class.– In many cases, but not all, tests created for output are

redundant to equivalence class tests created for inputs.

• Boundaries of an ordered output should also be tested.

Output Testing

30



Outline

31

• System Testing focuses on the whole system:– Consists of not only software, but possibly

hardware and other components. – Necessary to understand the solution / product as

a whole.• Encompasses functional and non-functional

testing:– Functional: Does the software work?– Non-functional: Performance, usability, security,

etc.

System Testing

32

• Scenario-Based Testing: Develop a scenario or a script that is important to execute.

• Scenarios tend to mimic realistic use of the system. – How would a normal user use the software?

• Include invalid input scenarios:– How would a user accidentally or intentionally

misuse the system?

Scenario-Based Testing

33

• Exploratory Testing: Test the program without constraint.– Also called ad-hoc testing.– No script or scenario.

• Allows the tester to creatively try tests as they explore the application.

Exploratory Testing

34



Outline

35

What makes a software product usable?

Usability Testing

36

• Usability Testing: Tests the usability of a software product.

• Typically done with representative samples of real users to try out the software.

• Objective criteria:– Time and number of operations to perform a task– Frequency of user error• blame user errors on the product!

Usability Testing

37

• Ask questions with the user in mind:– Does the user interface make sense?– Are the prompts / messages clear?– Is the response appropriate when an invalid input

is entered?– Is the program intuitive?

• May want to have someone outside the class try out your programs.– Don’t let them read the specifications.

Basic Usability Features

38

introduction to software testing. types of software testing unit testing strategies – equivalence...

Documents

nonfunctional testing

testing inputs

integration testing

testing approaches

test execution

test suite

test passes

software testingintroduction