1

Some Software Testing Quotes:“Testing proves a programmer’s failure. Debugging is the programmer’s vindication.” -Boris Beizer,

“The most likely way for the world to be destroyed, most experts agree, is by accident. That's where we come in; we're computer professionals. We cause accidents.” Nathaniel S. Borenstein,

“All programmers are optimists. Perhaps this modern sorcery especially attracts those who believe in happy endings and fairy godmothers. Perhaps the hundreds of nitty frustrations drive away all but those who habitually focus on the end goal. Perhaps it is merely that computers are young, programmers are younger, and the young are always optimists. But however the selection process works, the result is indisputable: 'This time it will surely run' or 'I just found the last bug'.” -Frederick P. Brooks

http://www.wired.com/software/coolapps/news/2005/11/69355

Time to cover: 30-40 minutes

2

Software Testing (all UWO notes, not from the text)

•Intro to Testing

•Unit Testing

•Drivers

•Stubs

3

Introduction to Testing

What is the purpose of testing?

• Older view: “Testing is done to show that the system works”

—Tend to “go easy” on program

—Programmer use same logic to test as they did to design or code

—Therefore some bugs do not get caught• More modern and useful view: “Testing is done to uncover bugs”

—We purposely take attitude of trying to break program

—Hence, more bugs discovered

—Result: a better systemWe will take the more modern view!

4

Test Cases and Test SuitesTest Case: A set or sequence of inputs used to test a program, along with expected output.

• Example: test case for an automated day planner:—A sequence of commands to add appointments, move around the

calendar, etc, with the expected output of each commandSimilar to use cases, however:

• Use cases tend to:—Concentrate on features of user interface—Show common patterns of use—Be used as a basis for design

• Test cases tend to:—Concentrate on program correctness—Test obscure or problematic patterns of use—Be chose in order to try to break program

Test Suite: set of test cases• Example: set of test cases to automate day planner, each testing for a

different set of requirements

5

Good Test Cases and Test Suites

A good test case is one that we think is likely to uncover a bug

A good test suite contains good test cases and tests the requirements thoroughly.

So:

• The better our test suite is, and

• The more of it our software passes,

• The more confidence we can have in our software

6

Failures, Faults and Errors

We often use the informal term bug

• In fact, this can mean several different things

• Sometimes it is more useful to use precise terminology

Failure: what the program does wrong (crashing, wrong answer)

Fault: incorrect code causing the failure (less than should be greater than)

Error: mistake that programmer made leading to the fault (assumed that less than was correct)

We will still often use bug when it is unambiguous

7

Can We Find all Bugs?

80% of the failures are caused by 20% of the faults• 20% of faults are easy to find

20% of the failures are caused by 80% of the faults• Less frequent and therefore harder to find• May be just as bad, if not worse than other errors

Some serious failures may be very hard to findMust push system in unexpected / unusual ways in order to find bugsIn large systems, it is likely we will never find all the bugsTherefore for (e.g.) airplane software, nuclear power plant software, etc. developers must devote much time and money to:

• Avoiding, finding, eliminating as many bugs as possible• Building in failsafe checks to alleviate effects of faults

8

Testing vs. Debugging

Testing: running test cases, finding failures

• Testing often can be done without looking at the code

• Can be automated or partially automated

—By putting a test suite in a shell script or makefile

Debugging: finding and correcting faults

• Need to work with the code

• Use a debugger

• Can’t be automated (of course!)

9

Levels of TestingUnit Testing: testing individual functions or modules

• Done by programmer, • Usually use drivers and stubs

System Testing: test the whole system• Uses real inputs to the whole system• Planned well in advance

In between those comes: integration testing: make sure the modules compile together and pass basic tests

• Should be easy if:—The detailed design was good—Everyone assumed the same interfaces between modules

We will study each of these levels in more detailSystem testing by client or evaluator usually referred to as acceptance testing

• Tells whether program is acceptable to the people you are doing it for

10

Building Good Test Cases

Two basic strategies:

• Look at requirements, construct cases which correspond directly to these

—Called functional testing, “black box testing”

—Can be written in parallel with coding

• Look at finished code, try to execute paths through it, see what code has been missed, think up test cases which test missed code.

—Called structural testing, “clear box testing”

—Tools exist to help with this

We usually need and use both strategies

11

Unit TestingLets imagine that:

• Each person has finished writing the code of their modules• Each person has got their modules compiling• Now, we just put everything together without testing the individual modules• After some interface problems, the whole system compiles• We go to test it

What happens if:

• A failure happens right away; and it takes a while to trace it• The fault is traced to person X’s code

—Person X has to fix—Everyone else stands around and waits for Person X to fix the fault

• The fault is fixed, the test now succeeds• Another failure happens, and it takes a while to trace it• The fault is traced to person Y’s code

—Person Y has to fix it—Everyone else stands around and waits for Person Y to fix the fault

In other words: system testing is long and frustrating

12

Unit Testing: Motivation

Now let’s imagine that:• Each person has finished writing the code of their module(s)• Each person has got them compiling• Each person does some separate testing on their modules before the

system is integratedWhat happens is:

• A similar pattern!…• However, most basic, obvious bugs have already been eliminated• They were eliminated while everyone was working on their modules

at the same time• Less frustration, less waiting around for other people

In other words: doing unit testing first makes system testing much smoother when we get to it

13

Unit Testing: Questions

How do we select test cases for the units (modules, objects)?

• Both structural and functional methods will work

• We will get to these topics a bit later

How to test one module in isolation at all, without having working implementation

• We will address this question first

14

DriversIn particular: how to test a module which is low in the module hierarchy, without the main program?

• Answer: a driverDriver: a simple main program which exists only in order to test a low-level function or moduleTo test a module using a driver:

• Write the driver• Compile the driver together with the module• Test the driver + module as if it is the program you wanted to test

Typical simple driver for single function:• Prompts user for input or reads it from command-line arguments• Calls the function, reports its result, terminates

Slightly more complex driver for module:• Reads simple commands from user (e.g. one command per function in

module)• Calls function corresponding to command, reports results• Terminates on quit command

15

Simple Driver: ExampleFunction to be tested (in CalendarImplementation.c): printCalendar(int y, int x)

• Function for printing out a calendar for year y, month m

Main program of driver (CalendarDriver.c):

• Interprets the first two command-line arguments as y and m

• Calls function on y and m, then terminates

To Test:

• Compile CalendarImplementation.c, CalendarDriver.c

• Issue command CalendarDriver 2002 10 to test Oct 2002

—Should display something like:

—Run on test cases to check output correctness

QUESTION: What test cases would you

supply to CalendarDriver to thoroughly test

the module you have written?

16

More Complex Driver: Example

Module to be tested (StackImplementation.c):• Integer stack ADT, with functions push, pop, printStack, etc.

Main program of driver (StackDriver.c)• Has one Stack variable s• Reads and interprets simple user command

—User command push n causes driver to callpush(s,n); printStack(s)

—User command pop causes driver to calln=pop(s); printf(“Popped %d\n”, n);printStack(s)

—Etc..

User: push 42System: 42User: push 56System: 42 56User: popSystem: Popped 56 46

17

StubsNow: how to test a module which is high in the module hierarchy, without any of the modules it calls?

• Answer: stubsStub: a simple fake function which stands in for another function, without much of the ability of the real functionTo test a module using stubs:

• Write a main program (this may be the real main purpose for a driver)• Write stubs for the functions you need to• Compile main program + module + stubs• Test the resulting program as if it is the program you wanted to test

Typical simple stub for a single function• Always prints out the same message; e.g.

— Function warpDrive being calledMore complex stub for a whole module

• A simple, slow implementation of the module which doesn’t have all the required functionality

18

Simple Stub: ExampleFunction to be “stubbed out” (in CalendarImplementation.c):

• Function for printing out a calendar for year y, month mStub:

• Function which always prints a message of the form

Main program initially compiled together with stubWhen higher-level functions tested, every time they call the stubbed-out function, this message will appear; e.g.

After real CalendarImplementation.c is available, stub will be replaced by the real thing

Calendar for YYYY/ MM goes here

Calendar for 2002/10 goes here

19

More Complex Stub: Example

Module to be tested: StackImplementation.c, will all usual stack functions

Stub module:

• Simple array implementation of a stack with 2 elements

• Will crash or complain if we try to put more than 2 elements in it

• However, this may be enough to test basic functionality of higher-level modules

Main program initially compiled together with stub module and then tested

After real implementation of StackImplementation.c is available; stub will be replaced by the real thing

20

Functions with Random Behaviour

So far:

• Examples for stubs were useful for functions that exhibit repeatable behavior

• What if this is not the case?

Example: a game program has functions that choose randomly from a number of alternatives

In this case, a stub might

• Always pick a particular alternative

• Read an alternative from a file or request one from the user

• Make a simple random choice, unbiased by other game factors

21

Functions with Random Behaviour: Example

A game program has a module that manipulates a deck of cards, DeckImplementation.cContains functions:•void shuffle (Deck d);•CardList deal (Deck d, int ncards);•void cut (Deck d);

Say, DeckType.h contains definitions”#define DECKSIZE 52typedef int Deck[DECKSIZE];typedef int CardList[DECKSIZE];

QUESTION: How might we write stubs for the above functions, perhaps to test a gameplay module?