cs 3773 software engineering basic concepts of software testing unit testing

27
CS 3773 Software Engineering Basic Concepts of Software Testing Unit Testing

Upload: arron-hardy

Post on 23-Dec-2015

241 views

Category:

Documents


1 download

TRANSCRIPT

CS 3773 Software Engineering

Basic Concepts of

Software Testing

Unit Testing

2

Why testing? Examples

Mars Climate Orbiter ($165M, 1998) Sent to Mars to relay signal from the Mars Lander Smashed to the planet: failing to convert

between English measure to metric values

Shooting down of Airbus A300 (290 death, 1988) US CG-49 shoot down a Airbus A300 Misleading output of the tracking software

THERAC-25 Radiation Therapy (1985) 2 cancer patients at East Texas Cancer

center received fatal overdoses Miss-handling of race condition of the

software in the equipment

3

Why testing? Numbers

On average, 1-5 bugs per KLOC (thousand lines of code) In mature software More than 10 bugs in prototypes

Windows2000 35MLOC 63K known bugs at the time of release 2 bugs per KLOC

$59.5B loss due to bugs in US 2002 (estimation by NIST)

It is not feasible to remove all bugs But try to reduce critical bugs

4

Answer is testing, why?

“50% of my employees are testers, and the rest spends 50% of their time testing”

---- Bill Gates, 1995

More reliable than inspection, relatively cheap Actually in the old days, when testing is expensive,

inspection was the major answer

You get what you pay (linear rewards) Compared to other 3 approaches

Inspection, static checking, formal proof

5

Testing: Concepts

Test case Test oracle Test suite Test script Test driver Test result Test coverage

6

Testing: Concepts

Test case An execution of the software with a given list of

input values Include:

Input values, sometimes fed in different steps Expected outputs

Test oracle The expected outputs of software by feeding in a

list of input values A part of test cases Hardest problem in auto-testing: test oracle

problem

7

Testing: Concepts

Test suite A collection of test cases Usually these test cases share similar pre-requisites

and configuration Usually can be run together in sequence Different test suites for different purposes

Smoke test, Certain platforms, Certain feature, performance, …

Test Script A script to run a sequence of test cases or a test

suite automatically

8

Testing: Concepts Test Driver

A software framework that can load a collection of test cases or a test suite

It can usually handle the configuration and comparison between expected outputs and actual outputs

Test Coverage A measurement to evaluate how well the testing is

done The measure can be based on multiple elements

Code Input combinations Specifications

9

Granularity of Testing: V-model

10

Granularity of testing Unit Testing

Test of a single module Integration Testing

Test the interaction between modules System Testing

Test the system as a whole, by developers on test cases

Acceptance Testing Validate the system against user requirements, by

customers with no formal test cases Regression Testing

Test a new version with old test cases

11

Unit testing

Testing of an basic module of the software A function, a class, a component

Typical problems revealed Local data structures Algorithms Boundary conditions Error handling

12

An example: writing test cases without a test framework

Class Definition Consider a class that handles and manages a Fibonacci array Initialized the class

Fibonacci f = new Fibonacci (); Extend to a certain length

f. extend (length) Provide the number at certain index

f. get(index) Get Length

f.getLength() Provide a range of numbers

f. getRange(start, end)

13

An example: writing automatic test cases without a test framework

public static void main(String args[]){ Fibonacci f = new Fibonacci(); if(f.getLenght != 0){ System.err.print(“wrong results …”); } f.extend(50); if(f.getLenght != 50){ System.err.print(“wrong results …”); } int num = f.get(50); if(num != 12586269025){ System.err.print(“wrong results …”); } int[] range = f.getRange(40, 50) if(!ValueEquals(range, …)){ System.err.print(“wrong results …”); } …}

Not Enough!What happened if

an exception is thrown

14

An example: writing automatic test cases without a test framework

public static void main(String args[]){ Fibonacci f = new Fibonacci(); if(f.getLenght != 0){ System.err.print(“wrong results …”); } f.extend(50); if(f.getLenght != 50){ System.err.print(“wrong results …”); } int num = f.get(50); if(num != 12586269025){ System.err.print(“wrong results …”); } int[] range = f.getRange(40, 50) if(!ValueEquals(range, …)){ System.err.print(“wrong results …”); } …}

Potential test interference!

15

Common Operations

Comparison between actual results and expected results

Report error messages Catch Exceptions Re-initialize the class to be tested

16

Unit test framework

xUnit Created by Kent Beck in 1989

This is the same guy we mentioned in XP, design patterns The first one was sUnit (for smalltalk)

Junit The most popular xUnit framework There are about 70 xUnit frameworks for corresponding

languages

Never in the annals of software engineering was so much owed by so many to so few lines of code

-------Martin Fowler

17

State machine of JUnit

Initialization Of Test Class

setUptry{ testMethod} catch {…}

tearDown

Why?Try and Catch?

18

Writing a Test Casepublic class FibonacciTest extends TestCase { protected Fibonacci fTest; protected static int groudtruth[] = {0, 1,...,12586269025}; public void setUp(){ fTest = new Fibonacci(); } @Test public testInitialize(){ assertEquals(“Length of fTest after initialization”, 0, fTest.getLength()); } @Test public testExtend(){ fTest.extend(50); assertEquals(“…”, 50, fTest.getLength()); } @Test public testGet(){ fTest.extend(50); assertEquals(“…”, groudtruth[49], fTest.get(50)); } …}

19

Benefits of using test framework

Clear structure of testing Each test method for one feature Common initialization to setup

Assertion methods You can always define new assertion

methods

Reduced Interference Try catch to run all test methods Always do re-initialization in setup

20

Demo

Writing a JUnit testcase for a class in your project

Run it as a JUnit test case

21

Tips of writing test cases

Bottom Up If you know that obj.m1 -> obj.m2, test obj.m2 first Why?

You may not understand the error in m1 if you don’t know whether m2 is working or not

Externalize data Externalize expected results to files and load them

at the test case constructor Why?

Easier to maintain if you want to reuse them in several places

22

Tips of writing test cases Use development Database

Do not use real database for testing Why? Obvious

Do enough logging in the code during testing Why?

Help debugging

Testing for exceptions How?

public testException(){

try{

do(invalid); fail();

}catch(xxxException e){}

}

23

Assertions

public void testCapacity() { // a test method

….

assertTrue(fFull.size() == 100+size); //assertion

}

If assertion fails:Assertion failed: myTest.java:150 (expected true but was

false)Not so good!

Try: assertEquals(100+size, fFull.size()); //expected value firstAssertion failed: myTest.java:150 (expected 102 but was

103)Better!

Try: assertEquals(“list length”, 100+size, fFull.size()); Assertion failed: myTest.java:150 (list length expected 102

but was 103)

24

Assertions

Extend TestCase and write your own assertions AssertStringContains, AssertArrayEquals, … Use fail(String) to fail a test with certain message

Feel free to change the fixture Each test will re-setup

public void testRemoveAll() {

fFull.removeAllElements();

fEmpty.removeAllElements();

assertTrue(fFull.isEmpty());

assertTrue(fEmpty.isEmpty());

}

Tear down Consider the following test code

void setUp() { File f = open(“foo”); File b = open(“bar”);}void testAAA() { use f and b}void testBBB(){ use f and b}

Problems?

void setUp() { File f = open(“foo”); File b = open(“bar”);}void testAAA() { try { use f and b } finally { clean&close f, b }}void testBBB() { try { use f and b } finally { clean&close f, b }}

Better?

26

Tear down Consider the following test code

void setUp() { File f = open(“foo”); File b = open(“bar”);}void testAAA() { use f and b}void testBBB(){ use f and b}void tearDown{ clean&close f, b}

Problems?

void setUp() { File f = open(“foo”); File b = open(“bar”);}…void tearDown{ try{ clean&close f }catch{ … } the same for b}

27

Tear down

Be careful about tear down If tear down is not complete, a test failure may

affect the following test cases Recover the changes done to global data that are

not well handled by the setup Database, files, network, global variables

Clean resources Caution of exceptions in tear down itself