Silver Bullets and Software Testing

CS3300Fall 2015

Fred Brooks

Author of Mythical Man Month -- Essays about development of OS360 (1975)

Overall Premise??

Essential ComplexityAccidental Complexity

Thoughts from the Retrospective Panel (OOPSLA 2007)Fred Brooks -- biggest benefit was OO

David Parnas -- substitute tools for learning your trade, everybody wants an easy answer to hard problems. Coding genius is actually bad since no one else can understand or fix it.

Linda Northrop – the hardest thing about software is what to say, not how to say it. We focus on technologies instead of understanding the needs of users. Focus on essence, not accident.

Dave Thomas – Universities don’t teach key concepts, they teach specific languages

Ricardo Lopez – we are the silver bullets

Martin Fowler – Buy vs. Build, Growing great designers, rapid prototyping and incremental development, Agile – syntax over semantics

Werewolf – Every time someone makes a CPM/PERT chart or a Gantt chart, I get to eat a kitten

Audience – How do you measure productivity?

Software Quality Assurance• Testing is not SQA.• SQA is a broad field that we will touch on later.

Vocabulary Failure Fault/Defect/Bug Error Test Case Test Suite Driver Stub / Mock

Testing Principles

• 1. Testing shows the presence of defects (cannot show absence of defects) (Dijkstra)

• 2. Exhaustive testing is impossible• a java routine to test ax2+bx+c exhaustively would take 10 billion

years to run 4 times.• 3. Confusing an absence of failures with product fitness is a fallacy• 4. Testing should begin as early as possible

Testing Principles

• 5. Defects are not spread uniformly in modules (Defect Clustering – Pareto Principle)

• 6. Using the same set of test over and over will cease to find bugs (Pesticide Paradox)

• 7. Vary testing efforts based on circumstances

Goals Uncover defects : Static and Dynamic Gain confidence in product and provide information about its quality Preventing defects : Uncover new defects after changes (regression

tests)

Testing is in entire lifecycleRequirements : Reviews, WalkthroughsDesign : Analysis, Review, WalkthroughCode : Unit Tests, Integration and System testsQA : Formal TestsDeployment : Alpha/Beta Tests, Acceptance TestsMaintenance : Regression Tests

How much testing?• Adequacy and Coverage criteria (measure)• Need to define a finite set of tests that is a subset of all possible

tests.• These tests need to be thorough, but how do we define

thorough?• So Adequacy and Coverage try to answer the question “Do I

have enough tests?”

• Adequacy: Our product should have all requirements tested.• Criteria Measure: Percent of requirements currently tested.

10

White Box Testing• Dynamic• AKA Basis Path testing or Structural Testing• Faults can be discovered by exercising the different execution

paths of the code.• Many errors deal with incorrect branching conditions or

variable assignment errors• Usually used at the unit level (individual functions and classes).• Coverage criteria = Statement, Branch, Path

11

A little more on CFG Basic Block Decision Point Junctions Compound Predicate Shapes don't matter, just graph Statement coverage = visit all nodes Branch coverage = visit all edges

1 public int foo(int argc, string[] argv) {2 int x, y;3 if (argc < 4) {4 usage();5 System.Exit(-1);6 }7 x=Int32.TryParse(argv[1]) + 34;8 for (y = 0; y < 5; y++) {9 y += x + Int32.TryParse(argv[2]);10 }11 }

Annotating CFG DataflowDefinition (d) : value of a variable is definedUse (u) : value of a variable is used

Computation (c) : used in right hand side of exprPredicate (p) : used in boolean expression

Definition-Use path (du path) : a path through the CFG from a d to a u.

Definition-Clear path : a path through the CFG from a d to a u that has no intervening d

Program Slicing Focusing on statements that contribute to the values of certain

variables.int i;int sum = 0;int product = 1;for(i = 0; i < N; ++i) { sum = sum + i; product = product *i;}write(sum);write(product);

int i;int sum = 0; for(i = 0; i < N; ++i) { sum = sum + i; }write(sum);

McCabe’s Measure

• AKA Cyclomatic Complexity• Based on idea that complexity is caused by complex control flow• Maximum complexity for a quality routine is 10• Three ways to compute:

• Number of Predicates + 1• Edges-Nodes+2 from CFG• Number of Regions in CFG

• Forms an upper bound on test cases required for branch coverage and lower bound on path coverage thus: branch coverage cases <= mccabe <= path coverage cases

A problem for WhiteBox tests

17

public int divide(int num, int den) { return num / den;}

test case: divide(10, 2) expect 5

Statement, branch and all paths coverage with one test case.

Black Box Testing• Dynamic• AKA Specification-Based Tests• We pretend not to know anything about how the code is

implemented

18

Equivalence Partitions• Dynamic• Divide the inputs into partitions such that for every input value

in a partition, the system would treat it the same way.• Alternatively, if one test case in a partition succeeds, then we

assume all the cases would have succeeded. If one case fails, we assume all the cases in that partition would have failed.

19

Equivalence Partitions Ranges = 1 valid, 2 invalid Set = 1 in set, 1 not in set

Example:Triangle analyzer: Returns Scalene, Isoceles, Equilateral or Not a

Triangle. valid lengths are 1 to 100.

TriType analyze(int a, int b, int c)

Boundary Values• We don’t just want to pick values in a partition, we want to

focus on those right at the boundary between partitions. • We pick a good value on the boundary, and one off on the

invalid side.

21

Decision Tables / Decision Condition Testing• Used when the inputs are not independent, but have some kind

of relationship• In our triangle example, there is a relationship between the

three inputs that governs the correct answer.• Testing Condition “A or B”

• TT and FF will provide both the true and false result• Is that adequate?

22

Decision Table Testinga < b + c F T T T T T T T T T T

b < a + c - F T T T T T T T T T

c < a + b - - F T T T T T T T T

a= b - - - T T T T F F F F

a = c - - - T T F F T T F F

b = c - - - T F T F T F T F

NOT XX XX XX

SCALENE XX

ISOCELES XX XX XX

EQUILAT XX

IMPOSSIBLE XX XX XX

Syntax Testing• For a particular grammar• Take each mandatory and optional path

• - name : string• name : string• - name : string = “Bob”

24

Mutation Testing Common errors occur in coding like off-by-one Create Mutations of a program Run tests Mutations should fail Real program should succeed Add tests as necessary to kill all mutants Useful to develop complete test suite, or to validate existing tests. May have to prove equivalence for some mutants.

Markov Chains, randomized testing Build Markov model of program Assign probabilities Generate and Run Tests

Combinatorial Testing Pairwise Tests Cannot test all possible combinations Assumption that pairs of vars cause problems Test all pairs Use tool to generate Manual use Orthogonal Arrays

Exploratory Testing• simultaneous learning, test design and test execution• style of software testing that emphasizes the personal freedom

and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project