algorithmic testing
DESCRIPTION
Algorithmic Testing. Doron Peled, University of Warwick. Why testing?. Reduce design/programming errors. Can be done during development, before production/marketing. Practical, simple to do. Check the real thing, not a model. Scales up reasonably. Being state of the practice for decades. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/1.jpg)
Doron Peled,University of Warwick
![Page 2: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/2.jpg)
Why testing?Reduce design/programming errors.Can be done during development, before
production/marketing.Practical, simple to do.Check the real thing, not a model.Scales up reasonably.Being state of the practice for decades.
![Page 3: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/3.jpg)
Part 1: Testing of black box finite state machine
Know:Transition relationSize or bound on size
Wants to know:In what state we started?In what state we are?Transition relationConformanceSatisfaction of a temporal property
![Page 4: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/4.jpg)
Finite automata (Mealy machines)
S - finite set of states. (size n)– set of inputs. (size d)O – set of outputs, for each transition.(s0 S - initial state). S S - transition relation. S O – output on edge.
![Page 5: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/5.jpg)
Why deterministic machines? Otherwise no amount of experiments would
guarantee anything. If dependent on some parameter (e.g.,
temperature), we can determinize, by taking parameter as additional input.
We still can model concurrent system. It means just that the transitions are deterministic.
All kinds of equivalences are unified into language equivalence.
Also: connected machine (otherwise we may never get to the completely separate parts).
![Page 6: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/6.jpg)
Determinism
When the black box is nondeterministic, we might never test some choices.
b/1a/1
a/1
![Page 7: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/7.jpg)
Preliminaries: separating sequences
s1
s3
s2
a/0b/1 b/0
b/1
a/0
a/0
Start with one block containing all states {s1, s2, s3}.
![Page 8: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/8.jpg)
A: separate to blocks of states with different output.
s1
s3
s2
a/0b/1 b/0
b/1
a/0
a/0
Two sets, separated using the string b {s1, s3}, {s2}.
![Page 9: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/9.jpg)
Repeat B: Separate blocks based on moving to different blocks.
s1
s3
s2
a/0b/1 b/0
b/1
a/0
a/0
Separate first block using b to three singleton blocks.Separating sequences: b, bb.Max rounds: n-1, sequences: n-1, length: n-1.For each pair of states there is a separating sequence.
![Page 10: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/10.jpg)
Want to know the state of the machine (at end). Homing sequence.Depending on output, would know in what
state we are. Algorithm: Put all the states in one block
(initially we do not know what is the state).Then repeatedly partitions blocks of states, as
long as they are not singletons, as follows: Take a non singleton block, append a
distinguishing sequence that separates at least two states.
Update all blocks to the states after executing .
Max length: (n-1)2 (Lower bound: n(n-1)/2.)
![Page 11: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/11.jpg)
Example (homing sequence)
s1
s3
s2
a/0b/1 b/0
b/1
a/0
a/0{s1, s2, s3}
{s1, s2} {s3}{s1} {s2} {s3}
b
b1 0
011
1
On input b and output 1, still don’t know if was in s1 or s3, i.e., if currently in s2 or s1.So separate these cases with another b.
![Page 12: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/12.jpg)
Synchronizing sequenceOne sequence takes the machine to
the same final state, regardless of the initial state or the outputs.
Not every machine has a synchronizing sequence.
Can be checked whether exists and can be found in polynomial time.
![Page 13: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/13.jpg)
State identification: Want to know in which state the
system has started (was reset). Can be a preset distinguishing
sequence (fixed), or a tree (adaptive).
May not exist (PSPACE complete to check if preset exists, polynomial for adaptive).
Best known algorithm: exponential length for preset,polynomial for adaptive [LY].
![Page 14: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/14.jpg)
Sometimes cannot identify initial state
b/1a/1 s1
s3
s2
a/1
b/0
b/1
a/1
Start with a:in case of being in s1 or s3 we’ll move to s1 and cannot distinguish.Start with b:In case of being in s1 or s2 we’ll move to s2 and cannot distinguish.
The kind of experiment we do affects what we can distinguish. Much like the Heisenberg principle in Physics.
![Page 15: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/15.jpg)
Conformance testing Unknown deterministic finite state system B. Known: n states and alphabet . An abstract model C of B. C satisfies all the
properties we want from B. C has m states. Check conformance of B and C. Another version: only a bound n on the number
of states l is known.
![Page 16: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/16.jpg)
Check conformance with a given state machine
Black box machine has no more states than specification machine (errors are mistakes in outputs, mistargeted edges).
Specification machine is reduced, connected, deterministic. Machine resets reliably to a single initial state (or use homing
sequence).
s1
s3
s2
a/1
b/0
b/1
a/1
?=
a/1
b/1
![Page 17: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/17.jpg)
Conformance testing [Ch,V]
a/1
b/1
Cannot distinguish if reduced or not.
a/1
b/1
a/1
b/1
a/1
b/1a/1
b/1
![Page 18: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/18.jpg)
Conformance testing (cont.)
ab b
aa
a
a b
b
b
a
Need: bound on number of states of B.
a
![Page 19: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/19.jpg)
Preparation:Construct a spanning tree
b/1a/1 s1
s3
s2
a/1
b/0
b/1
a/1
s1
s2s3
b/1a/1
![Page 20: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/20.jpg)
How the algorithm works?According to the spanning
tree, force a sequence of inputs to go to each state.
1. From each state, perform the distinguishing sequences.
2. From each state, make a single transition, check output, and use distinguishing sequences to check that in correct target state.
s1
s2s3
b/1a/1
Reset or hom
ing
Reset or hom
ing
Distinguishing sequences
![Page 21: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/21.jpg)
Comments1. Checking the different distinguishing
sequences (m-1 of them) means each time resetting and returning to the state under experiment.
2. A reset can be performed to a distinguished state through a homing sequence. Then we can perform a sequence that brings us to the distinguished initial state.
3. Since there are no more than m states, and according to the experiment, no less than m states, there are m states exactly.
4. Isomorphism between the transition relation is found, hence from minimality the two automata recognize the same languages.
![Page 22: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/22.jpg)
Combination lock automaton
Assume accepting states.Accepts only words with a specific suffix
(cdab in the example).
s1 s2 s3 s4 s5bdc a
Any other input
![Page 23: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/23.jpg)
When only a bound on size of black box is known…Black box can “pretend” to behave
as a specification automaton for a long time, then upon using the right combination, make a mistake.
b/1a/1s1
s3
s2
a/1
b/0
b/1
a/1
b/1
Pretends to be S3
Pretends to be S1
a/1
![Page 24: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/24.jpg)
Conformance testing algorithm [VC] The worst that can happen is a
combination lock automaton that behaves differently only in the last state. The length of it is the difference between the size n of the black box and the specification m.
Reach every state on the spanning tree and check every word of length n-m+1 or less. Check that after the combination we are at the state we are supposed to be, using the distinguishing sequences.
No need to check transitions: already included in above check.
Complexity: m2 n dn-m+1
Probabilistic complexity: Polynomial. Distinguishing sequences
s1
s2s3
b/1a/1
Words of length n-m+1
Reset or hom
ing
Reset or hom
ing
![Page 25: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/25.jpg)
Model Checking Finite state description of a system B. LTL formula . Translate into an automaton P. Check whether L(B) L(P)=. If so, S satisfies . Otherwise, the intersection
includes a counterexample. Repeat for different properties.
![Page 26: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/26.jpg)
Buchi automata (-automata) S - finite set of states. (B has l n states) S0 S - initial states. (P has m states) - finite alphabet. (contains p letters) S S - transition relation. F S - accepting states.Accepting run: passes a state in F infinitely
often.System automata: F=S, deterministic, one initial state.
Property automaton: not necessarily deterministic.
![Page 27: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/27.jpg)
Example: check a
a, aa
a <>a
![Page 28: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/28.jpg)
Example: check <>a
aa
a
a<>a
![Page 29: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/29.jpg)
Example: check <>a
Use automatic translation algorithms, e.g., [Gerth,Peled,Vardi,Wolper 95]
aa
a, a<>a
![Page 30: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/30.jpg)
System
c b
a
![Page 31: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/31.jpg)
Every element in the product is a counter example for the checked property.
c b
a
aa
a
a
s1 s2
s3 q2
q1
s1,q1
s1,q2 s3,q2
s2,q1a
b
ca
Acceptance isdetermined byautomaton P.
<>a
![Page 32: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/32.jpg)
Model Checking / Testing Given Finite state
system B. Transition relation of B
known. Property represent by
automaton P. Check if L(B) L(P)=. Graph theory or BDD
techniques. Complexity: polynomial.
Unknown Finite state system B.
Alphabet and number of states of B or upper bound known.
Specification given as an abstract system C.
Check if B C. Complexity: polynomial
if number states known. Exponential otherwise.
![Page 33: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/33.jpg)
Black box checking [PVY]
Property represent by automaton P.
Check if L(B) L(P)=.
Graph theory techniques.
Unknown Finite state system B.
Alphabet and Upper bound on Number of states of B known.
Complexity: exponential.
![Page 34: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/34.jpg)
Experimentsaa
bb cc
reset
a
ab
b
c
c
try b a
ab
b
c
c
try c
fail
![Page 35: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/35.jpg)
Simpler problem: deadlock?
Nondeterministic algorithm:guess a path of length n from the initial state to a deadlock state.Linear time, logarithmic space.
Deterministic algorithm:systematically try paths of length n, one after the other (and use reset), until deadlock is reached.Exponential time, linear space.
![Page 36: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/36.jpg)
Deadlock complexityNondeterministic algorithm:
Linear time, logarithmic space.Deterministic algorithm:
Exponential (p n-1) time, linear space.Lower bound: Exponential time (use
combination lock automata).How does this conform with what we
know about complexity theory?
![Page 37: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/37.jpg)
Modeling black box checking
Cannot model using Turing machines: not all the information about B is given. Only certain experiments are allowed.
We learn the model as we make the experiments.
Can use the model of games of incomplete information.
![Page 38: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/38.jpg)
Games of incomplete information Two players: player, player (here, deterministic). Finitely many configurations C. Including:
Initial Ci , Winning : W+ and W- . An equivalence relation on C (the player cannot
distinguish between equivalent states). Labels L on moves (try a, reset, success, fail). The player has the moves labeled the same from
configurations that are equivalent. Deterministic strategy for the player: will lead to a
configuration in W+ W-. Cannot distinguish between equivalent configurations.
Nondeterministic strategy: Can distinguish between equivalent configurations..
![Page 39: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/39.jpg)
Modeling BBC as gamesEach configuration contains an automaton
and its current state (and more).Moves of the player are labeled with
try a, reset... Moves of the -player withsuccess, fail.
c1 c2 when the automata in c1 and c2 would respond in the same way to the experiments so far.
![Page 40: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/40.jpg)
A naive strategy for BBC Learn first the structure of the black box. Then apply the intersection. Enumerate automata with n states
(without repeating isomorphic automata). For a current automata and new
automata, construct a distinguishing sequence. Only one of them survives.
Complexity: O((n+1)p (n+1)/n!)
![Page 41: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/41.jpg)
On-the-fly strategy Systematically (as in the deadlock
case), find two sequences v1 and v2 of length <=m n.
Applying v1 to P brings us to a state t that is accepting.
Applying v2 to P brings us back to t.
Apply v1 v2 n to B. If this succeeds,
there is a cycle in the intersection labeled with v2, with t as the P (accepting) component.
Complexity: O(n2p2mnm).
v1
v2
![Page 42: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/42.jpg)
Learning an automaton
Use Angluin’s algorithm for learning an automaton.
The learning algorithm queries whether some strings are in the automaton B.
It can also conjecture an automaton Mi and asks for a counterexample.
It then generates an automaton with more states Mi+1 and so forth.
![Page 43: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/43.jpg)
A strategy based on learning
Start the learning algorithm.Queries are just experiments to B.For a conjectured automaton Mi ,
check if Mi P = If so, we check conformance of Mi with
B ([VC] algorithm). If nonempty, it contains some v1 v2
. We test B with v1 v2
n. If this succeeds: error, otherwise, this is a counterexample for Mi .
![Page 44: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/44.jpg)
Complexity l - actual size of B. n - an upper bound of size of B. d - size of alphabet. Lower bound: reachability is similar to
deadlock. O(l 3 d l + l 2mn) if there is an error. O(l 3 d l + l 2 n dn-l+1+ l 2mn) if there is no error.If n is not known, check while time allows. Probabilistic complexity: polynomial.
![Page 45: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/45.jpg)
Some experimentsBasic system written in SML (by Alex
Groce, CMU).Experiment with black box using Unix
I/O.Allows model-free model checking of C
code with inter-process communication.Compiling tested code in SML with BBC
program as one process.
![Page 46: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/46.jpg)
Part 2: Software testing
Testing is not about showing that there are no errors in the program.
Testing cannot show that the program performs its intended goal correctly.
So, what is software testing?Testing is the process of executing the
program in order to find errors.A successful test is one that finds an error.
![Page 47: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/47.jpg)
Some software testing stages Unit testing – the lowest level, testing
some procedures. Integration testing – different pieces of code. System testing – testing a system as a whole. Acceptance testing – performed by the
customer. Regression testing – performed after updates. Stress testing – checking the code under
extreme conditions. Mutation testing – testing the quality of the
test suite.
![Page 48: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/48.jpg)
Some drawbacks of testing
There are never sufficiently many test cases.
Testing does not find all the errors. Testing is not trivial and requires
considerable time and effort. Testing is still a largely informal task.
![Page 49: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/49.jpg)
Black-Box (data-driven, input-output) testing
The testing is not based on the structure of the program (which is unknown).
In order to ensure correctness, every possible input needs to be tested - this is impossible!
The goal: to maximize the number of errors found.
![Page 50: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/50.jpg)
testingIs based on the internal structure of the
program.There are several alternative criterions
for checking “enough” paths in the program.
Even checking all paths (highly impractical) does not guarantee finding all errors (e.g., missing paths!)
![Page 51: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/51.jpg)
Some testing principles
A programmer should not test his/her own program. One should test not only that the program does what
it is supposed to do, but that it does not do what it is not supposed to.
The goal of testing is to find errors, not to show that the program is errorless.
No amount of testing can guarantee error-free program.
Parts of programs where a lot of errors have already been found are a good place to look for more errors.
The goal is not to humiliate the programmer!
![Page 52: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/52.jpg)
Inspections and Walkthroughs
Manual testing methods.Done by a team of people.Performed at a meeting
(brainstorming).Takes 90-120 minutes.Can find 30%-70% of errors.
![Page 53: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/53.jpg)
Code Inspection
Team of 3-5 people. One is the moderator.
He distributes materials and records the errors.
The programmer explains the program line by line.
Questions are raised. The program is
analyzed w.r.t. a checklist of errors.
![Page 54: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/54.jpg)
Checklist for inspectionsData declarationAll variables
declared?Default values
understood?Arrays and strings
initialized?Variables with similar
names?Correct initialization?
Control flowEach loop terminates?DO/END statements
match?
Input/outputOPEN statements
correct?Format specification
correct?End-of-file case handled?
![Page 55: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/55.jpg)
Walkthrough
Team of 3-5 people. Moderator, as before. Secretary, records
errors. Tester, play the role
of a computer on some test suits on paper and board.
![Page 56: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/56.jpg)
Selection of test cases (for white-box testing)
The main problem is to select a good coveragecriterion. Some options are:
Cover all paths of the program. Execute every statement at least once. Each decision has a true or false value at least
once. Each condition is taking each truth value at least
once. Check all possible combinations of conditions in
each decision.
![Page 57: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/57.jpg)
Cover all the paths of the program
Infeasible.Consider the flow diagram
on the left.It corresponds to a loop.The loop body has 5 paths.If the loops executes 20times there are 5^20
different paths!May also be unbounded!
![Page 58: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/58.jpg)
How to cover the executions?IF (A>1)&(B=0) THEN X=X/A;
END;IF (A=2)|(X>1) THEN X=X+1;
END;
Choose values for A,B,X. Value of X may change, depending on A,B. What do we want to cover? Paths? Statements?
Conditions?
![Page 59: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/59.jpg)
Statement coverageExecute every statement at least onceBy choosingA=2,B=0,X=3each statement will
be chosen.The case where the
tests fail is not checked!
IF (A>1)&(B=0) THEN X=X/A; END;
IF (A=2)|(X>1) THEN X=X+1; END;
Now x=1.5
![Page 60: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/60.jpg)
Decision coverageEach decision has a true and false outcome at least once.
Can be achieved using A=3,B=0,X=3 A=2,B=1,X=1
Problem: Does not test individual conditions. E.g., when X>1 is erroneous in second decision.
IF (A>1)&(B=0) THEN X=X/A; END;
IF (A=2)|(X>1) THEN X=X+1; END;
![Page 61: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/61.jpg)
Decision coverage
A=3,B=0,X=3 IF (A>1)&(B=0) THEN X=X/A; END;
IF (A=2)|(X>1) THEN X=X+1; END;
Now x=1
![Page 62: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/62.jpg)
Decision coverage
A=2,B=1,X=1
The case where A1 and the case where x>1 where not checked!
IF (A>1)&(B=0) THEN X=X/A; END;
IF (A=2)|(X>1) THEN X=X+1; END;
![Page 63: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/63.jpg)
Condition coverageEach condition has a true and false value at least once.
For example: A=1,B=0,X=3 A=2,B=1,X=0
lets each condition be true and false once.
Problem:covers only the path where the first test fails and the second succeeds.
IF (A>1)(A>1)&(B=0) THEN X=X/A; END;
IF (A=2)|(X>1) THEN X=X+1; END;
![Page 64: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/64.jpg)
Condition coverage
A=1,B=0,X=3 IF (A>1) (A>1) & (B=0) THEN X=X/A; END;
IF (A=2) | (X>1) THEN X=X+1; END;
![Page 65: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/65.jpg)
Condition coverage
A=2,B=1,X=0
Did not check the first THEN part at all!!!
Can use condition+decision coverage.
IF (A>1)(A>1)&(B=0) THEN X=X/A; END;
IF (A=2)|(X>1) THEN X=X+1; END;
![Page 66: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/66.jpg)
Multiple Condition CoverageTest all combinations of all conditions in each test.
A>1,B=0 A>1,B≠0 A1,B=0 A1,B≠0 A=2,X>1 A=2,X1 A≠2,X>1 A≠2,X1
IF (A>1)&(B=0) THEN X=X/A; END;
IF (A=2)|(X>1) THEN X=X+1; END;
![Page 67: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/67.jpg)
A smaller number of cases: A=2,B=0,X=4 A=2,B=1,X=1 A=1,B=0,X=2 A=1,B=1,X=1Note the X=4 in the firstcase: it is due to the factthat X changes beforebeing used!
IF (A>1)&(B=0) THEN X=X/A; END;
IF (A=2)|(X>1) THEN X=X+1; END;
Further optimization: not all combinations.For C /\ D, check (C, D), (C, D), (C, D).For C \/ D, check (C, D), (C, D), (C, D).
![Page 68: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/68.jpg)
Preliminary:Relativizing assertions
(B) : x1= y1 * x2 + y2 /\ y2 >= 0Relativize B) w.r.t. the assignment
becomes B) [Y\g(X,Y)]e(B) expressed w.r.t. variables at
A.) (B)A =x1=0 * x2 + x1 /\ x1>=0
Think about two sets of variables,before={x, y, z, …} after={x’,y’,z’…}.
Rewrite (B) using after, and the assignment as a relation between the set of variables. Then eliminate after.
Here: x1’=y1’ * x2’ + y2’ /\ y2’>=0 /\x1=x1’ /\ x2=x2’ /\ y1’=0 /\ y2’=x1now eliminate x1’, x2’, y1’, y2’.
(y1,y2)=(0,x1)
A
B
A
B
(y1,y2)=(0,x1)
Y=g(X,Y)
![Page 69: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/69.jpg)
Verification conditions: tests
C) B)= t(X,Y) /\ C)
D) B)=t(X,Y) /\ D)
B)= D) /\ y2x2y2>=x2
B
C
D
B
C
Dt(X,Y)
FT
FT
![Page 70: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/70.jpg)
How to find values for coverage?
•Put true at end of path.•Propagate path backwards.•On assignment, relativize expression.•On “yes” edge of decision, add decision as conjunction.•On “no” edge, add negation of decision as conjunction.•Can be more specific when calculating condition with multiple condition coverage.
A>1 & B=0
A=2 | X>1
X=X+1
X=X/Ano
no
yes
yes
true
true
![Page 71: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/71.jpg)
How to find values for coverage?
A>1 & B=0
A=2 | X>1
X=X+1
X=X/Ano
no
yes
yes
true
true
A≠2 /\ X>1
(A≠2 /\ X/A>1) /\ (A>1 & B=0)
A≠2 /\ X/A>1Need to find a
satisfying assignment:A=3, X=6, B=0Can also calculate path condition forwards.
![Page 72: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/72.jpg)
How to cover a flow chart? Cover all nodes, e.g., using search strategies:
DFS, BFS. Cover all paths (usually impractical). Cover each adjacent sequence of N nodes. Probabilistic testing. Using random number
generator simulation. Based on typical use. Chinese Postman: minimize edge traversal
Find minimal number of times time to travel each edge using linear programming or dataflow algorithms.Duplicate edges and find an Euler path.
![Page 73: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/73.jpg)
Test cases based on data-flow analysis
Partition the program into pieces of code with a single entry/exit point.
For each piece find which variables are set/used/tested.
Various covering criteria: from each set to each
use/test From each set to
some use/test.
X:=3
z:=z+x
x>y
t>y
![Page 74: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/74.jpg)
Test case design for black box testing
Equivalence partitionBoundary value analysisCause-effect graphs
![Page 75: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/75.jpg)
Equivalence partition
Goals: Find a small number of test cases. Cover as much possibilities as you can.
Try to group together inputs for which the program is likely to behave the same.
Specificationcondition
Valid equivalenceclass
Invalid equivalenceclass
![Page 76: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/76.jpg)
Example: A legal variable
Begins with A-Z Contains [A-Z0-9] Has 1-6 characters.
Specificationcondition
Valid equivalenceclass
Invalid equivalenceclass
Starting char
Chars
Length
Starts A-Z Starts other
[A-Z0-9] Has others
1-6 chars 0 chars, >6 chars
1 2
3 4
56 7
![Page 77: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/77.jpg)
Equivalence partition (cont.)
Add a new test case until all valid equivalence classes have been covered. A test case can cover multiple such classes.
Add a new test case until all invalid equivalence class have been covered. Each test case can cover only one such class.
Specificationcondition
Valid equivalenceclass
Invalid equivalenceclass
![Page 78: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/78.jpg)
Example
AB36P (1,3,5) 1XY12 (2) A17#%X (4)
Specificationcondition
Valid equivalenceclass
Invalid equivalenceclass
Starting char
Chars
Length
Starts A-Z Starts other
[A-Z0-9] Has others
1-6 chars 0 chars, >6 chars
1 2
3 4
56 7
(6) VERYLONG (7)
![Page 79: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/79.jpg)
Boundary value analysis
In every element class, select values that are closed to the boundary. If input is within range -1.0 to +1.0,
select values -1.001, -1.0, -0.999, 0.999, 1.0, 1.001.
If needs to read N data elements, check with N-1, N, N+1. Also, check with N=0.
![Page 80: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/80.jpg)
Test case generation based on LTL specification
Compiler ModelChecker
Path conditioncalculation
First orderinstantiator
Testmonitoring
Transitions
Path Flow
chart
LTLAut
![Page 81: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/81.jpg)
Goals Verification of software. Compositional verification. Only a unit of code. Parametrized verification. Generating test cases.
A path found with some truth assignment satisfying the path condition. In deterministic code, this assignment guarantees to derive the execution of the path.
In nondeterministic code, this is one of the possibilities.Can transform the code to force replying the path.
![Page 82: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/82.jpg)
Divide and Conquer Intersect property automatonproperty automaton with the
flow chartflow chart, regardless of the statements and program variables expressions.
Add assertions from the property automaton to further restrict the path condition.
Calculate path conditions for sequences found in the intersection.
Calculate path conditions on-the-fly. Backtrack when condition is false.Thus, advantage to forward calculation of path conditions (incrementally).
![Page 83: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/83.jpg)
Spec:¬at l2U (at l2/\ ¬at l2/\(¬at l2U at l2))
¬at l2
at l2
¬at l2
at l2
l2:x:=x+z
l3:x<t
l1:…
l2:x:=x+z
l3:x<t
l2:x:=x+z
XX==
![Page 84: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/84.jpg)
Spec: ¬at l2U (at l2/\ xy /\ (¬at l2/\(¬at l2U at l2 /\ x2y )))
¬at l2
at l2/\xy
¬at l2
at l2/\x2y
l2:x:=x+z
l3:x<t
l1:…
l2:x:=x+z
l3:x<t
l2:x:=x+z
XX==
xy
x2y
![Page 85: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/85.jpg)
Example: GCD l1:x:=a
l5:y:=z
l4:x:=y
l3:z:=x rem y
l2:y:=b
l6:z=0? yesno
l0
l7
![Page 86: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/86.jpg)
Example: GCD l1:x:=a
l5:x:=y
l4:y:=z
l3:z:=x rem y
l2:y:=b
l6:z=0? yesno
Oops…with an error (l4 and l5 were switched).
l0
l7
![Page 87: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/87.jpg)
Why use Temporal specification
Temporal specification for sequential software?
Deadlock? Liveness? – No! Captures the tester’s intuitionintuition about the
location of an error:“I think a problem may occur when the program runs through the main while loop twice, then the if condition holds, while t>17.”
![Page 88: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/88.jpg)
Example: GCD l1:x:=a
l5:x:=y
l4:y:=z
l3:z:=x rem y
l2:y:=b
l6:z=0? yesno
l0
l7
a>0/\b>0/\at l0 /\at l7
at l0/\a>0/\b>0
at l7
![Page 89: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/89.jpg)
Example: GCD l1:x:=a
l5:x:=y
l4:y:=z
l3:z:=x rem y
l2:y:=b
l6:z=0? yesno
l0
l7
a>0/\b>0/\at l0/\at l7
Path 1: l0l1l2l3l4l5l6l7a>0/\b>0/\a rem b=0
Path 2: l0l1l2l3l4l5l6l3l4l5l6l7 a>0/\b>0/\a rem b0
![Page 90: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/90.jpg)
Potential explosion
Bad point: potential explosionGood point: may be chopped on-the-fly
![Page 91: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/91.jpg)
![Page 92: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/92.jpg)
![Page 93: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/93.jpg)
![Page 94: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/94.jpg)
![Page 95: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/95.jpg)
![Page 96: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/96.jpg)
![Page 97: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/97.jpg)
![Page 98: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/98.jpg)
![Page 99: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/99.jpg)
![Page 100: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/100.jpg)
![Page 101: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/101.jpg)
![Page 102: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/102.jpg)
![Page 103: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/103.jpg)
![Page 104: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/104.jpg)
![Page 105: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/105.jpg)
Drivers and Stubs Driver: represents the program
or procedure that called our checked unit.
Stub: represents a procedure called by our checked unit.
In our approach: replace both of them with a formula representing the effect the missing code has on the program variables.
Integrate the driver and stub specification into the calculation of the path condition.
l1:x:=a
l5:x:=y
l4:y:=z
l3:z’=x rem y/\x’=x/\y’=x
l2:y:=b
l6:z=0? yesno
l0
l7
![Page 106: Algorithmic Testing](https://reader035.vdocuments.us/reader035/viewer/2022062305/56815d2c550346895dcb23f5/html5/thumbnails/106.jpg)
Conclusions Black box testing: Know transition relation,
or bound on number of states, want to find initialstate, structure, conformance, temporal property.
Software testing:Unit testing, code inspection, coverage, test case generation.
Model checking and testing have a lot in common:CAV 2004+ISSTA 2004 together, in Boston, MA.