lecture 6 model-based testing and automated test case generation
TRANSCRIPT
Lecture 6
Model-based Testing and Automated Test Case Generation
Automated Test-Case Generation (TCG)
• By now we have seen that manual construction of test cases is a difficult, time-consuming and error-prone activity that requires expertise.
• Automated Test-Case Generation (TCG) has been the “holy grail” of testing for some time.
• Is this even possible? If so, how?• Black/white-box testing• Want algorithms which generate test cases
(preferably with a known level of coverage)
Constraint Solving
• A constraint is a logical formula with one or more free variables
• Examples: x >= 0, x+1 > y, x2 = 2.0, etc.• A constraint solver is an algorithm which finds values
for the free variables, which make a formula true, aka. solutions
• Examples: – x=1 I= x >= 0 ( I= is the satisfaction symbol )– x=1.41421356 I= x2 = 2.0
• Exist many different kinds of constraint solvers
Specification-based Black-box Testing
1. System requirement (Sys-Req)2. System under Test (SUT )3. Test verdict pass/fail (Oracle step)
SUTTCG Oracle
Sys-Req pass/fail
Testcase
Output
Constraint solver Constraint checker
Language runtime
Example: ATCG for numerical codeNewton’s Square root algorithm
SUTTCG Oracle
precondition x ≥ 0.0Postcondition Ι y*y – x Ι ≤ ε
Input Output
x=4.0 satisfies x ≥ 0.0 Verdict x=4.0, y=2.0 satisfies
Ι y*y – x Ι ≤ ε
Newton Code
x=4.0 y=2.0
Constraint solver Constraint checker
TCG for MBT of Embedded Systems
• We can make a simple model of an embedded system as a deterministic Kripke structure (or Moore machine)
• M = ( Q, Σ, q0, δ, λ )• Where
– q0 Q, - initial state– δ : Q x Σ Q - state transition function– λ: Q Bn - n-bit output function
Example: 2-bit shift register
• Q = A, B, C, D, Σ = 0,1, q0=A
A,λ(A) =00
B,λ(B) =10
C,λ(C) =01
D,λ(D) =11
1 1
0 00 1
0 1
Another Representation
• Q = A, B, C, D, Σ = 0,1, q0=A
!Bit1 & !Bit2
Bit1 &!Bit2
!Bit1 &Bit2
Bit1 &Bit2
1 1
0 00 1
0 1
2-Bit Shift Reg in nuSMV format
MODULE main VAR
Bit1 : boolean; -- Boolean var Bit2 : boolean; input : boolean; state : {A, B, C, D}; -- scalar var
variableASSIGN
init(state) := A; init(Bit1) := 0; init(Bit2) := 0; -- var “input” is uninitialised!
next(state) := casestate = A & input = 1 : B;state = B & input = 0 : C;state = B & input = 1 : D;state = C & input = 0 : A;state = C & input = 1 : B;state = D & input = 0 : C;TRUE : state; -- why?
esac;next(Bit1) := case ?? esac;next(Bit2) := case ?? esac;
Propositional Linear Temporal Logic PLTL in nuSMV syntax
• Simple temporal logic to express automaton requirements• Requirements are statements about all possible simulations• Boolean variables • A, B, …, X, Y, .. MyVar, etc.• Boolean operators • !ϕ, (ϕ & φ), (ϕ ϕ) , (ϕ -> ϕ) ,…• LTL operators• Fϕ (sometime in the future ϕ)• Gϕ (always in the future ϕ) • ϕ U φ (φ holds until ϕ holds) • Xϕ (next step ϕ holds)• Write Xnϕ for X X … Xϕ (n times)
ExamplesToday is Wednesday
WednesdayTomorrow is Wednesday
X Wednesday(A) Thursday (always) immediately follows (a) Wednesday
G( Wednesday -> Thursday )(A) Saturday (always) follows (a) Wednesday
G( Wednesday -> F( Saturday ) )Yesterday was (a) Wednesday
G( Wednesday -> X Thursday) & Thursday• Exercise: define the sequence of days precisely, i.e. just one solution• Question: are there any English statements you can’t make in LTL?• Question: what use cases can you express in LTL?
Useful Logical Identities for LTL• Boolean identities
!!φ φ, !(ϕ φ) (!ϕ & !φ) , (ϕ -> φ) (!ϕ φ) etc.
• LTL identities!G(!ϕ ) F(ϕ)!X(ϕ) X(!ϕ)G(φ & ϕ) G(φ)& G(ϕ)G(φ) φ & X G(φ)
• Exercise: using these identities, prove:!F(!φ) G(φ)F(φ ϕ) F(φ) F(ϕ)F(φ) φ X F(φ)
• Remark TCG usually involves negating formulas, so its useful to understand what a negation means
Specifications in nuSMV files
LTLSPECG( Bit1 <-> X Bit2 ) –- the value of Bit1 now always equals the-- value of Bit2 in the next time step-- This is obviously TRUE!
G( Bit1 <-> X Bit1 ) –- the value of Bit1 now always equals the-- value of Bit2 in the next time step-- This is obviously FALSE!
Model Checking
• nuSMV is a temporal logic model checker• A model checker is a tool which takes as input
– An automaton model M– A temporal logic formula φ
• If φ is a true statement about all possible behaviors of M then the model checker confirms it (proof)
• If φ is a false statement about M the model checker constructs a counterexample (a simulation sequence)
• A counterexample to φ satisfies !φ• A simulation sequence can be executed as a test case
nuSMV Output Example-- specification G( Bit1 <-> X Bit2 ) -- is true -- specification G( Bit1 <-> X Bit1 ) -- is false -- as demonstrated by the following execution sequence Trace Description: LTL Counterexample Trace Type: Counterexample -> State: 1.1 <-state = A Bit1 = false Bit2 = false-> Input: 1.2 <- input = 1-> State 1.2 <- state = A Bit1 = true Bit2 = false
Automated White-box TCG
• We can use a model checker to generate counterexamples to formulas (i.e. test cases) with specific structural properties.
• This is done by inspecting the graph structure of the automaton
• i.e. white/glass box testing• “Most” use of model checkers concerns this.
Test Requirements/ Trap Properties
• Recall a test requirement is a requirement that can be satisfied by one or more test cases
• Basic idea is to capture each test requirement as an LTL formula known as a “trap property”
Example Suppose the test requirement is “cover state D”
Trap property is G!( Bit1 & Bit2 ) This formula is False and any counterexample must be a path that goes through state D.
Case Study: Car Controller
Velocity = stop
Velocity = fast
Velocity = slow
accelerate
accelerate
brake
brake
!accelerate
!accelerate
Basic idea (incomplete)
Module mainVaraccelerate: boolean;brake: boolean;velocity: {stop, slow, fast}
ASSIGNinit(velocity) := stopnext(velocity := caseaccelerate & !brake & velocity = stop : slow;accelerate & !brake & velocity = slow : fast;!accelerate & !brake & velocity = fast : slow;!accelerate & !brake & velocity = slow: stop;Brake: stop;TRUE: velocity;esac;
Trap properties for Structural Coverage Models
• Let us use nuSMV to automatically construct test suites that satisfy the different structural coverage models introduced in Lecture 2.
• Examples: – Node coverage NC– Edge coverage EC– Condition coverage PC
• How to interpret these concepts?
Node Coverage for CC• Want a path that visits each node• Simple approach: 1 trap property per node• General form: • G(!“unique_state_property”)• Counterexamples satisfy: • F(“unique_state_property” )• Examples:
G(!velocity = stop)G(!velocity = slow)G(!velocity = fast)
• Clearly this will give redundant test cases, but method is still linear in state-space size.
Edge coverage for CC
• Want to be sure we traverse each edge between any pair of nodes
• General form G( α & ϕ -> X !β )• Counterexample satisfies F(α & ϕ & X β )• Example (5 more cases):
G(velocity=stop & accelerate -> X velocity=slow)
• Exercise: define the remaining 5 trap formulas
α βφ
Condition Coverage for CC
• For each transition, and each Boolean guard v there exist two test cases for the transition: (1) guard is true. (2) guard is false
α β Boolean v;
G!(α & v) -> X(!β) G!(α & !v) -> X(!β)
Requirements-based TCG
• Can we also perform requirements-based test case generation?
• Want to test the requirements are fulfilled rather than explore structure of the code.
• Can look at negation of a requirement
Car Controller LTL requirements1. Whenever the brake is activated, car has to stop
G(brake -> X velocity=stop)2. When accelerating and not braking, velocity has to increase gradually
G(!brake& accelerate&velocity=stop -> X velocity=slow )
G(!brake& accelerate&velocity=slow -> X velocity=fast )
3. When not accelerating and not braking, velocity has to decrease gradually
G(!brake& !accelerate&velocity=fast -> X velocity=slow)
G(!brake& accelerate&velocity=slow -> X velocity=stop)
Safety Requirements
• A safety requirement describes a behavior that may not occur on any path.
• “Something bad may not happen”• To verify, all execution paths must be checked
exhaustively• Safety properties usually have the form G!φ
where φ is the “bad thing”• Counterexamples (test cases) are finite
Liveness Requirements
• A liveness requirement describes a behavior that must hold on all execution paths
• “Something good eventually happens”• Safety does not imply liveness• Liveness does not imply safety• Liveness properties may have the form
F(φ) or G(ϕ -> Xnφ ) or G(ϕ -> Fφ )
where φ describes the “good” thing and ϕ is some precondition needed for it to occur.
• Counterexamples may be finite or infinite (why?)
Infinite Counterexamples
• nuSMV describes infinite counterexamples as a “loop and handle”
• a,b,c,d,(x,y,z)ω or handle(loop) ω
• a, …, d, x, …, z are the individual states• We can generate these sequences, but can we execute
them?• Are they test cases at all?• We call a liveness requirement finite if all
counterexamples are finite• Use cases are examples of finite liveness requirements
TCG for Finite Liveness Requirements
• A common form of liveness requirement isG( φ -> Xn ϕ ) (1)
• Negating (1) and rewriting gives F( φ & Xn (!ϕ ) ) (2)
• Every counterexample to (2) is a solution to (1) i.e. an intended behavior.
• We can ask nuSMV for these intended behavior, and compare them against the actual behavior.
• Edit the sequence to delete all internal and output variables• Agreement = pass, disagreement = fail
Car Controller Examples
(1)F(brake & X !velocity=stop)
(2) F(!brake & accelerate & velocity=stop -> X velocity=slow)
Conclusions
• TCG is certainly possible using various existing algorithms.
• Generally requirements MBT is trickier than structural MBT using this approach.
• There are other approaches to requirements testing. See next lecture.