Formal Change Impact Analysis of Extended Finite State Machines Using a Theorem Prover

Mahadevan Subramaniam and Bo Guo University of Nebraska at OmahaAn Approach for Selecting Tests with Provable Guarantees12/15/2013TESTCOM-2009Multi-version QA11/30/20122UNORequirementsDevelopmentTeamTestingTeamcodesbugsR1No bug claimedTesting is very costly and time consuming [NIST] 1) in critical path for release 2) development deadline slips can reduce test time 3) development team needs time to fix bugsChangesDevelopmentTeamTestingTeamModified codesbugsR2No bug claimedTwo or Three weeks2Its hard to expect that developers can finish coding on time.2/15/2013TESTCOM-2009Multi-version QA Scope11/30/2012Predictive Changes: new tests to validate new featuresCorrective Changes: Regression testing of existing features

ProblemsRegression Test Minimization remove redundant tests Regression Test Prioritization fault-finding tests firstRegression Test Selection (RTS) select relevant tests

3UNO32/15/2013TESTCOM-2009Regression Test Selection -- Motivation11/30/2012 Regression testing is most frequent [NIST] and has significant impact on product quality.

System behavior re-validated in practice using large test suites.

Re-running the entire test suite each time is impractical.

Regression test selection identifies tests from a test suite to validate a change in an evolution step.

4UNO42/15/2013TESTCOM-2009Current State of Art Code based11/30/20125UNO

Current State of Art Model-based11/30/20126UNO

Current State of Art11/30/2012Store all test traces on original version

Static analysisCompare new and modified Control Flow GraphsIf a trace includes the differences, select test including the trace.Use data flow graphs and program dependency graphs7UNOProblems11/30/2012Storing all traces is expensive

Control flow graphs comparing may involve untested parts of programs

No guarantee on the regression test suiteRelevant tests may be skipped (Incomplete)Irrelevant tests may be selected 8UNOGoal11/30/2012Develop an automatic approach for regression test selection to create a regression test suite with provable guarantees and without using any history information.

Main ChallengesProvably predict test execution behavior without actually executing the test ? Keep regression test selection economicalSelection cost + regression test cost 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).

List of states The machine is in only one state at a time

Triggering condition for each transitionOverview of the ApproachAnalyze a given test description to determine if the test execution will exercise a given change and select the test.Identify descriptions having sufficient information about their execution (fully-observable tests)Analyze such descriptions to determine if the change will be executed.Patch other descriptions to obtain information about their execution.

Changes add / delete / replace transitions.

Test description is a sequence of input and output messages with parameters with values over booleans, integers, arrays, queues, and records.

1211/30/201212UNOOverview of the Approach (Contd.) From a given test description and a changefind sets of matching transitions that can process inputs in the descriptiondescriptions having a matching change transition are candidatesdetermine feasibility of an execution path formed by matching transitionsautomatically attempt to patch infeasible paths to obtain an execution path

Test selected if the change transition appears in the feasible execution path.

1311/30/201213UNOTests1: open(100)/ack(100), deposit(50)/ack(150), close/ack(150)2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150)3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50)

A Simple Example11/30/2012Initially a = 0; state is s01. open(v), v > 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).

7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a).Not Selected14UNOs0s11623s234572: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150)

A Simple Example 211/30/2012Find transitions matching each input of description 2{ 1 }{ 2 }{ 3, 7 }{ 6 } = [ ] 15UNO

1. open(v), v > 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a).

s0s11623s234572/15/2013TESTCOM-200915A Simple Example 2 (conti)11/30/2012 16UNO

2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) = [ {1} {2} {3, 7} {6} ]

s0s11623s23457Transition 2 can immediately follow 1 to process next input of 2First test input of 2 must be processed by transition 11. open(v), v > 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a).

2/15/2013TESTCOM-200916A Simple Example 2 (conti)11/30/2012 17UNO

2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) = [ {1} {2} {3, 7} {6} ]

s0s11623s23457Transition 7 can immediately follow 2 since (160 > 0) (160 > 150) satisfiable. Selected1. open(v), v > 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a).

2 has all information about transitions it will execute to process the first input2/15/2013TESTCOM-20091711/30/2012A Simple Example 3

18UNOs0s11623s234573: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) = [ {1} {2} {3, 7} {3, 7} {6} ]

First two inputs processed by transition 1 followed by transition 2. Transition 7 cannot immediately follow 2 since (100 > 0) ^ (100 > 150) is unsatisfiable. 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a).

1. For the first, second and forth test inputs, there are only one candidate transition.

2. However, in the third test input, both 3 and 5 are possible to be triggered. which one we will pick is determined by theorem prover.

2/15/2013TESTCOM-20091811/30/2012A Simple Example 3(conti..)

19UNOs0s11623s234573: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) = [ {1} {2} {3, 7} {3, 7} {6} ]

But 3 can immediately follow 2 since (100 > 0) (100 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a).

1. For the first, second and forth test inputs, there are only one candidate transition.

2. However, in the third test input, both 3 and 5 are possible to be triggered. which one we will pick is determined by theorem prover.

2/15/2013TESTCOM-20091911/30/2012A Simple Example 3(conti..)20UNOs0s11623s234573: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) = [ {1} {2} {3, 7} {3, 7} {6} ]

Neither 7 nor 3 can immediately follow transition 3. 7 can follow 3 after some transitions not using test inputs. ?Can we patch 3 to obtain a path with transition 7? 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a).

2/15/2013TESTCOM-20092011/30/2012A Simple Example 3(conti..)21UNOs0s11623s234573: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) = [ {1} {2} {3, 7} {3, 7} {6} ]

4 cannot patch since (50 > 0) ^ (50 < 50) is unsatisfiable.5 can patch since (50 >= 50) is satisfiable and 7 can immediately follow 5. Selected

?1. open(v), v > 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a).

2/15/2013TESTCOM-200921Salient Aspects11/30/2012Regression test selection for test descriptions with rich data types.

Fully-observable tests: descriptions with sufficient information.A simple structural invariant to identify fully-observable tests.

Automatically analyze failure of invariant to patch test descriptions.

Procedures to select tests guaranteed to exercise changes to EFSMs.

Experiments with 10 web services and protocolsStudy costs of running the full test case and selective test cases.Proposed approach reduces test running times in all examples.

22UNO 2/15/2013TESTCOM-200922Outline11/30/2012Preliminaries Fully-Observable TestsTests with Non-Observable RegionsExperimentsConclusion23UNOEFSMs11/30/2012E = (Ii, Oi, Si, Vi, Ti)Ii, Oi : input and output messagesSi: local statesVi: Data and queue variablesTi: Deterministic transition relation

Transition t: mj, Pt, st qt, ml, Atmi ml: parameterized input and output messagesPt: a predicate over variables from Vist, qt : states from SiAt: ordered sequence of assignments to variables from Vi

Explicit transition is transition having both input and output messages24UNOTest Descriptions and Changes11/30/2012Test description = < g0, [i1/o1, i2/o2, , in/on] >g0 is concrete initial global stateFinite sequence of input/output elements ik/ok

Test run r= g0t0tmgmg0 is an EFSM run produced by applying to the EFSM in state g0.All the global states in r are concrete global states.

Changes to the EFSM Performed at the transition level = < sign, tn >, sign { +, - } (addition, deletion) = < to, tn > (replacement)25UNO1. Each element is a sequence of assignments to the data & the message queue variables

2/15/2013TESTCOM-200925OutlinePreliminaries Fully-Observable TestsTests with Non-Observable RegionsExperimentsConclusion2611/30/201226UNO11/30/2012Transitions Matching a Test DescriptionTransition matches a test if an input-output element of test is an instance of input-output message of the transition and satisfies the input condition of the transition.Transition 3. withdraw(v), v > 0 v 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a).

1236711/30/2012Identifying Fully-Observable Tests29UNO122: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) = [ {1} {2} {3, 7} {6} ]371: conjunct 0: (a0 == 0) (100 > 0)1: conjunct 1: (a0 == 0) (100 > 0) (a1 == a0+100) (50 > 0) 1: conjunct 2: (a0 == 0) (100 > 0) (a1 == a0+100) (50 > 0) (a2 == a1+50) (160 > 0) (160 0) (a1 == a0+100) (50 > 0) (a2 == a1+50) (160 > 0) (160 > a2) Procedure:Success if there is a path labeled s from start to last level

Remove nodes (and corresponding edges) a) NOT have any predecessors (except start) b) NOT have any successors (except those at the last level)

Remove all outgoing edges from node tk with label c (existing s). If there is an outgoing edge from tk with label s (deterministic)4. The resulting graph is then processed iteratively level by level in an attempt to construct an executable path from start to some node in the last level

2/15/2013TESTCOM-200929OutlinePreliminaries Fully-Observable TestsTests with Non-Observable RegionsPreliminary ExperimentsConclusion3011/30/201230UNOTests with Non-Observable Regions3111/30/201231UNOIdentify a non-observable region (tk, tk+1) for a conjunct that fails to extend to transition tk+1EFSM paths from output state of tk to input state of tk+1.All intermediate transitions in all paths have no test inputs.

Eliminating regions without loopsMerge the intermediate transitions with tk to tmergePatch conjunct C = Pos([t1,, tmerge], g0) Pre(tk+1) and check if satisfiable. Fails if patched conjuncts for all paths in region unsatisfiable.Patching Failures Example3211/30/201232UNO3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50)s0s11623s2345712377?3571. open(v), v > 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a).

?ignoreRegions with Terminating Loops3311/30/201233UNORegions having loops cannot be eliminated by simple merging.Each loop iteration requires additional test input.

Merge intermediate transitions in the path until the loop end-points and attempt to construct a path by executing loop transitions.Process terminates since all loops are terminating over concrete global states.7Patching Involving Loop Example3411/30/201234UNO4: open(100)/ack(100), deposit(50)/ack(150), withdraw(50)/ack(100), withdraw(60)/ack(40), withdraw(60)/ack(50), close/ack(50)1237353s0s11623s23457?7??ignorePatch fails?1. open(v), v > 0, s0 s1, {a = a + v}, ack(a),2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a),3. withdraw(v), v > 0 v 0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {},6. close, true, s1 s0, {}, ack(a).7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a).

7457 = [ {1} {2} {3, 7} {3, 7} {3, 7} {6} ]OutlinePreliminaries Fully-Observable TestsTests with Non-Observable RegionsExperimentsConclusion3511/30/201235UNOExperiments

Regression Cost Model [Leung and White 91] C1: Cost of running the full test suite C2: Cost of running the selected tests C3: Cost of analyzing the regression test selectionAtm: automatic teller machine; Thp: third party call; Bnk: back web services; Ven: a vending machine(Cmp, Tcp, Cnf) : completion, two-phase commit, and conference protocol36

- >2/15/2013TESTCOM-200936Time Savings across Multiple Changes

3711/30/2012UNO

2/15/2013TESTCOM-200937Test Selected across Multiple Changes

3811/30/2012UNO

2/15/2013TESTCOM-200938SPG with Code-based approach11/30/201239UNOP1:

incr1 (x) {p1 while(++x