test suite minimization
TRANSCRIPT
![Page 1: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/1.jpg)
Test Suite Minimization
A Concept Analysis Inspired Greedy Algorithm for Test Suite
Minimization
http://llvm.org/pubs/2005-09-PASTE-GreedySuiteMinimization.pdf
![Page 2: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/2.jpg)
Motivation
• Large softwares typically have large number of tests and their complete runs take several hours
• For minimal changes a complex software undergoes daily, a lot of these tests are only testing what’s already tested
• These tests are run in nightly release process, for pre-check in requirements and check-ins consuming time, energy and dollars
![Page 3: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/3.jpg)
Problem
• How do we know which tests to run given a software change?– Test coverage information for each tests can
answer this question
• But we can do better. Do we need to run all the tests according to the test coverage?– But this problem is hard. NP-hard.
![Page 4: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/4.jpg)
Problem in an examplepublic boolean isWeekend(int day) { if(day > 5) { return false; //R1 } else { return true; //R2 }}
Test/Req R1 R2 R3
T1 X X
T2 X
T3 X X
//T1public void testIsWeekend(int ) {
AssertFalse(isWeekend(Friday.index))//5AssertTrue(isWeekend(Saturday.index))//6
}//T2public void testIsBefore() {
AssertTrue(isBefore(Monday.index, Friday.index));
}//T3 public void testMonday() { AssertFalse(isWeekend(Monday.index))//5
AssertTrue(isBefore(Monday.index, Friday.index));
}
public boolean isBefore(int day1, int day2) { return day1 < day2 //R3}
![Page 5: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/5.jpg)
Problem in an example• Let us consider two functions:
– IsWeekend: If days are numbered from Monday to Sunday, every day numbered less than 5 is a weekday, otherwise it’s a weekend. These are marked as requirements R1 and R2 respectively
– IsBefore checks if day1 comes before day2 and is our requirement R3
• Now also consider 3 test cases– T1 tests if Friday and Saturday is a Weekend day– T2 tests if Monday comes before Friday– T3 tests if Monday is weekend and comes before Friday.
• Let us also create a table to map which tests which of the requirements. Now from the table we can see that if I run T1 and T2, the T3 is redundant. If Friday is a weekday and Monday comes before Friday then Monday has to be a weekday
![Page 6: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/6.jpg)
Problem Definition
• Given a test-requirement matrix where– A test tests all the requirements listed against it– A requirement may be satisfied by any test listed
against it
• Find the minimum number of tests which can test all the requirements
![Page 7: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/7.jpg)
Outline
• Problem is NP Hard – Reducible from Set Cover
• Heuristics used to solve this problem– Greedy– HGS Algorithm
• Delayed Greedy Algorithm using Concept Analysis (this algorithm)
![Page 8: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/8.jpg)
NP Hard
If there are n different things, how many subsets of n can I create?
2n
Therefore, if there are n tests, how many tests should I run to have enough coverage?
![Page 9: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/9.jpg)
Convention
• In the following slides, circles denote requirements and the rectangles denote tests– Rectangles span multiple circles. This implies that a
test can test multiple requirements
• A requirement can be completely tested by any test in which it appears.
• Red rectangles represent selected tests
![Page 10: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/10.jpg)
Greedy Algorithm
• Assuming the given requirements and tests, Greedy algorithm tries to pick the smallest set of test cases
• To achieve this, Greedy algorithm choses a test case which covers the maximum number of requirements. It then removes the covered requirements and test from consideration
• The algorithm ends when no requirements are left
![Page 11: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/11.jpg)
Greedy
Choose the set with max uncovered element
![Page 12: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/12.jpg)
Greedy
Choose the set with max uncovered elements
![Page 13: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/13.jpg)
Greedy
Choose the set with max uncovered elements
![Page 14: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/14.jpg)
Greedy
The final solution of Greedy algorithm
![Page 15: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/15.jpg)
Optimal Solution
However, the optimal solution is given below
![Page 16: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/16.jpg)
Problem with Greedy?
• The algorithm missed the optimal solution. Why?
• Choice about picking the first set was made too early!
![Page 17: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/17.jpg)
HGS Algorithm
• Choose requirements which are covered by k tests cases starting with k = 1
• Pick the test case which covers the most requirements. If more than one tests qualify then pick randomly
• Remove the requirements by the picked test. Continue with the rest
![Page 18: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/18.jpg)
HGS AlgorithmPick requirements which are tested by k tests. k = 1
![Page 19: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/19.jpg)
HGS AlgorithmPick requirements which are tested by k tests. k = 1
![Page 20: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/20.jpg)
HGS AlgorithmPick requirements which are tested by k tests. k = 2
![Page 21: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/21.jpg)
HGS AlgorithmPick requirements which are tested by k tests. k = 2
![Page 22: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/22.jpg)
HGS AlgorithmPick nodes which are present in k subsets starting with k = 3. Pick test T’ arbitrarily among T and T’
T
T’
![Page 23: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/23.jpg)
HGS AlgorithmPick nodes which are present in k subsets starting with k = 3
![Page 24: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/24.jpg)
HGS AlgorithmNow find the remaining requirement (shown as red circle). Pick a test randomly which satisfies it
![Page 25: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/25.jpg)
HGS AlgorithmPick nodes which are present in k subsets starting with k = 3. Picked a test randomly among the three
![Page 26: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/26.jpg)
HGS AlgorithmThis makes one of the earlier chosen tests (dark blue border) redundant
![Page 27: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/27.jpg)
Problem with HGS?
Implications could not be derived while making choices
![Page 28: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/28.jpg)
Problems with other Heuristics
• Choice about picking the first set was made too early
• Implications could not be derived
![Page 29: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/29.jpg)
How Concept Analysis works?
Find and eliminate all the dependencies before choosing subsets
![Page 30: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/30.jpg)
How Concept Analysis works?
t3 tests requirements r2 and r5. While t5 tests r5 only. Therefore if t3 is run, t5 can be ignored
![Page 31: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/31.jpg)
How Concept Analysis works?
If r4 is tested, then r1 is tested also. Similarly, r6 and r3
![Page 32: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/32.jpg)
How Concept Analysis works?
Remove r1, r3 and t5
![Page 33: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/33.jpg)
How Concept Analysis works?
Similarly, t1 can be removed in favor of t3
![Page 34: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/34.jpg)
How Concept Analysis works?
Now apply Greedy minimum set cover algorithm on that rest to find the minimum number of tests to be run to test all the requirements
![Page 35: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/35.jpg)
How Concept Analysis works?
• Minimization of test-requirement matrix to remove redundancies
• Delayed sub-set selection to avoid picking sub-optimal tests
• Delayed Greedy algorithm performs at least as well as the Greedy algorithm
![Page 36: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/36.jpg)
Concept Analysis - Details
• Lattice• Concepts• Lattice of Concepts• Labels• Strong concept• Algorithm
![Page 37: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/37.jpg)
Latticehttps://en.wikipedia.org/wiki/Lattice_(order). For example show below is a lattice of subsets of a set {a,b,c}
![Page 38: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/38.jpg)
Concept• Consider an ordered pair (t, r) where t is the set of tests
and r is a set of requirements
• Such a set is called maximal grouping if t(or r) is the maximal set of tests(or requirements) related to all requirements(or tests) in r(or t)– i.e. t = ∩iri where ri r and r∈ i is a collection of tests from
reqruiements-test table.– i.e. r = ∩iti where ti t and t∈ i is a collection of tests from
requirements-test table.
• Then this maximal grouping (t, r) is called Concept
![Page 39: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/39.jpg)
Lattice of Concepts
![Page 40: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/40.jpg)
Lattice of Concepts• The lattice is ordered from top to bottom for tests i.e. at
each level from top to bottom, number of tests in the concept increases
• And from bottom to top for requirements i.e. at each level from bottom to top, number of requirements in the concept increases
• Not all the points of the concept lattice have a maximal requirement set and therefore, not all of them will be found on the CA lattice e.g. {t5}
![Page 41: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/41.jpg)
Labels
• Lattice points are labeled with test case (requirement) name if it is the smallest (according to the partial order) lattice point in which it appears e.g– t3 appears on c4 and t5
appears on c8. – Similarly, r1 appears on c5
and r4 appears on c1
![Page 42: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/42.jpg)
Strongest concept
• Those concepts which are immediately above the bottom– e.g. c1, c2, c3, c4
are strongest concept of this lattice
![Page 43: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/43.jpg)
Object implicationPick the overlapping test cases (e.g. t3, t5) and remove the one which is a subset of the other. From the lattice, these are labeled concepts ci and cj where ci < cj
![Page 44: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/44.jpg)
Attribute ImplicationPick the requirements which overlap and remove the one which is a superset of the other. From the lattice, these are labeled concepts ci and cj where ci < cj
![Page 45: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/45.jpg)
Object Reduction - 2
![Page 46: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/46.jpg)
Owner Reduction
Empty Table
Pick the strongest concepts from the reduced lattice
![Page 47: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/47.jpg)
Delayed Greedy Algorithm
• Repeat until table is empty– Repeat until table changes• Object Implication• Attribute Implication• Owner Reduction and choose strongest concepts
– One step of Greedy
![Page 48: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/48.jpg)
Experiments
• No such requirement-test matrix exists for most softwares. How can this approach be tested?
• Requirements: Branches and def-use pairs (using LLVM)
• Tests: Identify tests which hit these requirements
![Page 49: Test suite minimization](https://reader035.vdocuments.us/reader035/viewer/2022070600/58a3566b1a28ab14598b56e5/html5/thumbnails/49.jpg)
Summary
• Minimization of test-requirement matrix to remove redundancies
• Delayed sub-set selection to avoid picking sub-optimal tests
• Greedy Algorithm on the remaining table
• Never performs worse than other approximation algorithms.
• Minimize the number of tests to run for a given set of requirements
• The problem is NP hard. However, calculation needs to be done only once.
• Given a minimized requirement-test matrix, changes to a requirement will need only limited tests to be run
Problem Solution – Delayed Greedy Algo