optimal test suite generator tool
DESCRIPTION
General Overview: Create a tool that can generate test suites with desired time/coverage results. As input the tool takes a set of tests, runs them to determine the coverage and time needed for each test. When queried by the tester, the tool can calculate test suites that satisfy different coverage and time requirements. The tool should be able to handle multiple queries for different output test suites without rerunning and recalculating all of the input tests values. Tool input: T: set of test cases written in either Cucumber or as wget requests. A: java based web application in the form of a WAR C: tuple of constraints (cT, cS, cB) that specify the maximum desired time (cT) O should run, the desired statement coverage (cS) of O, and the desired branch coverage (cB) of O. Tool Output: O: set of test cases, such that O is a subset of T, that satisfy the input constraints CTRANSCRIPT
- Abhishek Rakshe
- Rajasa Savant
Testing & Analysis of Web ApplicationsUniversity of Southern California (USC)Under the Guidance of Professor William G.J. Halfond
Outline
Introduction to Test Case Generator
Design & Approach
Algorithm
Implementation
Results
Analysis
Conclusion
Future Scope
Introduction to Test Case Generator Test Case Generator generates test suite of test
cases with desired line & branch coverage.
The tool automatically executes the test cases using Selenium TestNG and generates XML reports using another tool Cobertura.
The test cases chosen by the tool have highest line coverage and branch coverage.
The tool uses approximation technique to minimize the search space for selecting optimum test cases.
Design & Approach Test Case Generator makes use of the following tools:
Selenium 2.2.5
Selenium Standalone Server 2.2.5
TestNG
Cobertura 1.9.4.1
Eclipse IDE
Execute Test Cases
• Using Selenium TestNG automate test case execution in java
Generate XML reports
• Using Cobertura to generate reports
Get application meta data
• Parse the XML report and build application schema
Parse all test cases
• Using SAX xml parser in java. Build Test Case Suite Map
Optimize the test case data
• Get the line and branch overlap weightage
• Build Overlap Coverage Hash Map
• Get the chosen test cases on basis of the Overlap Coverage, line and branch coverage constraints
Algorithm – Optimize test case data 1. Initialize variables 2. Create Log file stream and configuration file 3. Get List of Coverage Reports and mark test ID’s 4. Parse all Coverage XML reports and store each test case data in
TestCase class objects 5. Each test case object data has a map of Classes Hit, Lines Hit
and Branches Hit 6. BuildTestSuiteTable => Optimize the test case data and
generate a hash map if Overlap Coverage Data 7. Approximate the test case overlap coverage data by grouping
similar test cases which have same amount of line and branch covegare.
8. Generate results using mathematical approximation techniques
Inputs required by the tool Test Cases written in Selenium and exported in
TestNG format
Expected Coverage Constraints
Configuration for the application to be tested –
config.properties
Output obtained Test Case IDs which need to be run to obtain required
line and branch coverage
Implementation - Part I Use of TestNG methods
BeforeClass
AfterClass
AfterMethod
Test
Selenium Test cases exported in TestNG
Cobertura for report generation
Implementation - Part I
Selenium Test Cases in TestNG
format
Execution of test cases one by one
Creation of Cobertura reports
after each test
Replace cobertura.ser file
Map test cases with respective
test IDs
Store the coverage report in xml
format
Implementation - Part II GetTestCaseData – Entry point (Main class)
AppMetaData – Store all the application meta data
TestCase – Store individual test case data
BuildOverlapTable – Store the overlap coverage for each test case with
all other test cases
BuildTestSuiteMap – Store all the indiviual test case data in a Map
BuildClassMap – Store individual class data for each test case
ParseXML – Pares the XML coverage reports and store the data in test
case object
Implementation - Part II
InitializeBuild application
schema
Parse XML coverage reports for each test case
Get all the test case data
Build the overlap coverage map
Select optimum test cases to achieve the
desired coverage
OverlapTable
0 T1 T3 T6 T8
1 T1 T2 T14 T29
2 T3 T7 T12
3 T6 T10
4 T5 T8 T11 T36
5
6
7
Test Case IDs
Gro
up
ID
s
Higest Line/Branch Coverage Test Cases
1 2 3 4
Note: T1, T2, T3 …. T36 are Test Case ID’s
Results generated by manual testing - I
I Test Cases Line Coverage Branch Coverage
INPUT 1, 2, 3, 4, 5, 6 10% 5%
OUTPUT 3, 6 16% -
OUTPUT 1, 2, 3, 4 - 5%
II Test Cases Line Coverage Branch Coverage
INPUT 1 through 61 15% 5%
OUTPUT 3, 6, 9 19% -
OUTPUT 1, 2, 3, 4 - 5%
III Test Cases Line Coverage Branch Coverage
INPUT 1 through 61 22% 10%
OUTPUT 3, 6, 9, 10 19% -
OUTPUT 1, 2, 3, 4, 5, 6 - 6%
IV Test Cases Line Coverage Branch Coverage
INPUT 1 through 61 35% 10%
OUTPUT 3, 6, 9, 10, 11, 14, 16 29% -
OUTPUT 1, 2, 3, 4, 5, 6 - 6%
Results generated by manual testing - II
Results generated by manual testing - II
V Test Cases Line Coverage Branch Coverage
INPUT 1 through 61 40% 11%
OUTPUT 6, 3, 9, 10, 11, 14, 16, 18
36% -
OUTPUT 1, 2, 3, 4, 5, 6 - 6%
VI Test Cases Line Coverage Branch Coverage
INPUT 1 through 61 50% 15%
OUTPUT 3, 6, 9, 10, 11, 14, 16 36% -
OUTPUT 1, 2, 3, 4, 5, 6 - %
Results generated by tool - II Test Cases Line Coverage Branch Coverage
INPUT 1 through 60 10% 5%
OUTPUT 6, 3, 9 11.99% -
OUTPUT 3, 4. 28 - 6.42%
II Test Cases Line Coverage Branch Coverage
INPUT 1 through 60 15% 10%
OUTPUT 3, 6, 9, 10 16.77% -
OUTPUT 3, 4, 5, 6, 28 - 12.37%
Results generated by tool - IIIII Test Cases Line Coverage Branch Coverage
INPUT 1 through 60 22% 15%
OUTPUT 6, 3, 9, 10, 11, 14 23.855% -
OUTPUT 3, 4, 5, 6, 7,8, 28 - 16.71%
IV Test Cases Line Coverage Branch Coverage
INPUT 1 through 60 30% 20%
OUTPUT 6, 3, 9, 10, 11, 14, 16, 18
32.18% -
OUTPUT 3, 4, 5, 6,7,8,9, 28 - 21.44%
Results generated by tool - IIIIII Test Cases Line Coverage Branch Coverage
INPUT 1 through 60 40% 30%
OUTPUT 6, 3, 9, 10, 11, 14, 16, 18, 19 ,20
41.8% -
OUTPUT 3, 4, 5, 6, 7,8, 9, 10, 11, 12, 13, 14, 28
- 30.85%
IV Test Cases Line Coverage Branch Coverage
INPUT 1 through 60 55% 50%
OUTPUT 6, 3, 9, 10, 11, 14, 16, 18, 19 ,20, 21, 23, 24
57.97% -
OUTPUT 28, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21
- 50.06%
Observation The tool selects the test cases with higher line and
branch coverage.
The readings are very close to manual testing readings
For most of the scenarios the coverage by manual testing is greater than the tool generated coverage, hence the test cases selected by the tool will always fulfill the expected coverage.
Conclusion The tool selects the test cases with highest line and
branch coverage
The tool makes use of Hash Table and Hash Set to build the overlap coverage to efficiently calculate the expected line and branch coverage.
The tool uses approximation technique which we have implemented to reduce the search space
The tool results show that the total coverage for selected test case will fulfill the expected coverage constraints
Future Scope Test Cases could be directly read from java class files
and then automatically executed
Time constraint can be added for applications which have huge LOC and have more number of test cases
Algorithm can be optimized
Approximation techniques can be tuned to increase precision
The data can be synthesized and stored in a persistent storage like SQL DBMS