a white paper on augmented unit test generation · can help developers by generating test cases and...

A WHITE PAPER on Augmented Unit Test Generation

September 2014

© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.

Abstract ............................................................................................. 3

What are unit tests?.......................................................................... 3

How does automation of unit test case generation help? ............... 4

Approach behind test case automation............................................ 4

Example ........................................................................................... 11

Key benefits of this approach ......................................................... 13

Conclusion ....................................................................................... 13

References ...................................................................................... 14

Author Information ......................................................................... 15

TABLE OF CONTENTS


3

Abstract

Unit testing is done by developers who write tests while building

any software. It is a simple but effective idea that improves time-

to-market, flexibility and the quality of any software released in the

market.

Preferably, each developer should produce a set of tests to validate

each individual method and class as they are being written. It takes

time to identify all the scenarios, write tests, run test cases and

monitor the results. Also, writing unit tests by hand is expensive,

labor intensive, tedious, and still leaves the possibility that many of

the execution paths are not tested and remain incomplete. For

beginners there is also the problem of understanding unit tests and

how to write them.

This problem can be solved to a large extent by automation, which

can help developers by generating test cases and scaffolding code

that enables and supports developers to quickly write unit test

cases. But instituting an automated unit testing practice across a

software development team can also be technically challenging and

time consuming.

This paper elaborates a simple approach on how unit test case

generation can be automated, helping the developers in reducing

the amount of engineering hours needed, while at the same time

increasing both completeness of the test coverage and the

software quality.

What are unit tests?

Unit tests are test cases that ensure logic is tested, by invoking

methods with different combinations of input parameter values, to

drive the code through different execution paths, thereby

Writing unit tests by hand is laborious and expensive, and still leaves the possibility that many of the execution paths might remain untested.

Hence, if there is a mechanism to automate large parts of writing unit tests, this helps developers to concentrate more on code development, which is writing code and spending minimum efforts in testing the code, thereby ensuring that there is no compromise in the quality.


4

indicating to developers that each piece of code is tested and

ensures code quality..

How does automation of unit test case generation help?

Writing unit tests by hand is laborious, expensive and still leaves

the possibility that many of the possible execution paths remain

untested. To guarantee that as many important execution paths

would be exercised by a test suite, it requires a detailed analysis of

how the input parameters drive the code in various directions. This

is very time consuming and difficult for anything but trivial

functions. While integrating Code Coverage tools with the

development environment has alleviated this problem to some

extent, it still requires the developers to do a lot of probing in order

to write unit tests.

If there is a mechanism to automate large parts of writing unit tests,

this will help developers to concentrate more on the crux of

development - which is writing code - and they would exert the

minimum effort in testing the code, thereby ensuring that there is

no compromise in quality.

Approach behind test case automation

Automating test cases for each method inside classes or on test

suites is simply not an easy task. There should be a methodology

that defines the way, starting from parsing the information, finding

the execution paths, and creating test cases according to the

conditions to be tested.

Broadly, the approach that will clearly help in complementing the

efforts, and in creating a more exhaustive set of unit tests with

much less manual effort can be described as follows:


5

1. Determine the number of test cases to be generated

(Cyclomatic Complexity is a good place to start)

2. Find test data that can enable a test scenario to be fulfilled

for each test case

3. Generate test case along with test data

1. Determine the number of test cases to be generated

Consider a file containing “n” number of methods. The number of

test cases that are to be generated for each method depends upon

the cyclomatic complexity of that particular method.

Cyclomatic complexity overview

Cyclomatic complexity is a static software metric pioneered in the

1970s by Thomas McCabe. The cyclomatic complexity number

(CCN) is a measure of the number of paths there are in a method. It

serves as a rough measure of code complexity and as a count of the

minimum number of test cases that are required to achieve full

code-coverage of the method.

The equation for calculating the cyclomatic complexity number

comes from the theory of graphs where it refers to the number of

paths from any point in a topological space to any other point. It is

expressed as:

Where, E represents the number of edges on the graph, N the

number of nodes on the graph, and P the number of connected

components. In programming terms, E represents the code

executed as a result of a decision, N is the number of decision

CCN = E - N + P


6

points (conditional statements) and P is the number of ways to exit

the program. As a software metric it can be expressed as:

Formula behind the generation of test cases

As the cyclomatic complexity of a method is fundamentally the

number of paths contained in that method, general rule of thumb

states that in order to ensure a high level of test coverage, the

number of test cases for a method should equal the method's

cyclomatic complexity.

2. Determine the test data for each test case

Once the number of test cases has been identified, one has to

identify what type of test case scenarios are to be handled inside

each test case method respectively.

The following steps are applicable while calculating test data:

Identify the code paths - which means it can be conditional,

branching or looping statements

For each path traversal in a program we can have test

cases (satisfying respective true and false conditions)

For a sample logic flow as shown below:

CCN = "The number of decision points" + 1


7

The number of test cases to be generated should be 4, since CCN

=4

Then test scenarios covered under each test case are as follows:

1. Path 1: 1,2,3,4,5,7,8,10,11

2. Path 2: 1,2,3,4,6,10,11

3. Path 3: 1,2,3,4,5,7,8,10,11

4. Path 4: 1,2,3,4,5,7,9,10,3,4,6,10,11

Determine the

variables that are present in

the code paths whose value

determine the true or false of

the code path. Variables can

be inferred in some cases

where there is actually no

need to define a variable but

are return values of method

invocations.

Identify the type of variable (can be basic data type like

integer, floating point, double, byte, etc. or complex data

types that can be classes by themselves)

Identify the scope of the variables. Variables are typically of

various scopes:


8

o Method Scope – Variables that act as input

variables to the method under test

o Class Scope/Global Scope – Variables that are set

elsewhere (either using setter methods or by direct

assignment, but populated outside the scope of

the method under test)

o Computed Scope – Inferred Variables that are a

result (return types) of method invocations.

How can initialization or mocking of test data values be

done?

The following are the factors on which initialization or mocking of

test data values is done for each test case, covering its test scenario

1. Type

Depending upon the type of the variable, these are the two

classifications

1. Basic data type

2. Complex data type (Object references of some classes)

Basic data type handling

If a particular variable is a basic data type, then test data for it will

be determined based on the condition and the operand given in

the conditional or branching or looping statements.


9

For e.g., if the condition inside the loop is i<8, operand is “<”, then

for the generation of a true case scenario the value of “i” can be

initialized to 7 and can be “9” for a false case.

Non basic data type handling

If the variable present in the code path is non basic, i.e. reference

of some class, then these types of variables cannot be initialized as

straightforward as that of basic types. There comes a need for

object mocking for those reference classes and returning the values

based on the condition.

What is the need for mocking?

When there are objects or class references, there comes a need to

keep dependencies out of the unit test. Mocking helps in achieving

this.

For fetching any data from a database, one would normally need a

dependency to the database to do this. However, with object

mocking, one can simulate the interaction with the database with

a mock framework, so it might return a particular dataset which

looks like the one returned from the database and can then test

the code to ensure that it handles translating a dataset to a user

object, rather than using it to test that a connection to the

database exists.

When there are objects or class references there comes a need to keep dependencies out of the unit test. Mocking helps in achieving this.

To fetch any data from the database one would normally need a dependency on the database to do this.

However, with object mocking, one can simulate the interaction with the database with a mock framework, so it might return a particular dataset which looks like the one returned from the database.


10

Many mocking frameworks exist for different languages. In the Java

world, mocking frameworks like EasyMock and jMock are available

and in the .NET world, frameworks like NMock, EasyMock.NET, and

Moq are available.

2. Scope

Depending upon the scope of the variable, the test data

determined for that particular test case will vary, such as:

Method Scope

The initialized or mocked values for method variables are passed to

the test method by passing as the method parameter.

Class Scope/Global Scope

The initialized or mocked values for these types of variables are

passed to the test method by calling its setter or assignment

statement.

Computed Scope

Mocking should be carried on for all the dependent classes before

the test method call happens.

3. Generation of test cases with test data

Once the data along with test cases has been determined, the next

step is to generate the test cases with their respective test data.

Broadly, the steps needed for generation are as follows:

1. Deciding on the testing framework that is to be used for

writing unit test cases

Depending upon the scope of the variable, the test data determined for that particular test case will vary

Method Scope Class Scope/Global

Scope Computed Scope


11

2. Based on the chosen framework, following the naming

conventions for defining class names and method names

for the test cases

3. Configuring the initial set up that is needed for any test

case to run

4. With the information obtained about the number of test

cases to be generated for a file containing n number of

methods along with test data information, generate test

cases along with test data

5. In cases where the generation cannot be complete, provide

sufficient template code and test scaffolding (example

object initialization and method invocation, handling

failures, handling success, etc.) for developers to complete

test cases along with sufficient documentation on the

scenario to be tested.

The objective of generating test cases is not to completely

automate all test scenarios. It might be possible that certain test

cases cannot be generated for various reasons. The objective of

automation in such scenarios is to accelerate the time taken to

write the test cases by creating a test scaffold, which will provide

the developer with all the necessary code to write the test cases

but still allowing the developer to fill in details wherever required.

Example

If the above procedure is applied to a language like Java, the

process of augumented unit test case generation pans out as

below:

Frameworks:

Testing framework – Junit 4.x

Mocking framework – EasyMock


12

Eclipse modelling framework - Eclipse JDT (for

code parsing and generating)

A sample Java class


13

A sample of generated test case for the above Java class

Key benefits of this approach

Ensures methodical and reliable test case coverage Helps in

covering all code paths even in case of high complexity

Reduces the maximum time effort spent by developers in

identifying the code path and how conditions are to be

handled

Helps in delivering good quality software on starting

Helps developers who are not exposed to unit testing

Accelerates software development speed

Conclusion

This paper discussed the approach towards augumenting unit test

case generation in order to ensure that methods are tested

sufficiently as well as provide sufficient guidance to developers to

write unit test cases.


14

By enabling and guiding developers in understanding scenarios to

test, generating scaffolding code to help complete testing scenarios,

and by generating complete test cases wherever possible, the

approach ensures code quality and a shorter time for development

across projects.

References

Improved Software Development by Unit Test Automation

Why Unit Testing?

Code Metrics: Cyclomatic Complexity and Unit Tests

Unit_testing

http://www.testready.net/White_Paper.htm

http://www.agitar.com/solutions/why_unit_testing.html

http://www.rkcole.com/articles/other/CodeMetrics-CCN.html


15

Author Info

Saranya Duraisamy is working with the ERS Practice Team as a Senior Software Engineer. She has over 2 years of software experience in developing J2E based applications and code generation tools.

Rajesh Venkatesan is a Solutions Architect with the ERS Practice team and has over a decade of experience in the development of large scale distributed systems and multi tenant cloud based architectures for HCL customers, and has filed multiple patents in this space. He is also responsible for building HCL IPs that impact the way software is built.

Hello, I'm from HCL's Engineering and R&D Services. We enable technology led organizations to go to market with innovative products and solutions. We partner with our customers in building world class products and creating associated solution delivery ecosystems to help bring market leadership. We develop engineering products, solutions and platforms across Aerospace and Defense, Automotive, Consumer Electronics, Software, Online, Industrial Manufacturing, Medical Devices, Networking & Telecom, Office Automation, Semiconductor and Servers & Storage for our customers. For more details contact: [email protected] Follow us on twitter: http://twitter.com/hclers Our blog: http://www.hcltech.com/blogs/engineering-and-rd-services Visit our website: http://www.hcltech.com/engineering-rd-services

http://twitter.com/hclers

http://www.hcltech.com/blogs/engineering-and-rd-services

http://www.hcltech.com/engineering-rd-services

a white paper on augmented unit test generation · can help developers by generating test cases and...

Documents