15.1 Introduction Test execution is situated on the critical path to

product introduction. Test automation is used, for instance, to minimize the time needed for test execution. Automation can be profitable in terms of time, money, and/or quality.

In principle, the execution of almost every test can be automated. In practice, only a small part of the tests are automated. Test automation is often used in the following situations: tests that have to be repeated many times; a basic test with a huge range of input variations – the same

steps have to be executed every time but with different data (for instance, evolutionary algorithms);

very complicated or error-prone tests; testing requires specialized equipment that generates the

appropriate input signals, activates the system, and/or captures and analyzes the output.

15.1 Introduction Changes in the system, design, and require

ments are common and are also a threat to the usability of the automated tests. The consequences of these changes can be: additional test cases; changed test cases; different result checks; changed system interface; changed functionality; different signals; different target platform; different internal technical implementation.

15.2 The technique of test automation-15.2.1 Design for maintain

ability To avoid a test suite from becoming shelfware it is designed to be as independent as possible of changes – changes in the test object, the technical connection between the test suite and the system under test, and the tests and data used.

This is established by reducing the dependencies: test data is stored outside the test suite; the description of test actions is separated from the technical implementati

on of these actions; the process of how to handle an automated test is implemented independen

t of its environment and system under test – this explains why any test suite needs several essential components;

the communication between the system under test and the test suite is implemented independent of the central process of the test suite.

These issues will be discussed in more detail in the following sections. The following terms will be used: Automated testing. Executing tests using an automated test suite. Test suite. Everything necessary to execute a test by pushing just one butto

n. Data driven. An approach where physical test actions are separated from te

st data, and the test data is the impulse for the test. Framework. A library of reusable modules and design.

15.2.1.1 Test cases The test cases are stored outside the te

st suite (see Figure 15.1). They can be stored in a database or spreadsheet. An additional test case means adding only this test case to the storage of the test cases. The test suite doesn’t have to be changed.

15.2.1.2 Test actions This implementation should be hidden from the

normal user of the test suite – the only thing to know is which test actions are available (see Figure 15.2).

By choosing the right test actions the right test script can be composed. If the functionality of the system under test stays the same but the technical implementation is changed (for instance, another interface is used) only the technical implementation must be changed. The user can still use the same script as if nothing has happened. Table 15.1 shows an example where test data and actions are combined in one test script. This combination is quite common.

15.2.1.2 Test actions

15.2.1.3 Essential compone15.2.1.3 Essential componentsnts

Those basic components are part of the generic design oThose basic components are part of the generic design of a test suite (see Figure 15.3) which is called a blueprint f a test suite (see Figure 15.3) which is called a blueprint – this has three major sections. – this has three major sections.

The first is the input side with test scenarios and scripts. The first is the input side with test scenarios and scripts. The last is the output side with status reports and progreThe last is the output side with status reports and progre

ss reports. The processing unit, or test suite kernel, is sitss reports. The processing unit, or test suite kernel, is situated in between. uated in between.

This kernel makes it possible to test with one push of a bThis kernel makes it possible to test with one push of a button. utton.

The test suite continues working without human interveThe test suite continues working without human intervention until the complete test scenario (see section 6.4.3) ntion until the complete test scenario (see section 6.4.3) is executed. Appendix C provides a more detailed descriis executed. Appendix C provides a more detailed description of the blueprint.ption of the blueprint.

15.2.1.4 Communication The system under test can have the follo

wing forms: executable code on the same machine as th

at on which the test suite is installed; executable code on a test target (separated f

rom the test suite environment); system under test in a simulation environm

ent; system under test as pre-production system.

15.2.1.4 Communication The test case will stay the same while the communication, or t

echnical implementation of the communication, is changed. There has to be a communication layer (see Figure 15.3) for all forms.

15.2.2 Maintenance Due to certain changes, the automated tests

may not run correctly any more and maintenance becomes necessary. The blueprint is developed to keep maintenance to a minimum. The parts possibly affected by changes are: the test data; synchronization, error recovery, checks and initia

lization; application specific modules; the communication layer.

15.2.2 Maintenance In the introduction to this chapter, a list of possible changes was presented.

The impact of those changes on the test suite are listed below. Adding test cases. Only the data store has to be changed. Changed test cases. The data store has to be changed. Sometimes it is n

ecessary to add new test actions to the framework. Different result checks. Only the data store has to be changed if only th

e output prediction is changed. Otherwise checks and or test actions have to be changed or added.

Changed system interface. The communication layer has to be changed. If the signal definition is changed, the test data should also be changed if these test data contain information about the signals.

Changed functionality. This change can have an effect on the test data because new test cases have to be added. The synchronization, error recovery, checks, and initialization can be changed because the system dialog and sequence of functionality can have changed. New functionality means new test actions. The new functionality can also affect the communication layer.

Different signals. There have to be changes in the communication layer. Sometimes it is also necessary to change the test data. Different target platform. This can affect the communication layer.

Different internal technical implementation. If the external interface and functionality are not changed then this can only affect the software implementation which bypasses the external interface.

15.2.3 Testability To make test automation easier it is ess

ential that testability is a quality attribute recognized during the design and implementation of the system.

It is rather time consuming and expensive to built a dedicated communication layer because the behavior of the system is impossible to monitor.

15.3 Implementing test automa15.3 Implementing test automationtion

Test tools are used to enhance the quality, lower the costs of test execution, and/or save time.

The automated test suite must be “repaired” and “upgraded” to make it run once again and cover the changed system behavior that must be tested. In bad cases, the repair costs more than executing the test manually.

The objectives of quality, cost, and time are no longer realized and the overall perception is one of failure.

15.3 Implementing test automa15.3 Implementing test automationtion

Investment in the implementation of these kinds of tools is high and so the complete test set should be executed three to six times at least to achieve pay back.

Most problems can be overcome by using a lifecycle model with proper analyzing activities and decision points (see Figure 15.4). This lifecycle has three consecutive phases: initiation; realization; exploitation.

15.3 Implementing test automa15.3 Implementing test automationtion

15.3 Implementing test automa15.3 Implementing test automationtion

Test automation objectives are quantified, the test objectives are described in detail, and the technical implications of the system under test are investigated.

The output of this process biases the design process. According to the design, the automated test environment is developed.

The end product of this phase will be an implemented test automation environment capable of realizing the test automation objectives.

15.3.1 Initiation The test automation project starts with a feasibility study. The

main objective of this is to get enough information to decide if test automation is the proper thing to do. In order to get the right information, three items are analyzed: test automation objectives; system under test; test organization.

The test automation objectives are the expectations of the organization about test automation.

The objectives are described in terms of quality, cost, and time – there should be a balance between these. High quality generally leads to lower cost and time saving.

The objectives are also the criteria for the determination of the success of the test automation project.

Other objectives include in which development stage should automated testing be implemented, and what should be tested with this automated test environment.

15.3.1 Initiation The system under test is investigated tech

nically to find out how it should be tested and whether additional equipment, such as simulators and signal generators, is necessary.

It is also necessary to know whether the interfaces, internal technical structure, and the functionality of the system are regularly changed during development or consecutive releases.

15.3.1 Initiation It would be rather nice if this format could be used iIt would be rather nice if this format could be used i

n the test automation environment. n the test automation environment. The proper use of test tools demands a repeatable teThe proper use of test tools demands a repeatable te

st process. This means that there are well-described st process. This means that there are well-described test cases derived by using test design techniques atest cases derived by using test design techniques and a test strategy which decides on the relative impnd a test strategy which decides on the relative importance of the system parts and quality attributes.ortance of the system parts and quality attributes.

To use and develop the automated test environmenTo use and develop the automated test environment, special knowledge is necessary. If this knowledge t, special knowledge is necessary. If this knowledge is not available then employees have to be trained.is not available then employees have to be trained.

Another consequence can be that the usage of the aAnother consequence can be that the usage of the automated test environment is made as simple as posutomated test environment is made as simple as possible by developing an intuitive user interface. This sible by developing an intuitive user interface. This will hide the complexity of the automated test envirwill hide the complexity of the automated test environment.onment.

15.3.1 Initiation In addition to all these primary conditions one shou

ld also know if there is a tool available to use for the system itself. This will depend on the system itself, the target, the instantiation of the system, and also the program language.

Figure 15.5 shows the different stages in the test tool selection process.

15.3.1 Initiation The ultimate consequence is the development of a dedicated tool or no auThe ultimate consequence is the development of a dedicated tool or no au

tomated test execution. The following activities have to be carried out.tomated test execution. The following activities have to be carried out. 1. Set goals and define the scope. Goals can include improving the test process,1. Set goals and define the scope. Goals can include improving the test process,

shortening test execution time, diminishing test execution capacity, or enhan shortening test execution time, diminishing test execution capacity, or enhancing quality test execution. Be aware that goals such as solving organizational cing quality test execution. Be aware that goals such as solving organizational problems or attention-free testing can not be realized by tools. The scope shoproblems or attention-free testing can not be realized by tools. The scope shows whether the automated test environment should be used for only one partiws whether the automated test environment should be used for only one particular system, a range of systems, or across the company.cular system, a range of systems, or across the company.

2. Define knockout criteria. These criteria are mostly based on the technical i2. Define knockout criteria. These criteria are mostly based on the technical implications of the system under test. A tool that cannot meet these criteria is umplications of the system under test. A tool that cannot meet these criteria is useless and is immediately deleted from the list.seless and is immediately deleted from the list.

3. Long list and RFI. With the knockout criteria in mind, a long list of tools is p3. Long list and RFI. With the knockout criteria in mind, a long list of tools is prepared. All tool vendors get a Request For Information (RFI). This is a list of qrepared. All tool vendors get a Request For Information (RFI). This is a list of questions a vendor should answer. The accuracy of this answer is also an indicauestions a vendor should answer. The accuracy of this answer is also an indicator of the vendor’s service level – tool vendors that do not answer are discoutor of the vendor’s service level – tool vendors that do not answer are discounted.nted.

4. Detailed criteria. A detailed list of selection criteria is prepared. There are t4. Detailed criteria. A detailed list of selection criteria is prepared. There are three kinds of criteria:hree kinds of criteria:

technical (compatibility, stability, script language, etc.);technical (compatibility, stability, script language, etc.); functional (object recognition, synchronization, report functionality, operating systefunctional (object recognition, synchronization, report functionality, operating syste

m, target platforms, etc.);m, target platforms, etc.); demands to tool vendors (continuity, education, support, installed base, etc.). The crdemands to tool vendors (continuity, education, support, installed base, etc.). The cr

iteria are grouped by knockout criteria, demands, and wishes.iteria are grouped by knockout criteria, demands, and wishes.

15.3.1 Initiation 5. Short list and RFP. Based on the detailed criteria, tools ar

e selected from the long list. The selected tools are placed on a short list which is sent to the vendors with detailed criteria and a Request for Proposal (RFP). This gives an indication of the costs, and the possibility of support and training. Based on this information, two or three tool vendors are invited to show the capabilities of their tools on a demonstration test environment. A list of capabilities each vendor should show with their tool should be prepared.

6. Proof of concept. Before using the tool in practice, and investing much effort in its implementation, a pilot can be started. If this pilot is successful the tool can be implemented. Many tool vendors offer the chance to use the tool on trial for a certain period.

7. Decision. The results of the proof of concept give enough information to decide if an implementation of the tool will meet expectations. The outcome could be that the objectives can only be met with a dedicated developed test tool.

15.3.1 Initiation The benefits can also be quantified in terms oThe benefits can also be quantified in terms o

f saving money by spending less time and labf saving money by spending less time and labor on testing. or on testing.

The enhancement of the quality of the test prThe enhancement of the quality of the test process is the subjective part of the benefits. Be ocess is the subjective part of the benefits. Be aware that the direct cost of tools is high but taware that the direct cost of tools is high but the cost of labor during implementation, usaghe cost of labor during implementation, usage, and maintenance are usually much higher.e, and maintenance are usually much higher.

This whole process should aid the enhancemThis whole process should aid the enhancement of awareness and positive support for iment of awareness and positive support for implemention as a major side effect.plemention as a major side effect.

15.3.2 Implementation The first is the preparation of a plan of

action – the purpose being to show the activities, people involved, and time estimation. Then the design and development will start. In this process, six activities are identified (see Figure 15.6).

15.3.2 Implementation Based on this design, the engineer develops the autBased on this design, the engineer develops the aut

omated test environment and is responsible for impomated test environment and is responsible for implementing lementing the necessary functionality to execute the delivered test cases. The functional testers develop the test cases based on test design techniques. In the last step, a final test of the automated test environment is performed.

Now, the design and all other documentation about the automated test environment is brought up to date. Together with the automated test environment, this is deployed and used in the maintenance phase.

In additional to these major activities, parallel activities can be executed as necessary.

15.3.3 Exploitation The automated test environment is

now ready for use. The impact on the test process of the availability of an automated test environment is shown in Figure 15.7.

15.3.3 Exploitation During specification, the test cases should be During specification, the test cases should be

described in a readable format for the automadescribed in a readable format for the automated test environment, and the environment shted test environment, and the environment should be adjusted due to changes in the test objould be adjusted due to changes in the test object. ect.

The test execution is partly covered by the autThe test execution is partly covered by the automated test environment and in the completioomated test environment and in the completion phase all changes are frozen and stored. It is n phase all changes are frozen and stored. It is essential that a complete copy of the automateessential that a complete copy of the automated test environment is subject to configuration d test environment is subject to configuration management.management.

15.3.3 Exploitation The major objective of the exploitation phase is to

maintain and use the automated test environment. During this phase, the organization can profit from

the benefits of the automated test environment. To meet the test automation objectives and to be

able to benefit from this environment, in the next release the focus must be on maintenance of the environment. By neglecting this part of the job the environment will be useless after just one or two releases – and that is probably before the investment is paid back.