A SmartBear White Paper

Selecting the right automated software testing tool involves having a list of requirements to review during your evaluation. Without a list of requirements, you may waste time downloading, installing and evaluating tools that only meet some of your requirements. This White Paper outlines common requirements for automated testing tools which you can build from to create a list that reflects the specifics of your software testing needs.

Selecting Automated Testing Tools

Contents

Introduction ........................................................................................... 2

Support for Various Applications and Platforms ..................... 2

Support for Various Operating Systems ...................................... 3

Support for Various Testing Types ................................................. 3

Creating Automated Tests Without Programming ................ 4

Automated Test Scripting................................................................. 5

Recording Automated Software Tests ......................................... 5

Creating Automated Tests that are Resistant to Changes ............................................................................................. 6

Support for Controls and Objects .................................................. 6

Support for Various Test Data Sources ........................................ 7

Running Automated Software Tests and Synchronization .....................................................................................7

Automatically Compare Results of Automated Test Actions .............................................................................................8

Ensuring Software SuccessSM

www.smartbear.com/testcomplete

Logging Automated Test Results ..............................................8

Exporting Reports ...........................................................................8

Additional Needed Automated Testing Tool Features .....9

Testing Multi-Language Applications .....................................9

Automated Testing Tool: Support for Team Work .............9

Command Line and OLE Automation Support ................ 10

Integration with Team System and Build Software ....... 10

Technical Support ........................................................................ 10

Pricing Policy .................................................................................. 10

Free Trial Version .......................................................................... 11

Introduction

Testing is a critical part of the software development process. There are a lot of different automated software

testing tools currently on the market. Some of these tools are only able to perform specific kinds of testing and

only work with specific languages, like, for example, Java application unit testing. Other products support a wide

range of applications and offer more features and functionality.

When you start your search for the right automated software testing tool, it’s important to create a list of

requirements to review when choosing a tool for evaluation. If you don’t have a list of requirements, you may

waste time downloading, installing and evaluating tools that only meet some of your requirements, or may not

meet any of them.

Creating a list of requirements is not easy, since you have to take all of your testing needs into account. We have

prepared a list of requirements, which are described below. Of course, the list does not include every situation,

but this list is a good start. You can modify and add to it to get a list that reflects the specifics of your software

testing environment.

Support for Various Applications and Platforms

Some automated testing tools only support one type of application, like, for instance, only Java or .NET ap-

plications, while other tools support more application types. In general, the best option is to choose a tool that

supports all development tools currently used in your organization, or all tools that you plan on using in the near

future. Even if you build your application with one compiler, you might use other compilers in the future.

Selecting one “multi-compiler” software testing tool instead of several “compiler-specific” tools may save you

money, but you must decide which is more important and whether it will save you time and energy when creat-

ing tests for future application versions. Purchasing and using several software testing tools requires additional

time for training and creating tests. Do you have that much time? Does your company have enough budgeted

to pay for this?

TestComplete is one of the automated software testing tools that support the testing of any Windows program

created with any compiler. It works with both 32-bit and 64-bit Windows, .NET, Java and web applications. It

can also test applications running on Windows Mobile-based PDAs.

TestComplete’s automated test engine provides extended support for the following applications:

¿ C#, Visual Basic .NET, VCL .NET and any other .NET application for any version of .NET Framework (1.0, 1.1, 2.0, 3.0, 3.5 or 4.0)

¿ WPF (XAML) and XBAP applications (.NET Framework 4)

¿ Java applications (including Swing, SWT, AWT and WFC applications)

¿ Visual C++ 6.0-9.0 applications (including ATL, MFC and WTL applications)

¿ Visual Basic 6.0

¿ Delphi and C++Builder applications (both VCL and CLX applications)

2

¿ Flex, Flash, Silverlight and JavaFX applications

¿ Windows Mobile applications

¿ Qt applications

¿ wxWidgets applications

This extended support means that TestComplete can access the internal objects, methods and properties of the

application under test. QA engineers can use all of these elements in their automated tests. For instance, your

tests can execute an object’s internal methods to initiate a unit test.

TestComplete also provides access to methods and properties of web page elements displayed in Internet Ex-

plorer, Firefox, Netscape Navigator or any browser based on the Microsoft Web Browser control. It can test web

pages on the client computer and is not dependant on the server technology. You can use it to test web pages

generated by IIS, ASP, ASP.NET, PHP, CGI, AJAX, Atlas and other applications.

Support for Various Operating Systems

Choose an automated testing tool that supports as many operating systems as possible. Make sure that the

tool supports recent operating system versions. The fact that your customers do not run your application under

Windows Vista does not mean that they will not use it in the future. If your application works under this operat-

ing system, your software testing tool should be able to work under it as well. In other words, the more Windows

versions the software testing tool supports, the better.

TestComplete supports both old and new Windows operating systems: Windows 2000, Windows XP, Windows

Vista, Windows 7, Windows Server 2003 and Windows Server 2008, including both 32 and 64-bit versions of

these operating systems. TestComplete also provides automated testing support for Windows Mobile applica-

tions running on PDAs. The range of supported operating systems on PDAs includes Windows Mobile 2003

(Pocket PC 2003), Windows Mobile 2003 SE (Pocket PC 2003 SE), Windows Mobile 5.0 and Windows Mobile 6.0.

Support for Various Software Testing Types

When choosing an automated testing tool, check whether it supports the test types that you are going to run.

Can it simulate user actions over the tested application’s user interface, in other words, perform functional

testing? Can it run unit tests built into the tested application? Can it run tests created with NUnit, JUnit, DUnit

or MSTest? Does it support distributed testing? Does it have special features for synchronizing tests that run

simultaneously on several network workstations? Does it support load testing?

Also, if you test your products using both manual and automated methods, you may want to choose a tool that

provides specific support for manual tests. Also, don’t forget other important factors, such as, data-driven testing

support and the ability to use the tool for regression testing. There are a lot of automated software testing tools

that support only one or two testing types, for instance, only unit testing or functional testing. Some vendors

require separate products for different testing types. For instance, support for load or distributed testing can be

implemented with separate products or add-ons. If you wish to support these testing types, you will have to add

the price of these products or add-ons to the price that you are going to pay for the product.

3

TestComplete supports unit, functional (or UI), regression, data-driven, manual and distributed testing of Win-

dows applications. It provides special support for unit tests created in NUnit, JUnit, DUnit, MSTest, which allows

you to run these tests directly from your TestComplete projects. This lets you use unit tests that were created

with these tools earlier in your development cycle. With TestComplete you can also perform load, stress and scal-

ability testing of Web servers and services. To perform coverage testing, check the application for resource and

memory usage and you can find bottlenecks in your application by using another one of SmartBear’s products

called AQtime. AQtime is tightly integrated with TestComplete, so you can run coverage, performance and

memory leak tests directly from your TestComplete projects.

Creating Automated Tests Without Programming

An automated testing tool must let you create tests fast and effectively. The test creation techniques must be

sufficiently powerful to allow performing various automated testing tasks. At the same time the automated

testing tool must be easy-to-use and clear even for inexperienced QA personnel. A good automated testing tool

may support several ways for creating automated tests. For instance, it may provide a simple, easy-to-use visual

automated test editor and some other features, like automated test scripts, for performing advanced tasks.

Many automated testing tools only provide scripting capabilities for creating automated tests. Creating auto-

mated test scripts require programming experience and may be difficult for inexperienced testers. TestComplete

provides a simple but powerful alternative to automated test scripts: keyword tests (also called keyword-driven

testing).

Keyword tests consist of operations that simulate user activities on the application under test (like clicking the

mouse buttons or typing on the keyboard) or perform other automated testing actions: launching tested ap-

plications, posting messages to the test log, running other automated tests (for instance, low-level procedures or

unit tests), use checkpoints to verify specific application functionality and more.

Keyword tests are very simple and easy to create. Testers do not need programming skills to create them. You

can record these keyword-driven tests or modify them visually in a special editor. The editor’s intuitive UI, wiz-

ards and dialogs help even inexperienced users create powerful automated tests with ease. Keyword tests are an

ideal feature for creating functional tests quickly.

But keyword tests are not the only automated tests that can be recorded or created visually. TestComplete can

record automated test scripts, load tests and low-level procedures (special procedures that simulate low-level

events like pressing and releasing of mouse buttons or keyboard keys). Load tests and low-level procedures can

then be modified visually with special editors. Recorded automated test scripts can also be modified in the code

editor.

4

Automated Test Scripting

Many automated software testing tools also provide scripting capabilities. Scripts are very important when per-

forming automated testing and they let users create automated tests for specific application features, provide a

variety of automated testing actions and perform those operations that cannot be done with standard features

built into the automated testing tool. Automated test scripts are useful even if the automated testing tool pro-

vides easier ways to create automated tests, because script code typically provides more powerful functionality,

it lets you perform those actions that other automated testing types cannot.

If you prefer scripting for your automated testing, it is important to pay attention to the scripting language the

automated testing tool uses. Using standard languages, such as VBScript or JScript rather than their dialects,

will decrease the time needed to get acquainted with a product. Another benefit is that you can find information

on these languages from third-party web sites as well as MSDN.

Two other important features to think about when choosing a tool that uses scripting is the code editor and

script debugger. Check whether the editor supports features that simplify and speed up script writing: syntax

highlighting, outlining, code completion, bookmarks, Ctrl-click navigation between script routines and many

others. Check if the tool’s script debugger includes the needed debugging functionalities, such as breakpoints,

step-over and step-into execution, Call Stack window and Evaluate dialog, etc. Using all of these features you

can easily find and correct bugs or find workarounds.

TestComplete supports several scripting languages: VBScript, JScript, DelphiScript, C++Script and C#Script, so,

you can choose the language you know best and thus save more time.

TestComplete includes a powerful code editor that supports all modern code editor features: syntax highlighting,

outlining, code completion, code templates, bookmarks and many others. TestComplete’s built-in script debug-

ger supports ordinary and conditional breakpoints, step-over, step-into and run-to-the-cursor features, the Watch

and Call Stack windows and special features for evaluating variables and objects during the script run.

Recording Automated Software Tests

Functional (or user interface) testing implies simulating user actions on the application under test. Implement-

ing these tests even for a small application is a tiresome process. When choosing an automated testing tool,

make sure the tool can record and play back user actions on the tested application. This feature will significantly

streamline test development.

TestComplete records user actions as keyword tests or automated test scripts, which you can then edit and play

back an unlimited number of times. Fast test development is not the only benefit. Recording also helps you

understand how to create a keyword test or how to write automated test script code, how to address application

objects and how to perform testing actions on them. We’d like to note that TestComplete not only records and

plays back tests; it also records and plays back HTTP traffic. This feature lets you easily create load tests for web

servers and services. TestComplete can also filter out traffic sent and received by specific applications during

traffic recording.

5

6

Creating Automated Tests that are Resistant to Changes

Normally, you start testing your application while it is still in development. So, it is quite possible that after you

make changes to the application, the current tests will become obsolete and you’ll have to update the test to

work with the changed application windows and controls. Modifying the tests may take the same amount of

time as modifying the application itself. That’s why smart teams like to create flexible tests that will be resistant

to the changes made to the application under test.

One approach that decreases the dependence on changes made to the tested application is the object-oriented

testing model. Each window or control of the application under test is represented by an appropriate object with

the name, for instance, [WinFormsObject(“Form1”)], so, to simulate user actions on this window or control, you

call the object’s methods and properties. For instance, simulating a mouse click on a button can be represented

as a call to the Click method of the test object that corresponds to this button. If you use this approach, there is

no need to change your test after the tested application has changed. For instance, if developers change a but-

ton’s width, height or position, the Click method will still work. If you simulate a click by coordinates, changing

the button’s size or position would cause changes in the automated test. Nevertheless, in some cases, you may

need to work with coordinates as well, so, it is nice if the automated testing tool supports both object-oriented

and coordinate-oriented methods.

Another approach that lets you create robust tests is object mapping. As a rule, automated software test-

ing tools address applications’ objects by their names or captions. However, during application development,

programmers may change controls’ captions, so you will have to modify your tests in order for them to match

the changed controls. That is why a good automated testing tool should be able to map an object name to a

custom name and use this custom name to address the object in tests. However, the tool must be able to map

an object not only by its caption or identifier. In this case, if a developer changes the tested application, you will

only have to modify the mapping conditions while the scripts and other tests remain the same.

TestComplete includes special features that let you map application windows, controls and objects by their prop-

erties. Testers can map a short name to the tested object and then use this mapped name in automated tests.

This simplifies working with objects, makes the created tests shorter and easier to understand. There is another

benefit of name mapping - it makes automated tests independent of object properties, which may change from

one build to another. Also, TestComplete supports the object-oriented approach to simulate user actions. Simu-

lating mouse events by coordinates is also supported.

Support for Controls and Objects

Windows applications use controls to display information and interact with users. The operating system provides

dozens of various controls that can be used in tested applications: edit boxes, list views, check boxes, menus,

buttons and much more. This number is also extended with custom components offered by third-party vendors.

These can be non-standard components, like a tree list or grid, or third-party analogs to standard components:

custom menus, toolbars and many others. When choosing an automated testing tool, make sure it includes spe-

cial features that simplify the work with controls used in your application. It is important to determine whether

7

these features are for both standard and third-party controls.

Another important factor is access to methods and properties of the application’s internal objects and controls.

This access provides methods and properties for simulating user actions and checking the results of test actions.

For example, you can use the internal methods and properties of a grid control to change the value in a grid’s

cell and then check if the change was successful.

TestComplete provides specific methods and properties that greatly simplify your work with standard Win32

controls, web page controls, WPF controls and third-party components provided by Microsoft, Borland, Com-

ponentOne, Developer Express, Infragistics, Syncfusion, Telerik and other vendors. In addition, TestComplete’s

automated test engine can access the internal methods and properties of application objects and use these

methods and properties to perform the desired testing tasks.

Supports Various Test Data Sources

When performing functional testing, it’s a common practice to simulate user input into the tested application’s

data forms. Thorough testing implies that you feed varied data to the input fields and that the tested applica-

tion processes it correctly. When choosing an automated testing tool, check which data formats it can use, such

as text files, XML files, database tables, and others.

With TestComplete, you can obtain data from text and XML files, Excel worksheets and databases like SQL

Server, Oracle and MySQL. TestComplete includes special program objects and test operations that allow you to

iterate through data records in external files and easily create data loops in your automated tests.

You can create and populate these data storages manually, or you can use TestComplete’s Data Generator to

generate the desired test data automatically and save it either to TestComplete’s Table variable or to the speci-

fied Excel file. This approach helps you decrease the time spent on preparing test data. For more information on

using this feature, see the Using Data Generators section in TestComplete’s help.

TestComplete also allows you to use script arrays, variables and constants to keep data for automated tests.

Running Automated Software Tests and Synchronization

The state of the application under test and the execution environment may differ from the state and conditions

you expected. For instance, some network resource may be unavailable, or the desired window is not created; the

web page loads very slowly or an unexpected modal dialog box appears. These and other similar factors may

cause the automated software testing tool to decide that the test has failed and it should be stopped. A good

automated testing tool must be able to handle these situations in the appropriate manner. For instance, it must

be able to “wait” for the needed windows, “close” unexpected dialogs automatically and perform other actions

that a human would do to continue the test.

TestComplete can handle these situations properly. For example, methods that work with windows and web

pages can wait until a window appears on screen or until a web page is loaded; methods that work with process-

es can wait until the desired process starts and so on. TestComplete also includes special features that let you

8

specify how the tool should handle unexpected windows without interrupting the test run.

Automatically Compare Results of Automated Test Actions

An automated testing tool should include features for automatically comparing the results of test actions. For

instance, if during the test run the tested application saves a customer list to a file, the automated testing tool

should be able to compare this file with a baseline copy and determine whether the save was successful and the

saved data is correct. The automated testing tool should be able to compare objects’ and controls’ properties,

images, files and other objects.

TestComplete includes special tools, objects and functionality that you can use to compare files, images and

object properties. The comparison can be implemented manually, or generated by TestComplete when you are

inserting a checkpoint in your test. TestComplete includes special dialogs and wizards that can be used to create

checkpoints for properties, objects, files, images, databases, tabular controls, XML documents, web services and

web pages. By using the powerful test implementation tools, you can create automated tests that will compare

complex objects and can also use regular expressions for comparisons.

Logging Automated Test Results

An automated software testing tool should keep a log of all testing actions and provide a detailed report for

them. The log of a high-end software testing tool must contain different types of messages: errors, warnings and

informative messages. In this case, you can easily determine what went wrong and why the test failed.

TestComplete includes a powerful test log that contains warning, error or informative messages as well as files

and images. The images are captured by the Test Visualizer subsystem and are automatically posted to the log

during the test run, or they can be posted using special operations or scripting methods. The captured images

illustrate test commands and simplify the analysis of the automated test results.

Exporting Reports

An automated testing tool should give users the opportunity to export test results to external files. For instance,

it may be useful, when you need to view test results on other computers that do not have this testing tool

installed. Also, you should check whether the testing tool can export the test log so that you can send the results

to your boss or colleagues.

TestComplete stores test results in XML files and allows you to easily export the test log data to several formats

(XML+HTML, Excel, MHT), pack the results with WinZip, WinRar or some other archive tool and send the results

via e-mail. You can perform these operations from the TestComplete UI or from tests.

Also, TestComplete provides access to the stored results. Since the test log uses the XML format, you can parse

the results and retrieve the desired information.

TestComplete also allows you to create a bug report for test log data and post the report to an issue-tracking

system.

9

Additional Needed Automated Testing Tool Features

To create powerful software tests, the test recording and playback features may not be enough. Choose a tool

that provides many additional features. For instance, you may need the following:

¿ A library of useful objects and functions to perform various operations. For example, reading data from and writing it to files, working with databases, working with date and time values, regular expressions and many others.

¿ The ability to call Win32 API functions.

¿ The ability to call functions that reside in DLLs and .NET assemblies.

¿ The ability to call routines that reside in Java classes.

¿ The ability to work with COM servers.

¿ Access to the shell (the WSH scripting object) and the ability to call the operating system commands.

¿ OCR functionality to retrieve data from applications’ windows and controls.

¿ The ability to create custom add-ons that will add specific functionality to your software test projects.

TestComplete includes all of these features and offers many others.

Testing Multi-Language Applications

If you have customers throughout the world and your company develops or is planning to develop localized ver-

sions of your applications, you have to check whether the software testing tool can work with localized applica-

tions and operating systems and whether it supports Unicode. If the tool does not support foreign languages,

then, most likely, you will not be able to use it to test localized software.

TestComplete supports Unicode and, therefore, it is possible to test multi-language software with it.

Automated Testing Tool: Support for Team Work

Testers are members of the development team. They work together and communicate with developers, manag-

ers and other various people. When choosing an automated testing tool, you may be interested in features that

support team work such as concurrent work with project files, sharing script code among several projects and

integration with source code control and issue-tracking systems.

TestComplete supports all of these features. It allows you to share your test projects with several testers. You

can easily link script units from one project to another. You can check project files in and check them out from a

source control system directly from TestComplete. Finally, you can add a bug report to an issue-tracking system

like Microsoft Visual Studio Team Foundation, Bugzilla, OnTime or JIRA, directly from the test log viewer, and

thus notify developers about the problem immediately.

10

Command Line and OLE Automation Support

Choose an automated testing tool that you can launch automatically, for instance, at a certain time or after the

application build is over. You will be able to do this only if your automated testing tool has the command line or

OLE interface where you can specify run parameters.

TestComplete supports command line arguments and can work with COM. Other applications, for instance,

SmartBear’s Automated Build Studio, use these features to call TestComplete automatically.

Integration with Team System and Build Software

Automated software testing tools are used along with build programs. For instance, the application “builder” can

run tests at the end of each build. When choosing an automated testing tool, check whether it can be used with

build programs.

TestComplete can be integrated with build software, for example, with SmartBear’s Automated Build Studio and

with Microsoft Visual Studio, including version 2010 of this popular Microsoft IDE. Also, TestComplete projects

can be called from Visual Studio test projects. TestComplete also includes a special assembly that you can use to

run TestComplete’s software tests from MSBuild projects.

Technical Support

A very important aspect of selecting an automated software testing tool is the quality of the technical support

provided. Prompt responses to your messages prepared by a professional support team will assist in resolving

problems and will save you time and energy.

Pricing Policy

When choosing an automated software testing tool, you have to determine whether its price is reasonable.

You should take into account not only the product price, but also the price of add-ons (if they are needed), the

support fee as well as training and upgrade prices. To get this information, you may need to call the company’s

office. Some vendors have special prices that you can only learn about when talking to a representative from

their sales department over the phone. SmartBear offers a clear pricing policy. You can find TestComplete prices

on our web site: TestComplete Pricing.

As you can see, SmartBear offers a feature-rich product at an affordable price. As a result, not only

large companies use TestComplete in their everyday work, but also small teams can afford it. Also, you can save

money by purchasing TestExecute licenses rather than TestComplete licenses for all of the computers that you

are going to run software tests on, but that are not for creating and debugging software tests.

Training is a paid option. For more information on this, see our TestComplete Training Web page.

Free Trial VersionAn important factor is the availability of a trial version. Even if the product seems to meet your requirements, the

best way to determine this is to download and use it before purchasing. Since you are new to the product, you

may get questions and need some assistance with performing tasks. So, another important factor is whether the

company provides support for trial-version users.

SmartBear provides support for trial-version users. The trial version is a full-featured version, but it has a 30-day

limitation and there are limitations on the size of your scripts.

You may also enjoy these other resources in the SmartBear Software Quality Series:

¿ 11 Best Practices for Peer Code Review

¿ Uniting Your Automated and Manual Test Efforts

¿ 6 Tips to Get Started with Automated Testing

Be Smart and join our growing community of over 100,000 development, QA and IT professionals in 90

countries.

Try TestComplete Free for 30 Days

Scan and download your free trial to see why

testers choose SmartBear for automated testing.

11

SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915 +1 978.236.7900 www.smartbear.com ©2012 by SmartBear Software Inc.Specifications subject to change. TC-WP-SATT-042012-WEB

About SmartBear Software

SmartBear Software provides tools for over one mil-

lion software professionals to build, test, and monitor

some of the best software applications and websites

anywhere – on the desktop, mobile and in the cloud.

Our users can be found worldwide, in small businesses,

Fortune 100 companies, and government agencies.

Learn more about the SmartBear Quality Anywhere

Platform, our award-winning tools, or join our active

user community at www.smartbear.com, on Facebook,

or follow us on Twitter @smartbear.