-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
1/44
......
MAGAZIN
OCT/NOV 2012$8.95
How To Make theMost of Model-based Testing
PRACTICAL TIPS AND CASESIN MODEL-BASED TEST
IMPLEMENTATION
CROSSING OVER:
Implement Cross-browser Testing in 4 Simple Steps
Job RotationINVOLVING USERS IN
AUTOMATOR TASKS ANDVICE VERSA
Randomness& Rotations
Randomness& Rotations
SOFTWARETESTINGAUTOMATEDAUTOMATED
AnAUTOMATEDTESTINGINSTITUTEPublication - www.automatedtestinginstitute.com
Random TestsSENDING RANDOM INPUTSTO THE SYSTEM UNDERTEST
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
2/44
Free Demo
& Trialwww.gotosca.com/testkit
TOSCA puts the "A" in Agile
AGILE processes are strictly risk-oriented
TOSCA risk-based approach
is the ideal complement
AGILE requires early test automation
TOSCA OneView (UI/Non-UI)
is the perfect solution
AGILE is based on specification by example
TOSCA helps provide the right test cases
Ask us how TOSCA can empower youto increase your test automationcoverage up to 85% while minimizingyour maintenance effort up to 80%.
One Solution For GUI, non-GUI, Manual,
Automated, X-Browser and Mobile.
Passion for Testing
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
3/44
AUTOMATEDSOFTWARETESTINGOct/Nov 2012, Volume 4, Issue 3
Automated Software Testing Magazinewww.automatedtestinginstitute.comOct/Nov 2012
CONTENTS
COLUMNS& DEPARTMENTSEDITORIAL 4
AUTHORS AND EVENTS 6 MOBILE HOSTING & AUTOMATION
SOLUTIONS ARE HOT! 41
I BLOG TO U 36Randomize and Rotate
A Host of Choices in Mobile
Meeting the challenges posed by system dynamics withsome dynamics of our own.
A look at commercial mobile hosting and automatiosolutions.
Read featured blog posts from the web.
Learn about AST authors and upcoming events.
OPEN SOURCERY 12Randys Favorite Free and Cheap Tools
An update to a previous article on free and cheap toolswith information shared at the TestKIT 2012 Conference.
GO ON A RETWEET 38Read featured microblog posts from the web.
RANDOMNESS& ROTATIONSGiven the highly dynamic nature of software test applications, its imperative that our approaches not become to
static. This issue focuses on improving test automation implementation and application quality through increase
data and tests randomness and job rotations.
FEATURESRANDOM TESTS 14Random Test is a special test type that has been proven to be a very good complement to ordinary functional tests. Read this article to learn ho
this technique may prove useful, particularly for systems with high criticality such as marketplace systems. By Lars Wahlberg
HOW TO MAKE THE MOST OF MODEL-BASED TESTING 20Model-based testing (MBT) has the potential to resolve many of the challenges of test automation including maintainability, cost contr
and keeping pace with development. Read this article to obtain a workable MBT workow. By Stefan Mohacsi and Armin Be
TESTKIT TIP 8Crossing OverBy Goran Begic
Implement cross-browser testing in 4 simple steps.
JOB ROTATION: THE ANSWER TO QUALITY AUTOMATED TESTING? 30
Automated testing could benet greatly from more involvement by the front-end user if we could gure out how to get around the facthat the front-end user is often not very technical. This article provides a possible way around this hurdle. By Ursula Friede
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
4/444 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
Often test automators push automated test
scripts into combat with the assumption
that the application is simply going to
let the script win. We assume that data
never gets corrupted, scripts never need
to be maintained, unexpected events never
happen, and each test objective has only a
single path. For this reason, scripts often fail
easily, require excessive maintenance, and
repeatedly overlook simple defects.
Application behavior is dynamic, so its
time that we train our automated scripts to be
more dynamic via the techniques provided inthe automation martial arts known simply as
Taekwondo-mation! Taekwondo-mation
comprises relatively simple techniques that
can be employed without the implementation
of an overly complex framework.
This began a four-part series of articles
I wrote a few years ago entitled
Taekwondo-mation - Training Your
Scripts for Dynamic Combat. A
few months ago, I met a guy that was
familiar with some of my writings, and
specically expressed his partiality formy Taekwondo-mation series. He also
indicated how much hed like for me
to continue the series with some new
articles. Aside from being attered that
some of my writings have made a lasting
impact, I was also intrigued by this
impact. It made me begin to question
why, out of all of the works that Ive
done, did the gentleman specically
bring up Taekwondo-mation? The more
I thought about it, the more I began to
come to the conclusion that it has todo with the fact that it addressed the
dynamic nature of software. And since
software has only gotten moredynamic
since Taekwondo-mation, and promises
to continue in that direction, the articles
are in a sense, timeless. Change is the
only constant when it comes to software,
so people are constantly looking for
information that will help them to deal
effectively with it. So, I will denitely
consider writing some more of my own
articles in the future that present more
aspects of the test automation martial
arts, but in the meantime, this issue of
the AST Magazine has enlisted some
other automation martial artists to
present their spin on effectively handling
software and automation dynamics.
The rst feature entitled Random
Tests by Lars Wahlberg describes how
one team introduced a new automation
approach, whose name is equivalent
to the fedatures title, in an effort to
supplement testing in a marketplace
systems environment where even
thousands of functional tests werent
sufcient for validating the application.
Following behind that feature is an even
more in-depth look at a similar approach
known as model-based testing (MBT),
which is a type of automated testing
approach that dynamically generates
and executes tests in real-time. This
feature article, entitled How to Make
the Most of Model-based Testing,is coauthored by Stefan Mohacsi and
Armin Beer, and presents a workow
for MBT that considers the traceability
between requirements, the test model,
the GUI, and the test cases. Finally,
Ursula Friede presents an article
entitled Job Rotation: The answer
to quality automated testing? In this
article, Friede provides an approach for
improving quality and cross-training
resources in a cost effective manner.
Randomize and Rotateby Dion Johnson
Editorial
It made me begin to question why, out
of all of the works that Ive done, did
the gentleman specically bring up
Taekwondo-mation?
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
5/44
Celebrating Excellence in the
Discipline of Software Test
Automation
Winners Now Announced!
www.atihonors.automatedtestinginstitute.com
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
6/446 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
Managing Editor
Dion Johnson
Contributing Editors
Donna VanceEdward Torrie
Director of Marketing and Events
Christine Johnson
A PUBLICATION OF THE AUTOMATED
TESTING INSTITUTE
CONTACT US
AST Magazine
ATI Online Reference
Stefan Mohacsi studied computerscience at the Technical University of Vienna.
In 1997, he joined Siemens and became project
manager of the research project IDATG. Today,
Stefan is senior consultant for model-based
testing at Atos. In addition, he is a member of the
Austrian Testing Board and has held numerous
lectures at international test conferences. Stefans
email address is [email protected].
Whos In This Issue?
Goran Begic is a senior productmarketing manager at SmartBear Software. He
brings 13+ years of professional experience
with various code and design verication tools
ranging from manual testing and test automation
to static analysis and formal verication. Goran
has extensive experience with integratingdevelopment and testing tools into end-user
processes including agile and Model-Based Design. He is an avid customer
advocate and a proponent of visual thinking. Goran holds a masters degree
in electrical engineering from the University of Zagreb, Croatia. Connect
with him via Twitter @gbegicw.
Authors and Events
AUTOMATEDSOFTWARETESTING
ATI and Partner Events
First Quarter 2013
TestKIT ONDEMANDVirtual Conferencehttp://testkitconference.com
November, 2012
AST Magazine Special Editionhttp://astmagazine.automatedtestinginstitute.com/
Ursula Friede has 10 years ofexperience testing software. She worked as
a performance specialist and automated test
analyst, but is currently on maternity leave.
Friede has contributed a chapter to the new test
automation book by Mark Fewster and DorothyGraham and she has a publication about Case
based reasoning in a German IT book.
The Automated Software Testing (AST) Magazine is an
Automated Testing Institute (ATI) publication.
For more information regarding the magazine visit
http://www.astmagazine.automatedtestinginstitute.com
Armin Beer has been working in thearea of test management and test automation for
about 20 years. He is currently an independent
consultant for the test management group at
the social insurance institution BVA. He is alsoparticipating in the research project SoftNet of the
Technical University of Graz and lecturing at the
University of Applied Sciences in Vienna and the
Technical University of Graz. Armins email address is [email protected].
Lars Wahlberghas been working withtesting of marketplace systems for about 10 years
in roles such as test engineer, test manager/leader,
and test strategist. Before that, during the 1990s,
he worked with testing of ight-control systems
for military aircrafts and has a MSc degree in
aerospace. He also teaches ISTQB test foundation
courses in Stockholm with Hans Schaefer.
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
7/44www.automatedtestinginstitute.comOct/Nov 2012 Automated Software Testing Magazine
Are You ContributingContent Yet?
The Automated Testing Institute relies heavily on the automated testing
community in order to deliver up-to-date and relevant content. Thats why
weve made it even easier for you to contribute content directly to the ATI
Online Reference! Register and let your voice be heard today!
CommunityComments Box
As a registered user you can submit content directly to the site, providing you w
content control and the ability to network with like minded individuals.
>>COMMUNITYCOMMENTSBOX- This comments box, available on the home pagethe site, provides an opportunity for users to post micro comments in real time.
>>ANNOUNCEMENTS& BLOGPOSTS- If you have interesting tool announcementsyou have a concept that youd like to blog about, submit a post directly to the A
Online Reference today. At ATI, you have a community of individuals who would lo
to hear what you have to say. Your site prole will include a list of your submit
articles.
>> AUTOMATION EVENTS - Do you know about a cool automated testing meetwebinar or conference? Let the rest of us know about it by posting it on the ATI s
Add the date, time and venue so people will know where to go and when to be the
Announcements &
Blog Posts
Automation
Events Learn more today at http//www.about.automatedtestinginstitute.com
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
8/448 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
Crossing OverImplement Cross-browser Testing in 4 Simple Steps
TestKIT Tip
Differences in web browsers may cause an application that
behaves correctly in one browser to behave incorrectly in a
different browser or browser version. The structures of different
vendors browsers are different as are their user interfaces and
inner object hierarchies often even for different versions of
the same browser. Application developers must therefore take
these differences into account and so too must the tests that
verify web applications. Otherwise a test recorded in a version
of Internet Explorer will most likely not work right out of the
box in a version of Mozilla Firefox.
These differences are more than skin deep. Tests that only
generate a series of screen-shots of applications running
in various browsers and operating systems can only verify
that everything displays correctly. They cant verify how
the program logic itself behaves or reacts to user actions
in different browsers. Testing the logic means testing an
applications various objects, methods and properties, either
directly (i.e., with conditional logic), with generic object
mapping, or with a combination.
Testing how program logic performs in multiple browsers
requires four steps, each of which corresponds to one of the
reasons the same application might behave differently in a
different browser or browser version. If we assume that thetest knows how to get a list of installed browsers from the
Windows registry and launch them programmatically, these
four steps are:
1. Test non-application-related differences
2. Test application-related differences (GUI)
3. Test application-related differences (non-GUI)
4. Test use case differences
In these examples we will use JScript, although the same
principles apply using any object oriented scripting languageor (as covered in step 4), a keyword based testing tool.
STEP1: TESTNON-APPLICATION-RELATEDDIFFERENCES
Some browser actions are unrelated to how a web application
is constructed. If the test encounters one of these actions
and has not taken that possibility into account, the test will
probably crash or report erroneous results. An example of
this type of behavior is Internet Explorers check for active
content. If a page contains active content the Internet Explorer
browsers may display a notication message that asks the user
TestKIT is the name used by ATIfor describing ones testing toolkit.A TestKIT is lled with knowledge,information and tools that go withus wherever we go, allowing ourprojects and organizations toquickly reap the benets of thepractical elements that weveamassed. This section provides tipsto add to your TestKIT.
By Goran Begic
Figure 1: StockTrader Website
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
9/44www.automatedtestinginstitute.comOct/Nov 2012 Automated Software Testing Magazine
if the content is permitted, whereas the Mozilla browsers do
not notify the user in the same situation.
Another example is the update modal dialog that appears when
Firefox launches if it detects that a software update to the
browser is available for download and installation. This dialog
does not appear in either Internet Explorer or Chrome. The test
must rst determine which browsers methods and propertiesare currently available. For example, if the possibilities are
just a version of IE or Firefox, you can simply test for Firefox
since anything else will be IE:
Depending on your tools, you can either use browser-specic
names for objects or else use wrappers to provide commonnaming to access equivalent objects that have different names
in different browsers. In general, conditional logic is a lot
easier to code, and that code tends to be much more reliable,
if you use generic (i.e., non-browser-specic) methods and
properties. Thats why TestComplete provides a unied object
model (used here) with generic objects mapped to browser-
specic objects.
In the code above, aqStringis an object, whose method, Find,
looks for occurrences of specied strings. It works with the
Page wrapper object that has methods and properties that
are common for all on-screen objects and methods across
browsers. It adds a number of specic methods and properties
(e.g., browser.FindChild, modalDlg) that let you navigate
through web pages in a browser, search for elements of a web
page and perform other operations.
The isMozillafunction enables you to build another function,
PrepareForBrowser , that tests (in the case of Firefox only)whether the update notice is displayed:
If this is Firefox and the value Firefox Update is found, the
test re-launches the browser (which will install the update
automatically) and the test can then continue.
TestKIT T
Figure 2: Discussion Forum Page
Because different
browsers have
different GUIs, the
way they interact
with users will be
different, even to
accomplish the
same task.
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
10/4410 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
STEP2: TESTAPPLICATION-RELATEDDIFFERENCES(GUI)
Because different browsers have different GUIs, the way they
interact with users will be different, even to accomplish the
same task. For example, the location and program hierarchy of
the Addressedit box is different for each browser. So opening
a page by entering the desired URL via the user interface isGUI-specic and requires separate instructions for each GUI
each of which would need to be tested. But, again, by using
a Pagewrapper method in this case, Page.ToUrl you can
open each supported browser with a single script instruction.
Or suppose you want to record a click over the Backbutton,
this will also be a GUI-specic action and you would need
separate instructions for each GUI. However, if you refer to
the engines methods, then only two code branches will be
needed.
STEP3: TESTAPPLICATION-RELATEDDIFFERENCE(NON-
GUI)
Other aspects of an application are not GUI-related (hence,
beyond the scope of the screenshot type tests mentioned
earlier). A test, for example, might need to verify that links on
a web page actually open the correct pages or that a request for
a browser modal dialog is successful.
Here is a test script portion that accounts for differences in
how various browsers respond to an applications request for
a modal dialog. The code uses conditional logic to identify
the current browser. It then asks if the object, modalDlg,
exists and, depending on the answer, writes either a success
or fail message to the test log. As in the previous test steps,
its an example of how using generic objects across browsers
streamlines test coding.
The code takes into account the following browser specics:
in Firefox version 3 and earlier the modal dialogs have the
JavaScript Application caption and are Window objects;
in Firefox 4, 5 and 6 modal dialogs have the same caption
but are vbox XUL objects; in Internet Explorer modal dialogs
are entitled as Explorer User Prompt. Due to Microsofts
Loosely Coupled IE (LCIE) concept, the modal dialogs in
Internet Explorer 8 and 9 are invoked in the helper tab process.
Hence, we need to search for the dialog object within the entire
Sys object rather than only within the browser process.
STEP4: TESTUSECASEDIFFERENCES
Some differences in how applications behave in different
browser only show up because users behave differently in
different browsers. For example, the user might always click
on a certain button when using Internet Explorer, but not click
on it when using Chrome. Suppose we want to verify the value
of Cash Balance in the StockTrader website (Figure 1)
always equals $89,984.05 in all browsers. But for IE only, we
also want to verify that clicking on the Discussion Forum
button loads that page (Figure 2). Further, we want to test that
the Ask a Question box is enabled once the page loads.
To show how the same general approach conditional logic with
generic object naming applied works in a keyword-driven test
tool with no manual scripting, this time well use TestComplete 9.Figure 3: Recorded Keyword Web Test
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
11/44www.automatedtestinginstitute.comOct/Nov 2012 Automated Software Testing Magazine
Figure 3 shows the test, which TestComplete has constructed
automatically by recording user actions in the applications
GUI: entering a user name and password, clicking the
submit button, clicking the log out button and closing the
browser window. The test is browser-independent because
TestComplete employs a unied object model for web
browsers and pages. The Browser node contains Page objectsthat provide browser-independent access to the pages (e.g.,
pageHomePage) opened in that browser.
To test that the content of Cash Balance is correct, the
TestComplete user inserts a property checkpoint after the
pageHomePage loads. (The user simply drags and drops an
Insert Checkpoint icon on top of the appropriate item on
the web page, and another insert Checkpoint icon at the
appropriate step in the test itself. This action opens a dialog
panel where the user enters the value to be check for.) Note
that the generic property contentText, automatically maps to
either innerText or textContent, the browser-specic propertyfor text on a web page in Firefox or Chrome, respectively.
Figure 4 shows the same test, this time with conditional logic
(indented) that checks whether the browser is IE and, if so,
clicks on the Discussion Forum Button, and waits for the
page to load. A property checkpoint inserted here then checks
whether the Ask a question box on the forum page is
enabled. No other changes were made in order to include this
special IE case in the test.
So, is a cross-browser test of a web application ever really
complete even if you cover all four steps covered here?Since technology and user requirements keep changing, test
completeness is probably more of a goal than an actual result.
How far you get toward that goal, however, depends on the
scope of your testing to begin with whether it does, in fact,
address all four reasons why applications behave differently in
different browsers. It also depends on whether you incorporate
strategies like generic object mapping for addressing as many
potential differences as fast as you can.
Figure 4: Recorded Keyword Test with IE Special Case
How far you get
toward that goal,
however, dependson the scope of your
testing to begin with
whether it does,
in fact, address
all four reasons
why applications
behave differently
in different
browsers. It alsodepends on whether
you incorporate
strategies like
generic object
mapping for
addressing as many
potential differences
as fast as you can.
TestKIT T
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
12/4412 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
Randys Favorite Free andCheap Tools
The above quote is from an article
that Randy Rice wrote for the June
2010 issue (Volume 2, Issue 2) of
the Automated Software Testing Magazine
entitled, Free and Cheap Test Automation.
In that article, Randy Rice not only describes
the importance of and process for getting
started with free and cheap tools, he also
identies and describes some of his favorites.
In his new tutorial at the TestKIT 2012
conference, Randy provided an updated list
of tools in the following categories:
Test Design Test Execution
Test Frameworks Performance Testing and
Monitoring
Test Management Test Data Test Evaluation
Defect Tracking Special Applications
This article identies some of the execution
tools Randys list focuses on.
CUBICTEST
CubicTest is a graphical Eclipse plug-in for
writing Selenium and Watir tests. It makes
web tests faster and easier to write, and
provides abstractions to make tests more
robust and reusable.
CubicTests test editor is centered around
pages/states and transitions between these
pages/states. The model is intuitive for
both Ajax and traditional web applications
and supports most user interaction types.
CubicTest features an innovative test recorderand test runner based on Selenium RC which
are fully integrated with the graphical test
editor. Tests can also run standalone from
Maven 2.
BADBOY
Badboy is a powerful tool designed to aid in
testing and development of complex dynamic
applications. Badboy makes web testing and
development easier with dozens of features
including a simple yet comprehensive
capture/replay interface, powerful load
testing support, detailed reports, graphs and
much more!
CUCUMBER
Cucumber is a tool that executes plain-text
functional descriptions as automated tests.
The language that Cucumber understands is
called Gherkin.
While Cucumber can be thought of as atesting tool, the intent of the tool is tosupport Behavior Driven Development
(BDD). This means that the tests are typicallywritten before anything else and veried bybusiness analysts, domain experts, and othernon-technical stakeholders. The productioncode is then written outside-in, to make thestories pass.
Cucumber itself is written in Ruby, but itcan still be used to test other code writtenin Ruby and/or other languages includingbut not limited to Java, C# and Python.Cucumber does require minimal use of theRuby programming language, but Ruby is arelatively simple language.
SIKULI
Sikuli is an open source visual technology to
automate and test graphical user interfaces
(GUIs) using images (screenshots).
Sikuli includes Sikuli Script, a visual scripting
API for Jython, and Sikuli IDE, an integrated
development environment for writing visual
scripts with screenshots.
You can programmatically control a web
page, a Windows/Linux/Mac OS X desktop
application, or even an iPhone or Android
application running in a simulator or viaVirtual Network Computing (VNC).
ICUTEST
ICUTest is a free GUI Unit test tool for
Windows Presentation Foundation.
Test Generator, included with ICUTest, is a
tool that automatically creates GUI unit tests
for an application. Just specify your assembly
(executable or DLL) and Test Generator will
inspect your assembly for UI components and
generate unit tests automatically.
Open Sourcery
As a tester, I have always found it helpful to have my own toolkit that I understand andcan take with me on projects. If my client has other tools, or does not want me use my owntools, thats fine. I still like to be prepared with my own tools that I can use in a pinch if theopportunity arises.
The problem is that I dont have the same deep pockets as some companies do. So, I lookfor affordable ways to get the job done. Sometimes my clients dont have the budgets formajor commercial test tools, either. In that case, I can offer other alternatives that add to my
value as a consultant.
Randy Rice
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
13/44www.automatedtestinginstitute.comOct/Nov 2012 Automated Software Testing Magazine
ATILocal Chapter
Program
ATI- Meeting Local Needs In Test Automation
ATIs Local Chapter Program is established to help better
facilitate the grassroots, global discussion around test
automation. In addition, the chapter program seeks to
provide a local based from which the needs of automation
practitioners may be met.
ENHANCE THE
AWARENESS OF TEST
AUTOMATIONas a discipline
that, like other disciplines,requires continuous education
and the attainment of a standard
set of skills
OFFER TRAINING AND
EVENTSfor participation by
people in specifc areas around
the world
HELP PROVIDE
COMPREHENSIVE, YET
READILY AVAILABLE
RESOURCESthat will aid
people in becoming more
knowledgeable and equipped to
handle tasks related to testing
and test automation
Start a Local Chapter TodayEmail contact(at)automatedtestinginstitute.com to learn more
www.automatedtestinginstitute.com
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
14/4414 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
M arketplace SystemFor context, a short introduction to marketplace systemsis necessary. Figure 2 is a simple diagram of a marketplace
system illustrating the trading of a nancial instrument on a
market in an order book. Examples of instruments are stocks,
bonds, indexes and commodities etc. A marketplace system
supports automatic matching of incoming orders in real time
and determines trades and order events (orders are cancelled,
updated or deleted).
An order put into the order book is either a bid (buy) or an ask
(sell). There are several types of orders, including:
Limit Order: A limit order is an order with a price. It willbe traded at the specied price or better.
Market Order: Is a buy or sell order with no price. It
will be executed at best available price.
Iceberg Order:Is an order where only a part of the order
is visible. When that quantity has been traded another
portion is put into the market.
All-Or-None Order (AoN):The order will only trade if
the total volume can be executed.
Additional order types include: Combination Orders, Linked
Orders, Fill-Or-Kill (FoK), Fall-And-Kill (Fak), Stop Loss,
Pegged, Quotes etc.
Random Testis a
special test type thathas been proven
to be a very good
complement to ordinary functional
tests for testing marketplace
systems. Figure 1 below shows an
overview of the concept. The main
idea is to send random input to the
system under test (SUT). A (Test)
Oracle listens to the output and
(but not necessarily) the randominput. The Oracle can then decide if
the actual output deviates from the
expected output.
By
LARSWAHLBER
G
it is time to analyze why andhow random tests can be used
It is a fundamental
test principle that you
cannot test everything.
Figure 1: RANDOMTESTCONCEPT
TestOracle
RandomInput Output
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
15/44www.automatedtestinginstitute.comOct/Nov 2012 Automated Software Testing Magazine
NDOMTESTS
Actors,
Oracles and
Analysis of
Random
Tests in a
Marketplace
System
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
16/4416 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
The following notation will be used to
describe an order:
Bid/Ask quantity @ price
For example, a Limit Bid order to buy
200 shares for a stock at price 21.2
could be written as: Bid 200 @ 21.2.
An order book is represented as a table
with two columns, containing the bid
in the leftmost column, and the ask inthe rightmost column. For example, a
stock order book for companyXYZ may
appear as follows:
BID ASK
200 @ 21.2 300 @ 22.3
Now, lets walk through a sample order
event ow. Given the above order book,
if a newBid 200 @ 23.0order is entered,
in most markets, a Trade 200 @ 22.3will occur, with the resulting order book
appearing as follows:
BID ASK
200 @ 21.2 100 @ 22.3
Trading clients sometimes subscribe to
an order event ow to build up a Shadow
order book of the real order book in
the marketplace system. In the above
example, a client would have receivedthe following order events:
Insert: Bid 200 @ 21.2, ID=1
Insert: Ask 300 @ 22.3, ID=2
Insert: Bid 200 @ 23.0, ID=3
Cancel: ID=3
Update: ID=2, 100 @ 22.3
In some marketplace systems the Insert
and the immediate Cancelevents of the
order with theID=3are not published to
the clients (to save bandwidth).
With this brief introduction to marketplace
systems behind us were ready analyze
why and how random tests can be used.
R andom Test forMarketplace Systems
Marketplace systems are very often
considered critical, thus customers and
authorities require these systems to be
subjected to extensive testing. One of
the major challenges faced by those
responsible for testing these systems,
however, is the explosion of order book
situations that need to be veried. So,
while it is a fundamental test principle
that you cannot test everything, the
criticality and importance of correct
functionality for marketplace systems
often still results in thousands offunctional tests (2). Even thousands of
tests may not cover all pertinent areas,
though, which is why random tests are
an excellent complement.
The framework used on my project for
automated execution of random tests
for trading simulations was the same
framework wed already implemented
for automated functional tests; with
a few simple modications. It used
the same API libraries that were usedby the customers for development of
trading clients (i.e. tests were done with
transactions, not via a client GUI). For
more information about the specic
technical makeup of this framework,
read chapter 18 of the Experiences
of Test Automation book listed in the
References section. This article focuses
primarily on the logical design and use
of two major elements of a random test
framework: actors and oracles.
Actor DesignActors are dened as a set of actions that
can be performed on the system. While
it is the frameworks responsibility
to select actors during a simulation,
the actors are responsible for how thatsystem is actually exercised. Some of
the actors that we created included:
Trading Actor:Enters modify,
suspend/activate, and cancel limit,
Fak, FoK, market orders.
Iceberg Actor:Enters modify,
suspend/activate, and cancel
Iceberg orders.
Trading Control Actor:Suspends/
Releases order books, ush orders,
trade halt order books, suspendsother actors, etc.
State Actor:Changes trading
states between pre-trading,
continuous trading, post trading,
market closed, etc.
A probability factor was assigned to
each action belonging to an actor to
increase the likelihood of some actions
occurring. These probability settings
were congurable, however, and could
therefore be easily adjusted.
O racle DesignAn oracle is a component that reviews
the output from a test and determines,
based on that tests input, what the
expected output should be and whether
or not the actual output matches what
was expected.
There are many different approaches tothe design of the Test Oracle depending
on how advanced the situation requires
the random tests to be. Hoffman (1)
suggests the following types of Oracles:
1. No Oracle: Can only detect
crashes of the SUT, but is
simple in design.
2. True Oracle:An independentcopy of the system. Detects all
deviations, but is expensive to build.
Figure 2: MARKETPLACESYSTEM
Orders Trades OrderEvents
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
17/44www.automatedtestinginstitute.comOct/Nov 2012 Automated Software Testing Magazine
3. Consistency:Veries current
run results with a previous run.
Cannot be used to test new
functionality.
4. Self-Referential: Embeds
answer data in the input data.
5. Heuristic:Veries some
outputs. Can miss systematic
errors.
In addition to the types described by
Hoffman, lets add the following:
6. Fuzzy: Can detect that a
deviation may have occurred.
The most used oracle design principle
by us was the No Oracle and Heuristic
Oracle.
The No Oracle design is simple and
nds severe defects. Some defects leadto a crash or controlled stop (critical
assertion failure), which also ends the
simulation. After each simulation the log
les were scanned for critical errors and
exceptions. It was a cheap and easy way
to nd complicated and severe defects.
Different strategies were used for the
Heuristic Oracle. Simple checks on the
output included:
Prices for Trades are always
positive (for stocks)
Prices of Trades must always
be within the range of the
min/max prices of orders sent
into the system
Traded quantity is always
positive
Traded quantity can never be
more than the total quantity
sent into the system (the
lowest from either bidor askside)
A slightly more complicated oracle is
one that listens to the order event ow
and builds up a Shadow order book. If
any of the following situations occurs
something is clearly wrong:
Insertion of an order ID that
has already been inserted
Update of an order ID that
has never been inserted
Cancel of an order ID that
has never been inserted or has
already been cancelled
Self-referential Oracles were used to
verify:
An order is always traded
at the limit price or better
(exceptions are market
orders)
The order can never
participate in a trade with
a larger quantity than was
inserted.
Some attempts were made on a True
Oracle. In the development of one of the
trading systems, a copy of the matching
engine (core logic) was made in Java
and tested against the production code
implemented In C. The Java engine
was used to nd defects early in the
development project and was later
abandoned as the complexity increased.
S imulationsOur simulations used the actors and
oracles we designed to implement
random tests. For a typical simulation,
a mix of the various actor types was
chosen in advance and the simulation
was started with a dened random seed,
to ensure no simulation was identical.
During the simulation, the following set
of steps were continuously performed:
1. The framework selected an actor.
2. Based on the dened
probabilities an action was
selected and executed.
3. An oracle was triggered to verify
the results.
It was possible to run the simulation
either as a single session/thread or asmultiple sessions/threads. In a single
session/thread simulation, each selected
action was performed one after another
in a round robin scheme. This made
it easier to reproduce the sequence
of actions performed by the actors.
The advantage of a multiple sessions/
threads simulation is that a higher load
could be achieved, but it increased the
difculty associated with reproducing
the sequence of actions.
A nalysis of resultsThe simulation results were measured
using mostly defects found vs.
transactions sent into the system; a
transaction being dened as an inserted,
updated or cancelled order. Around one
million transactions (1 MTxN) were sent
each day in production. A test goal for
the simulations was to have at least 20
RandomTests
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
18/4418 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
simulated trading days with no new defects. A typical graph
that was always included in the test report of a new release
to production is shown in Figure 3. The simulation typically
found between one and ve severe defects that were not found
with ordinary functional tests. If a defect was found, it was
immediately corrected and the simulations continued until the
reliability goal was reached.
In one release, an actor was incorrectly designed so it did
not exercise a newly implemented order type. As a result of
the oversight, the real production system went down within
an hour of deploying the new release of the software. Upon
correcting the actor, the simulation was re-executed and found
the problem after just 15 minutes of simulation time! After this
incident we improved the test process by:
Conducting a more extensive review of actor and
oracle code
Analyzing code coverage after running simulations Using fault injection to see if the actors & oracles
could trigger and nd expected errors.
The lesson learned was that simulations (or random tests) are
an important complement to traditional testing techniques for
critical systems.
A dvanced ConceptsSome advanced random test techniques we plan on
implementing in the future include:
Oracle using constraint programming techniques: The
oracle using constraint programming succeeded in modeling
a large part of order types (but not all). It showed that
techniques used in, for example, the aerospace industry
could cost effectively be applied to the nancial industry.
During the design of this almost true oracle, a fault in the
requirements was also found! For more information read
reference 3 identied at the end of the article.
Fuzzy Oracle using Neural Networks:The oracle had the
ability to learn normal behavior of the system using
neural networks. The Oracle could then warn when
something deviated too much. A simplied model of a
marketplace system using z-transforms is shown in Figure 4.
Fuzzy Oracle using Machine Learning:This oracle type uses
the same approach as the Neural Network type but uses
Machine Learning techniques (5).
S ummaryRandom tests are a very good complement to ordinary
functional automated tests. If a framework for automatic
functional tests is in place, it is a small and easy task with a
high payoff to simply add random tests. This article is based
on real tests performed since 1999.
R eferences1. Hoffman, Douglas. Using Test Oracles in Automation.
Software Test Automation Conference Spring 2003.
Presentation
2. Fewster, Mark, and Dorothy Graham. Experiences of
Test Automation. Boston: Addison Wesley, 2012. Print
3. Lozano, Roberto C. Constraint Programming for
Random Testing of a Trading System. MA Thesis.
KTH Royal Institute of Technology, Stockholm. http://
riunet.upv.es/bitstream/handle/10251/8928/roberto_
report_upv.pdf. Web. 28 January 2010
4. Bckmark, Torbjrn. Neronnnts baserad agent. MA
Thesis. http://www.addq.se/document/Referensjobb/
Random-Neuron.pdf. Web. 5 October 2001
5. Stenman, Peter. Learning the transaction ow at an
Exchange. MA Thesis. 2002
Figure 4: Z-TRANSFORMMODEL(4)
Figure 3: SIMULATIONRESULTSGRAPH
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
19/44www.automatedtestinginstitute.comOct/Nov 2012 Automated Software Testing Magazine
Training Thats Process Focused, Yet Hands On
Software Test Automation Trainingwww.training.automatedtestinginstitute .com
Public and Virtual Trainin Available
Come
participate in a set
of test automation courses
that address both fundamental
and advanced concepts from a
theoretical and hands on perspective.
These courses focus on topics such
as test scripting concepts, automated
framework creation, ROI calculations
and more. In addition, these courses
may be used to prepare for the
TABOK Certification
exam.
PUBLIC COURSES
Software Test Automation Foundations
Automated Test Development & ScriptingDesigning an Automated Test Framework
Advanced Automated Test Framework Development
Mobile Application Testing & Tools
VIRTUAL COURSES
Automated Test Development & Scripting
Designing an Automated Test Framework
Advanced Automated Test Framework Development
Mobile Application Testing & Tools
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
20/44
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
21/442Automated Software Testing Magazine
By Stefan Mohacsi and Armin Beer
Though the introduction of test automation has its
benets in promoting regression testing, we also
observed some weaknesses concerning the creation
and maintenance of the test cases:
Automated test cases are designed by hand, often
without taking specic test case design methods into
account.
The implementation of automated test scripts requires
advanced programming skills.
The maintenance of automated test cases after changes
to the SUT (Software Under Test) is difcult and time-
consuming.
Model-based testing (MBT) has the potential to resolve these
issues. However, it is often still regarded as too academic and
therefore as not suitable for testing in a real-world project
where resource, cost and time constraints apply.
The most common way of getting a test model for MBT is to
reuse a system model that has been created for development.
However, this method has serious disadvantages:
If both the source code and the test cases are generated
from the same model, no deviations will be found.
Defects in the system model will remain undetected.
The system model is created with a focus on
development, not on testing. Thus, it includes a wealth
of information that is irrelevant to testing while missing
other data that is required.
Many models in notations like UML lack the level of
formality required for automated test case generation,
e.g., because information is contained in plain text
annotations. UML extensions can mitigate this problem,
but add a further level of complexity to the model.
Test automation based on tools like HP QUICKTEST
PROFESSIONAL is well established and successfully usedin many software organizations, but...
www.automatedtestinginstitute.comOctober 2012
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
22/4422 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
For all these reasons, it is recommended
to create a separate model focused on
the purposes of testing. It should only
contain information relevant to testing
and be just detailed enough to generate
test cases and test data. If the notation
used for MBT is understandable for
system testers, they are more likely to
use it. Most system testers nowadays
are familiar with use cases, activity
diagrams and Boolean expressions.
Systematic test design techniques like
equivalence partitioning, boundary value
analysis, and cause-effect analysis are
often not used exhaustively enough by
the testers. MBT fosters the application
of these techniques and as a consequence
improves the capability of the test cases
to detect specic categories of defects.
Despite the initial effort needed to create
a test model and integrate MBT into the
test process, the improved effectiveness
and maintainability of the test cases will
be appreciated by all stakeholders.
This article is structured as follows:
In the next section we present the
recommended workow and give
practical tips for the efcient usage of
MBT. Subsequently, we introduce the
MBT tool Atos TEMPPO Designer, and
show how it supports the MBT workow
using a practical example. Finally, we
give a conclusion and an outlook on our
future work.
Workow and
Practical Tips
for Model-based
Testing
In this section we describe the steps
required to successfully introduce MBT
into your project and also provide a
number of practical tips. But before
reading on, you should be aware of
the rst pre-requisite for applying
test automation in general, and MBT
in particular: a well-dened software
development process that guarantees the
adequate management of project tasks;
requirements; test artifacts; and change
requests. If the process in place is not
at the required maturity level, there is
a strong probability that MBT will not
work. Fast and reliable communication
channels between the testers and the
rest of the project team are an essential
feature of such processes (see Figure 1).
Ensure Testability of
Software Requirements
While requirements engineering is
usually not considered a part of the test
process, it is, nonetheless, crucial for
the subsequent testing phases. There
are some points that have to be taken
into account already at this early stage
to ensure the success of the testing
activities, especially if MBT is going to
be applied.
Testability comprises two aspects:
1. Ensuring that the input documents
for the test process have the
required quality level
2. Ensuring that the system under
test fullls all technical pre-
requisites for test execution
Roles: Requirements Engineer, Test
Manager, Test Analyst, Test Automation
Specialist
Input: User Requirements
Output: SW Requirements, Testability
Analysis Report
Practical Tips:
Involve test experts in early
reviews of requirement and
design specications in order to
avoid testability problems later. A pre-requisite for MBT is that
the requirements are detailed and
clear enough that a formal model
can be derived from them.
Try to use activity diagrams, state
diagrams, usage scenarios etc.,
instead of plain text.
Make sure to plan interfaces
that can later be used by test
automation tools to access the
system under test.
Figure 1: MBT-related activities in relation to the ISTQB test process
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
23/44www.automatedtestinginstitute.comOct/Nov 2012 2Automated Software Testing Magazine
Defne Test Structure
Based on the requirements, a test
structure consisting of test packages and
test cases is derived. This should be done
with a professional test-management
tool that provides convenient interfaces
to the other tools involved in the test
process (requirements management,model-based test design, test execution,
defect management, etc.)
Roles: Test Analyst
Input: SW Requirements
Output: Test Structure
Practical Tips:
Dont only focus on the required
functions, also plan tests for the
non-functional requirements.
Decide for which test casesmodel-based design shall be
used. Typical candidates are tests
for requirements that can easily
be formalized as a model. On the
other hand, for some requirements
MBT might not make sense.
In addition, dont forget to plan
some experience-based tests in
addition to the systematic tests. A
test model is a good thing but it
cannot replace human experience
and intuition!
Create Model-based
Test Design
In contrast to an ordinary test project
where test cases are designed e.g., in
a tabular format, a model-based test
project requires the creation of a test
model. This model depicts the system
behavior from a testing perspective and
contains all scenarios, constraints, and
dependencies that are relevant to testing.
Modeling can require considerableeffort, especially if the input documents
do not have the required level of
detail and frequent clarications are
required. On the other hand, this is an
excellent opportunity to detect defects
and inconsistencies at an early stage
even before the implementation phase.
Therefore, this initial investment helps
to avoid serious problems in later stages
of the project and greatly enhances the
maintainability of the tests. Nonetheless,
it has to be checked beforehand when the
ROI will be reached and if MBT makes
sense for a certain test item.
Roles: Test Analyst, Requirements
Engineer, Software Designer
Input: SW Requirements, Test Structure
Output: Test Model
Practical Tips:
Avoid reusing a model that
has also been used for code
generation. If the code and
the tests are generated from
the same model, no deviations
will be found! Try to create a
specic model from the testing
perspective instead.
Creating a test model is a
challenging task that requiresmany different skills (test design,
abstraction, understanding
of requirements, technical
background). Dont expect end
users to perform this task but
employ specialists instead!
Start modeling at an abstract
level, then add more detail step
by step
Make sure that requirements
engineers and software designers
are available to clarify any
questions from the test designers.
Pay attention to the reusability
and maintainability of the
model components, e.g., use
parameterizable building blocks
Link Test Model to
Executable Test Scripts
If the test execution shall be done
automatically, the abstract test model
has to be linked to a test script containing
concrete instructions for a test executiontool. A possible way of doing that is
to implement an adapter or script for
each keyword that is used in the model.
However, this requires considerable
effort and advanced programming skills.
A much better alternative is to use
an MBT tool that can import GUI
information and generate executable
scripts automatically. In this case, each
step in the model only has to be assigned
to the affected GUI element.
Roles: Test Analyst, Test Automation
Specialist
Input: Test Model
Output: Detailed Test Model/Test Script
for each Keyword
Practical Tips:
If using a professional test
generator that generates complete
scripts (e.g., TEMPPO Designer),
record GUI information and
assign steps to GUI elements.
If using a generator that only
produces abstract keywords,
implement an adapter or a script
for each keyword.
Prepare Test DataDepending on the requirements it has
to be decided whether test data can be
imported from an existing source, has to
be created by hand or will be generated
automatically. In the latter case, some
MBT tools provide the possibility to
specify data dependencies as part of the
model and to produce data accordingly.
Roles: Test Analyst
Input: SW Requirements, Test Structure
Output: Test Data
Practical Tips:
Try to cover both test data and test
sequence-related aspects in your
model (e.g., dene equivalence
partitions for test data, activity
diagrams for sequences). Finally,
connect the two aspects by
specifying which data is used in
which step of the sequence.
Simple test data design methods
like equivalence partitioning and
boundary value analysis should
only be applied for data elds that
do not have any dependencies
with others. For all other cases,
methods like cause/effect analysis
or CECIL should be applied.
Generate Test Cases
Based on the test model, test cases can be
generated automatically. Depending on
ModelBased
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
24/4424 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
the model type and the generation tool,
the user can choose between various
generation strategies and coverage
criteria. While many tools can only
generate abstract test cases consisting
of keywords or instructions for manual
testing, others can produce complete,
executable scripts.
Roles: Test Analyst
Input: Detailed Test Model-/-Test Script
for each Keyword
Output: Executable Test Scripts
Practical Tips:
Choose test coverage based on
model type and criticality of the
associated requirements.
Combine different test methods
to increase defect detectionpotential. Use random generation
in addition to systematic methods.
The integration should make it
possible to transfer the generated
test cases directly to the test
management tool.
Execute Test Cases
In some projects, the test generator
produces textual instructions for manual
testing. However, more frequently, the
test execution is performed automaticallyby a tool. In both cases, appropriate
interfaces for sending test commands to
the system under test and verifying the
results have to be available.
Roles: Tester, Test Analyst
Input: Executable Test Scripts
Output: Test Protocols, Defect Reports
Practical Tips:
Dont expect test automation to
work immediately -- its ratheran iterative process. Verify ifthe generated test cases work asexpected (GUI object recognition,timing, etc.). Correct the modelwhere necessary until the tests arerunning smoothly.
Test scripts derived from a modelusually require that the systemunder test is in a well-denedstate at the beginning of each testand behaves in a predictable way.For instance, alterations in the
database or unforeseen pop-upmessages may cause the scripts tofail if they are not considered inthe model.
In cases of deviation from the
expected behavior, it has to bedetermined whether the problemwas really caused by a bug in thesoftware or on the contrary by a
defect in the test model.
Create Test Report
The test protocols created during the
test run have to be transformed into
a concise test report that informs the
responsible managers about the current
project status.
Roles: Test Manager
Input: Test Protocols, Defect Reports
Output: Test Report
Practical Tips:
For MBT it is important thatstatistics of the model coverageare included (e.g., the percentageof states, transitions, or scenariosthat have been covered by thetests).
Many test management tools cangenerate such reports, providedthat they have access to all therequired data. Again, a good
integration with other tools is
important.
Maintain Test Model
For each new version that has to be
tested, the tests have to be adapted and
repeated. Updating each single test script
by hand is impractical as it can require
an enormous effort. On the other hand,
updating a model consisting of reusable
components is much less tedious, since
only a few parts have to be changed
before generating perfectly updated test
scripts.
Roles: Test Analyst
Input: Test Model, Change Requests
Output: Updated Test Model
Practical Tips:
It is essential that the test team
is informed about all changes in
the system under test. Even small
modications that are invisible to
the user can cause an automatedtest run to fail.
The effort invested into a well-
structured model design will pay
off now. If designed properly,
only some parts of the model
have to be adapted. Afterwards,
all test cases can be updated
automatically by re-generating
them.
Model-based
Testing with Atos
TEMPPO Designer
While it is relatively easy to nd tools
for automated test execution, there
are signicantly fewer that focus on
model-based test generation. They can
be categorized according to the method
used for test generation: data-oriented
tools and sequence-oriented tools. A tool
that unites both features and can also
Figure 2: Model-Based Testing with TEMPPO Designer
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
25/44www.automatedtestinginstitute.comOct/Nov 2012 2Automated Software Testing Magazine
be easily integrated into a projects tool
framework is Atos TEMPPO Designer;
this tool was previously called IDATG
(Integrating Design and Automated Test
case Generation).
When the research project IDATG started
in 1997, the aim was to develop a simple
tool for facilitating the maintenance of
SilkTest and WinRunner scripts for
GUI testing. Over the years, IDATG
was steadily expanded to support an
increasing number of test methods and
output formats, as well as testing via
non-GUI interfaces. In 2004, IDATG
became part of the test framework of
the European Space Agency ESA (this
success story can be found in Dorothy
Grahams new book Experiences of
Test Automation [Graham et al. 2012]).
Interfaces for Model
Creation
Many MBT tools do not have a model
editor of their own, but instead rely on
other sources from which models areimported. However, reusing models
that have not been created with a focus
on testing has several disadvantages.
Quite often such models are not formal
enough (e.g. plain text annotations) or
they contain a lot of information that
is irrelevant to testing while missing
important data. To overcome these
problems, TEMPPO Designer has a
built-in model editor that provides
an independent way of creating test
models. It uses a specic notation that
is simple to learn, and it focuses on the
necessities of testing (see Figure 4).
Test sequences are represented as task
ow diagrams that consist of simple test
steps (blue) and parametrizable building
blocks (yellow) that represent reusable
sub-sequences of steps. For GUI testing,
steps can be assigned to GUI objects
using the built-in GUI Spy.
Apart from common testing methods
like equivalence partitioning, a number
of innovative test strategies have
Fig ure 3: Sample Application
Figure 4: Task Flows in TEMPPO Des igner
ModelBased
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
26/4426 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
been developed for the tool. These
include CECIL (Cause-Effect Coverage
Incorporating Linear boundaries [Beer &
Mohacsi 2008]) and a hybrid algorithm for
random testing (Mohacsi & Wallner 2010).
Interfaces for Test
Execution
A major drawback of many testgenerators is that they only produce
abstract test cases that still require
manual completion, for instance
by implementing a script for each
keyword. A distinguishing feature of
TEMPPO Designer is that its output
are complete, executable scripts for a
variety of popular test execution tools
including HP QuickTest Professional
and the Micro Focus tools SilkTest
and TestPartner. However, the tool can
also be used for producing test cases thatare executed manually or over a non-
GUI interface.
This direct integration between test
generation and execution has proven to
be a considerable advantage in that it
delivers a signicant reduction in test
maintenance costs; i.e., instead of having
to update each single test script for every
new version of the system under test, it
usually sufces to change a few building
blocks inside TEMPPO Designer and let
it generate new test cases. A case study
from an ESA project has shown that the
additional effort for introducing model-
based test automation paid off after
only four test repetitions (Graham et al.
2012).
Interfaces for Test
Management
Integration of test generation with
test and requirements management is
equally important. As its new name
suggests, TEMPPO Designer is a part of
the TEMPPO suite that also includes the
test management tool TEMPPO Test
Manager. Information can be passed
in both directions: Test structures and
requirements created in Test Manager
are passed to Designer. After the testmodel has been created, the generated
test cases are passed back to TEMPPO
Test Manager which ensures proper
versioning, traceability, execution
planning, and reporting.
There is also an interface for HP Quality
Center that allows the user to import
TEMPPO Test Designer test cases via
Excel.
Practical
Example
We are going to demonstrate the most
relevant steps of the MBT workow
using a simple application for managing
tennis players and tournaments. While
this is a hypothetical example, it includes
many experiences from real projects.
It also introduces the usage of the Atos
TEMPPO tool suite, and shows how to
efciently use it for model-based test
design and generation.
Ensure Testability of
Software Requirements
In the rst application design, the
use of the GUI library FlashyGUI
was envisaged by the designer whohad found it on the Internet and was
eager to try it out. Fortunately, the test
automation specialist was involved in
the design review and pointed out that
this technology was not supported by
any test automation tool on the market,
thus making automation impossible.
Finally, it was decided to use a standard
Win32 GUI instead.
Defne Test Structure
Once the requirement and designspecications had been reviewed, the
test analyst began creating a tree-like
test structure in TEMPPO Test Manager.
He started with typical usage scenarios
like Create Player, then added tests
for non-functional requirements (e.g.,
performance of the search function) and
error cases (e.g., assigning a male player
to a female tournament). Based on his
ample experience from past projects he
also planned a few scenarios simulating
unusual user behavior.
Create Model-based
Test Design
Using TEMPPO Designer, the test
analyst began to break down each
scenario of the test structure into a
rough sequence of test steps. While
doing this, he identied recurring
steps like Edit Player and dened
Figure 5: Excerpt from a Generated Script in QTP
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
27/44www.automatedtestinginstitute.comOct/Nov 2012 2Automated Software Testing Magazine
them as re-usable building blocks with
parameters. These rough building blocks
were further rened, for instance, Edit
Player consisted of the blocks Edit
Personal Data, Edit Career Data, and
Edit Ranking. This renement was
iteratively repeated until ending up with
single-user actions like entering the rst
name or clicking the OK button.
The user action represented by each
step could be entered in a generic
event language, for instance Click
or Input (Serena). However, instead
of concrete texts, mostly references
to parameters were used to make
the model more exible: Input (#@
EditPersonalData:FirstName#)
Semantic conditions were also
considered in the model. For instance,a middle initial was only entered if
the corresponding parameter was not
empty. This could simply be specied in
the form of a Boolean expression: #@
EditPersonalData:Initial# != .
Also, the expected results and reactions
of the system were specied. This could
easily be done with events like Verify
(CAPTION, Player data has been
saved.).
However, the specication documentson which the model was based seemed
to be incomplete. For instance, it was not
clear how the system should react if a
man was added to a female tournament.
Maybe the designer had assumed that
the programmer would surely consider
this obvious case and it was therefore
unnecessary to mention it. Fortunately,
the test analyst could contact the designer
directly and ask him. It turned out that
this case had simply been forgotten in
the design and that the programmer, who
had no interest in tennis at all, hadnt
thought of it either. Since this defect
had been found in an early stage of the
development, it could quickly be xed
and there was little harm done. However,
if it had found its way into the nished
product, this embarrassing mistake
could have damaged the companys
reputation.
Link Test Model to
Executable Test Scripts
Once the GUI of the tennis application
had been implemented, its window
could be recorded with the built-in
GUI Spy of TEMPPO Designer. Each
blue test step was assigned to a certain
object in the screenshot, for instance,
EnterFirstName to the input eld
FirstName. Since the GUI Spy did
not only capture the appearance but also
the technical details of the GUI objects,
no further information was needed toproduce a complete object repository for
the test execution tool (in this case, HP
QuickTest Professional).
Also, the generic events like Input would
later be translated automatically into the
tool-specic language (for QuickTest,
Input would be translated into the Set
command). Only a few steps involved
more complex actions that could not be
expressed in the generic language but
had to be dened in QuickTest code. An
example was checking the alphabetical
order of the player list, a verication that
had to be implemented as a QuickTest
function. Apart from this, no other
action was required to link the model to
the nalized test scripts.
Prepare Test Data
Most information about the tennis
players was imported from an Excel
sheet showing the current ATP ranking.
In addition, equivalence partitioningand boundary analysis were applied
to generate test data, thus covering
all relevant cases for elds like the
Prize Money. In more complex cases,
dependencies were specied, e.g., that
the start date of a tournament must
lie before its end date. Finally, it was
specied which data was used in each
test step simply by inserting references
to the data tables.
Generate Test CasesOnce the rst version of the model wascompleted, our test analyst was able
to start the script generation. He chose
This direct integration between test generation
and execution has proven to be a considerableadvantage in that it delivers a significant reduction
in test maintenance costs
ModelBased
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
28/4428 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
a combination of graph-oriented tests
covering all step connections, data-
oriented tests covering all tennis players,
and a few random sequences each
consisting of 1000 steps. The output
was a complete set of test scripts and
an object repository for HP QuickTest
Professional. Also, information about
the generated test scripts was transferred
to TEMPPO Test Manager.
Execute Test Cases
Via TEMPPO Test Manager, QuickTest
was opened to start a rst trial run of
the scripts. In the course of this it was
noted that a few scripts failed because
sometimes a timeout occurred when
the tennis application connected to its
database. The test analyst had to insert a
higher delay value in the correspondingstep of the test model and re-generate the
scripts.
In the next trial run it turned out that the
tree control displaying the tournaments
had changed its ID and could thus not be
found any more by QuickTest. The test
analyst solved this by removing the ID
from the set of properties that was used
for object identication. Again, this was
done in the model, not in QuickTest
itself. Finally, after re-generating
the QuickTest les, the tests did runsmoothly.
Once the test analyst decided that the
scripts were ready, the tester could
begin his task. He ran the entire test
suite and evaluated the test protocols
that were produced by QuickTest.
Several deviations from the expected
behavior were found, for instance, it
was possible to enter a negative value
for the prize money. One of the random
sequences even caused the application
to crash when it tried to assign Serena
Williams and her sister Venus to the
same tournament.
Create Test Report
The test results and defect reports were
then comprised by the test manager into a
test report using the reporting capabilities
of TEMPPO Test Manager. The test
manager also included information
about the model coverage, especially the
percentage of covered scenarios, step
connections, and data records. This gave
him an excellent position in the next
meeting with the customer who wanted
to know exactly about the test progress
and was impressed by the accurate data.
Maintain Test Model
Every two weeks, a new version of
the tennis application had to be tested.
Since there were more than 500 test
cases, adapting them by hand for each
version would have been a nightmare.
But thanks to the model-based testing
approach, the test analyst only had
to adapt a few building blocks in the
model and re-generate the scripts. Even
when suddenly a major change of the
rules for Grand Slam tournaments wasannounced by the International Tennis
Federation, it sufced to adapt a few
steps and conditions.
Conclusion
In this article, we presented a workow
for MBT that considers the traceability
between requirements, the test model,
the GUI, and the test cases. The
workow is supported by the Atos
TEMPPO framework which has theadvantage of using a simple model
notation specically designed for the
purposes of testing. Also, instead of
generating only abstract keywords from
the model, the tool produces complete,
executable test scripts for tools like HP
QuickTest Professional (QTP).
The generation of test cases is
systematically increasing the
effectiveness of nding relevant defects
in the SUT. Thus, the condence in
the product quality is increased and
the number of unplanned releases and
hot xes can be reduced signicantly.
The effort required for dening and
maintaining the test cases can be limited
to a reasonable level.
Applying MBT using TEMPPO
Designer has become a major focus of
Atos. A global Competence Center for
MBT has recently been established with
the task of promoting the topic within
the company. Our future plans include
a closer coupling of TEMPPO Designer
with other test tools. The newest version
already supports the import of object
repositories from QTP. The next step will
be the import of recorded actions that
can then be converted into parametrized,
reusable building blocks.
Acknowledgment
The research herein is partially
conducted within the competence
network Softnet Austria II (www.soft-
net.at, COMET K-Projekt) and funded
by the Austrian Federal Ministry of
Economy, Family and Youth (bmwfj),
the province of Styria, the Steirische
Wirtschaftsfrderungsgesellschaft mbH.(SFG), and the city of Vienna in terms of
the center for innovation and technology
(ZIT).
Some parts of this work are based on
an article we published in the March
2012 edition of the Testing Experience
magazine.
References
[Graham et al. 2012] D. Grahamand M. Fewster, Experiences of Test
Automation, Chapter 9, p. 155-175,
Addison-Wesley, 2012.
[Beer & Mohacsi 2008] A. Beer
and S. Mohacsi, Efcient Test
Data Generation for Variables
with Complex Dependencies,
Proceedings of the IEEE ICST,
Lillehammer, Norway, 2008.
[Mohacsi & Wallner 2010] S.
Mohacsi and J. Wallner, A Hybrid
Approach to Model-Based Random
Testing, Proceedings of the Second
International Conference on
Advances in System Testing and
Validation Lifecycle, 2010, pp. 10-
15, 22.-27. August 2010.
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
29/44www.automatedtestinginstitute.comOct/Nov 2012 2Automated Software Testing Magazine
CrowdamationCROWDSOURCEDTESTAUTOMATION
Its OK To Follow the CrowdInquire at [email protected]
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
30/44
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
31/44www.automatedtestinginstitute.comOct/Nov 2012 3Automated Software Testing Magazine
We all want to enjoy the benets
offered by automated testing, but
somehow the workload, project
stress or other parameters interfere, and weare not able to turn our perfect plan into
practice.
This article, based on practical experiences
in insurance companies in Germany and
the United Kingdom, provides one possible
approach to addressing these challenges to
test automation.
An alternativeapproach
Sometimes with automated testing one gets
carried away with optimizing the code for
the test tool or nding more efcient ways
to run a script, but we should never lose sight
of the aim of testing itself. That aim is to
improve the quality of the software, and to
minimize the number of defects found in the
live environment.
With this in mind, our automated testing may
benet greatly from more involvement by the
front-end user.
In a perfect world an automated test analyst
is provided with manual test cases that may
be used as a basis for test automation. All
the test cases are quality checked and can beused 1:1. In my experiences, the real world
presents a different situation. There often
only exists a rough idea as to which tests
should be automated, and this idea has to be
used as the basis for building the automated
tests step-by-step. For this reason, knowledge
from the front-end user is important to the test
automation process. In Dorothy Grahams
and Mark Fewsters book Test Automation,
I described how the lack of user knowledge
causes several show stopping mistakes in
test automation. Following is an excerpt from
my chapter in the book:
Typical beginner mistake: we just assumed allpeople are working with the same screen resolution.This, of course, was not the case. Automated testcases passed, because we had on our screen thescreen resolution we assumed everyone else inthe world has. But the real user was not able tonavigate to some pages, because a different screenresolution caused graphical items to be drawn offscreen and therefore become invisible to the user.
Very funny was the fact that the submit buttonwas missing on a specifc screen resolution. Well,not really funny for the user world and this wasdefnitely a lesson learnt.
Even though test automationhas developed a lot over the last years,
we are still at the beginning of an
exciting journey.
What goes wrong in modern day test automation
implementations? Where is the room for
improvement? Are we just wasting our time? These
are questions we need answered. Developing
effective test automation packages is a complex
process. There are very good best-practice guides out
there, many good tools, different kinds of models,
and still our test automation pack is not at its best.
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
32/4432 Automated Software Testing Magazine www.automatedtestinginstitute.com Oct/Nov 2012
Where is the front-end user
usually involved?
1. Acceptance testing performed by
the customer is known as a User
Acceptance Test (UAT). A UAT
is composed of a selection of
high priority tests and is often animportant nal step for ensuring an
application is ready to be rolled out.
I have worked for several insurance
companies and a UAT was usually
performed at the end of every release.
If major problems are found during
a UAT, it can be extremely costly to
the organization. Hence, involving a
user for the rst time at this stage is
way too late.
2. User stories are created at the
beginning of the developmentprocess, and are a good attempt
to involve the front-end user very
early. Generally there is room for
improvement in terms of the quality
of user stories and they tend to require
many reviews and amendments. User
stories are often documented by a
test analyst representing a front-end
user, however, which means that
the user may not actually be getting
very involved early in the process.
This also means that pertinent steps
in the workow unknown by the test
analyst may fail to make it into the
user stories.
Approach
A new and maybe unusual approach
to getting the front-end user involved
earlier in the testing process is job
rotation. Wikipedia quotes job rotation
as follows:
Job rotation is a management technique
that assigns trainees to various jobs and
departments over a period of a few years.
Surveys show that an increasing number
of companies are using job rotation to train
employees. There are both positive and
negative effects involved with job rotation
that need to be taken into consideration whena company makes the decision to utilize this
technique.
One of the main reasons job rotation has
not been employed for automated testing
is that workers are specialized due
to technology, and automated testing
requires highly skilled workers. For
example, a claims manager from a claims
department in an insurance company
cannot program an automated test
script without some extensive training.
Therefore, getting a non-technicalperson involved in an inherently
technical task such as test automation
could be extremely expensive, unless
done using the job rotation approach.
The job rotation scenario well be using
in this article rst examines a front-
end user conducting test automation
tasks, then examines a test automator
conducting front-end user tasks.
Front-end User JobRotation Scenario
Lets look for example at a front-
end user - i.e. claims manager in an
insurance company - who is rotated
onto an automated test team for a couple
of weeks. The training for coding and
automated test execution takes too long,
and is therefore not a suitable task. A
more tting task is reviewing old test
automation packs. Usually there are alot of meetings and reviews involved in
making sure the pack gets altered and
updated correctly. Although reviewing
TASKTIME-
CONSUMINGFACTOR
CAN TIME BEREDUCED
WITH MORE
CAN IT BEEXECUTEDBY A JOB
ROTATOR?
Automate test
casesYes No No
Review old test
packsYes Yes Yes
Running test cases No No Yes
Analyzing script
error messagesYes No No
Review manual
test cases and
prepare for auto-
mation
Yes Yes Yes
Writing user guides Yes No No
TABLE1: TASKSOFANAUTOMATEDTESTER
...a different perspective can help
make the process more effective.
-
8/10/2019 2012.11 AutomatedSoftwareTestingMagazine November2012
33/44www.automatedtestinginstitute.comOct/Nov 2012 3Automated Software Testing Magazine
these test packs does involve minimal
training, the overall reduction in time
that results from having a business
person closely involved will be worth it.
Running test cases is not very time
consuming, but it is also a task which
can be taken over by a front-end user.
Reviewing manual test cases and
preparing them for automated test is
another ideal task. The front-end user
can verify the test steps and prepare
the test scenario so