e-Informatica Software Engineering Journal, Volume 5, Issue 1, 2011, pages: 51–63, DOI 10.2478/v10233-011-0030-4

Experience with instantiating an automatedtesting process in the context of incremental and

evolutionary software development

Janusz Górski∗, Michał Witkowicz∗

∗Departament of Software Engineering, Gdansk University of Technologyjango@pg.gda.pl, miwi@eti.pg.gda.pl

AbstractThe purpose of this article is to present experiences from testing a complex AJAX-basedInternet-system which is under development for more than five years. The development processfollows incremental and evolutionary lifecycle model and the system is delivered in subsequentreleases. Delivering a new release involves both, the new tests (related to the new and/or modifiedfunctionalities) and the regression tests (after their possible refactoring). The article positionsthe testing process within the context of change management and describes the applied testingenvironment. Details related to documenting the test cases are given. The problem of automationof tests is discussed in more detail and a gradual transition from manual to automated tests isdescribed. Experimental data related to the invested effort and the benefits resulting from testsautomation are given. Plans for further development of the described approach are also presented.

1. Introduction

Testing can serve both, verification and valida-tion purposes. It can generate considerable costs(according to [1] it is not unlikely that testingconsumes 40% or more of the total developmenteffort) and therefore defining testing objectivesand strategy belong to the key decisions to bemade in a software development project. Thisincludes not only the answer to how much wewant to spend on testing but also what, when andhow to test to maximize benefits while keepingthe costs in reasonable limits. It is well knownthat testing cannot prove absolute correctness ofa program [2] and can consume (practically) un-limited resources. On the other hand, testing canprove (and does it in practice) that the programincludes faults. Therefore, an important crite-rion to be used in practice is to drive the testingprocess in a way that increases the likelihood offault detection. In practical situations it means

that we are interested in such test cases selectioncriteria which result in tests that have the highestpotential for detecting significant faults withingiven time and budgetary constraints. Signifi-cant faults are these which have highly negativeimpact on the program behavior in its target en-vironment. For instance, a fault which is never orvery rarely activated within a given operationalcontext of a program is less significant than afault which is activated in (almost) every usagescenario, unless the former fault is consideredto be ‘catastrophic’ in which case it should beeliminated even for a very high price.

Current tendency is that the number of ap-plication programs developed for use in Internetincreases rapidly. The complexity of such appli-cations grows. As they are often used to supportbusinesses, commerce and other services, the ex-pectations related to their dependability increase.This requires employment of more sophisticatedassurance processes. To maintain their usabil-

c© Copyright by Wrocław University of Technology, Wrocław 2011

52 Janusz Górski, Michał Witkowicz

ity, such programs have to follow the evolutionof their requirements and target environments.Therefore, in addition to the common correctivemaintenance practices, they have to be subjectedto the perfective and adaptive maintenance pro-cesses [3]. A program, instead of being consideredas the final result of its development process, isbetter understood as an object which appears intime in subsequent incarnations, following an evo-lutionary process. An important question then isnot only how to ensure the expected dependabil-ity of the program, but in addition how to main-tain the assumed level of guarantees throughoutthe program evolution. Without such change inthe attitude we can end up with a product whichquickly disintegrates in time and in consequence,the users lose their interest in it.

Changes of a program undermine the confi-dence in its reliability. Even a very small change(for instance one single bit) can result in a dra-matic loss of reliability (for instance, the pro-gram stops to work entirely). A widely adoptedsolution is to have regression testing in place,meaning that the program is subjected to a des-ignated set of test cases each time it has beenmodified. With careful selection of these testcases, a positive result of all tests supports theclaim that the reliability of the modified programremains unchanged. Depending on the scope ofthe changes involved, the regression test suite isbeing modified to reflect the program evolution.

In present business environments, time isconsidered a very valuable asset and shorteningtime-to-market of new products and services isamong the highest priorities. This is reflected inthe growing popularity of incremental deliveryof software products, where the product is deliv-ered as a series of subsequent increments, eachincrement representing some added value for theusers. From the software assurance viewpoint weare facing here the same situation as in softwareevolution. The product is growing and each sub-sequent increment is expected not to decrease itsdependability comparing to its predecessor.

Combining incremental development withsoftware evolution is the common situation whichcalls for strengthening software assurance prac-tices and building them into the software process

from the very beginning. This has been reflectedin the agile approaches to software developmentwhere testing is brought to the front of the pro-cess by integrating it with the requirements spec-ification (specifying requirements by test cases)and running tests as soon as the first incrementsare coded.

The purpose of this paper is to present acase study involving instantiation of an auto-mated testing process during development of asubstantial Internet application TCT [4, 5] basedon AJAX technologies. The application followsthe evolutionary and incremental developmentprocess model. The data presented in this paperwere collected during the period of more thanfive years of development and evolution of TCT.

The application is based on AJAX technolo-gies [6, 7]. AJAX radically changes the protocolof interaction between the Internet browser andthe server. The granularity of exchanged mes-sages drops down from a full page to a pageelement and such elements are exchanged asyn-chronously in a way which is highly transparentto the user. The result is that the user has afeeling of working interactively without delayscaused by page reloading.

Moving to the AJAX technologies has sig-nificant impact on program testing. Numeroustechniques, e.g. these described in [8, 9] becomenon-applicable or can be applied only partially.In Table 1 we assess some of them following [8].

Table 1 demonstrates that only selected tech-niques (in particular, these based on the so calledtest recording and replaying) and some tools ofthe xUnit type (e.g. squishWeb or Selenium) aresuitable for testing AJAX-based software. Othertechniques are not applicable or require signifi-cant modifications.

The TCT application considered in this pa-per is following the incremental and evolutionarydevelopment process. The objective is to deliversubsequent increments while maintaining a sat-isfactory level of reliability and keeping the de-velopment effort in reasonable limits. To achievethis we had to invest in automation of tests whichproven to be particularly beneficial.

The paper first sets the scene, explaining theobject of testing, its architecture and the pro-

Experience with instantiating an automated testing process in the context . . . 53

Table 1. Web testing techniques applied to AJAX-based applications [8]

Testing Adequate Problems ToolsModel-based no Web models extracted are par-

tial; existing Web crawlers are notable to download site pages

research

Mutation-based no Mutant operators are never beingapplied to client Web code; theapplication of mutant operatorsis difficult

not-existing

Code Coverage partially It is difficult to cover dynamicevents and DOM changes; cov-erage tools managing a mix oflanguages are not available

Javascript: Coverage val-idatorJava: Cobertura, Emma,Clover, etc.Languages mix: not avail-able

Session-based no It is impossible to reconstruct thestate of the Web pages using onlylog-files

research

Capture/Replayand xUnit

yes Javascript, asynchronous HTTPrequests and DOM analysis arenot always supported

not ok: Maxq, HTTPUnit,InforMatric, etc.partially ok: Badboy,HTMLUnit, etc.ok: squishWeb, Selenium,etc.

cess of its development. Then we describe thetesting process and explain how it developed intime. Next, we present subsequent steps towardsautomation of the testing process together withthe collected data which characterize the perfor-mance of the automated tests. We also highlightthe main factors which, in our opinion, had themost significant influence on these results. Inconclusion we also present the plans for furtherimprovement of the process as the applicationgrows and its usage context becomes richer.

2. Object of testing

The object under test was an Internet applica-tion called TCT, being a part of the Trust-ITframework [4, 5]. The objective of Trust-IT isto provide methodological and tool support forrepresentation, maintenance and assessment ofevidence-based arguments. From the user’s per-spective, TCT implements a set of services. They

cover eighteen groups of key system functionali-ties accessible by a user (listed in Table 6). Multi-ple instantiations of the services are deployed fordifferent groups of users (presently we run somesixteen such instantiations). Each instantiationsupports different ‘projects’ which are used bydifferent users. The users work independently andin parallel, accessing the services by an Internetbrowser.

Trust-IT together with the TCT tool has beendeveloped in a series of projects supported byEU 5th and 6th Framework Programmes1. Thetool was already applied to analyze safety, secu-rity and privacy of IT systems and services fromdifferent domains, including healthcare, automo-bile and business. Presently TCT is being usedto support processes of achieving and assessingconformance to norms and standards, in particu-lar in the medical and business domains (moreinformation can be found in [10]). Further appli-cation domains are being investigated, includingmonitoring of critical infrastructures.

1 5th FR UE Project DRIVE, IST-12040, 6th FR UE Integrated Project PIPS IST-507019, 6th FR UE STREPProject ANGEL IST-033506

54 Janusz Górski, Michał Witkowicz

2.1. Application architecture

The architecture of TCT follows the richclient-server model which is illustrated in Figure1 [11]. The model includes three layers: databaseserver PostgreSQL [12], application server JBoss[13] and a client written in JavaScript [14] in ac-cordance with AJAX (Asynchronous JavaScriptand XML) [6]. The client is automatically up-loaded to the browser of the end user.

Figure 1. The architecture of TCT

The lowest layer (the database) implementsthe business logic as a set of stored procedures.The intermediate layer is based on J2EE [15] andlinks the database with the client. Communica-tion between these layers is based on web servicesand SOAP (Simple Object Access Protocol) [16].

Each layer is a complex program: presentlythe database stores some 135 procedures, theintermediate layer and the client layer have 141classes and 139 classes correspondingly. The twohigher layers include additional structure (inter-nal layers) to provide for better understandabilityand maintainability.

2.2. Increments and evolution

Following earlier prototypes, the development ofthe present TCT tool was initiated in December2005. At the beginning, the objectives and thescope of the required functionalities were identi-fied and the delivery of the functionalities wasplanned as a series of increments. The intention

was to follow the incremental development model[17, 18] by producing deliverables in subsequentiterations, where each iteration involves require-ments gathering and analysis, design, implemen-tation and testing. In effect, each iteration resultsin the release of an executable subset of the finalproduct, which grows incrementally from itera-tion to iteration to become the final system.

However, it has been soon realized that fol-lowing the incremental model in a strict senseis not relevant. As TCT was being developed inthe context of on-going research and the researchobjectives (and consequently the results) wereshaped and scoped by the results of the experi-ments and the feedback received from the partic-ipants of these experiments, the requirements forTCT were changing, following a learning curve.Therefore we had to switch to a more complex,incremental and evolutionary development model,which can be characterized as follows [19]: it im-plies that the requirements, plans, estimates, andsolutions evolve and are being redefined over thecourse of the iterations, rather than being fullydefined and ‘frozen’ during the major up-frontspecification step before the development iter-ations begin; evolutionary model is consistentwith the pattern of unpredictable discovery andchange in new product development.

From the testing perspective the incremen-tal and evolutionary development process posesimportant challenges: (1) to maintain reliabilityof the subsequent increments it was necessary tohave the regression testing in place from the verybeginning, and (2) to follow the evolution of theapplication, the set of regression tests could notbe treated as monotonic (i.e. extended by the newtest cases reflecting the current increment whilepreserving the already used test cases); insteadit had to evolve following the changes introducedto the already existing functionalities.

So far, TCT has been delivered in eight re-leases: four of them were related to the majorchanges and the remaining four were related tothe localized changes of the application. To reflectthis scope of change, the former are also calledmain releases and the latter intermediate releases.The difference between the two is based on the as-sessment of the impact of the introduced changes

Experience with instantiating an automated testing process in the context . . . 55

Figure 2. The history of development of TCT

Figure 3. Testing in the context of change management process

and the resulting need for the thoroughness of re-gression tests. The history of TCT releases is illus-trated in Figure 2. The releases are numbered bytwo digits, where the first one denotes the numberof the main release, whereas the second denotesthe intermediate release related to the main one.

Throughout the whole evolution, the archi-tecture of the application remains stable. Thechanges were mainly related to functionality(adding new functions, changing existing func-tions, removing obsolete functions) and to theapplied technologies.

3. Testing in the context of changemanagement

The testing process is embedded in the broaderprocess of change management. The model of thechange management process is shown in Figure 3.

The process implements mechanisms for re-porting the issues related to the application and

for maintaining a related repository of issues.The repository is based on the MantisBT plat-form [20]. The repository is being periodicallyreviewed, which results in selecting the issuesto be resolved. The selected issues are assignedpriorities and then are grouped together in pack-ages, each package containing the issues whichcan be solved by a common maintenance action.Not all issues require changes in software, forinstance some are related to the way a user in-teracts with the system and can be solved byimproving user’s documentation and training.The issues which call for software changes aredealt with in the next steps of the process. First,the necessary changes are subjected to planningand then an explicit decision about implement-ing the plan is bring made, after assessing theresources needed for such implementation andthe resulting impact. The plan for the changeand the change implementation process provideinput to the step of updating the test cases. Thenew and updated test cases are then stored in

56 Janusz Górski, Michał Witkowicz

the tests repository forming the new suite of thetests to be performed. The repository of testcases is implemented using Subversion (SVN)[21]. After implementing the change, the testsare being applied.

For a given release of the application, depend-ing on if it is related to a main release or to aintermediate release, the scope of related testingdiffers. In the former case, the tests include both,the tests covering the new functionality and thefull set of regression tests. In the latter case, thetests cover the new/changed functionality andonly a subset of the regression tests is included.Limiting the scope of regression tests for a local-ized change is based on the assumption that theimpact of the change is limited and is unlikelyto affect the whole scope of the functions.

The testing process is illustrated in Figure4. It starts with building the current repositoryof test cases which is a subset of the tests main-tained in the main repository of test cases. Theselection criteria depend on if we are testing amain or an intermediate release of the system.In the former case the current repository is sim-ply a copy of the main repository. In the lattercase it is a proper subset of the main repository.Then, the selected test cases are run and theresults are collected in the repository of test re-sults. The results of the tests are then analyzedand assessed. In case of failed tests, there aretwo possibilities: (1) inserting new issues to therepository of issues (for further processing) or(2) updating the current repository of tests. Theformer possibility takes place if removing thecause of the test failure involves a significantchange; then introducing this change is left tothe next releases of the system. The latter possi-bility is in place if (due to a negative test result)an immediate and localized change is introducedto the software which affects the correspondingtests kept in the current repository. The decisionon which alternative is chosen is taken by thetests manager and involves consultation with therepresentatives of key groups of the users.

Figure 4. Two phase testing process

After assessing the test results, the decisionis being made concerning the continuation of thetesting process. If all the issues detected duringthe previous phase are deposited to the reposi-tory of issues, the testing process stops and thenext release is delivered to the users. If however,some immediate changes were introduced to thesoftware, the tests kept in the current repositoryare executed again.

3.1. Specification of test cases

Each test case is represented in accordance witha predefined structure. It includes the followingelements:– Identifier - a unique name of the test case.

The name suggests the tested functionality;– Author - identification of the person respon-

sible for this test case;– Feature - brief, intuitive description of the

tested feature;– Scenario - description of the related testing

scenario including:– summary of the scenario,– description of the initialization phase,

Experience with instantiating an automated testing process in the context . . . 57

– the list of actions necessary to completethe test case,

– description of the closing phase of the testcase;

– Success criterion - specification of how toassess that the test case completed success-fully.An example test case specification is given

in Table 2. The objective of this test case is tocheck if authentication of the users assigned todifferent roles works correctly.

3.2. Manual execution of test cases

At the beginning, all test cases were executedmanually. The testers were following the testscenarios specified for each test case. If a testcase does not pass the success criterion, the testerimmediately reports this as an issue to be solved.The issue specification includes details of thesequence of actions which led to the failure.

If the currently reported issue is similar toan issue already reported, then the existing issuedescription is being updated instead of insertingthe new one. Assessment of this ‘similarity’ wasleft to the tester. And this appeared to be a weakpoint in issues reporting. Because analyzing theexisting descriptions was boring, the testers oftendid not go deeply into the details and just con-cluded that the issue has been already reportedand its description did not need any update. Theresult was that sometimes an important infor-mation was not included in the issue description,which adversely impacted fault diagnostics andcorrection.

4. Automation of test case execution

Manual execution of the process illustrated inFigure 4 consumed significant resources for both,complete regressions tests (full suite of regressiontests performed for each main release of the sys-tem) and partial regression tests (for the releasesrelated to the localized software changes). Thishad negative impact on the delivery time of subse-quent releases and slowed down the developmentprocess.

To deal with the above problems we decidedto invest in automation of test case execution. Aseparate testing system was developed based onthe TestNG library [22] and the server and thelibrary offered by Selenium Remote Control [23].The testing system maintains the TCT systemmetaphor which is being updated in parallel tothe subsequent releases of TCT. The metaphoris used to activate these functions of TCT whichare callable from an Internet browser. In con-sequence, these functions are being activatedautomatically.

Each result of an automated test case is struc-tured in the XML format (Extensible MarkupLanguage) [24] and inserted to a file. Such reportsare periodically reviewed and the detected issuesare inserted to the issues repository.

Figure 5 illustrates an example fragment ofthe code implementing the testing scenario shownin Table 2. Method 1 attempts to log a user in,assuming the role ‘viewer’. Method 3 implementsthe logging sequence. It includes a check if therequired element has been visualized. Method 2finalizes the test scenario and logs the user out.

5. Experiences

So far, there were eight system releases (see Fig-ure 2) including four main releases and four inter-mediate ones. Testing of the three first releases(two main and one intermediate) was performedby a team of five testers. The process was manualand consumed significant resources. Testing ofthe second main release was performed by a teamof four testers and with partial automation oftest cases (automated testing did not play animportant role yet and was just experimentedwith). The third main release was tested with30% test cases already automated. The testingprocess involved two testers.

The results achieved were so encouraging thatthe effort in test case automation was increasedwhich resulted in that for the fourth main re-lease (Release 4.0 in Figure 2) the number ofautomated test cases reached 95%. This resultedin radical decrease of the effort to execute test

58 Janusz Górski, Michał Witkowicz

Table 2. An example of test case specification

Identifier SystemFunctions_LoginAuthor Michał WitkowiczFeature Access to system functions for different user roles - system loginScenario Summary:

Make sure that all possible roles have accounts in the system (viewer, developer,assessor and admin). Then login to the system as a user assuming differentroles.

Initialization:Check if login screen is properly displayed (find DOM element withid=“tct_login_data_form”). If not, the user is likely to be logged in; log theuser out (SystemFunctions_Logout). Then, if the login screen is properlydisplayed - do nothing.

Actions:1. Input user login and password.2. Press “log in” button.3. Check if the root node named “Projects” of the projects tree is displayed

(max. waiting time = 10 sec.).4. Log out the user.5. Check if the login screen is correctly displayed (find DOM element with

id=“tct_login_data_form”; max. waiting time = 10 sec.).6. Repeat the steps 1-5 for every possible user role: admin, developer, assessor

and viewer.

Finalization:Repeat the Initialization phase again.

Success criterion Users assigned to all different roles are able to log in to the system

Method 1. test@Test (groups = {"TCT","viewer"})public void test() throws InterruptedException {

cmdContainer.loginPage.logIn(viewerUser.getLogin(), viewerUser.getPasswd());}

Method 2. tearDownTest@AfterMethodpublic void tearDownTest() throws InterruptedException {

cmdContainer.mainMenuBar.logout();}

Method 3. logInpublic void logIn(String userName, String password)throws InterruptedException, SeleniumException {

selenium.type("login_username", userName);selenium.type("login_password", password);selenium.click("button_logIn");cmdContainer.waitForElementPresent("dom=document.

getElementById(’projects_root’).parentNode.childNodes[3].firstChild",cmdContainer.loadPageDelay);}

Figure 5. An example test case code

Experience with instantiating an automated testing process in the context . . . 59

cases and in significant shortening of the deliverytime for this release.

The progress in test cases automation is il-lustrated in Figure 6.

Figure 6. The progress of the test cases automation

Automation of test cases is not free, how-ever. In our experience, one man-day resulted inautomation of approximately five test cases (toautomate 67 test cases we needed 14 man-days).However, comparing to the effort needed for man-ual execution of test cases during testing of subse-quent system releases, this effort was acceptable(more details are given in Table 4 and 5).

When it comes to the cost of maintainingautomated test cases, it depends on a scope ofchanges in the application in the next release. Ob-viously, the maintenance of test cases was moreexpensive while preparing a main release of thesystem. For example, for the fourth main releaseit has been observed that one man-day resulted inapproximately three updated test cases and total16 man-days were needed for tests maintenance(see Table 4 and 5). For intermediate releases,usually only few test cases needed to be updated,and the total effort was significantly less.

The total numbers of test cases for the sub-sequent main releases of the system are given inFigure 7.

Tables 3, 4 and 5 summarize the effort neededfor testing the four main releases and illustratethe gain (in terms of effort) resulting from testcases automation.

Figure 7. The numbers of all test cases for the mainreleases of the system

Test cases were following system developmentand evolution. This was not only because newtest cases were being introduced but also be-cause some test cases became obsolete and someother were merged together or modified. As theresult, periodic refactoring [25] of test cases wasnecessary, to maintain test cases integrity andunderstandability. In particular, such refactor-ing was performed before testing the release 4.0which resulted in reducing the number of testcases from 236 (in release 3.0) to 133.

The coverage by test cases of the key systemfunctionalities of the release 4.0 is shown inTable 6.

Tables 7 and 8 illustrate the numbers of issuesreported during testing of subsequent main re-leases (Table 7) and during exploitation of thesereleases (Table 8).

During the exploitation phase, the reportedproblems were classified in accordance with thedifferent categories of the maintenance objec-tives [26]: adaptive, corrective, preventive andperfective. On the other hand, all issues detectedduring testing were classified as corrective.

Figure 8 compares numbers of different issuesdetected during exploitation and Figure 9 com-pares the issues of corrective category betweentesting and exploitation phases.

From figures 8 and 9 we can see that for re-lease 2.0, testing detected some 30% of correctiveissues whereas the remaining 70% were detectedin the exploitation phase. For the release 3.0 thisproportion looks better and may indicate thepositive influence of tests automation. For therelease 4.0 this proportion looks much better:automated tests detected nearly 100% of correc-tive issues. However, it should be noted that the

60 Janusz Górski, Michał Witkowicz

Table 3. Number of test cases and number of testers involved in testing of main releases

Release # # of test cases # of manual testcases

# of automated testcases

# of testers

1.0 185 185 0 52.0 222 222 0 43.0 236 169 67 24.0 133 6 127 2

Table 4. Distribution of testing effort for main releases

Release # Application de-sign

Specification/Implementation

Maintenance Execution Total effort

1.0 - 5 man-days 0 15 man-days 20 man-days2.0 - 1 man-day 6 man-days 18 man-days 25 man-days3.0 10 man-days 15 man-days 5 man-days 20 man-days 50 man-days4.0 0 14 man-days 16 man-days 3 man-days 33 man-days

Table 5. Distribution of testing effort for automated tests

Release # Application de-sign

Specification/Implementation

Maintenance Execution Total effort

3.0 10 man-days 14 man-days 0 2 man-days 26 man-days4.0 0 14 man-days 16 man-days 2 man-days 32 man-days

Table 6. Test case coverage

Functionality Number of test casesCopy, cut and paste functions 27Accessibility of basic system functions 14Tree of projects and versions 11Management window for administrators 10Tree of trust cases 9Access rights management 9Appraisal mechanism 8Management on links 7Tree element expand and collapse 6Refresh function 5Import and export 5Login and logout 5Management of user settings 4Behaviour of tree icons 4Management of repositories 4Reference nodes 3Report generator 1Traversal tool 1

Experience with instantiating an automated testing process in the context . . . 61

Table 7. The numberof reported issuesduring testing ofmain releases

Release # Issues1.0 152.0 563.0 204.0 63

Table 8. The number of reported issues during exploitation of main releases

Release # Adaptive Corrective Preventive Perfective Sum of issues1.0 1 47 0 38 862.0 4 173 18 75 2703.0 5 39 1 32 774.0 0 1 0 2 3

Figure 8. Issues distribution during maintenance of the TCT system

Figure 9. Corrective issues detected during testing and exploitation

exploitation period of release 4.0 amounts for justthree months (whereas for release 2.0–12 monthsand for release 3.0–10 months). To make thesedata more comparable we calculated the ‘issuesdensity’ metrics (dividing the number of detectedissues by the system exploitation period). Theresult is shown in Table 9.

The above numbers should not be over in-terpreted, however. To assess the influence ofthe testing process on the reliability of the TCTsystem we would have to take into account otherfactors for which we have no quantifiable data

Table 9. Corrective issues density for the last threemain releases

Release # Issues per month2.0 14,413.0 3,94.0 0,33

yet. This involves for instance the influence ofthe ‘size’ system change or the operational pro-file during system exploitation. Nevertheless, atleast one relationship can be clearly observed: asthe number of different users of the subsequent

62 Janusz Górski, Michał Witkowicz

system releases increases, in the light of the datapresented in Table 9, the claim of increasingreliability of the system gains more credibility.

The presented results suggest that there isstill a considerable opportunity to improve theeffectiveness of the testing process. However, wecannot ignore the fact that the defects detectedduring testing are usually the ‘big’ ones (i.e. suchthat disable or significantly disturb the usage ofthe system), whereas the defects detected duringexploitation are usually ‘small’ and rarely preventthe users from using the system. However, it isalso worth to note that the difference between‘big’ and ‘small’ defect is context dependent andwhat is ‘small’ from one user’s perspective (notnoticeable at all or slightly disturbing) can beconsidered ‘big’ from the perspective of anotheruser with different operational profile. Althoughwe did not yet collected enough data to differen-tiate between the different impact of the defects,we are well aware of this problem and intend toexploit it while planning for the next steps oftest process improvement.

6. Conclusions and plans for thefuture

The decision about automation of test cases, inparticular with respect to the regression tests,has been positively verified in practice as it re-sulted in considerable reduction of the tests exe-cution effort and contributed to removing subjec-tivity from execution and interpretation of testsand their results. Despite relatively high cost ofthe implementation and maintenance of auto-mated tests, the total testing effort decreasedand a significant gain in system reliability hasbeen observed.

In our particular case we could observe thatperfective maintenance had a considerable sharein system changes. This is because the systemis being developed in the context of a researchprocess which generates new ideas and discoversnew ways of system usage. It can be expectedthat for systems developed in a business contextthe influence of this type of changes would beless significant.

In the near future we plan for delivering thenext (intermediate) release of the system. Thisinvolves the ongoing effort of designing new testcases (checking the new functionalities) and refac-toring the existing test cases. Nevertheless, thegoal of having 100% regression tests fully auto-mated seems to be not realistic due to the presentlimitations of the Selenium platform [23].

To exploit the potential of improving the ef-fectiveness of the testing process (illustrated inFigure 9) and to take into account the differ-ences between ‘big’ and ‘small’ faults we planfor introducing to our testing process the riskbased selection of test cases [27, 28, 29, 30]. Thiswill take into account different usage scenariosand the consequences related to system failurewithin these scenarios. This information will bethen traced back to the system functionalitiesand reflected in ‘weighting’ of the related testcases. These weights will be taken into accountwhile planning for the test coverage of criticalfunctions.

The next main release of the system will in-volve a significant change of technology, espe-cially in relation to the client layer (see Figure1). To deal with this change we also plan forextending the scope of unit testing of systemcomponents. For better control of tests coverage,mutation testing [31] is also considered.

References

[1] I. Sommerville, Software Engineering, eighth edi-tion ed. England: Pearson Education, 2007.

[2] R. Patton, Software Testing, second edition ed.United States of America: Sams Publishing,2006.

[3] P. Grubb and A. A. Takang, Software Mainte-nance Concepts and Practice. Singapore: WorldScientific Printers, 2003.

[4] J. Górski, “Trust-it - a framework for trust cases,”in Proc. Workshop on Assurance Cases for Secu-rity - The Metrics Challenge. Edinburgh, UK:The 37th Annual IEEE/IFIP International Con-ference on Dependable Systems and NetworksDSN, 2007, pp. 204–209.

[5] J. Górski et al., “Trust-it research project,” in-formation Assurance Group, Gdansk Universityof Technology (18.10.2011). [Online]. Avail-able: http://iag.pg.gda.pl/iag/?s=research&p=trust_cases

Experience with instantiating an automated testing process in the context . . . 63

[6] D. Crane, E. Pascarello, and D. James, Ajax inAction. Manning Publications Co., 2006.

[7] J. Eichorn, Understanding AJAX: UsingJavaScript to Create Rich Internet Applications.Prentice Hall, 2006.

[8] A. Marchetto, P. Tonella, and F. Ricca, “Test-ing techniques applied to ajax web applications,”2007, workshop on Web Quality, Verificationand Validation (WQVV), at the InternationalConference on Web Engineering.

[9] A. Mesbah, “Analysis and testing of ajax-basedsingle-page web applications,” Ph.D. disserta-tion, Delft University of Technology, 2009.

[10] NOR-STA, “Support for achieving and as-sessing conformance to norms and stan-dards,” (04.11.2011). [Online]. Available:http://www.nor-sta.eu/

[11] L. Cyra, J. Miler, M. Witkowicz, and M. Ol-szewski, “Advanced design solutions of a richinternet application,” in Zwinność i dyscyplinaw inżynierii oprogramowania, A. Jaszkiewicz,B. Walter, and A. Wojciechowski, Eds., Politech-nika Poznańska. Poznań: Nakom, 2007, pp.35–47, (In Polish).

[12] PostgreSQL. (10.06.2010). [Online]. Available:http://www.postgresql.org/

[13] JBoss. (10.06.2010). [Online]. Available:http://www.jboss.org/

[14] D. Flanagan, JavaScript: The Definitive Guide.O’Reilly, 2001.

[15] SunMicrosystems, “Developer resources for javatechnology,” (10.06.2010). [Online]. Available:http://java.sun.com/

[16] W3C, “Simple object access protocol,”(10.06.2010). [Online]. Available: http://www.w3.org/TR/soap/

[17] C. Larman and V. R. Basili, “Iterative and incre-mental development: A brief history,” Computer,vol. Volume 36, no. 6, pp. 47–56, June 2003.

[18] PCMagazine-Encyclopedia, “Iterative devel-opment,” (06.04.2011). [Online]. Available:http://www.pcmag.com/encyclopedia/

[19] C. Larman, Agile and Iterative Development: AManager’s Guide. Addison-Wesley Professional,

2003.[20] Mantis. (10.06.2010). [Online]. Available:

http://www.mantisbt.org/[21] Subversion. (10.06.2010). [Online]. Available:

http://subversion.tigris.org/[22] Testng. (10.06.2010). [Online]. Available:

http://testng.org/[23] Selenium, “Selenium web application test-

ing system.” [Online]. Available: http://seleniumhq.org/projects/remote-control/

[24] W3C, “Extensible markup language,”(10.06.2010). [Online]. Available: http://www.w3.org/XML/

[25] A. V. Deursen, L. Moonen, A. Bergh, andG. Kok, “Refactoring test code,” in Proceedingsof the 2nd International Conference on ExtremeProgramming and Flexible Processes in SoftwareEngineering. Sardinia, Italy: XP2001, 2001, pp.92–95.

[26] E. B. Swanson, “The dimensions of maintenance,”in Proceedings of the 2nd international confer-ence on software engineering, San Francisco,1976, pp. 492–497.

[27] R. Black, Advanced Software Testing - Vol. 2:Guide to the Istqb Advanced Certification as anAdvanced Test Manager. USA: Rock Nook Inc.,2009, vol. 2.

[28] R. Black and N. Atsushi, “Advanced risk basedtest results reporting: putting residual qualityrisk measurement in motion,” Software Test &Quality Assurance, vol. Volume 7, no. issue 8, pp.28–33, 2010.

[29] R. Black, K. Young, and P. Nash, “Acase study in successful risk-based test-ing at ca,” (06.04.2011). [Online]. Available:http://www.softed.com/resources/

[30] F. Redmill, “Theory and practice of risk-basedtesting,” Software Testing, Verification and Re-liability, vol. Volume 15, pp. 3–20, 2005.

[31] Y. Jia and M. Harman, “An analysis and surveyof the development of mutation testing,” CRESTCentre, King’s College London, Technical ReportTR-09-06, 2009.