effective regression testing strategy

1. IntroductionThis paper focuses on the area ofregression testing. An industry-widedefinition of regression test is “retest-ing of a previously tested program fol-lowing modification to ensure thatfaults have not been introduced oruncovered as a result of the changesmade.”

We have adapted this slightly for thetype of testing we perform to give thefollowing definition: “a test designed toconfirm that an existing feature stillperforms correctly after any softwaremodifications.”

Both these definitions provide the read-er with a sense of the importance ofregression testing. It is of no benefitproviding your valued customer withlots of new and exciting features if youmanage to break features they havebeen using successfully for the lastnumber of years. In fact, some feedbackreceived from a customer several yearsback indicated that they expect to haveproblems with new features when theyfirst get a new release of software, but

that they cannot tolerate existing fea-tures not working on the new release.So, regression testing is recognised asbeing very important, yet it oftenremains the test phase given the leastattention in the planning of test activi-ties. Too often, organizations take theeasy way out and regression test phasessimply consist of reruns of existing testsuites in the hope that these will catchall the problems. This approach has anumber of potential flaws:

• If you are dealing with a large com-plex system, you can end up withthousands upon thousands of regres-sion tests as you continue to add fea-ture test suites from previous releas-es to the regression program for yourcurrent release. This is all well andgood if you have a fully automatedand stable test environment, whichallows you to run all these testsquickly without using a lot ofresources.

• You may not have the required cov-erage with your existing test suites tofind problems being introduced as

new features or other code changesare being added.

• Although you may build up a libraryof regression tests over time, and thislibrary can be quite extensive, theycan also prove troublesome as theycontain both redundant test suites &test cases which themselves containerrors. There can be a huge overheadin maintaining this library of regres-sion suites and test cases.

So, an unplanned approach to regres-sion testing can lead to inefficient use ofresources. If you take the time to planyour regression test program, there are anumber of questions you need to ask:

• What do I need to test?• How long is it going to take?• What resources are required?• What tools are required? • Are there any other special

requirements?

The first question on this list is the mostimportant one to answer, as it will helpanswer the remaining questions. Apart

September/December 2001 http://www.testinginstitute.com Journal of Software Testing Professionals 1

A Strategy for Designingand Executing an

Effective RegressionTest Program


from being the most impor-tant question it is also themost difficult to answer.

2. Understanding OurProblemWe are involved in sub-sys-tem and system testing ofsoftware releases for aMobile Switching Centre(MSC). We test in both anMSC-only configuration andalso in a full cellular systemconfiguration with other net-work elements. Over the past3 years the complexity of thefeatures being developed forthe MSC has increasedtremendously as we move away fromMSC-only features to features that spanseveral network elements. Our regres-sion testing is a mixture of automated(30%), semi-automated (30%) andmanual (40%).

On our current release of MSC soft-ware, we decided to spend some timeanalysing the way we approached ourregression testing. In general we hadbeen taking an ad-hoc approach to plan-ning our regression test program forprevious releases and our analysisuncovered the following facts:

• We are dealing with a legacy system;there has been a lack of consistencyin the way features developed overthe years have been documented andcontrolled. This has led to a lack oftraceability from existing testcases/suites to old feature/functionalrequirements.

• Consistently on all our releases, ourProblem Report (PR) arrival rateshowed a straight-line increase withno levelling off towards the end ofour cycle as shown in Figure 1. A‘normal’ PR arrival rate is one whereyou are finding the majority of yourproblems early in your cycle, thus

allowing them to be fixed early. Thisthen reduces the risk of impactingquality with fixes having to be madeto the release late in the cycle prior tocustomer deployment.

• We had an increasing number ofregression tests per release, as wesimply added new tests to cover newfeatures developed on the previousrelease we had tested. This increas-ing number of tests had turned ourregression cycle into a 12-weekinterval on average and this wasincreasing with every release. In fact,over a 6-year period our regressioncycle had increase from 6 weeks tothe current average of 12 weeks.

• There was a lot of frustration amongtest engineers, as they were simplyrunning regression tests for no appar-ent reason except we did it on the lastrelease so must do it again! Moreoften or not, these tests would noteven find any problems, so the frus-tration increased.

• Over the years we have built up anextensive library of regression teststhrough the addition of tests to covernew functionality. Even with thislibrary of tests, we can never be

100% sure that we havefull coverage, as themethod of selecting theregression tests waspurely to try and coverthe basic functionality ofthe MSC.

We have been perform-ing Escaped DefectAnalysis & Root CauseAnalysis on problemsthat escape to our cus-tomers, and this showedus a number of things inrelation to our regressiontesting:

• We were good at testing new featureand enhancements but problemsescaped in other areas that shouldhave been covered by our regressiontesting.

• By simply re-running existingregression suites, we could not hopeto catch most of the problems thatwere escaping. Regression suitesoften covered the basic functionalityof a particular feature, so if a codechange resulted in an impact outsideof this basic functionality, we did nothave the required coverage to catchthe problem.

• Quite often, the solution to theEscaped Defect Analysis was just toadd a test to provide coverage for theproblem that escaped, thus increas-ing the number of regression testseven further. Clearly it would be bet-ter to try and detect the problem inthe first place rather than adding atest just in case it re-occurred.

3. A Time For ChangeOnce we had done the analysis, it wastime for some change. Quite simply wecould not continue the way we weregoing.

Figure 1. Problem Report(PR) Arrival Rate

0

10

20

30

40

50

60

1 3 5 7 9 11

Week

# PR

s

Normal Actual

30 Journal of Software Testing Professionals http://www.testinginstitute.com September/December 2001

Due to the effort involvedfor older features and func-tionality, we could not goback and resolve the prob-lem of poorly documentedand controlled requirementsand the lack of traceability,so we had to work aroundthis in some way.

Why did our typical prob-lem arrival rate show astraight line for all ourreleases? Some of it wasdue to churn on the release,i.e., features being addedlate in our cycle causingnew problems to be intro-duced and found late in ourcycle or sometimes not atall. However, the majorfinding was that we simplyhad no strategy in terms ofthe way we executed our regressiontests—engineers simply took theirsuites, went into our labs and startedexecuting. We decided that we needed away of prioritising our tests and then toexecute based on the priority.

While looking at the issue of theincreasing number of tests per release, itwas clear that we had no regressionselection criteria, except that we weretrying to show that existing features andfunctionality still worked. We decidedwe needed some way of reducing theamount of tests we were running byidentifying the tests that were actuallynecessary to run. When testing new fea-tures you have requirements fromwhich to work from in developing testsand have a methodology that can be fol-lowed to create these tests. For regres-sion we had nothing, so there was aneed to create some form of require-ments, which we could then use to iden-tify the required regression testing. Todo this we needed specific informationfrom the development organization onthe changes they were implementing in

the software, both for new features/enhancements and any other changesthey were making (problem fixes, re-engineering work etc.).

We used 2 ways of collecting this infor-mation:

3.1 A Regression Impacts Matrix.We created a simple RegressionImpacts Matrix that lists all existingfeatures and areas of functionality. Eachdevelopment team working on a newfeature or enhancement was then askedto complete the matrix indicating thefeatures and functionality both directlyand indirectly impacted by their codechanges. Table 1 shows a sampleRegression Impacts Matrix.

It is quite simple in format and lists allexisting features and functionalityalong the top row with all new featureson the release being listed in the firstcolumn. Where a new feature has anexplicit impact on an existing feature,an ‘X’ is used to indicate the impact.Where the impact is not explicit, an ‘R’is used to indicate a related impact.

Related impacts can be consid-ered to be of lower priority andtests for these features can beincluded in the Secondaryphase (refer to Section 4.).Where there are no impacts,the cell can be left blank.

3.2 A Test Impacts ChecklistWe used an existing TestImpacts Checklist. The devel-opment engineers are requiredto complete this checklist forevery submission of code theymake to the software release.The checklist asks specificquestions that the developerhas to answer, and the testengineers can then use thisinformation to identify regres-sion impacts due to codechanges not associated with anew feature or enhancement

(i.e. fixes, re-engineering). In our case,the checklist was broken down into sev-eral areas as follows:

Area of Impact: We were able to identi-fy 8 main functional areas (includingcall-processing, Billing, Fault Man-agement & Statistics), and the develop-er first has to indicate which area wasbeing impacted by their code change.This will then give the testers an idea ofthe type of regression test they mightrequire.

Description of Impact: They are thenasked to describe the impact in theirown words. This information helps fur-ther define the regression test that mightbe required.

Call Scenarios: Because a lot of ourtesting involves call processing, devel-opers are asked to identify specific call-scenarios that are impacted. Examplescenarios include: three-party confer-encing, call forwarding, and hand-offs.Knowing the call-scenarios impactedallows the tester to select these scenar-ios from our existing suites of regres-

Paging Service

Call Completion

Validation

Bandit Handling

. . . . .

SMS Overload X

Message Retrieval Service

X R

Authentication X X

.

.

.

.

Existing Features & Functionality

New Features

sion tests if they exist, or be in a posi-tion to create new regression tests ifrequired.

Area of Code Change: Developers areasked whether the code change is in afrequently executed leg of code, i.e.,would every call-scenario be affected.This gives testers an idea of the likelycomplexity of the required regressiontest, as a code change in a less frequent-ly executed leg of code may requireadditional effort to create the regressiontest.

Recommended Regression Test: Thedeveloper is asked to recommend whatregression testing would be required toexercise both the code change itself andalso functionality around the codechange. Depending on the developer’sexperience, this information can againbe very useful to the tester.

Features Impacted by Code Change:Similar to the Regression ImpactsMatrix, the developer must indicatewhat features are impacted by the codechange. This will allow the tester tocross-check this with the RegressionImpacts Matrix and they can determineif additional feature-specific regressiontesting is required.

External Message Changes: As we dealwith a product that interfaces to a largenumber of other products, there can beexternal message changes involved inthe code change. If so, this can oftenadd complexity to the regression testingrequired, as it can either mean that:

a) Another product will also need to bemodified and we may need that prod-uct modification to fully regressiontest, or

b) We will need to make modificationsto our test tool in order to update it toinclude the new external messagechange.

By using all the information providedby both the checklist and the matrix,test engineers can then decide on theregression testing required and allowedthem to identify any required test envi-ronment changes. This gives them asense of ownership and purpose, and achance to help design the regressionprogram and make decisions them-selves rather than being simply toldwhat regression suites to execute.

4. The Regression StrategyA strategy was needed to pull every-thing together so that:

a) We were consistent in our approach, b) It was repeatable, andc) It led to focused regression testing

taking place.

We came up with the followingapproach based on grouping our suitesinto 3 main types and then executingour regression tests in 3 phases:

4.1 PrimaryThe Primary group consists of suitesand test cases that are selected for exe-cution based on the following criteria:

• Providing coverage for areas directlyimpacted by new features and otherknown impacted code changes – allthe information provided in theRegression Impacts Matrix and TestImpacts Checklist is used to identifyeither the individual regression testsor suites of regression tests requiredto test the impacted areas.

• A small subset of tests covering basicfunctionality for all features – herewe simply select smaller subsets ofsuites from our regression library. Itis much easier to then maintain thissmaller subset.

The idea behind identifying Primarytest suites is that these will focus on theimpacted areas of the software and so

will increase the possibility of detectingproblems due to those impacts. Thesesuites would also cover overall basicfeature functionality thus giving a highdegree of confidence in the overallquality of the software release.

4.2 SecondaryThe Secondary group consists of suitesand test cases that are selected for exe-cution based on the following criteria:

• Suites that have repeatedly foundproblems on previous releases – thisinformation was readily availablefrom our Final Test Reports for pre-vious releases.

• Suites that cover customer-sensitivefunctionality such as billing and sta-tistics.

4.3 TertiaryThe Tertiary group consists of suitesand test cases that are selected as fol-lows:

• Hot-spot suites – these are selectedbased on problems found by earlierPrimary and Secondary testing, i.e.you might wish to increase test cov-erage due to large number of prob-lems found in specific areas.

• Additional testing due to late codeadditions – once again we would usethe information provided in the TestImpacts Checklist to identify thetests required.

• Clean-run/Sanity testing prior todeployment of software release tocustomers.

Primary and Secondary phase testingcan be identified early in the releasecycle, and can therefore be planned for. Tertiary tests are more ad-hoc andunplanned, but we allow time for themin the schedule when Primary andSecondary testing is complete. We can


then plan and report on their status on ashort-term basis.

5. The BenefitsThis new approach to regression testinghas just been used on our current releaseof software, and we have seen the fol-lowing benefits:

• We have reduced the number of testsin our regression program to 2000from a figure of approximately 6000on previous releases. While this is alarge decrease in the number of testsbeing executed, we were confidentthat there would not be an impact toquality as the strategy allowed us tofocus on covering all basic function-ality, plus the areas where changeswere being made. This was clearly abig improvement on simply rerun-ning tests from previous releaseswithout any sound reason.

• Our problem arrival rate was less lin-ear - Figure 2 shows our CurrentRelease Problem Report ArrivalRate. For this release we had a totalregression test interval of 28 weeks,and the graph shows that we opened30 PRs during the first half of thisinterval. This is 65% of the totalnumber of PRs we opened duringregression test. This compares toapproximately 30% of PRs beingopened during the first half of ourinterval on previous releases. Thisdifference can be attributed to testingof impacted areas earlier in our cycle.

• Test engineers have increased satis-faction because they see a purpose towhat they are doing as they areactively involved in designing theregression program. They also get achance to learn more about the fea-ture and functionality they are test-ing, as they are not executing a suitesimply because they were told to doit.

• We are focusing on high-risk areasearlier, increasing the chances offinding the more serious problemsearlier in the cycle.

• The strategy is flexible and there isscope for additional testing ifrequired.

• There is an increased level of aware-ness from the development commu-nity with regards to the impacts theircode changes can have. We have alsoincreased their interaction withtesters in helping to define theregression program.

• It had led to improved maintenanceof existing regression suites giventhat impacts and feature interactionsare identified earlier and changes/updates to suites can then be plannedduring the test development phase ofthe release.

The current release has just beendeployed on a customer site for the firsttime. When we compare the level ofescaped defects during this deploymentphase to the previous release, we see areduction in the amount of problemsfound, from 21 to 8. This reduction can-not however be directly attributed toour new regression strategy as otherdefect reduction activities have also

taken place on this release earlier in thelifecycle. These activities focused onimprovements to the formal inspectionprocess including: improved inspectionchecklists and training, along with reg-ular analysis of inspection data, leadingto reinspection of out-of-bound inspec-tions.

6. IssuesWhile we have seen many benefits ofthis new strategy, there are some issueswe have noted:

• Depending on the number of testengineers available, it can be difficultto schedule execution of the testsuites in each phase so that eachphase completes fully before theother begins. It is recommended thattest managers aim to complete a cer-tain percentage of the Primary phasebefore beginning the Secondaryphase and similarly with theSecondary phase before moving ontothe Tertiary phase. It is suggestedthat between 80 and 90% of testingin each phase should be completedbefore starting the next phase.

• There is effort involved in planningand executing this regression strate-gy, and this effort needs to be esti-mated, resourced and scheduled inthe project plan accordingly. It needs

32 Journal of Software Testing Professionals http://www.testinginstitute.com September/December 2001

Figure 2. Current Release PR Arrival Rate

0

10

20

30

40

50

1 4 7 10 13 16 19 22 25

Week #

# PRs

to be seen as an integral part of theproject.

• The benefits of this strategy do needto be explained clearly to the devel-opment community so that they cansee the importance of their contribu-tions. We have been moving to anorganizational structure where devel-opers and testers belong to the sameteam. For those organizations wherean independent test team exists, theremay be some friendly persuasionrequired!

7. Summary & ConclusionRegression suites can often containthousands and thousands of tests as theyare built up over releases of software,especially when dealing with a legacyproduct. When looking to focus regres-sion testing, requirements are neededalong with a strategy that is consistent,repeatable and flexible. Developmentand Test Engineers need to be activelyinvolved in defining the requirementsfor the regression program and thendesigning the program that will testthese requirements. This strategy does

not provide for 100% coverage of thesystem being tested. What it does isidentify what regression tests need to beexecuted based on the ‘requirements’we have defined using the strategy, i.e.,the tests in the Primary, Secondary andTertiary Phases. Given the complexityof the system we are testing, we don’tknow for sure that we have exact cover-age but we are sure that we are coveringall basic feature functionality, plusfocusing on areas where changes arebeing made.

For operational reasons, my organiza-tion is moving towards a combineddevelopment/test team. As a result ofthis we are going through some ‘test re-invention’ at the moment. As we havesuccessfully reduced the amount of sub-system & system regression testing weneed by using this strategy, increasedfocus is now being placed on lowerlevel testing such as unit & integrationtesting, and the idea of regression test-ing is now being applied to our integra-tion testing, where we will now regres-sion test individual low-level functionalareas of the MSC. We will now be using

this strategy for our low-level function-al area regression testing. We have seenseveral benefits on the first releasewhere we used this strategy to designour regression program and feel it canbe adapted to suit different applicationsand products.

About the AuthorMartin Gilleran graduated with a B.Sc.

in Electrical/Electronic Engineeringfrom Trinity College, Dublin in 1986and has spent the last 13 years workingin industry. He spent 4.5 years withMeasurex, Inc. as a software develop-ment engineer responsible for develop-ing, testing and installing measurementand control systems for the paperindustry. Mr. Gilleran has spent 8.5years working with Motorola in differ-ent test roles. He has 4 years experienceworking as a test project. He then spent2 years as a Test Manager responsiblefor testing a range of Motorola'sNetwork Products including MobileSwitching Centre (MSC) and HomeLocation Register/Visitor LocationRegister (HLR/VLR).


effective regression testing strategy

Documents