regression testing in a migration project: challenge or nightmare? · 2019-07-25 · regression...

Regression Testing in a Migration Project:

Challenge or Nightmare?

Wim Demey, CTG Belgium, Belgium

Europe’s Premier Software Testing EventStockholmsmässan, Sweden

WWW.EUROSTARCONFERENCES.COM

“Testing For Real, Testing For Now”

Copyright © CTG, Inc.

Organize regression testing in companywide migration project

Challenge or nightmare?

Eurostar 2009 – Wim Demey


Agenda

Context

Project

Challenges

Lessons learned

Questions


Context

Similar project in timeframe of 2 years

Customers within same sector

Organisation

Business

Unit A

Business

Unit B

Business

Unit D

ITS

Infrastructure ApplicationsDivision A

Business

Unit …

Division …

Project


Context

Project mode -> service mode

Project mode

Initiate Define Design Build Implement Close

- Project methodology needs to be followed

- A lot of deliverables (project + specific test deliverables)

+ Test strategy

+ Test requirements matrix

+ Test scenarios

+ Test reports (intermediate – progress – final)

+ Defect management

Project leader

Test coordinator

Test team

Service mode

Business

Unit B

Service

Delivery

Manager

Business

Unit …

Business

Unit A

SLA based support for application(s)

Maintenance budget


Context

Test awareness within organisation Pool of test resources // separate team

Mainly test coordinator profiles

Methodology forces projects to use pool/test team Especially for large projects

Being compliant with methodology requires a lot of time

Internal/external audits

Difference in focus on testing Infrastructure <-> application development

Separate test environments

Development Validation Production


Context

Work station concept -> standard environment By default no admin rights

Default software for every user

Additional software/rights on demand and dependent of business unit

Application software

Default software

Operating System

(SP’s, hotfixes)


Agenda

Context

Project Content

Test approach

Challenges

Lessons learned

Questions


Project >> Content

Initiate

WHO HAS INITIATED PROJECT?

-> Infrastructure

-> Similar project methodology as

application development

WHAT IS PROJECT SCOPE?

-> Default software upgrade

(e.g. Office2007, IE7, SapGui 7.2)

-> Front-end project

-> No upgrade of OS

STRUCTURE OF PROJECT TEAM?

-> Infrastructure architect team has lead

-> Organized in tracks

-> Involvement of test coordinator

(build <> define phase)

WHAT IS IMPACT FOR USERS?

-> All business units (company wide)

-> All workstation types (12.000 – 3.400)

-> Way of migration

(upgrade <> re-install)

Define Design Build Implement Close


Project >> Content

What is new in Office2007?

Ribbon (with tabs -> menus)

Office button

New and xml based formats

-> .docx, xlsx, .accdb

-> possibility to save as PDF


Project >> Content

What is impact of software upgrades? Huge number of Office related applications

Mainly Access applications

Heavy Excel files (VBA, macro,…)

New behaviour of Word <> regulations

Compatability of third party software with new Office formats

New software versions as prerequisite for ongoing projects

Design decisions Out of the box installation <-> Office configured to needs of BU’s

Increase security-level (VBA, macro) in Office and IE

What about applications in project mode?


Project >> Test approach

WHO PERFORMS THE TESTS?

-> SDM + team

-> Business

-> (support by internal test team)

HOW IS TESTING ORGANIZED?

-> Mini test cycle

(ad hoc, on preliminary build)

-> Major test cycle (3m -> 6 months)

(frozen build)

WHICH TEST ENVIRONMENT IS USED?

-> Sandbox idea (=virtual pc)

-> SDM is responsible for installation of

application + apps/test data

WHAT IS ROLE OF TEST COORDINATOR?

-> Follow up testing by SDM’s

-> Follow-up defects/issues

-> Ensure tight schedules are followed

-> Deliver required test documents

MAIN GOAL REGRESSION TESTING

What works today, still works tomorrow on migrated software

SDM = Service Delivery Manager


Project >> Test approach

Overview test environment Sandbox contains clean installation of components

CORP LCORP: XPLV90000xx

SANDBOX - Internet Explorer

- Outlook RepositoryInstall applications

admin

Remote Desktop Connection

Production account

Test account

ValidationProduction


Agenda

Context

Project Content

Test approach

Challenges How to apply test approach?

How to coordinate/motivate SDM organisation?

How to gather/report information -> project?

Does test environment fit the needs?

Can we keep business out of scope?

Lessons learned

Questions


Challenge 1: Test approach

Late involvement of test

coordinator

Very rough estimations

of test effort

Confusion about what is

in the project budget

No official/central list of

applications to test

Project methodology /

test deliverables not

100% fit for project

“Tour of SDMs”

-> Public - individual

-> Get overview of portfolio

-> Planning

-> Roles + responsibilities

-> Deliverables

-> Support

Methodology adapted

-> Pragmatic but structure

is essential

-> Risk application test matrix

-> Test scenario matrix

-> MoSCoW principle

Clear out objections

-> Refine estimations

-> Provide enough information

-> Escalation to steering

committee


Challenge 1: Test approach

Risk & Application Test Matrix

Sap GUI 7.0 IE 7.0 Word 2007 Excel 2007 PowerPoint 2007 Outlook 2007 Access 2007

0001 APPLICATION A SDM X N/A X N/A N/A N/A N/A X

0002 APPLICATION B SDM X X N/A X X N/A N/A N/A

COMPONENTSID Application SDM

Impact Severity Risk

exposure

Priority Overall Status

1 - Low 3 - High 3 Should Not started

3 - High 2 - Medium 6 Must Preparation

Link TSM Comment


Challenge 2: Test coordination

No dedicated test

resources available for

this project

No test cases available No Office2007

knowledge to adapt

application(s)

Daily activities

interfere with test

execution

For some applications

business has to perform

tests

Technical support

-> Analysis of Office2007 issues

-> Document solutions

-> Assist developers during

implementation

-> Functional mailbox for business

Pragmatic follow up

-> Planning on weekly basis

-> Network driven approach per SDM

-> Test scenario template

-> Ad hoc meetings

-> Simple web based defect form


Test scenario matrix

Challenge 2: Test coordination


How detailed must info

be collected/reported?

Main focus on

planning/milestones

No time to read long

reports

Which feedback do we

give to SDM/business?

Several documents as

input for reporting

Challenge 3: Info gathering & reporting

Dashboard

-> Overview of progress (total, per SDM)

-> Overview of defects (type, severity,

trends)

-> Small VBA code to gather

automatically status update

-> All information in one sight

-> Basis for status meeting with project

team

Test news letters

-> Only sent out if relevant & necessary

-> Information about general issues &

solutions

-> Changes in test strategy

-> Status about builds

-> Keep it as short as possible


Challenge 3: Info gathering & reporting


Mismatch between

theory and practice

Test machines needed

in production

environment

Teams create own test

environment

Some applications could

not be tested remote

Need to travel around

with test environment

Security causes a lot of

problems

Physical machines

-> Test accounts

-> Separate container in Active Directory

-> Availability managed by test

coordinator

-> Production like install of applications

-> Staged by project team

-> Pick up & return concept

Security level reviewed

-> Decreased to current level

-> Alternative of exception policy doesn’t

work

-> Digital signing/increased security

outscoped to separate project

Challenge 4: Test environment


Unknown number of

applications developed

by business

Often very critical

applications

No support by

ITS/helpdesk

No experience /

knowledge to solve

issues

Problem of resources

even higher than SDM

teams

Not always clear what is

expected

Challenge 5: Business

Active involvement

-> Same roadshows as for SDM

-> Inventory made + prioritisation

-> Critical applications are taken over by SDM

-> Technical support

-> Test scenario template provided

-> Same defect tool + procedure

-> Business representatives are SPOC


Agenda

Context

Project Content

Test approach

Challenges

Lessons learned Some facts & figures

Conclusions

Questions


Lessons learned >> Facts & figures

Applications 1052 ITS supported applications identified (420)

650 to test (Must, Should, Could)

402 rest (Won’t) -> not impacted by components

All applications tested in major test cycle (3 months) One team had 6 months time due to number of applications (420)

Issues 180 issues reported (both test cycles)

Mainly Excel + IE7 related (70% of all issues)

Compatibility issues with third-party software (e.g. SAS/SAP)

REMARK: only for ITS supported applications

No need to rebuild applications from scratch

Some issues led to delayed migration of users/divisions


Lessons learned >> Conclusions

• No reason to minimize impact, approach, effort

• Risk to forget things (e.g. environment, preparation)

Testing is important in migration projects

• Needs to be involved from beginning

• Explain very clear what is expected

• Collect information from testing (issues, remarks)

Business can’t beignored

• Project is focused on milestones

• SDM teams + business have other prioritiesBuffer function of test coordinator

• All (apps) problems related to migration

• Hardware problems (memory / video cards)Side affects during rollout


Lessons learned >> Conclusions

• Other behaviour of applications

• Rarely asked to test deployment scenario

Test environment deviates fromdeployment method

• Familiar with test deliverables

• Have test focus

• Easier to get status update

Dedicated test resources per team is more comfortable

• Infrastructure project has own characteristics

• Full approach is overkill

• Structured but pragmatic

Need for variant of test methodology

• 2 versions of applications to maintain

• Trigger to migrate earlier

Phased roll out causestransition period


Questions / Remarks

Wim Demey ([email protected])

mailto:[email protected]

regression testing in a migration project: challenge or nightmare? · 2019-07-25 · regression...

Documents