Quality Strategies In AGILE

Gurudas Sable (Test Lead)

Somnath Chaskar (Test Analyst)Tejaswi Landge (Test Analyst)

Shubhashree Senapati (Test Analyst)

Datamatics Global Services Limited

Abstract

The maturity of quality strategies is to provide a gap analysis and evolve toward more effective quality best practices. In Agile testing, Quality Strategy is used to define the quality techniques and standards to be applied, and the various responsibilities for achieving the required quality levels, during the project. Agile practitioners’ endeavour validates their work to the best of their ability. As a result, they are constantly finding ways to bring testing and quality assurance techniques / processes into their work practices as much as possible to improve the overall efficiency and quality. These processes provides quality deliverables within stipulated timeframe and can assure better quality deliverables to the business by mitigating risk factors. Quality of test process is also dependent on Time, Efforts and Coverage and results in direct / indirect implications on the cost of the entire exercise. Therefore, the quality of the information system and the risks for the business are very challenging to determine.

This white paper overviews the test strategies and processes which disciplined agile testers are applying in practice and putting them into the context of the agile projects.

Contents

Abstract.................................................................................................................2

Introduction..........................................................................................................3

Scope.....................................................................................................................4

Important elements of the Quality Strategies......................................................4

Test Review Process..............................................................................................4

Quality Risk Analysis.............................................................................................7

Project Planning....................................................................................................................8

Sprint 0 activities.................................................................................................................. 8

Sprinting 1 to N (During Project).........................................................................................11

RAID Principles....................................................................................................12

Automation Regression Test...............................................................................14

Functional Knowledge Transfer across the Teams.............................................15

Benefits...............................................................................................................15

Statistics..............................................................................................................16

Conclusion...........................................................................................................17

Author Biography................................................................................................18

About Datamatics Global Services......................................................................20

Introduction

Quality is the degree to which the product meets customer’s expectations. To deliver quality product, it must be a process from start to end, not as a goal itself. It must be a part of entire project cycle starting from requirement analysis through implementation. A set of quality strategy are carried out in every sprint and release cycle for projects. It includes Manual, Automation, Reviews, Team Collaboration, and Stakeholders Involvement etc. All these strategies provide guidance to set the objectives, milestones, delivery schedules for the Manager and Stakeholders in terms of reducing risk and tighten release cycles.

Scope

The purpose of this white paper is to bring out some of the best practices of quality strategies that Datamatics is following while working on agile projects. It offers guidance for best practice approaches.

Following are some of the key things that are addressed in this document:

Quality strategies or processes in Agile to ensure high quality testing, zero leakage of bugs, quality products,

Why it’s important, How to implement it in scrum methodology.

Important elements of the Quality Strategies

System Test and UAT Test Review Process Quality Risk Analysis Process RAID Principles Automation Regression Test Functional Knowledge Transfer across the Teams

Test Review Process

Test case review process is an important process to follow in software testing. Test case ensures the functionality mentioned in Software Requirement Specification is covered. To success and completeness of any test cases every test case should be reviewed.

Test Case – Self Review followed by Peer-Review

We have Test Case Peer Review process in place initiated by Datamatics Test Leads. We have peer review checklist for testers and according to the list testers does peer review of the test cases for other testers in the team to verify the test coverage and can provide comments based the knowledge and experience with respect to application and testing concepts. Post peer-review, test cases are reviewed by Test Lead considering acceptance criteria is covered as a part of tests. Test Lead will review tests based on test coverage, end to end flow, test case quality, dependencies, test conditions, test data etc. In the mean time (during test case review) system testers identifies/creates test data based on the different conditions.

We have introduced below checklist for peer-review which is appreciated by customer and being followed in all agile/scrum projects.

This document is a guide for the Manual Checklist as a part of Peer- Reviewing Process.

SR No Test Case Peer Review Check list1 Each point of AC is converted into Test Scenario/Test Case i.e. AC coverage2 Happy path tests/Negative Path tests/Regression test3 Scenarios based on conditions (Permutations and combinations)4 Test In scope/Test Out of scope5 Test Title and Test Summary is self explanatory6 Test step details (end to end flow), spell checks, attachments, notes, examples etc.7 Test Case Readability (Clarity for Language and Grammar, step by step user flow)8 Test Suite/Test Case configuration (environment version)9 Test Case Details like Functional Area, Test Type, Regression Scope/Candidate etc.

10 Test Case Summary : Pre-conditions, AC for respective tests, Environment details, other comments etc.

11 Additional details (If any)

Once peer-review completed, Test Lead reviews test cases based on test coverage, end to end flow, test case quality, dependencies, test conditions, test data etc. We get below benefits out of these processes.

Conceptual and Technical Coverage: Requirements – Usability ensures the coverage from the Concept point of view and Developer ensures the coverage from the

Technical Point of view. The traceability coverage track is assumed by traceability tools such as Quality Centre, MTM etc.

Defect prevention while reviewing test cases: If the developer has the opportunity to check the test cases while implementing code, it is possible that this will help him to realize codes that may cause potential defects.

System Test Case Review (Scrum Team and Stakeholders)We involve UAT early in the sprint. UAT engagement early in the sprint is very useful in terms of end to end testing from business perspective. As a part of the process, test lead conducts session with UAT, including BA/PO, stakeholders and of course scrum team. As a part of

this session, discussions happen within the team including stakeholders based on the knowledge, experience and functionality awareness. BA/PO and UAT will review the scenarios and suggest amendments if required.

This session is useful for development team to understand all possible scenarios to be verified as a part of unit test and to prevent defects at early stage. This session gives an opportunity to system testers to know test coverage and no discrepancy would be there which helps to achieve the goal and all are lying on same page. Based on final agreement with all stakeholders on the requirements and consequently on the test cases written.

UAT Test Case Review

User Acceptance Testing is not trivial or easy. UAT can be one of the most critical and risky types of test on a project, which means that a great deal of care should be taken when planning, executing and evaluating the results of UAT.

Having said earlier, we involve UAT early in the sprint. UAT engagement early in the sprint is very useful in terms of end to end testing from business perspective. Basically, UAT is involved in entire project and not just for test. UAT can also be defining acceptance criteria and can be involved in requirement reviews and inspections.

UAT Test is based on real-world conditions, not user requirements. There are two sides to testing -Verification and Validation. Verification is testing based on specifications and

requirements. Validation is testing based on real-world operational conditions. UAT need to test from both perspectives. Validation is necessary because requirements have defects. As a part of scrum process, we used to have UAT Test Case Review session with Stakeholders and Scrum team conducted by UAT Tester for each sprint. In this session, end to end test cases (based on the business scenarios) are being discussed and finalised based on final agreement with Stakeholders and Scrum team. UAT Test is to determine if the system or application will correctly support the real world conditions.

Now, we will come to the interesting part of this paper which is the most important thing to include in your projects. This best fits with Agile / Iterative development life cycle where team starts with a vision rather than concrete requirements.

Quality Risk Analysis

All risks have both probability and impact. Probability is the likelihood that a risk event will occur, and impact is the significance of the consequences of the risk event. Impact

typically affects the project elements like schedule, budget, resources, deliverables, costs, quality, scope, performance.

The QRA lifecycle is intended to be a continuous holistic process, and each phase of the project lifecycle is to include:

Identification of known and foreseeable issues associated with a product, process, or system

Estimation and evaluation of associated Risks Control of Risks Monitoring the effectiveness of the control Communication of Risks to the appropriate stakeholders

We follow Quality Risk Assessment process extensively for our agile projects. As a part this we are defining test approach to mitigate specific quality risks.

Process Flow to implement Quality Risk Assessment

Project PlanningIn the project planning, TDM or BA provides walkthrough of the project and takes team to entire backlog available. This is a proactive process in which upfront support is required from BA during project planning phase. Later on light touch is expected from BA as this document will be maintained by Test Lead.

As a part of Quality Risk Assessment Workshop, BA supposed to provide high level functional overview of project, EPICS / Features in TFS, User Stories are good to have. It helps Test Lead to identify possible risks associated with product backlog / user stories which then can be discussed with BA, Technical Lead, TDM, Scrum Master Etc. from project perspective. Basically, Risk Communication happens during this phase.

Sprint 0 activitiesIn Sprint 0, we prepare Quality Risk Assessment document based on the project overview provided during project planning. Risks are identified and mapped with User Stories / PBI’s.

A Risk Assessment Matrix (RAM) helps to determine which risks we need to develop a risk response for. The first step in developing a RAM is to define the rating scales for Aggregate Risk, Risk Priority and Extent of Testing. Aggregate Risk is measured using a relative scale.

Mapping Relative Quality Risks to the Extent of Testing

Aggregate Risk Risk Priority

Extent of Testing Comments

Very Low 01 – 05 Report Bugs only

Only report bugs observed in these risk areas.

Low 06 – 10 Opportunistic Leverage other tests or activities to run a test or two of an interesting condition in the related risk area, but only if it involves a very small investment of time and effort and only if the opportunity presents itself.

Medium 11 – 15 Cursory Run a small number of tests that sample the most interesting conditions in the related risk areas.

High 16 – 20 Broad Run a medium number of tests that exercise many different conditions in the related risk areas.

Very High 21 -25 Extensive Run a large number of tests that are both broad and deep, where deep tests exercise many combinations and variations of interesting conditions.

Risk Priority is decided based on Technical Risk and Business Risk. Technical Risk is identified by Technical Lead and Business Risk is identified by BA for User Stories / PBI’s.

Risk Priority Number = Technical Risk x Business RiskTechnical Risk The likelihood of risk to realize. It can be any number from 1 to

5. Where 1 is very low and 5 is very high.Business Risk The impact of a risk on business if it gets realized. It can be any

number from 1 to 5. Where 1 is very low and 5 is very high.

Extent of Testing is calculated based on the nature of the Risk.

Extent of Testing Extensive Tester should run a 'Large Number' of tests that are both

broad and deep, exercising combinations and variations of interesting conditions.

Broad Tester should run a 'Medium Number' of tests that exercise many different interesting conditions.

Cursory Tester should run a 'Small Number' of tests that sample the most interesting conditions.

Opportunistic Testers should 'leverage other tests' or activities to run a

test or two of an interesting condition but invest very little time and effort. Experience based and defect based test design techniques should be followed.

Report Bugs only Testers should not test it but if bugs related to this risk arise during other tests, report those bugs.

Based on the Risk Assessment Matrix (RAM), Risk Priority and Extent of Testing, we decide testing levels to be performed as a part of those risks as below.

Test Level DefinitionsTest Level Owner Objective (s) Key Areas of TestingUnit Development

(Scrum) - Detect Defective Code in units - Reduce Risk of unit failure in Production

Functionality and system resource utilization

Integration Development (Scrum)

- Detect Defect in unit interface - Reduce risk of dataflow and workflow failure in production

Functionality, data quality, unit interoperability and compatibility, performance

System System Tester (Scrum)

- Detect defects in use cases and end-to-end scenarios - Assisting mitigating risk of unmet business requirements in production

Functionality, data quality, performance, reliability, usability, system resource utilization, maintainability, instability, portability and interoperability

Acceptance UAT (Business)

- Demonstrate readiness for deployment - Detect defects in user workflows

Functionality

All above matrices will be used to prepare QRA document in which identified risks are mapped with respective PBI’s and same is shared with BA and TL for Business Risk Assessment and Technical Risk Assessment. BA and TL provide Business Risk and Technical Risk ratings and related comments. At initial level, Risk Mitigation column is set to NO, however it will be set to YES once we deliver the PBI as a part of future sprints. This QRA document will look like as per below.

Sprinting 1 to N (During Project)Testing estimations for PBI’s are provided based on the basis of Extent of Testing which is calculated as per the above matrices. As Risks are mapped with PBI’s as a part of Quality Risk Assessment, Risk Mitigation Status can be reported during sprint review sessions and if risk is mitigated as per the plan, then status would change to Risk Mitigated = Yes for related PBI’s which we deliver for that sprint.

Summary is, using RAM (Risk Assessment Matrix) and Rating Scales, we can analyse the likelihood of each risk event occurring and its impact to determine what Risk Level it is at. This gives the information we need to prioritize our list of project risks.

A Risk Assessment helps to determine if there are any specific types or categories of risks that would require special attention or any risk events that need to be handled in the near-term.

The most challenging aspect of performing a Risk Analysis is defining your rating scales. But once that has been done, you can use them for the duration of the project to effectively manage project's risks in a timely manner.

RAID Principles

Another core part of our strategic process is to follow RAID Principles in our agile process which helps to build concrete quality strategy. RAID is stands for Risks, Assumptions, Issues, and Dependencies.

Assumptions – Assumptions are things we judge to be true, but cannot be certain. We use these assumptions to plan out test strategy. However if they are not true then our strategy could fail. We therefore must test assumptions as soon as possible to see if they are true. If they are true they can be accepted as a fact, if not they become an issue. Each assumption must have; A Test Description An Owner A Deadline for the Test to be completed An Outcome – Fact or Issue

Dependencies – Dependencies are things we need to be present before we can action something that we need to do. We use these dependencies to be in place to carry out the activities that are detailed in the test strategy. They may be in place when we create our plans but if they do not continue to be in place when we carry out our test activities then our strategy could fail.We therefore monitor these dependencies to ensure that they continue to be in place. Each dependency must have;

An Owner A Continuous action to monitor the dependency

Risk – Risks are things that may happen that will stop us performing the activities that we need according to our test strategy. We therefore need to construct a plan on how we will stop this problem from happening in the first place. This plan is called mitigation. Each risk must have; An Owner A Plan of Mitigation

Issues -

Issues are things that have already happened that will stop us performing the activities that we need according to our test strategy. We therefore need to construct a plan on how we will resolve this problem. Each risk must have a resolution plan. An issue will need; An Owner A Plan that will describe how we will resolve the problem A Deadline of when the problem will be resolved by

Below are few examples of RAID principles in detail.

Owner can be TDM, Scrum Master, Test Lead; UAT Lead etc. based on the RAID type and impacted area.

Automation Regression Test

Automation is a critical component in agile projects. It would be impossible to keep pace with the Agile development schedule otherwise. Basically, automation is used to run regression testing during the sprint and prior to any release.

We follow a dedicated automation tester for each scrum or project, who will be responsible for automating the test cases/scenarios. Generally, automation testing lags behind one sprint as automation tester automates system tested PBI’s i.e. functionality has been verified as a part of system test. For automation we identify candidates to be automated. Thumb

rule of automation is that the requirement should be stable so we have to wait for system testing to be completed to start automation. However we have our own Automation Testing Framework which has improved a quality of delivery earlier in the cycle. This helped us to automate test scenarios in the same sprint and we need not to wait until system test done.

Test Lead identifies scenarios to be automated based on the functionality for which system testing is completed. Scenarios are identified depending on Business priority, Testing complexity, scripts reusability, technology adaptability etc. Identified scenarios are assigned to automation tester. System tester’s supports automation tester for test case, test flow, test data etc.

Automation scripts are re-usable and enable faster execution for the most important test cases which facilitates greater test coverage and regression testing. All possible test scenarios get automated and are handed over to centralize regression (Automation) team to include test scripts into regression pack. Automation team adds these new scripts and test all functionalities are working fine pre-release. Once that is done, Regression (Automation) team sends confirmation that all scripts are incorporated as a part of the project into the final automation pack.

Functional Knowledge Transfer across the Teams

Another key are of our strategy is Functional Knowledge Transfer session to be held with the scrum teams post every sprint.

We have Scrum of Scrum meeting in every sprint during which Scrum Masters, Test Lead share scrum activities and functionality they are working on. They also share any impediments and issues they are facing which may impact delivery. As a part of Scrum meeting we also conducts session on Knowledge Transfer across all scrum teams which are working on different functional area. This session happens post every sprint i.e. after delivering the PBI’s in previous sprint. System testers from every scrum team explain the functionality they have implemented as a part of their scrums / projects. Functional / Business processes and Functional / Business scenarios are discussed within teams in terms of gaining knowledge.

Benefit of this session is that every system tester will be aware of what have been implemented and delivered as a part of functional or non functional changes and if they come across the functionality while working on their projects they will be aware of the same and can directly access the related documents or WebEx’s parked at shared path for reference. For complex requirements, teams help each other in achieving the goal. Test Lead monitors the sessions. System testers gain knowledge from other modules and utilize their knowledge on new projects. Test Lead and system testers actively participate in these sessions and spread the application awareness.

Benefits

Better Test Estimation and CoverageWe upfront know extent of testing so can do better estimation of PBIs during sprint planning.

Business Case for NFT We can identify level and types of testing required for project by analyse Quality Risk and can support business case of Non Functional testing (where applicable).

Improved Regression Pack Quality Risks would be a great input to regression team as it will have list of PBIs with Risk Priority Number so test cases can be prioritise in regression pack.

Statistics

Initially there were various gaps in our existing test processes resulting in some defect slippages to UAT, later on after implementing test process and strategies we manage to see the positive results in upcoming releases. Below is the example wherein we can see that we have reduced defect count leaked to UAT.

Kindly refer the below release wise test statistics showing the significant increase in test performance.

In 7th release of Year 2016 there were 9 defects leaked to UAT.

In 3rd release of Year 2017 there were 0 defects leaked to UAT.

Conclusion

Producing high quality software requires thought, discipline, and the ability to adapt and improve processes. It includes:

Clearly understanding the characteristics of high quality software Identifying and applying appropriate quality strategies throughout the software

testing process Building appropriate feedback loops into the process to reduce the time between

introduction and detection of defects A desire to continually raise the bar, to improve internal quality to make it easier

to maintain and improve external quality We produce high quality deliverables Maximum test coverage of functionality at Unit and System Test Level and

maximizing the control on the defects leaked to UAT and production

The costs of poor quality are tangible; they cost customers and money, and ultimately affect the success of the business. Quality is not an optional extra in any part of what you do. If a customer/client experiences a lack of quality in one area they are likely to jump to the conclusion that issues they are experiencing in another area are due to lack of quality rather than their own mistakes.

The result of these bad assumptions is increased calls to technical support, more time spent in technical support investigations and escalations, delays to projects and general loss of confidence in your software. It is far more cost effective to ensure quality is high in all areas of what we are delivering.

Test Strategies and processes have been beneficial to our organisation in terms of increase business and end user trust on the delivery team.

Author Biography

Gurudas Sable Test Lead

Gurudas is working as a Test Lead in Datamatics Global Services Ltd (DGSL). He has more than 9 years of experience in Testing Practices and working with DGSL since last more than 6 years. Being a Test Lead, he independently handling multiple off-shore testing scrums. He has completed a Post Graduate in Computer Science and has completed Diploma in Software Testing. He is ISTQB and CAT certified professional. Before joining Datamatics, he had worked with Agile Financial Technologies & Zeus Systems Pvt Ltd. He has sound expertise in Healthcare Insurance, Life-Non Life Insurance, e-Learning domains.

Somnath ChaskarTest Analyst

Somnath is currently working as Test Analyst in Datamatics Global Services Limited (DGSL). He has more than 6 years of experience in the field of Software Testing and working with DGSL since last more than 3 years. He is handling his work independently and assists new testers in their queries. He holds BSc in Information Technology and has completed certification in ISTQB. Before joining Datamatics, he has worked with Quality Kiosk Technologies. He has good knowledge in Agile Scrum methodology and sound knowledge in Insurance, Banking and E-commerce domains.

Shubhashree SenapatiTest Analyst

Shubhashree is working as Test Analyst in Datamatics Global Services Limited (DGSL). She has 4 years of experience in Testing Field and working with DGSL since more than 2 years. She has completed Bachelor of Engineering in Information Technology. She has achieved CPSAT certification in Automation. She has good knowledge and experience of agile process. Recently she has started implementing selenium automation tool for web based application.

Tejaswi LandgeTest Analyst

Tejaswi is working as Test Analyst in Datamatics Global Services Limited (DGSL). She has 3 years of experience in Testing Field. She has completed Bachelor of Engineering in Information Technology. She has achieved CPSAT certification in Automation. She has good knowledge and expertise of agile process. She has well versed knowledge in Health insurance domain.

About Datamatics Global Services

Datamatics is a trusted partner to several Fortune 500 Companies globally for managing their

End-to-end Application Life Cycle & Business Critical Processes. Datamatics is a global

Information Technology (IT) and Knowledge Process Outsourcing (KPO) organization focused

on delivering smart, next-generation business solutions that help enterprises across the world

overcome their business challenges. These solutions leverage innovations in technology,

knowledge of business processes, and domain expertise to provide clients a competitive edge.

As a strategic partner, Datamatics helps its clients improve their business competitiveness and

achieve operational efficiencies. The benefits delivered by the Datamatics’ unique solutions

include accelerated time-to-market for new products and services, maximized productivity

and efficiency in business processes, shorter turn-around times for service delivery and lower

total cost of ownership.

Datamatics’ next-generation solutions and the services span:

A cross section of industries including Banking & Finance, Insurance, Publishing,

Manufacturing, Research and Retail.

Varied business processes that transcend F&A, Enterprise Document Management,

Portal Management, Publishing Solutions, Application Development, Support & Testing,

Data warehousing & Analytics, Engineering & Embedded solutions, Billing solutions,

Enterprise Content solutions.

Datamatics has a geographically diverse portfolio of Fortune 500 clients with a footprint

across four continents- Americas, Asia, Australia and Europe.

THANK YOU!