Analytics driven testing

Anusha Mukkavilli – Associate ConsultantNibhanupudi Suguna- Associate Consultant

Abstract:Digitalization is fundamentally changing the business climate. To be successful, organizations must develop and adopt agile business models that respond faster to changing market and economic conditions. To support this business shift, the software industry re-invented itself to allow for fast-

paced software systems evolution. This evolution has been made possible by many innovative technologies, architectures and tools (such as Cloud infrastructures, Open platforms, Building Blocks Method, etc.) combined with new practices and methodologies

In this context, development cycles are increasingly shorter while testing resources often remain at best unchanged. As consequence, testing became a big challenge mainly for large and scale software systems. IT professionals seek for industrial ways to maintain the quality of their systems as whole while they continuously and independently evolve.

Analytics driven testing provides answer to the question, How to be selective in testing without compromising on quality. It provides practical ways to industrialize a software testing approach based on software metrics which consists of gathering data from various metrics as inputs and then building a business intelligent system that shows the test gaps as outputs. The proposed approach which turns the task of measuring test coverage into an easy and instantaneous process proves to be not only effective in saving manual effort, time, and energy but also helps in improving product quality.

The Goal of analytics driven approach is to

Determine a quantitative measure of test coverage, which is imperative to measure quality. Find untested areas of the software application. Identify redundant test cases that do not increase test coverage.

Analytics driven approach in QA

Gives insights on the customer sentiments that help in customer-centric QA Provides insights to start, stop, and prioritize testing Increases testing efficiency and predictability Reduces overall cost due to early defect detection Accelerates time to market

Going beyond the traditional QA methodologies and taking an analytics-based approach has become key factor in the next generation of QA. This helps in predicting the future failure looking at the past data and taking the proactive measures for future.

Analytics driven testing

For many software testing firms to accelerate growth in competitive world it is imperative for them to transform their business models.

This is not possible until and unless the companies reinvent themselves to allow for fast-paced software systems evolution.

This evolution has been made possible by many innovative technologies (Agile, DevOps etc.), architectures and tools combined with new practices and methodologies which drastically shorten their time-to-market

IT professionals seek for industrial ways to maintain the quality of their systems as whole while they continuously and independently evolve

Development cycles are increasingly shorter while the challenges around managing and driving down the cost of testing is growing.

The continued requirement to find efficiencies at every level in QA and Testing remains evident

These are some of the issues that complicate testing process:

Every stakeholder has a different expectation in terms of KPI reports, Test results, Audit reports, Test management reports, other metrics, etc.

Not able to produce desired and on-demand analytical reports because data resides in different sources and the process is very time consuming.

Is the project going in the right track? Who is the right tester for this assignment? In my testing practice, where exactly I am incurring more cost? With the current pace of the project, will I be able to meet the deadline? Project got deviated! What measures should I take to accomplish the project with in the

set deadlines?

According to World Quality Report 2016-17- the industry’s largest research study and analysis of testing and quality assurance states the top priorities of QA and Testing Executives

How to be selective in testing without compromising on quality?

Data-driven decision making is one of the biggest trends in startup and design culture right now

What is Analytics?

Analytics is defined as the process of measuring, recording, tracking, and analyzing data to study real-time patterns and usage of an application or a workflow. Some of the key players in this field are Google Analytics, Yahoo Analytics and many others.

We need to switch from “gut feel” to evidence-based-testing, or simply put, adopt the practice of analytics to drive testing

Analytics data gives you better performance insights and also helps you understand, prioritize and in decision making on what needs testing and optimizing first on your application/website

Analytics processLet us discuss on how we can streamline the analytics process, maximizing the extraction of meaningful insights

The analytics process must be structured and well planned

Following are some of the key points to be noted

1. Analytics process components:

A communications platform. A feedback system. Training. A text analytics engine.

2. Real time versus right time – information and insights must be provided at the right time and not necessarily continuously.

3. Insights – the process of interpretations must be standardized so that different managers come to the same conclusion after analyzing the data.

What should be in the analytics toolbox?

Tools that extract meaningful insights that support:

Descriptive analytics – creating simple counts, distributions, visualizations describing your data.

Predictive analytics – predicting organizational and process behavior, e.g., can you predict at the beginning of a release how it will end?

Prescriptive analytics – prescribe corrective actions and suggest mitigations after identifying aberrations, risks, etc.

What is the best way to develop and design a dashboard?

‘Dashboarding’ is defined as the development of an effective dashboard – a tool that supports the way to gain insights, allowing the information consumer to think business instead of data. Some considerations for dashboarding:

1. Follow an accepted methodology for dash boarding.2. The dashboard must support a specific process/goal in order to be useful.3. Do not mix strategic metrics with operational metrics, since this causes ambiguity

and confusion.4. Provide drill down capabilities to better focus on and target your issues. Define

threshold indicators and color coding to add intelligence.

What is the best way to develop ‘meaningful’ metrics?

The de-facto methodology that provides the most cost effective list of metrics is the GQ (I) M (goal-question-indicator-metric) methodology, where:

Goals need to be defined based on organizationally adopted quality frameworks (TMMI, ISO, IEEE, etc.).

Questions need to be defined so that their answers meet the goals. Defined metrics are a natural outcome of the questions.

What are the major issues and misconceptions surrounding the data asset?

The two major issues are:

1. Data quality and standardization:

Organizations are not aware of the low quality of their data. Analytics will not solve the data quality issue, but only highlight where data quality

is weak. Improve the quality of the data asset and form a trustworthy base for analytics. Data quality issues are treated through process improvement initiatives.

2. Data structures:

Data must have a stable and consistent unified data structure defined in entity-relationship-diagrams reflecting business entities, relationships and attributes.

Data structures inherently contain natural hierarchies which are translated to and defined for filtering and drill down capabilities when dash boarding.

On which processes should organizations focus when considering the analytics initiative?

The three major quality processes that directly impact the analytics initiative are:

1. Requirements management – requirements must be modelled. Requirements modelling techniques include use cases and ERDs. In the absence of good requirements, test case documentation is used.

2. Test automation – used to decrease the delivery time of software by decreasing the time and effort allocated to quality. Effective test planning can dramatically simplify the test automation effort. Considerations for test automations include:

Identify high risk business areas. Do not try to automate everything. Define clear criteria for automation. Automate test cases that always pass or always fail, test cases with a high number

of runs.

3. Defect management – effective management of the defect lifecycle will vastly increase the quality of delivered applications. Metrics in this domain are usually filtered by defect severity and include:

Counts and average times between defect statuses to identify bottlenecks. End-to-end times along the defect status network, e.g., open-close. Counts and average times between cycles of inefficiencies on the network – e.g.,

number of defects and average times on the open-reject-open path.

Get a better testing tool to make better use of your analytics data!

And lastly, to make your analysis and testing even more powerful, make sure you are using a testing tool that integrates with your web analytics tool. For the best example of this, if you are using Adobe SiteCatalyst, you can easily use your segments in that tool to target your tests in Adobe Test&Target, and this seamless integration allows you to perform detailed test analysis in SiteCatalyst. For a much simpler version of this tool integration, you can now use your Google Analytics segments to segment your tests in their new Content Experiments feature (the old Google Website Optimizer).

How can we get organizational and management support for the QA analytics initiative?

The organization defines a framework that drives the QA improvement function. Management understands that their success is directly proportional to the quality of the delivered applications. Management always needs to balance between committing resources to deliver quality applications and lowering QA costs. Budgets are justified by continuously demonstrating the quality of application delivery through efficient quality processes.

Advantages of Analytics driven approach in Software Testing

It helps in identifying right tester for particular task

It helps you discover, monitor and set targets for your key success metrics

It helps in identifying issues impacting various areas of project

It helps in proactively identifying the risks and mitigating the risks at the earliest

stage

It helps in identifying where the delay is and what is the issue

It helps in Improve Planning, Quality and Delivery

It Predictive Analytics helps in making right decisions at right time.

Gartner’s Hype Cycle for emerging trends in testing 2013

Business benefits-Real world exampleTricentis Tosca a small Austrian startup raised $165M to deliver continuous testing at the

speed of DevOps

Tricentis being named a Leader in Gartner’s Magic Quadrant for Software Test Automation for

the second year in a row. According to the Forrester report, “Tricentis Tosca provides top test

automation and optimization design capabilities, test assets reuse, and combined

automation.”

They used analytics technique for reporting and dashboard. Tracking and reporting critical

KPIs during testing and ensuring the alignment of business objectives. Tricentis Tosca has a

rich set of capabilities to track and report on various metrics related to testing.

With Tricentis Tosca, customers can describe the status of testing at any point in time. The

reporting can be done from a variety of perspectives including risk coverage, the number of

test cases and their execution results, or based on current as well as historical data (status

charts and trends).

Conclusion

There are many heuristic approaches which are emerging to push the boundaries beyond

pure analytics leading to continuous and touch less automation at the speed of thought. For

example, MBT (model based testing) and Test Coverage, PBT (Theorem Proving Based

Testing), AI, Machine Learning, Deep learning etc.

The proposed approach proves to be not only effective in saving manual effort, time, and

energy but also in helps in reducing the testing costs and deriving better ROI early in the

testing life cycle. Ultimately, it improves efficiency and effectiveness of the testing operations.

References & Appendix

https://www.linkedin.com/pulse/analytics-driven-software-testing-hussein-jibaoui

http://qa.siliconindia.com/qa-expert/Analytics-Driven-Test-Coverage--Part-I-eid-211.html

http://www.softwaretestingnews.co.uk/

https://www.capgemini.com/thought-leadership/world-quality-report-2016-17

https://www.tricentis.com/tricentis-tosca-testsuite/supported-technology/

http://www.gartner.com/newsroom/id/3412017

Author Biography

Anusha Mukkavilli

IT professional with 4+ years of work experience, currently working for Progressive Insurance account

Nibhanupudi Suguna

IT professional with 4+ years of work experience, currently working for Progressive Insurance account

THANK YOU!