test report dashboard – art of showcasing data...

STC - 2013

Test Report Dashboard – Art of Showcasing data

graphically, dynamically

Prepared by: Indium Software India Ltd.

Name : Poornima Gopalan & Vishnupriya B

Email : [email protected]

[email protected]

Contact # : 91-98409 81449, 91-90925 25934

Test Report Dashboard – Art of Showcasing data dynamically, graphically 1

-------------------------------------------------------------------------------------------------------------------------------------------

Indium Software | www.indiumsoft.com

TABLE OF CONTENT

1 ABSTRACT ........................................................................................................................................................ 2

2 WHY WE NEED................................................................................................................................................. 2

3 THE IMPORTANCE ......................................................................................................................................... 2

4 REPORT TYPES ................................................................................................................................................ 3

4.1 DAILY / WEEKLY / MONTHLY STATUS REPORT ............................................................................................ 4

4.1.1 Contents ............................................................................................................................................... 4

4.1.2 Report Designing ................................................................................................................................. 4

4.1.3 Benefits ................................................................................................................................................ 6

4.1.4 Recipients ............................................................................................................................................ 6

4.2 PRODUCTIVITY / PROJECT MANAGEMENT METRICS REPORT ........................................................................ 6

4.2.1 Contents ............................................................................................................................................... 6

4.2.2 Report Design ...................................................................................................................................... 7

4.2.3 Benefits ................................................................................................................................................ 8

4.2.4 Recipients ............................................................................................................................................ 8

4.3 OVERALL TEST SUMMARY ............................................................................................................................ 9

4.3.1 Contents ............................................................................................................................................... 9

4.3.2 Report Design ...................................................................................................................................... 9

4.3.3 Benefits .............................................................................................................................................. 10

4.3.4 Recipients .......................................................................................................................................... 10

5 CONCLUSION ................................................................................................................................................. 11

6 REFERENCES ................................................................................................................................................. 11

7 AUTHOR BIOGRAPHY ................................................................................................................................. 11


-------------------------------------------------------------------------------------------------------------------------------------------


1 Abstract

One of the important facets of software development life cycle is testing. Software testing is an area that is being

considered and given utmost importance in the world of fast changing technology. There are various stages at which

testing are done to ensure quality of delivery.

Reports are the basis for many processes used to manage and communicate results from business operations. In

business today, the reporting process is often incredibly complex. It can include reports derived directly from

information systems, custom reports developed from data warehouses, and reports based on spreadsheets that

compile various system and manual inputs, among many others.

Most key reports not only assist corporations in understanding pertinent financial information, but also are critical

components of the control framework of the organization.

2 Why we need

Visibility, Accountability & Transparency (VAT) are essential to any project/product team at different levels such as

Project/Product level, Team level, Task level and also at the Team members level. An effective status report is the

only way that can make sure that these attributes are met

We do send reports on a day to day basis to communicate the accomplishments / tasks performed for the day,

however as a product owner or product team member do we get a consolidated list of these at will? The Answer is a

big “NO”. The team has to either go through a bunch of status reports or wait for the test team to consolidate and

send a weekly status report / test summary report.

Our solution is to inculcate all the objectives of a daily / weekly / summary report into a single report which not just

reduces time to prepare but also helps the project team to have access to all the data that they seek for at their will.

3 The Importance

Is it sufficient to give a single line certification that, the product is ready/not ready for implementation?

A detailed test reporting should bring in transparency to test coverage / effectiveness / defect metrics / test

confidence which the customer would appreciate most. This could help the customer to look for alternate solutions

sometimes

An effective test report should highlight

Test Scope for release / sprint / product

Test Coverage

Test Effectiveness

Test Metrics

Strengths, Limitations and Recommendations

Impediments

Key Issues with solutions

Other aspects such as performance, platform dependence etc.

Test Confidence


-------------------------------------------------------------------------------------------------------------------------------------------


4 Report Types

Testing teams send out various reports at different phases of the STLC,

Planning It is enough to communicate with the rest of the project teams, when a test plan is created

or when a major change is made to it based on the overall product / project plan devised

Designing

Reports in these stages will highlight information on the High level test scenarios (HLTS)

/ Low Level Test cases (LLTC), Traceability Matrix (RTM), Test Data generation and

other activities to all the project team stakeholders.

These will not only let them know about the progress of the task but also signal the teams

that need to review and provide signoff on the artifacts, that they are up next

Execution

Execution is the phase of a project when the testing team is the primary focus – positively

and negatively.

The test execution stage consists of:

Setting up the test environment as detailed in the test case specification

Selecting and running relevant tests from the test case spec

Recording test results

Recording and managing defects

Final Reporting of the test execution phase

A typical day during a test cycle is not done, unless the daily status report is sent out. In

some teams, they could agree on a weekly report, but having it sent daily is the norm. It is

also not uncommon to have a status meeting everyday (or week) to present the QA team‟s

status to the concerned parties. Hence, the mode of a status report can be:

Email/document

Meeting/presentation

The following are the different types of test reports:

Daily / Weekly Status Report

Productivity / Project Management Metrics Report

Monthly Summary / Release (or) Sprint summary Report

Overall Release / Test Summary Report


-------------------------------------------------------------------------------------------------------------------------------------------


4.1 Daily / Weekly / MonthlyStatus Report

Status report templates are used by teams needing frequent check-ins against progress. Product development and

project management teams often use daily status reports (sometimes in the form of a scrum) to keep everyone

coordinated and on track toward team goals.

During the execution phase, test case results and any defects will need to be recorded in order to track and manage

the test progress until all planned tests are completed or the final acceptance milestone has been reached. This

information is often stored via a test management tool such as Quality Center or an Excel spreadsheet such as the

Test Execution Dashboard which contains the following

4.1.1 Contents

On a day to day basis, we provide information on the four parameters listed below

What did we do today?

What is for tomorrow?

Impediments we faced with resolutions (if any)

Dependencies

In addition, what we propose is to include contents of what exists generally in the test summary reports such as

Test Scope – highlighting the features (or) areas in scope for the release

Test Approach – Depicts the strategy planned for the release / test cycle

Test Deliverables – Lists all the deliverables to be sent for review / submission

Test Coverage – Indicating the progress made everyday (Designing / Execution stage)

Test Effectiveness – Indicating status of each of the LLTC (Execution stage only)

Defect Metrics - By Status / Priority / Severity / Module (or) Verticaletc. to name a few

Key Issues with resolutions – This helps to conduct postmortem on what went wrong and how we resolved

the issue and serves as data point for future reference. E.g. capturing environment downtimes with reasons

Test Resource Productivity – Capturing information on how much time was spent by each resource on

which task / activity to capture the root cause analysis on any schedule deviation or effort variance

The same report can include additional information such as Build Metrics capturing the number of release number or

version, number of features implemented, number of bugs fixed etc. to have more insight on the number of builds

worked on by the test team for a Product / Project Release

4.1.2 Report Designing

It becomes difficult when all the above mentioned has to be represented into a single component and still not be

looking as a troublesome process from an understanding perspective.

The below snapshot depicts how to represent the status report effectively and efficiently in MS Excel 2010 using

some of the key features such as Pivot tables & Slicers which help us to generate data dynamically.

Header: Includes the Client name & the Reporting Period, the current date is listed below, on the right side of the

header is a synopsis of the test approach

Middle Section: Includes the day wise activities along with the areas in scope & resources used. The right pane of

the report includes graphical representations of the overall defects in the product / project along with the slicers

which allow us to filter and view the graphs dynamically based on the selection. For instance the user can choose to

view the current list of defects that are critical and active. This can also support applying multiple filters

This section also includes areas for deliverables in scope, action items/ Impediments and also helps to capture key

issues with resolution notes


-------------------------------------------------------------------------------------------------------------------------------------------


Fig.1 Daily Status report Dash board

Bottom Section: Includes Test design metrics on how many High level test cases are designed by module or vertical

and how many low level test cases were drafted for the HLTS identified. This can include the coverage /

effectiveness graphs in the execution phase and can have slicers for the users to filter and view the report

Fig.2 Test Case Completeness by Module

Fig.3 Test Coverage & Test Effectiveness


-------------------------------------------------------------------------------------------------------------------------------------------


Test Coverage:A measure of the proportion of a program exercised by a test suite, usually expressed as a

percentage. This will typically involve collecting information about which parts of a program are actually executed

when running the test suite in order to identify which branches of conditional statements which have been taken.

Test coverage matrix determines the test completeness. It measures the percentage of test execution details with

respect to the no. of high level cases designed

Test Effectiveness: This is a metric to captures the effectiveness of the test cases executed based on the number of

passes/ failures of the documented test scenarios. This determines the test confidence on the application.

Execution & Coverage reports answer the following questions:

How much work is complete, how much work is left, and is progress being made as expected?

Which module / vertical have high numbers of failed points or test case execution records?

Which module / vertical plans have high numbers of defects?

Which systems have high numbers of failed points or test execution records?

Which test case categories have high numbers of failed points?

Monthly Status reports shall include consolidated weekly sheets, wherein we pull in data from different weeks and

create a data sheet from which project team can pull information on any specific week or release.

4.1.3 Benefits

The objective of capturing this data on a day to day basis is to

Save timeby not having to spend repeated efforts of preparing multiple reports through consolidation of

your daily tasks. Instead updating this sheet on a daily basis converts the same into a weekly status

report with few additional sections and minimal effort

Enhance Tracking mechanism by bring in Transparency & Visibility on the progress made from the

designing stage to the execution stage and thereafter as well

Improve Communication by communicating risks / issues on a regular basis

Improves efficiency by analyzing the productivity / root cause analysis on the delays

4.1.4 Recipients

Generally, the recipient of this email/report is generally the manager, also the team members can be CC‟ed in some

cases – this depends on the communication protocol the team follows but we recommend this to be made available in

a repository (Cloud / Server) for any of the project stakeholder to have access to the document at his/her own will.

4.2 Productivity / Project Management Metrics Report

The core objective of this report is to get insight on the productivity of the team members on various tasks /activities

performed in different stages of the STLC which can be used to transform how we can manage resources and derive

sustainable growth in future which can help the organization.

4.2.1 Contents

In today‟s fast world, getting timesheet information filled is considered as strenuous ask , developing an internal

application or procuring one from market, customizing , maintaining is even more challenging. To avoid all such we

developed a simple 4 step process to record the task / activities executed by the team members with the timelines.

The productivity report will include the following

Day wise task/ activities with the time spent

Schedule Deviation with RCA

Effort Variance with RCA

Milestones Achieved

Upcoming Milestones

Top Risks with mitigation plan

http://dictionary.reference.com/browse/test%20suite


-------------------------------------------------------------------------------------------------------------------------------------------


4.2.2 Report Design

Summary sheet will include project specific information such as Lead, customer contact,Domain, start & end date

along with resources allocated by module / verticals. The productivity of the resources will be captured as a

consolidated list against modules if there are multiple resources working on the same module.

Additional information such as the number of billable / shadow resources and new resource request shall also be

comprised as part of this section

Fig.4 Project Management Metrics Dashboard

The second half of the report includes the Delivery related metric such as capturing the planned deliveries against the

actual deliveries made and includes information on the root cause of the deviances (if any)

Fig.5 Effort Variance

Some of the common RCA‟s are as listed below

RCA

Change Request Coding Issue Coding Standard Issue

Analysis Issue Availability Change in Priority

Configuration CR Data Issue

Design Issue Duplicate Issue Environment issues

Estimation Framework Issue Incompetency

Not an Issue Planning Skill

Unclear Requirement


-------------------------------------------------------------------------------------------------------------------------------------------


The second sheet in this report includes the detailed productivity hours per resource as depicted below. This serves

as an input to the productive hour‟s column in the summary sheet

Fig.6 Productivity Hours Table

The last section of the summary report will include additional metrics specific to the team members and their

contributions towards the project on weekly basis or activity wise

Fig.7 Weekly basis Metrics

4.2.3 Benefits

Helps organization to identify the effective productivity of resources

Helps in analyzing areas where the project team had spent long hours and helps improve resource

optimization and planning for future

RCA helps the customer and the project team in determining the milestones for future releases

Improves test estimation for future projects with similar domain/ technology

Helps customer to realize ROI for any kind of delivery engagement esp. T& M projects as this brings in

transparency on the output of the individuals in the team

4.2.4 Recipients

The recipient of this email/report is to all project stake holders Internal & Customer. This report can also be

maintained in the repository (Cloud/ Server).


-------------------------------------------------------------------------------------------------------------------------------------------


4.3 OverallTest Summary

Test summary report will be different for different kind of testing. If the project is risk based testing then the report

should contain the test coverage and unresolved defects against the product risks, and if it is requirements based

testing then the test summary report should measure coverage against the requirements.

In addition to test coverage and unresolved defects test summary reports should also contain test strategy, test

objectives and overall result of test effort.

4.3.1 Contents

Test Summary Report Includes:

Objective

Test Scope

Variances

Comprehensive assessment

Evaluation

Executive Test Summary

Our Challenges & Findings

Project Metrics

Project Deliverables

Summary of activities

Summary of results

Next Steps

Approvals

4.3.2 Report Design

Test Summary report can be best showcased in MS presentation. It shall contain following sections along with

pictorial representations.

Objective: Includes the testing focused areas, system requirements and features in scope for the testing.

Fig.8 Test Summary - Objective

Test Process: Includes the process that the QA team followed to streamline the application under test and to enhance

the test effectiveness. Also includes analyzing of test Results, to determine the effectiveness of testing that

QAperforms

Challenges and Key findings: Includes the challenges we faced during this Sprint/Release. The Major issues of this

Sprint/Release will come under Key findings.


-------------------------------------------------------------------------------------------------------------------------------------------


Value Adds:Includes the recommendations / suggestions or any other „extra‟ features beyond the standard

expectations that might have given good ROI. This can also include following:

Total number of defects raised

No. of recommendations given

Defect Metrics:Includes the overall project defect metrics in a pictorial representation based on following

categories, in the same format as DSR

i. Status

ii. Priority

iii. Severity

iv. Bug Type

v. Module / Component wise

vi. Also, the statistics of defects based on two categories can be given as in following graphical representation

Fig.9 Test Summary – Resolution vs. Priority

4.3.3 Benefits

There are many benefits that can be gained by using Test Reports. They are:

Clear visibility of work in progress

Reduction of time spent

Greater consistency and repeatability

Objective assessment

Ease of access to information about tests or testing

Helps development team to achieve better quality in future releases

Helps customers with key decision making with respect to release of products

4.3.4 Recipients

The recipient of this email/report is to all project stake holders Internal & Customer. This report can also be

maintained in the repository (Cloud/ Server).


-------------------------------------------------------------------------------------------------------------------------------------------


5 Conclusion

Test reporting has no end to the value additions that one could provide with good report dashboard. But it is

prominent to trail test report from start of any project until its closure

A dynamical and accurate test report dashboard helps its recipients obtain better visibility for key decision

with respect to release of products

Dynamic test reports can be showcased well in MS Excel along with graphical data

All future risks, mitigation & contingency plan can be arrived at by tracking pertinent test reports. Future

milestones for any project specific proceedings also stands as „easy to arrive‟ with virtuous report

dashboard

6 References

http://istqbexamcertification.com/what-is-test-coverage-in-software-testing-its-advantages-and-

disadvantages/

http://www.softwaretestingmentor.com/test-deliverables/test-execution-reports/

7 Author Biography

Poornima Gopalan has an overall IT experience of 9 years on delivering IT solutions on several technologies over

different domains through Software testing. She is a Quality Analyst in Indium Software and was associated

previously with Cognizant Technology Solutions. Her current work includes coordinating and managing multiple

test teams to deliver quality at its best. She has passion in training testers to go beyond test functionality and

coverage. Her interests also include test reporting and metrics. She is also passionate about Professional

Management Professional and a member of pmhub.net

Vishnupriya comes with 2.5 years of experience and ISTQB-BCS certified professional. She started her career as

Assistant Engineer QA in SmarTek Consultancy Services and worked in various projects related to mobile

technology. Currently she is working in Indium Software as Test Engineer for mobile application testing. She did her

Bachelor degree in Electrical and Electronics from Anna University, Tiruchirappalli. Her hobbies are listening music

and crafts work.

http://istqbexamcertification.com/what-is-test-coverage-in-software-testing-its-advantages-and-disadvantages/

http://istqbexamcertification.com/what-is-test-coverage-in-software-testing-its-advantages-and-disadvantages/

http://www.softwaretestingmentor.com/test-deliverables/test-execution-reports/

test report dashboard – art of showcasing data...

Documents