03. october 2013 stephan obbeck kroll consulting ag wiesbaden, germany test status reporting: focus...

03. October 2013

Stephan ObbeckKROLL Consulting AG

Wiesbaden, Germany

Test Status Reporting: Focus Your Message for Executives

Agenda

Firmenvorstellung 2KROLL Consulting AG

View this presentation as animated Prezi at

http://prezi.com/j0ufgev9cukz

Management - and Test Reports

• Too Many Information Management members pretend to have not much time -

keep them thinking like this! Your status report is not the only one a manager has to read

• Too Technical Details Most management members do not understand the

technical details of a test project If they do not understand, they rarely ask for more

information Decisions are made on data which is not understood

• Contradictory Information Data is tricky - it might fight back if you have contradictory

information Irregular reporting mechanism and content during a project

(this week reporting about progress, next week about defects)

3KROLL Consulting AG

Status Reports - Layers of Communication


• Information Transfer FACTS Obvious Status of the project Transfer of Information Actual Situation and next steps

• Non-Information Transfer EMOTIONS Project advertisement Underline Requests you might have by data Influence Management decisions regarding your test project

• Reduced Complexity COMPACTYour recipient does not have much time

Focus your message - bring it to the point key facts on one page statistics understandable without additional information Make sure the right decision is taken from the report

Management Summary Report


• Status of your project Colored status

• Progress of your work Tasks during reporting period Tasks for the next reporting period

• Maximum two statistics that describe: Progress of the test project Situation of your main project risks

Detailed Report


• Status of your project Additional statistics and features Overview of bugs Statistics for specific stakeholders …

Status Reports - Structure of Test Reports


Keep report as short as possible

Use graphics and charts whenever possible

Avoid technical details needing explanation

Focus on what you really want to say

Prepare Your Test Report: Recipients

During Test Planning – Plan Your Reporting

•To whom do you report? What is his / her position? Which information does he / she need? How deep is the technical understanding? To whom do you HAVE to report? To whom do you WANT to report?


Prepare Your Test Report: Content

During Test Planning – Plan Your Reporting

•What do you want to report? There are tons of data information you can report. Which numbers really give value to management? How can you influence management?

•How do you want to report? Plan your project and keep your reporting in mind. Reporting the progress? Reporting the risks or better the tasks?


Stakeholder Analysis (WHO?) - Questions


Stakeholder are all persons / organizations who might be affected by your project.

•This analysis helps you to find out, who might have interest / or requests reports from your test project:

What is the position of the stakeholder? How is the stakeholder affected by your project? What is the stakeholder's opinion to your project? How does the stakeholder see the project?

Stakeholder Analysis (WHO?) - Rating


• Rate each stakeholder by Interest Power

• Use a matrix to evaluate your results.• Visualize your stakeholders and

their relation to the project.

Stakeholder Analysis (WHO?) - Matrix


Risk Analysis (WHAT?) – for Benefit

A risk analysis identifies areas of danger to your project and how they can be defeated.

•Report your success. (What have you already achieved?)

•Report your areas of risk. (What can make your project fail?)

•How can you measure your risk? •How can numbers and diagrams illustrate your risk?


Remember

You want something from your management:

SUPPORT

Risk Analysis (WHAT?) - Matrix

• Analyse your risks and evaluate Likelyhood (How likely is it that this risk occurs?) Consequences (How big are the consequences of this risk?)

• Use a matrix to group your risks• Visualize the most important risks for your project


Risk Analysis (WHAT?) - Visualization


Example Risk: High number of defects might be found

Your team can handle 20 defects per week. If more occure, you need additional testers.

Report number of defects from the beginning on with a bar chart.

Plan Your Work Packages Right (HOW?)


Try to build work packages that have the same size by either

•Testcases with equal number of steps.•Group Testcases to bundles of same estimated working time.•Use parameters for level of completenessfor each working package (see later slide).

•If your work packages have a comparable size You ease your test planning. You have better control on your project status. Your status report can use linear progress information.

• How to present the overall project status?

• Some of the most common used status methods: “Signal Status” with red, yellow, green Project tendencies Subjective project progress

General Status Reporting


Remember

The first impression counts!

Status Reports often use a “signal status“

•Green: Project on track•Yellow: Yellow: Slight delays - project re-planning necessary•Red: Main project targets can not be reached within project parameters

•Problem: Yellow (middle) is a safe harbor. I have seen projects being on yellow for weeks and months.

The Yellow Comfort Zone


Signal Status with Even Number

BETTER:Use a reporting system with an even number of status fields (e.g. 4 or 6)

•Green: Project on track•Yellow:Yellow: Slight delays - no management attention necessary (e.g. setup daily defect meetings)•Orange: Significant problems - management attention necessary (e.g. additional resources)•Red: Main project targets can not be reached within project parameters


Avoid Project Tendency

Often a tendency is added to the status:Which direction the project heads to.

"Status will go up next week, when we work over the weekend“

"Status will go further down, if we do not get another tester“

•Do not report trends - manage your risks! Every manager should know that a risk coming to reality

will have a negative impact on the project. Report the risk with the actions need to be taken. If a risk is resolved by actions, your project will be

in scope again! Add a risk management matrix to your report. Ensure management can see the risks and their actions

on a first look at the report / risk matrix.


Risk Management

Example of a Risk Management Matrix:

A simple way how to manage a risk (Risk #01):• Risk: Time schedule tight. If many bugs will be found during

testing and need to be documented, time for testing might be too short.

• Action taken: We will work next weekend.• Result: At the weekend we can test 25 test cases and the

project will be back on track on Monday.


90% Syndrom

Joe has to write a test concept.

At the end of every day you ask him about the status.

What answer will you get as the days went by? 90%

Longer tasks often stick in an incomplete status of 80 - 95%.

There are always some remaining things to be done...


Level of Completeness

• Split longer tasks ( > 1 day) Sub tasks (<= 1 day) Assign level of completeness

to each sub task.

• Add a constant"time elapsed" fraction for each reporting periodduring the project: If your project is 4 days long (=100%) and

you report every day, your time fraction is 25%

• Report elapsed time and level of completeness• At least one of these two should have a constant progress to

manage your test report by progress status reports.


Manage your Test with Status Reports


Keep in Mind• Think about it, before you use pre-defined reports of your test

management tool!• Analyze if these reports really contain the message you want to send out!

=> Export the data to excel or a database and create the report with numbers and charts you really need!

Improve your Status Reports


By using different Charts• Excel - and other tools - offer dozens of different chart types.• Which one to choose?

Support the message you want to transfer to management. One Eye Catcher for your message. Use colors to emphasize your main message.

• Green = OK

• Red = Attention I prefer basic chart layouts,

do not be too creative.

Pie Chart and Bar Chart


• Pie Chart Basic chart to show different status information Low expressiveness Just a snapshot status

• Bar Chart Same information available as in pie chart for each bar History information available (PAST) History information to follow a trend (FUTURE)

Test Progress Evaluation


If your goal is to finish 2000 test cases in calendar week 32 the history in the bar chart can help you to justice if you are on track.

Green arrow: Your project is on track.

Red arrow: Your project is not on track and you need to take corrections (new target date)

Yellow / orange arrow: You look fine but you should be careful...

Test Progress


In general your test progress •does not follow a straight line •follows an S-Curve:

Starting flat Increases in the middle Flats down at the end

Test Progress Analysis Methods


• This analysis method works if you have a constant work progress for each of your reporting periods.

• If this is not the case (different number of testers, etc.):

Use a time elapsed progress report. Add an expected target line to your diagram to identify your

planned progress during the project.

Test Automation


Three reports should be created out of your test automation

REMEMBERIt is easy to gather all sorts ofdata in a test automation.But it's challenging to edit andfocus the data in the right way.

Management Summary

• Purpose: Total test results of the test run Basis for decisions

• Adressee: Managers Decision Makers

• Structure: Not more than one or two pages Use graphics to evaluate the result quickly Use colors to illustrate important messages Just a passed / not passed information No (or very short) explaining texts for errors


Error Report

• Purpose: List of errors found during the test run. Checklist for analysts to work through all

anomalies and review it.

• Addressee: Developers, Testers and Software Analysts Project Manager

• Structure: Show all results that are not OK. Show the test case and the object under test. State a text explaining the anomaly: The IS

state and the expected SHOULD result.


Inspection Report

• Purpose: Complete list of all test results for each test

cases and objects under test. Complete overview of the test run - all test

cases have been executed.

• Addressee: Project Manager QA Inspector Auditing Department

• Structure: Show ALL results of the test run

(OK and not OK). Show all test cases and all objects under test. Explaining text might be used as well. Might be created as a data file for a test

database.33KROLL Consulting AG

Summary - You‘re almost there…


Don’t forget

•Keep your report compact.•Plan your reporting before kicking offthe project:

WHO? (Stakeholder Analysis) WHAT? (Risk Analysis) HOW? (Project Planning)

•Avoid reporting mechanism needing explanation.•Use your project control mechanism in your report.•Organize your reporting from test automation.

I appreciate your feedback


THANK YOU

•The presented methods are taken from various management certification workshops and are my own experiences in different projects.

•If you pick up one of these ideas and try it in your organization, feel free to drop me an email and share your experiences with me:[email protected]

Thank You

STEPHAN OBBECK

KROLL Consulting AGSchenkendorfstraße 5D-65187 Wiesbaden

www.kroll-consulting.de [email protected]

03. october 2013 stephan obbeck kroll consulting ag wiesbaden, germany test status reporting: focus...

Documents

kroll consulting agkeep

kroll consulting agview

kroll consulting agprepare

test projectsituation

test reportstoo

test projectif

week reporting

reporting periodtasks