Ronald Kohlman

THE VALUES OF TESTING TECHNOLOGY

© 2011

2

Contents

Introduction What is the Value of Testing to

Business Value of Testing to Technology

Where does Testing fit in? Further Questions Importance of Testing Enemies of Testing

Questions for the Test Manager What do you bring to the

table? What does the Organisation

bring to the table? Approaches

Purpose of Testing Long Term Goals Test Objectives What are you testing?

Testing Architecture Four Stages Test Phases Test Management Test Planning

Closing

3

Introduction

This Presentation This presentation has the aim to generate discussion and

participation by you, questions are encouraged The ideas presented here are focussed on some of the

attributes from a testing framework

About Presenter – Ronald Kohlman Brings a wealth of many years of experience in IT, Program

Management and Testing from numerous major programs and projects across a number of industry sectors.

Granite Consulting Granite Consulting is a specialist human capital solutions

provider in the areas of technology and business project services. Our experienced team has extensive global experience and we provide innovative solutions that work.

4

Where does Testing fit in? The testing process should begin early in the

systems development life cycle, not just at the traditional testing phases.

Testing activities should be integrated with systems development activities

Where does testing fit within your organisation?

5

Where does Testing fit in?

6

Importance of Testing

A poor testing program can: cause mission failure significantly impact operational performance and reliability require significant rework double or triple field support and maintenance costs

It is a significant project cost Testing activities can be more than half the total program effort –

averages of 25-40% is a common benchmark To be effective, time up front is necessary to plan and organize

it A good testing program forces you to face and deal with

problems as the work is done, and when the cost of rework and fixes are much lower

A good testing program can’t make up for poor software quality, but it does help prevent many ills and will at least let you know you are in trouble early

Testing is insurance

7

What is the Value of Testing to Business

Is it to save money or risk?

Is it a “degree” of comfort?

Is it insurance?

Is it because we are supposed to do it?

Is it to stop “bugs” getting into production?

Where does Testing add value and what is its value?

8

Value of Testing to Technology Is it to prove something to ourselves and others

about the quality of what we have built/delivered?

What do you think is the value for Technology?

Is testing valued at all?

9

Further Questions What is your definition and understanding of Testing?

What are the main testing issues in your organisation?

What is the driver for testing in your organisation or organisations you have worked within?

What do you think is your organisation’s SDLC methodology and where does testing fit in?

To plan for your Testing, what do you think you would need?

10

Enemies of Testing

Vague requirements

Undocumented design

Poor code, or configuration

Compressing the development schedule risks introducing problems

Compressing the test interval risks not finding all critical problems

11

Questions for the Test Manager

Have I defined, and do I understand, my overall strategy for software testing?

Are the test planning requirements clearly defined, consistent, assigned for development, and supportable by the staff responsible for their implementation?

Are the test methods and techniques defined, are they consistent with the strategy, and can they be implemented in the software environment?

Is each test activity traceable to a controlled set of requirements and/or design data?

Are the configuration management control, and quality disciplines adequate to support the test strategies, methods, and techniques, and are they really in place?

Do I know when to quit?

What do you bring to the table? What works for you?

What doesn’t work you?

What is the one thing that you would suggest as “a must

have” in your

tool kit?

Where do you go to source information to help you?

Own knowledge base? Colleagues? Training?

Organisation?

Internet? Books? Papers? Conferences?

What knowledge do you pass on (what do you leave behind)?

How do you approach the work?

What do you look for?

How do you know you have all the information you

need?

What does the Organisation bring to the table?

Their own processes, culture, ideas, practices

and templates

Their own problems, issues, and perceptions

Organisation Maturity Level

Ability to accept your inputs, or not

Inability to clarify scope

Inability to do it the “right way”

Inability to understand testing (“Education”)

Wanting to “cut corners” & “squeeze” time

14

Approaches

Testing can be manually executed, automated, or a combination of both

Can be done with resources who are on-shore, off-shore, or a combination of both

Test every conceivable option possible to be satisfied of complete thoroughness, or run testing as a Risk Assessment – “Risk Based Testing” where you test only the highest risk areas first and subsequent areas as time and resources permits

Test multiple time across “test phases” e.g. Unit, systems, integration, performance, user acceptance, etc. Or, minimise the number of test phases to a lesser degree

15

Purpose of Testing The purpose of testing is to verify technical

performance, operational effectiveness, operational suitability; and it provides essential information in support of decision making by the business.

MISSION FOR TESTING To validate that the technology applications and solutions

delivered meet the businesses requirements and needs.

VISION FOR TESTING Operational excellence dictates the need to complete

more testing that is efficient, cost effective, and targeted; proving the quality of solutions being delivered to Project's customers is superior; thereby reducing the overall risk to the business.

16

Long-Term Goals Transfer knowledge and ownership of deliverable’s

and solution(s) to appropriate business stakeholders. Ensure that the testing methods used can be

leveraged for future use beyond the completion of any program or project.

Ensure that automated testing tools, project documents, and other processes, procedures and systems introduced by testing are retained and useable beyond the implementation of an application.

Continuous improvement of testing effort, processes, and practices.

17

Test Objectives

Testing is focused on exercising the solution to obtain confidence in its readiness for use and to demonstrate that it is working as specified.

Testing is the means by which objective judgments are made regarding the extent to which the system meets, exceeds, or fails to meet stated objectives.

Early, structured reviews are to be used to help prevent problems from ever being produced by work streams. Demonstration, detection, and prevention are important objectives and benefits of testing.

18

What are you testing?

Software? Infrastructure? Data? Interfaces?

Useability? Suitability? Robustness? Performance?

Functional / Non-Functional attributes?

Requirements – Business and Technical?

Processes?

All of the above

19

Testing Architecture

20

Four Stages

Planning Preparation Execution

Reporting

Planning: produces test strategies, approach, objectives, schedules and plans

Preparation: produces detailed test plans and test cases / scripts based on the test objectives and conditions; and establishes the test environment

Execution: the execution of the cases / scripts against the application(s) and therecording of test results

Reporting: the reporting of test progress and results, defects found, metrics, and testassessment reports

21

Test Phases

The test phases to be conducted by any particular Project may vary; but the primary test phases for are generally: unit test, systems test, integration test, user acceptance test.

There are other test phases (e.g. performance, operational readiness, etc.) that may be added to these primary ones, which will be up to you and the project to determine and identify within the Test Strategy.

The next diagram graphically represents these test phases.

22

Test Phases – Who does What?

Performed by: Program/Project Test Teams

Systems, Integration & E2E Testing

Ap

plic

atio

n C

on

fig

ura

tio

n

Sys

tem

s In

teg

rati

on

Sec

uri

ty /

Pen

etra

tio

n

E2E

Fu

nct

ion

al

Inte

gra

tio

n

Dat

a C

on

vers

ion

Pro

du

ctio

n R

ead

y T

o U

se

Performed by: Vendor / Developer

Unit Testing

Systems Testing (Vendor)

Ap

plic

atio

n P

erfo

rman

ce, S

tres

s &

Vo

lum

e

Tec

hn

ical

& F

un

ctio

nal

Co

nfi

gu

rati

on

Ap

plic

atio

n F

un

ctio

nal

In

teg

rati

on

Regression Testing

Performed by: Business

User Acceptance

Performed by:

Operations Team

Pre-Production

(Operational Readiness)

Tests

Bu

sin

ess

Sce

nar

ios

Bu

sin

ess

Pro

cess

Hig

h R

isk

Ttr

ansa

ctio

ns

COTS Solution (including any Customisations)

Project A

Project B

Program X

Program Y

Program / Project n...

Dep

loym

ent

Reg

ress

ion

Dep

loym

ent

To

Pro

du

ctio

n (

Pre

-Pro

du

ctio

n)

Ap

plic

atio

n D

eliv

ered

to

Pro

gra

m/P

roje

ct f

or

Tes

tin

g

E2E

Fu

nct

ion

al P

erfo

rman

ce

Dis

aste

r R

eco

very

Regression Testing Regression Testing

23

Test Management

The aim is to deliver quality applications in less time.

The process begins with managing requirements and test planning.

The test management process includes the preparation of the plans for execution of the test processes.

The test management activity will monitor and evaluate all test results and is performed during all life cycle processes and activities.

24

What are the key benefits of test management?

Reduce defects by capturing issues earlier in the planning process

Understand the impact changes in requirements have on testing by creating relationships through traceability

Ensure full coverage of requirements by your test cases and test plans, so everything is properly tested

Keep your planning and testing teams in sync throughout the process with open and ongoing collaboration

25

Test Planning

Like many elements of project management; testing is a discipline in itself and embodies a project in its own right.

Testing must be planned to ensure it delivers on its expected outcomes.

26

Verification Methods

Verification of test results is generally carried out through visual checking, by a tester, of actual results against expected results.

Not all parts of a solution features or characteristics are testable in a technical sense, but must be verified using an alternate method, such as:

27

Acceptance Criteria

The purpose of setting acceptance criteria is to lay down a clear and concise set of goals for the “project” to enable all the stakeholders and participants to understand exactly what the final product and deliverables should be measured against.

This is a necessary step in the life cycle for testing; because it clearly establishes the goals of testing at each phase and avoids confusion between stakeholders and other parties.

28

Testing Completeness

Testing will be complete if 100% of test cases have been attempted during a test phase or cycle and have a pass rate of greater than 95%. 

95% is a minimum pass rate, a higher % is better, e.g. 97%, 98%, 99%, and of course 100%.

This translates to mean that the solution is “fit for purpose” with minimal risk, assuming that the 5% of failed tests do not equate to any outstanding critical defects.

  If for example 1,000 test cases were run, that would mean; 50 failed and are still outstanding or were not run.

(Survivable) Reality though is that testing completeness will be a trade off between time, resources, and level of comfort that what is outstanding is “manageable”

29

Closing comments

Bring attention to opportunities and risks in testing

Championing of quality in testing

Raising awareness of test issues and opportunities

Advocating more effective testing practices

Proactively tackling quality and testing issues

Should we do technology testing?

30

Any Questions