13 14-test planning and management and rbt

Introduction to Test Planning, Process and Project

Management:Risk Based Testing

Prof Gopalaswamy [email protected]

04/10/23 Test Planning and Management Slide : 2

Test Planning and Management:Agenda of the Session

Testing phase as a project Planning a testing project Test Management Test Process Test Reporting


Essential Reading http://www.testingeducation.org/BBST/BBSTRisk

-BasedTesting.html

Google search on “Risk Based testing” and read a few articles

Rest of the outline from Software Testing book…


Perspectives of a “Project”

Projects

Products Services

EngineeringActivities

“In-stream”Activities

“Umbrella”Activities

Technology

Process

People


Characteristics of a Project (in Software context)

Originates from organizational vision Focused in achieving a very specific requirement (to create a unique

product or service) Targeted, controlled execution (planned, executed and

controlled) Has a definite beginning and an end (temporary) Usually in a stand alone group (performed by people) Normally has to obey some constraints (budget, time …) and

dependencies Operates in the context of various stakeholders Life period ties back to product if applicable

All these apply to Testing also


Parts of Test Planning

Comm’nMgmt

RiskMgmt

Test WBS And

Scheduling

IdentifyingTest

Deliverables IdentifyingEnv needs

IdentifyingSkill sets /

Trng

SettingEntry / Exit

Criteria

DecidingTest

Strategy

ScopeMgmt

PreparingA TestPlan

Test Planning

Start Here


Scope Management

Understanding what constitutes a release / engagement Breaking down the release / engagement into features (may

be given to you already) Prioritizing the features for testing (may be given to you

already) Deciding which features will be tested and which will not be

(may be given to you already) Gathering details to prepare for estimation


Estimation

Challenges in Estimation for testing services Risk Based Testing as a method of scope

determination and resource allocation Briefly mentioning other scope determination /

resource allocation methods


Challenges in Testing Projects for Testing Services

Testing is done with incomplete information (compared to a product organization)

Testers are more insulated from developers Testing is a crunch time job Testing has no definite end and hence you need

to decide when to stop, but… Before you stop, you better have tested the right

things because… Testing is highly visible when incomplete or

ineffective!


What are the “right” things to test? Are you testing an entire product or some components? Which functions are critical for a product? Which functions are highly visible when they fail? Which functions have been highly error-prone? Which functions are most often used? Which functions are changed most often / recently? Which functions suffered turnover of employees? Where is high geographic distribution of work? Which functions require new skills / technology?


Risk Based Testing –Basic Principles

Is a method to objectively decide the scope of testing to maximize effectiveness

Basic Premise # 1: Testing is never exhaustive – Dijkstra’s doctrine: “Program Testing can never prove

the absence of bugs – only their presence” Basic Premise #2: There is never enough time

for testing Basic Premise #3: The purpose of testing is to

find serious defects before customers find them, and find them early

Basic Premise #4: The purpose of testing is to minimize the impact of the risk of bugs


The Essence of Risk Based Testing

HowCould a program

Fail?

Can you DesignTests to make it

Fail this way

Evaluate theLikelihood &

Consequences

Prioritize &Develop Test cases

Risk Based Test Design

Risk Based Test Management


Visualizing How a Program Could Fail…


So the steps are… #1: Build a Risk Taxonomy #2: Prioritize the risks by likelihood and impact #3: Decide which are testable #4: Design and test those test cases which are

high priority and can be testable #5: Continually re-assess the risks and re-order

the list


Start with an organization-wide list of factors– An Initial “Risk Brainstorming” to go into your Process

Database

Add project-specific failure types– Do the “Risk Brainstorming” as a part of the Project

Initiation

Update as needed from industry-standard taxonomy lists available

Building a “Risk Taxonomy” or“Risk Catalog” How

Could a program

Fail?


Risk/Bug Taxonomy:Organization-wide Risks

Infrastructure Attrition / People Factors Distribution of team in multiple locations …


Risk/Bug Taxonomy:Project Specific Risk Factors

New technology / environment / version New people Customer-specific risks Third party code


Risk/Bug Taxonomy:Identifying specific ways how products can break

“Installer” (or Installation product)– Wrong files installed– Temp files left behind– Required files missing– Extra files installed– Installation in wrong location– Older files replace newer files– User data loss– Affects other applications


Risk / Bug Taxonomy:Categorize Risks

Categorization groups “logical” risks together Categorization can be by

– Function (Validation, Reporting, etc.)– Functional Area (Order Processing, AP, etc.)– Quality Characteristic (Visibility, Reliability, etc.)


Risk/Bug Taxonomy:Check with Industry Standard Lists


Prioritize the List of Possible Defects Ways to Prioritize

– By Usage Frequency– By Visibility of Failure– By Impact on Business– By complexity of function– …

Any dimension(s) you use should eventually result in a scale of 1 to n (typically 1-5) for likelihood and a similar scale for impact

The risk value is likelihood * impact


Another Method of Identifying the Risk Prone Areas to be Tested

Identify the characteristics to be tested (Functionality, Reliability, Usability, Maintainability and Portability)

Alternatively, identify the functional areas / functions to be tested

Divide a fixed booty of 100 points across these characteristics


What do you do when you don’t know anything about the product?

Take a breadth-first view A little of everything Use the information gained to fine tune


Some Other Methods of Scope Determination / Estimation

Estimating with Use Case Points– http://www.methodsandtools.com/archive/archive.php

?id=25

Requirements Based Testing – http://www.stsc.hill.af.mil/CrossTalk/2003/03/mogyoro

di.pdf

Test Driven Development– http://www.agiledata.org/essays/tdd.html


Having Got The Scope of Testing…


Deciding the test approach / strategy for the chosen features

What type of testing would you use for testing the functionality of each feature?

What are the configurations or scenarios for testing the features?

What integration testing would you do to ensure these features work together?

What localization validations would be needed? What “non functional” tests would you need to do?


Setting up criteria for testing Criteria can be at phase level

– See example in page 143, 163, 164 Entry Criteria

– Parallelism Vs Ready for use Exit Criteria

– Completeness Vs Risk of release Suspension Criteria

– Show stopper bugs– Crossing a threshold number of bugs– Developers producing a new version making the old one

redundant Resumption Criteria

– Above hurdles being cleared


Identifying Test Deliverables The test plan (master test plan, and various other

test plans for the project) Test case design specifications Test cases, including any automation that is

specified in the plan Test logs produced by running the tests Test Summary Reports


Activity Breakdown and Scheduling


Scheduling of Activities Identifying external and internal dependencies among

the activities Sequencing the activities, based on the expected

duration as well as on the dependencies Identifying the time required for each of the WBS

activities, taking into account the above two factors. Monitoring the progress in terms of time and effort Rebalancing the schedules and resources as necessary


External Dependencies Availability of the product from developers Hiring Training Acquisition of hardware / software required for

training Availability of translated message files for testing


Internal Dependencies for Testing

Completing the test specification Coding / scripting the tests Executing the tests


Parallelism Constraints for Testing

Machines People Software dependency Test serialization


Test Management

IntegratingWith

ProductRelease

PeopleMgmt

TestInfrastructure

Mgmt

TechnologyMgmt

Test Mgmt


Test Technology Management

Naming and storage conventions for test artifacts– Product functionality --- Test suites mapping– Documentation standards– Test coding standards– Test reporting standards (covered separately)


Test Infrastructure Management


People Management(Covered in more detail next session)


Integrating with Product Release

Sync points between development and testing as to when the different types of testing can commence. For example, when integration testing could start, when system testing start, etc could.

Service Level Agreements between development and testing as to how long kit would take for the testing team to complete the testing

Consistent definitions of the various priorities and severities of the defects

Communication mechanisms to the documentation group to ensure that the documentation is kept in sync with the product in terms of known defects, workarounds, etc.


Test Process Putting together and baselining a Test Plan (discussion of template) Getting detailed test specifications

– Purpose of test– Items being tested, with their version numbers– Environmental setup needed– Input data required– Steps to be followed– Expected results– Any relationship to other tests

Update of RTM Identifying possible automation candidates Developing and baselining test cases Executing tests and keeping RTM and DR current Collecting and analyzing metrics (next session) Preparation of Test Summary Report (refer IEEE template and two

slides later) Recommending product release criteria


Test Reporting Test Incidence Report

– Update to DR– Id, product info, component info, defect description, fix info,

etc Test Cycle Report

– a summary of the activities carried out during that cycle – defects that were uncovered during that cycle, based on

their severity and impact– progress from the previous cycle to the current cycle, in

terms of defects fixed– outstanding defects that are yet to be fixed in this cycle– any variations observed in effort or schedule (that can be

used for future Test Summary Report – phase-wise summary and final test

summary


Test Summary Report A summary of the activities carried out during the test cycle or phase. Variance of the activities carried out from the activities planned. This

includes – the tests that were planned to be run but could not be run (with

reasons),– modifications to tests from what was in the original test

specifications (in this case, the TCDB should be updated)– additional tests that were run (that were not in the original test

plan), – differences in effort and time taken between what was planned

and what was executed– any other deviations from plan

Summary of results– tests that failed, with any root cause descriptions – severity of impact of the defects uncovered by the tests

Comprehensive assessment and recommendation for release– “Fit for release” assessment

Recommendation of release


Recommendation for Release Dijkstra’s Doctrine Again! Testing team gives management (and customer)

an idea of what defects exist and what risks to be expected

Management makes a final call whether to release as is or expend extra resources to fix the reported defects


Test Planning and Management:Agenda of the Session - Recap
