agile designer™: hp alm less test cases, more coverage webinar

The science behind software testing

© Grid Tools Ltd 2014

HP ALM: Less Test Cases more Coverage

Huw PriceMD Grid Tools Ltd

OR

Huw PriceMD Grid Tools Ltd

How Many Testers does it take to screw in a light bulb?

The Answer is 3!

What is Agile Designer?

Requirements

Clear LogicalSpecs

Storyboards

Expected Results

OutsourceWork

Packages

Test Cases

Data-2

DRAW STORY BOARD TEST CASES TEST DATA CLEAR SPECIFICATIONS

Data andVirtualization

Agile

Outsourcing Data

Flow

Logic

Agenda

• Different Coverage Techniques

• Building the perfect set of test cases from Requirements

• Updating Test Cases when Requirements Change

• Optimizing Existing Test Cases

Common Challenges

• 56% of software defects can be traced back to ambiguities in requirements1

• Detecting and resolving production defects is over 40x more expensive than in requirements2

• Industry standard for manual test case design coverage is between 10-20%

• Over-testing of some functions by factors of 40 is not uncommon

• Change requests to existing software requirements can take up to as much as 60% of the SDLC

(Stats: 1 and 2 - Bender RBT, Others: Grid-Tools)

What is Coverage?

It is not:

• Code Covered

• Number of Test Covered – Tests Run

• Percentage of use cases

• All Paired Combinations

It is:

• Designing Sufficient Tests To VERIFY That The Design And Code Correctly Implement The Requirements

• Did you get the right answer for the right reason - Two or more defects may sometimes cancel each other out - Observability

Copyright 2008 Bender RBT Inc. 9

What is Coverage?

If the customer is a business client or a preferred personal client and they have a checking account, $100,000 or more in deposits, no overdraft protection and fewer than 5 overdrafts in the last 12 months, set up free overdraft protection otherwise do not give overdraft protection.

This function has sixty-four possible combinations of input from which to select test cases:

So which ones to choose?

Different Coverage Techniques

Combinatorial – All Pairs – Constrained All Pairs

• Does not support Expected Results - You have to work it out for each combination which is very time consuming

• Combinatorial is something that ‘accidentally’ increases functional coverage (and only to a point)

• Combinatorial does not give you any solid metrics on how good your testing actually is – you can only infer that x out of y combinations have been covered, but it has no bearing in actual functional logic.

• You end up with lots of false errors that have to be checked by hand

Copyright Bender RBT Inc. 2009 11RBT24261

Cause-Effect Graphing Observable Events and Path Sensitizing

• Assume C and F are not observable events.

• Assume A is stuck at FALSE.• Enter as a test case A(T), B(T),

D(T), E(T).• Results should be C(T), F(T)

and G(T).

T

T

TT

T

T

T

Different Coverage Techniques

Different Coverage Techniques

Optimized Flow Chart Modelling

• Advanced Graph Optimization

• Most projects already have a flow chart

• Similar results to Cause and effect

• Supports Looping

• Supports Constraints

• Not applicable for non sequential based logic

Agile Designer – Test Case Techniques

Coverage Feature Agile Designer

Combinatorial Yes

Flow Chart Yes

Cause and Effect Yes

Constraints Yes

Observability Yes

Looping Yes

Complexity Metrics Yes

Typical Examples of Test Cases built Manually from Requirements

Baseline Agile Designer

Test Cases Created 14 17

Time Taken 5 hours 2 hours

Functional Coverage - % 16% 100%

Baseline Agile Designer

Test Cases Created 150 19

Over Testing x18 x2

Time Taken 5 hours 40 mins

Functional Coverage - % 80% ~100%

Baseline Agile Designer

Test Cases Created 3 12

Time Taken 4 hours 30 mins

Functional Coverage - % 5% 100%

Leg Before Wicket Rule

Demo 1

Visio VersionNode Types

Existing CoverageHP ALM Data Packs

Screw in a Light Bulb

Demo 2

Different OptimizationLooping

Path HintsTesting Importance, risk based

Business Analyst

Programmer

Tester

User

Initial Project

Test Cases

Program

Requirements Business Analyst

Programmer

Tester

User

Change Request

Test Cases

Program

Change request

Business Analyst

Programmer

Tester

User

New Process

Requirements

Tester

Test Cases

Program

RED=HARD WORK

Evolving to Greater Efficiency with Agile Designer

A day at the Races

Demo 3

Changes to RequirementsFix Broken Test Cases

Return to a 100%

The HP Test Case Factory

Baseline Optimized Reduction in Testing

Existing Test Cases 8 9

Test Coverage - % 20% 100%

Functional Nodes 78 31 60%

Edges 76 44

Baseline Optimized Reduction in Testing

Existing Test Cases 15 7

Test Coverage - % 30% 100%

Functional Nodes 113 39 65%

Edges 117 50

N.B. Node = design step in test case / Edge = Journey from one design step to the next

Typical Examples of Optimized Test Cases Built with No Requirements

Leg Before Wicket Rule

Demo 4

Reverse Engineer

Test cases designed manually by hand usually have quite a static flow.

As you can see by these examples there are a few flows which cover the same functionality.

A->B->C->D

A->B->C->E

R->A->B->C->D

Agile Designer Test Cases

Agile Designer Test Cases

Each has its individual starting point and set of steps.

G->F->A->B->C

Q->P->R->G

But often cross paths in terms of Testing the same features several times.

We want to avoid this.

When we have imported these into Agile Designer we can see we have 5 paths but as we mentioned before some cover similar paths.

A occurs 4 timesB occurs 4 timesC occurs 4 timesR occurs 2 timesG occurs 2 times

This gives us the perfect opportunity to optimize and improve the test cases

Agile Designer Test Cases

From this example we have 5 test cases but have we covered each functional step in the most optimal of fashions?

5 Test Cases24 Nodes (Design Steps)22 In/Out Edges (Journeys)

Agile Designer Test Cases

Agile Designer Test Cases

So now we optimized the flow and get something like this.

Optimize:- Removed duplicate design

steps and revaluated how the test cases should link together.

During this evaluation process we are able to navigate through the diagram from Start->End without duplicates causing us problems.

Agile Designer Test Cases

Post Optimize you can see that the number of Nodes/Edges we are testing has reduced significantly.

- 12 Nodes (Design Steps)- 20 Edges (Decisions)

This shows us a 50% reduction in steps we need to take whilst giving us 100% in functional coverage.

We’ve also reduced the number of decisions by 2. This highlights that although a smaller reduction our decision process is much more intelligent and with just 2 fewer we are covering more functions.

Agile Designer makes Continuous Integration = “Idea to Production”

Optimized Tests

Existing Test Cases

Test Data

Automation

Continuous Development

Agile Designer - Summary

Think of a Requirement like a circuit board

• The Test Cases will reflect the requirements

Static Testing

• You can Accurately Measure your Coverage

• You can Automatically create the MinimumTest Cases

With Maximum Coverage

• You can reduce and optimize your existing test cases

• You can predict how long software will take to develop

Agile Designer – Next Steps

• There is a free limited download

• Contact sales for a full copy trial – Linked to HP ALM and Datamaker

• We offer half day training courses (Free T shirt!)http://info.grid-tools.com/agile-designer-training-workshops

• We offer an Agile Designer Certification

• We are looking for partners to roll out the test optimization factory

• We are looking for local partners

Huw.price@grid-tools.com

agile-sales@agile-designer.com

Huw PriceMD Grid Tools Ltd