agile designer™: hp alm less test cases, more coverage webinar

31
The science behind software testing © Grid Tools Ltd 2014

Upload: georginat

Post on 28-May-2015

244 views

Category:

Technology


1 download

DESCRIPTION

These are the slides from the webinar HP ALM Less Test Cases, More Coverage which took place on the 28/05/2014. The slides discuss: • Agile Designer™'s ability to reduce the number of your test cases and improve your functional coverage • How test cases can be created in a fraction of the time • How you can quickly find which test cases need to change when a requirement changes and automatically repair them • Agile Designer™'s ability to quickly find and reserve test data as well as build sophisticated pools of data to drive test cases • How Agile Designer™ can eliminate error prone and manual test case design, reduce testing time and improve quality.

TRANSCRIPT

Page 1: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

The science behind software testing

© Grid Tools Ltd 2014

Page 2: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

HP ALM: Less Test Cases more Coverage

Huw PriceMD Grid Tools Ltd

Page 3: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

OR

Huw PriceMD Grid Tools Ltd

How Many Testers does it take to screw in a light bulb?

Page 4: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

The Answer is 3!

Page 5: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

What is Agile Designer?

Requirements

Clear LogicalSpecs

Storyboards

Expected Results

OutsourceWork

Packages

Test Cases

Data-2

DRAW STORY BOARD TEST CASES TEST DATA CLEAR SPECIFICATIONS

Data andVirtualization

Agile

Outsourcing Data

Flow

Logic

Page 6: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Agenda

• Different Coverage Techniques

• Building the perfect set of test cases from Requirements

• Updating Test Cases when Requirements Change

• Optimizing Existing Test Cases

Page 7: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Common Challenges

• 56% of software defects can be traced back to ambiguities in requirements1

• Detecting and resolving production defects is over 40x more expensive than in requirements2

• Industry standard for manual test case design coverage is between 10-20%

• Over-testing of some functions by factors of 40 is not uncommon

• Change requests to existing software requirements can take up to as much as 60% of the SDLC

(Stats: 1 and 2 - Bender RBT, Others: Grid-Tools)

Page 8: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

What is Coverage?

It is not:

• Code Covered

• Number of Test Covered – Tests Run

• Percentage of use cases

• All Paired Combinations

It is:

• Designing Sufficient Tests To VERIFY That The Design And Code Correctly Implement The Requirements

• Did you get the right answer for the right reason - Two or more defects may sometimes cancel each other out - Observability

Page 9: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Copyright 2008 Bender RBT Inc. 9

What is Coverage?

If the customer is a business client or a preferred personal client and they have a checking account, $100,000 or more in deposits, no overdraft protection and fewer than 5 overdrafts in the last 12 months, set up free overdraft protection otherwise do not give overdraft protection.

This function has sixty-four possible combinations of input from which to select test cases:

So which ones to choose?

Page 10: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Different Coverage Techniques

Combinatorial – All Pairs – Constrained All Pairs

• Does not support Expected Results - You have to work it out for each combination which is very time consuming

• Combinatorial is something that ‘accidentally’ increases functional coverage (and only to a point)

• Combinatorial does not give you any solid metrics on how good your testing actually is – you can only infer that x out of y combinations have been covered, but it has no bearing in actual functional logic.

• You end up with lots of false errors that have to be checked by hand

Page 11: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Copyright Bender RBT Inc. 2009 11RBT24261

Cause-Effect Graphing Observable Events and Path Sensitizing

• Assume C and F are not observable events.

• Assume A is stuck at FALSE.• Enter as a test case A(T), B(T),

D(T), E(T).• Results should be C(T), F(T)

and G(T).

T

T

TT

T

T

T

Different Coverage Techniques

Page 12: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Different Coverage Techniques

Optimized Flow Chart Modelling

• Advanced Graph Optimization

• Most projects already have a flow chart

• Similar results to Cause and effect

• Supports Looping

• Supports Constraints

• Not applicable for non sequential based logic

Page 13: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Agile Designer – Test Case Techniques

Coverage Feature Agile Designer

Combinatorial Yes

Flow Chart Yes

Cause and Effect Yes

Constraints Yes

Observability Yes

Looping Yes

Complexity Metrics Yes

Page 14: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Typical Examples of Test Cases built Manually from Requirements

Baseline Agile Designer

Test Cases Created 14 17

Time Taken 5 hours 2 hours

Functional Coverage - % 16% 100%

Baseline Agile Designer

Test Cases Created 150 19

Over Testing x18 x2

Time Taken 5 hours 40 mins

Functional Coverage - % 80% ~100%

Baseline Agile Designer

Test Cases Created 3 12

Time Taken 4 hours 30 mins

Functional Coverage - % 5% 100%

Page 15: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Leg Before Wicket Rule

Demo 1

Visio VersionNode Types

Existing CoverageHP ALM Data Packs

Page 16: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Screw in a Light Bulb

Demo 2

Different OptimizationLooping

Path HintsTesting Importance, risk based

Page 17: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Business Analyst

Programmer

Tester

User

Initial Project

Test Cases

Program

Requirements Business Analyst

Programmer

Tester

User

Change Request

Test Cases

Program

Change request

Business Analyst

Programmer

Tester

User

New Process

Requirements

Tester

Test Cases

Program

RED=HARD WORK

Evolving to Greater Efficiency with Agile Designer

Page 18: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

A day at the Races

Demo 3

Changes to RequirementsFix Broken Test Cases

Return to a 100%

Page 19: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

The HP Test Case Factory

Page 20: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Baseline Optimized Reduction in Testing

Existing Test Cases 8 9

Test Coverage - % 20% 100%

Functional Nodes 78 31 60%

Edges 76 44

Baseline Optimized Reduction in Testing

Existing Test Cases 15 7

Test Coverage - % 30% 100%

Functional Nodes 113 39 65%

Edges 117 50

N.B. Node = design step in test case / Edge = Journey from one design step to the next

Typical Examples of Optimized Test Cases Built with No Requirements

Page 21: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Leg Before Wicket Rule

Demo 4

Reverse Engineer

Page 22: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Test cases designed manually by hand usually have quite a static flow.

As you can see by these examples there are a few flows which cover the same functionality.

A->B->C->D

A->B->C->E

R->A->B->C->D

Agile Designer Test Cases

Page 23: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Agile Designer Test Cases

Each has its individual starting point and set of steps.

G->F->A->B->C

Q->P->R->G

But often cross paths in terms of Testing the same features several times.

We want to avoid this.

Page 24: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

When we have imported these into Agile Designer we can see we have 5 paths but as we mentioned before some cover similar paths.

A occurs 4 timesB occurs 4 timesC occurs 4 timesR occurs 2 timesG occurs 2 times

This gives us the perfect opportunity to optimize and improve the test cases

Agile Designer Test Cases

Page 25: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

From this example we have 5 test cases but have we covered each functional step in the most optimal of fashions?

5 Test Cases24 Nodes (Design Steps)22 In/Out Edges (Journeys)

Agile Designer Test Cases

Page 26: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Agile Designer Test Cases

So now we optimized the flow and get something like this.

Optimize:- Removed duplicate design

steps and revaluated how the test cases should link together.

During this evaluation process we are able to navigate through the diagram from Start->End without duplicates causing us problems.

Page 27: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Agile Designer Test Cases

Post Optimize you can see that the number of Nodes/Edges we are testing has reduced significantly.

- 12 Nodes (Design Steps)- 20 Edges (Decisions)

This shows us a 50% reduction in steps we need to take whilst giving us 100% in functional coverage.

We’ve also reduced the number of decisions by 2. This highlights that although a smaller reduction our decision process is much more intelligent and with just 2 fewer we are covering more functions.

Page 28: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Agile Designer makes Continuous Integration = “Idea to Production”

Optimized Tests

Existing Test Cases

Test Data

Automation

Continuous Development

Page 29: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Agile Designer - Summary

Think of a Requirement like a circuit board

• The Test Cases will reflect the requirements

Static Testing

• You can Accurately Measure your Coverage

• You can Automatically create the MinimumTest Cases

With Maximum Coverage

• You can reduce and optimize your existing test cases

• You can predict how long software will take to develop

Page 30: Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar

Agile Designer – Next Steps

• There is a free limited download

• Contact sales for a full copy trial – Linked to HP ALM and Datamaker

• We offer half day training courses (Free T shirt!)http://info.grid-tools.com/agile-designer-training-workshops

• We offer an Agile Designer Certification

• We are looking for partners to roll out the test optimization factory

• We are looking for local partners