form and structure of test case matters!

Form and Structure of Test Case MATTERS.

Webinar: Dec 7, 2011, 1430-1530 IST

T [email protected]

in.linkedin.com/in/AshokSTAG ash_thiru

mailto:[email protected]

mailto:[email protected]

© 2011. STAG Software Private Limited. All rights reserved.

Form is about the external shape, the way it is presented.

Structure is about the composition of the elements of a system.

Courtesy http://www.pcars.us

Courtesy http://carwallpapers.in

2

http://www.pcars.us/viewalbum.php?a=321





The Roller Coaster

From http://gadgets.gunaxin.com/fifty-roller-coasters/65403

Form & Structure is the key to:

Creating the twists & turnstoExperience the thrill of high G forcesyetEnsuring safetyand enableit to be built from desired materials

3

http://gadgets.gunaxin.com/fifty-roller-coasters/65403












Test cases should excite the system in interesting ways to shake out the bugs that matter.

i.e. Effective

4


Test cases or Stimuli is the “combination of inputs”. We know that these are combinatorial in nature.

The key is to generate the smallest yet sufficient set and be optimal.

i.e Efficient

5


The underlying architecture i.e. Form & Structure plays a vital role in ensuring a “scaffolding” that is light yet strong.

Effective & Efficient

6


Form & Structure of Test Cases must ENABLE:

What level of quality will it help me get to?

Clarity of the entity that I am testing?

The types of defects am I looking for - Clear objective

The stage when should I execute these

Frequency of execution - How frequent?

Objective of assessment - Conformance or Robustness?

The ease of conversion into automated scripts?

The importance of these- Clear prioritisation

Rapid generation of test cases

A logical & methodical review

Foster just-enough documentation

7


A good “Form & Structure” fosters simplicity, robustness and most importantly enables clear thinking.

Clarity of thinking is key to producing good artifacts that can be objectively assessed to ensure confidence.... or shall we say Guarantee?

8


Properties to be satisfied by a Good Form & Structure

Enable setting a clear baseline for testEnable good design

Enable objective evaluation of adequacyAid in checking if it is designed well

Enable quick yet flexible and robust automation

Foster rapid design & testAllow for choosing test cases to execute logically

Enable clear & objective assessment of “cleanliness”

Allow for continual improvement and “de-weeding”

Design

Review

Automation

Execution

Assessment

Improvement

9


Hypothesis Based Testing - HBT 2.0 A Quick Introduction

HypothesizePotential Defect Types

Nine Stage Defect Removal FilterCleanliness Assessment

SetupCleanliness Criteria

SUT

Personal, scientific test methodology.SIX stage methodology powered by EIGHT disciplines of thinking (STEMTM).

Click here to know more about HBT.http://slidesha.re/qBMNiy

10

http://slidesha.re/qBMNiy%0D

http://slidesha.re/qBMNiy%0D

http://slidesha.re/qBMNiy

http://slidesha.re/qBMNiy


Cleanliness Levels in HBT 2.0Nine-Stage Filter to Detect Defects

Input cleanliness

Input interface cleanliness

Structural integrity

Behaviour correctness

Environment cleanliness

Attributes met

Flow correctness

Clean Deployment

End user value

L1

L2

L3

L4

L5

L6

L7

L8

L9

That inputs are handled wellInput data handling

That the functional behaviour is correctFunctionality

That the internal structure is robustInternal structural issues

That the user interface is cleanUI issues

That end-to-end flows work correctlyBusiness flow conditions, Linkages

That it does not mess up the environmentResource leaks, Compatibility...

That the stated attributes are metPerformance, security, volume, load...

That it deploys well in the real environmentCompatibility, migration

That user expectations are metUser flows, experience

Objective Issues

11


HBT Test Case ArchitectureNine-dimensional FORM

Organised by Cleanliness levels sub-ordered by items (features/modules..), segregated by type, ranked by importance/priority, sub-divided into conformance(+) and robustness(-), classified by early (smoke)/late-stage evaluation, tagged by evaluation frequency, linked by optimal execution order, classified by execution mode (manual/automated)

A well architected set of test cases is like a effective bait that can ‘attract‘ defects in the system.

It is equally important to ensure that they are well organised to enable execution optimisation and have the right set of information to ensure easy automation.

Level

Item

Type

Priority

Focus

Stage

Frequency

Order

Mode

12


Input cleanliness

Input interface cleanliness

Structural integrity

Behaviour correctness

Environment cleanliness

Attributes met

Robustness

Clean Deployment

End user value

L1

L2

L3

L4

L5

L6

L7

L8

L9

Focused Scenarios enable Rapid and Easy automation

Level based test scenarios foster“single-path” script.

... yields shorter scripts

... enables rapid scripting

... makes it more flexible

... eases maintenance

Form

13


Clear Assessment of Cleanliness

Quality reportQuality reportQuality reportQuality reportQuality reportCC1 CC2 CC3 CC4

R1R2R3R4R5

Met

Not met

Partially met

PDT3PDT2

PDT1

TT2

TT1

PDT5PDT4

PDT8

PDT7PDT6

TT5

TT4

TT3L1

L2

L3

L4

Stage

PDT10

PDT9

PDT9 TT6

TT7

TT8

Cle

an

lin

ess

Form

14


Entity under test

Cleanliness Criteria

Potential Defect Types

Test CasesRequirementstraceability“what to test”

Faulttraceability“test for what”

should satisfy impeded by Fault Traceability

It is necessary for test cases to be purposeful/goal-focused.Tracing test cases to requirement is a necessary condition for this, but not sufficient!Two way tracing i.e. Requirements + FAULT makes this sufficient.

Better Assessment of Adequacy Structure

15


Better Assessment of Adequacy

PD1

PD2

PD3

...

PDn

R1

R2

R3

...

Rm

PD1

PD2

PD3

...

PDn

TC1

TC2

TC3

...

TCi

Requirements traceability

Faulttraceability

Faulttraceability

That test cases for a given requirement shall have the ability to detect specific types of defects

FAULT COVERAGE

Fault Traceability

What to test?

Test for what?

Structure

16


C1

C2

C3

C4i1

i2

i3}

Outputs1

2

3

45

Inputs Conditions

• What an input(stimuli) is an therefore its values depends on the cleanliness level.• Conditions govern behaviour, what a condition is also depends on the cleanliness level

Behavioural scenarios StimuliEntity

under test Structure

17


Structure of a Test Scenario in HBT

[EUT-ID + TS-ID] [Behaviour evaluation goal]

R1.TS1 Ensure that the system does/does-not

or Check that.../Validate that...

A clear and concise one-liner. A collection of these shall cover all the behaviours (intended or un-intended behaviours)

The structure of test scenario “forces” clarity of thinking and is sharply goal-focused i.e. what do we want to check in the behaviour.

Create a behavioural model and use this to generate test scenarios.

Techniques landscape

18


Structure of a Test Scenario (Detailed)

[EUT-ID + TS-ID] [Behaviour evaluation goal]

[Evaluation pre-conditions]

[Evaluation steps]

[Evaluation post-conditions/oracle]{

Useful for1. Ensuring consistency in execution

Useful for2. Improving execution efficiency automation

Clarity of purpose of a scenario is critical. Having a set of steps without clarity of purpose is like listing the directions without being clear about the destination.

19


Structure of Test Cases

[EUT-ID + TS-ID + TC-ID] [Combination of values of pertinent inputs] [Expected value]

{I11 , I21, , I31}

{I12 , I22, , I32}

A set of test cases for a scenario is the optimal cartesian product.Document in a tabular form.

What an “input” is is key here.The values for each input depend on “what an input is”.

Remember Basic Types & Abstract Data Types in programming languages?

Techniques landscape

Input granularity principle

20


Results of using this Form & Structure by Applying HBT

Ability to question requirements better1. Find defects in requirements2. Ensuring adequacy in scenarios/cases.

Improved existing scenarios/cases by 2x-3x increasing defect yield

Reduced in-process defect escapes by 30% due to better dev test criteria

Able to convince customer logically about completeness

Lessen the dependency on the experience as key factor to good test design

Lower maintenance on automation and better ROI on functional automation

21


Summarising...

Form and Structure of Test Case MATTERS.

FORM = External Shape

STRUCTURE = The way it is composed

NINE Dimensional Form

Level

Item

Type

Priority

Stage

Frequency

Order

Mode

Structure EUT-->TS-->TC

Fault traceability

Cleanliness index

BenefitsClarity of test

Test adequacy

Logical review-ability

Shorter scripts

Rapid design

Better criteria

Prioritisation

22


Thank you!

HBT is the intellectual property of STAG Software Private Limited.STEMTM is the trademark of STAG Software Private Limited.

www.stagsoftware.com

@stagsoft

blog.stagsoftware.com

Connect with us...

http://www.stagsoftware.com

http://www.stagsoftware.com

form and structure of test case matters!

Education