To Infinity and Beyond!

Simple Sampling Techniques for Test Coverage Dawn Haynes

dhaynes@perftestplus.com Twitter: dawnmhaynes

LinkedIn: linkedin.com/in/dawnhaynes/

With delivery notes added from session!

© 2013

Agenda

Key concepts

Common test development methods

Enhancing common test design techniques

More design tips and ideas

Designing test sets for coverage and reuse

Wrap-up

© 2013

A testing parable …

Once upon a time there was a project – my first project. There were too many tests and not enough time. It was sad.

The End

© 2013

The Problem of Testing

• Infinite possibilities • Projects have constraints • Teams have various notions

of Quality • Unclear missions and goals

for testing

“Testing is Intractable”

(Robert Sabourin™)

© 2013

All Testing is Sampling For a universe of possibilities …

we actually choose a small set of tests.

© 2013

What is Test Design?

A way to choose tests (sampling scheme)

A way to comprehend choices

A way to see overlaps, gaps & connections

A way to communicate and review choices

A way to determine sufficiency

© 2013

The Problem with Design

Design is hard Often bypassed (test everything) Done poorly (test the same things over &

over)

Good design is harder Requires goals and known targets Requires knowledge, techniques, and skills Requires the ability to critically evaluate

outcomes and make frequent adjustments

© 2013

Common Test Development Methods

Sit & Spin

Pros • Excellent for brainstorming test ideas Cons

• Lack of a concrete coverage goal • Heavily impacted by knowledge, experience, biases, energy, etc. (inconsistent)

• Redundancy is easy and hard to see • Not a method for prioritizing or weeding out test ideas

Just *my* name for commonly used

informal analysis.

© 2013

Common Test Development Methods

Work breakdown structures

Pros • Provides a basis for analysis • Yields an organized structure Cons

• The breakdown elements structure the analysis, and therefore, can limit vision and span

• Dependent on attributes of requirements (and or system)

- Inputs / Outputs / Processing - Inputs / Constraints / Outputs / Methods - Suppliers / Inputs / Process / Outputs / Customers - Users / Functions / Attributes / Constraints

© 2013

Common Test Development Methods

Requirements-based testing

Pros • Provides a basis for analysis Cons

• Requirements can be ambiguous or have gaps • Limits testing scope (reqs vs deliverable system)

• When designing tests early, changing requirements create risk & rework

• Non-functional requirements are rare…ly useful • Goals (acceptance criteria) are rarely stated • Reqs coverage (traceability) is often misused

Discussion on Requirements Coverage

Question: If you have created & run tests that “cover” all the requirements, have you tested “everything?” - How much of the “system” is exercised

(evaluated) with 100% of your requirements-based tests? (one study says 30-35%)

- What’s missed? How do you articulate it? - Where are the overlaps/redunancies? Gaps? - How do you champion the rest of the testing

that’s risky not to do? - How many tests does it take to “cover” a

requirement? (1 by default = RISK)

© 2013

Common Challenges

When do you STOP creating tests? NEVER!?! Out of ideas, exhausted, bored … Out of coffee Ship happens! (Scott Barber™)

How many tests is enough? The answer is … 3

• (so says Mr. Owl)

© 2013

Dawn’s Rule of 3 - Positive test(s)

Happy path / basic scenarios - Negative test(s)

Unknown / unexpected paths, inputs, etc. Error flows, scenarios & conditions

- Crazy test(s) Stress & break scenarios Challenge software robustness

© 2013

Crazy Tests

Stress and break conditions Corner cases (combining extremes, limits … ) Attacks Target constraints (load, volume, repetition,

concurrency)

Known issues, previous defects, error guessing

Extreme user and usage scenarios Sorcery and voodoo

ATM Features / Transactions Priority Scope (intensity) of Testing

Withdraw cash High Positive / Negative / Crazy

Transfer funds Medium Positive / Negative

Deposit Med / Low Positive / Neg?

Check balance Low Positive

- Use it to sample tests when time constrained, or for shrinking regression testing span

- Obviously, can use it to guide/direct exploratory testing - Use it to enhance light regression tests when needed ---------------------------------------------------------------- - Think about using it to derive tests in a risk-based setting

Discussion on Applying the Rule of 3

© 2013

Enhancing Common Test Design Techniques

Equivalence Partitioning

Boundary Value Analysis

Decision Tables

State-Transitions

Pairwise / All Pairs

Think about applying the ‘Rule of 3’ to

structured techniques to increase test span

and variety.

Examples follow …

© 2013

Equivalence Partitioning: Concept

Target Items

Partition (Class)

Partition (Class)

Inputs Valid / Invalid

Outputs Approved / Declined Report / File / Record types

Modes Off / Idle / Heating / Cooling

States New / Renewed / Suspended / Cancelled

- Quick overview of Equivalence Partitioning (if needed)

- I call this hard-core sampling! Take 1

representative sample from each class

Some examples

© 2013

Value needed for testing: Birth Month (numeric)

Equivalence Partitioning3

< 0 1 – 12 13 + Text/char (e.g. Feb) Special characters (&, delimiters, etc.) Blank / Space / NULL Decimal, percent, scientific notation … Commands (SQL injection, HTML script)

Invalid Valid Invalid Out of Range / Other

Think about applying the ‘Rule of 3’ and

“amp” up the test set!

Negative

Positive

Crazy

Starting GOAL: develop a small, representative set of tests that achieve partition coverage

© 2013

Value needed for testing: Birth Month (numeric)

Boundary Value Testing3

< 0 1 – 12 13 + MIN / MIN-1 MAX / MAX+1 Combinations of values at legal boundaries Data type or data structure limits

Invalid Valid Invalid Out of Range / Other

Apply the ‘Rule of 3’ to find other interesting

boundaries

Negative

Positive

Crazy

Starting GOAL: test changes in behavior triggered by values at the edges of partitions

© 2013

Decision Tables: Concept

Business logic and business rules typically warrant rigorous testing Complexity and importance drive up risk

Decision tables are a useful way to document rules for development and testing Built by identifying key conditions and

triggered actions Form of cause-effect mapping that reveals

interdependent conditions

Overview of Decision Tables (if needed)

Starting GOAL: execute all combinations of triggering conditions and resulting actions

© 2013

Decision Tables3

Condition 1 T T F F

Condition 2 T F T F

Action 1 X X

Action 2 X X

Error 1 X

RULES 1 2 3 4 CONDITIONS

ACTIONS

Positive Negative

Crazy - Consider ways to subvert rules (soap opera testing) - Effect of repetition, longevity, delays, and abandoning actions - Effect of concurrency, load, volume … - Check behavior across systems, components, services, etc. - Break or block logic

Example table

Apply the ‘Rule of 3’ to boost confidence that key business

rules are robust and functioning effectively.

© 2013

State Transitions: Concept

Systems and objects can have states that control various aspects of execution Flow or sequence; order of operations Limit or enable access at particular times

States are identifiable and finite System: Running/Shut Down, On/Off GUI: Enabled/Disabled, Selected/Deselected Workflow: New > Pending > Dispatched

Transitions trigger state changes Events, inputs, actions, transactions, etc.

Overview of State Transition testing

(if needed).

© 2013

Testing State Transitions

Choosing a set of tests to cover a state model is not a factor of the model alone

First establish a coverage goal, then build the tests to meet the goal Actual tests and total number of tests will vary

Be able to explain the pros & cons of the

goal you establish, and the derived tests

© 2013

Common Coverage Goals

Confirm all states are reachable Check that all legal transitions can be

triggered Follow a common or specific sequence / flow All possible sequences of transitions All possible combinations of states and

transitions

What about invalid transitions? What happens if there’s a error?

© 2013

State Transitions3

Pay money

and verify age

Upgrade period expires

Free Player

Registered Player

Upgraded Player

Banned Player

Positive

GO Crazy! - Make zombies! (escape from a terminating state) - On the web, use the BACK button (confuse state mgmt) - Try to jump states (illegal transitions)

Apply the ‘Rule of 3’ to boost confidence that critical state

models are robust and functioning effectively.

Suggested practice for State Models State models keep software safe and are often

undocumented or documented poorly/incompletely Checking state management can be critical to business (and

can be fairly complex to implement and test well = RISK) If you have testing challenges around state models, try this

exercise on your own:

1. Review the table (on the next two pages). 2. Compare it to the diagram. 3. Identify the differences in the tests you would derive for

each (without the Rule of 3). a. Then apply the Rule of 3 to illustrate the difference from a test

design perspective. Did it help you find other useful tests? 4. After reviewing the model in this way, do you have any

concerns about how the system will behave in production?

© 2013

State Table Example (1/2)

State Event Action Resulting State Initial Create character Success Free player

Create character Error Initial Free Player Register & verify email Success Registered player

Register & verify email Error Free player Registered player Pay money & verify age Success Upgraded player

Pay money & verify age Error Registered player Upgraded player Upgrade time period expires Success Registered player

Upgrade time period expires Error Upgraded player

Active P layer States and Events

© 2013

Banned Player Triggers

State Table Example (2/2)

State Event Action Resulting State

Free Player Violate usage T&C Success Banned player

Violate usage T&C Failure Free player

Registered player Violate usage T&C Success Banned player

Violate usage T&C Failure Registered player

Upgraded player Violate usage T&C Success Banned player

Violate usage T&C Failure Upgraded player

© 2013

Pairwise Testing: Concept

GOAL: manage the selection of tests when all possible combinations of options is impractical Testing combinations is rich and interesting Selecting less than all combinations is risky

and often difficult A subset of tests can be generated with math

(and brain off! Yikes!!) Generates a significantly reduced test set Need domain expertise to evaluate validity

and usefulness of the tests

Overview of Pairwise testing

© 2013

Pairwise Testing: Methods

Manual methods Combinatorial analysis Orthogonal arrays

Pairwise algorithms (tools) Satisfice.com > All Pairs (James Bach) Microsoft.com / MSDN > PICT (Bj Rollison) Hexawise.com (Justin Hunter)

© 2013

Pairwise Testing: Coverage

Coverage is not necessarily better, but it is predictable and different All pairs tests create an evenly distributed

pattern of coverage across the target space Biased (informed) selections often create

clusters of coverage

VS

© 2013

Other Considerations

3-way, 4-way, or more way interactions Handling illegal combinations & dependencies Invalid values Cautions Not necessarily “better” tests, just a way to

choose some Blind usage may lead to overconfidence about

coverage

Think about enhancing the basic set of pairwise tests with

the ‘Rule of 3’ … adding Negativity and Crazyiness!

© 2013

Agenda

Key concepts

Common test development methods

Review of common test design techniques

More design tips and ideas

Designing test sets for coverage and reuse

Wrap-up

© 2013

Design Tips

Avoid over-testing Understanding what you’re covering with any

particular test (or set of tests) helps • Map out test coverage against the code, modules,

system components, business objects, applications, screens, transactions, etc.

Perform “stingy” test design Would you pay 50 Euros out of your next

paycheck to add the test you are considering? • Think about the cost of the test over its lifetime • Be able to justify why it is unique and needed

Get designs reviewed

© 2013

When you’re stuck …

Pair up with others to do design Other testers Developers, BAs, PMs, Product Owners … Customers, users

Rotate targets or assignments

Use DARTS! Random way to generate

test ideas

© 2013

D

A R T S

D Data

Areas A

Rules

R

Trails

T

States

S

Sections of the board are partitioned based on size of

testing space (change labels or rings – size of area on board –

based on your needs).

© 2013

Data Areas Rules Trails States

Follow the popcorn trail

of data …

Same data, different

areas

Check rules in various states

Evaluate interacting state models

Now, make it more interesting (inspire

deeper and different tests) by combining

them (all pairs).

© 2013

Designing Test Sets: Considerations

Logical or linear chunking and grouping Looping and repetition Sequences and states Data and flow dependencies Interactions, timing, concurrency effects Global and persistent elements Combining opportunities Data, inputs, variables, parameters, etc. Features and functions

© 2013

Designing Test Sets: Challenges

Over-testing Reducing redundancy is tricky and a good

design will help!

Tuning and optimizing for efficiency Streamline execution when realistic Increase coverage with variability

Understanding coverage and gaps

© 2013

Map It!

If at all possible, use some sort of visual model or mapping tool, or matrix

Look for places where the same thing is required Restructure or object-orient (chunk)

Use to solicit feedback (facilitate reviews)

Only document more when required/needed

© 2013

Designing for Reuse

Retesting and regression testing Manual test documentation Automation

Challenges Overhead of creation and maintenance Propensity to get stale quickly Creates testing and coverage ruts

• Find fewer or no new issues • Running more tests without significantly increasing coverage

© 2013

Design for Reuse: Quick Tips

Avoid navigational steps & methods Unless you are “testing” navigation Make navigation a variable

Vary any data possible After first passes, shift to data combinations

where applicable (use the Rule of 3) Start in small chunks and group over time Vary and randomize execution flows Vary groupings and enhance with repetition Vary timing and durations (speed & size)

© 2013

References & Acknowledgements

Books I may have mentioned Why Software Sucks –David Platt How to Break Software –James Whittaker Perfect Software –Gerald Weinberg

Many thanks to the folks who have worked

with me, and inspired my work & thinking James Bach, Jon Bach, Scott Barber, Cem

Kaner, Lee Copeland, Elisabeth Hendrickson, Karen Johnson, Mike Kelly, James Lyndsay, Robert Sabourin, Ted Squire, Gail Tenney, and other kindly souls I did not intend to omit.

© 2013

Wrap-up

Open season

Feedback Suggestions Comments Something to share or add?

Thank You & Happy Testing!!!