why?

Apr 22, 2023Test Management Summit 2007

2

Why?

When we test we are trying to answer a big question: Can we release this (system, feature, code)?

This question can be broken down into many sub-questions:

Have we tested all the code? Have we tested all the features? Have we found all the bugs we expected to find? Have we fixed all the bugs we need to fix?

Definition: the exercise of some system, feature, component, or code such as to exhibit a bug


3

Feature coverage

Question: what is a feature?A1: anything stakeholders say it isA2: something users can use to do a jobA3 some collection of screens and

screen icons which users use to fulfil some unique task


4

Feature definitions

Feature definitions vary from the ludicrous (verbal assertions) through user stories, to the highly-structured (UML). Somewhere in between is text. Any definition is good if it lets you: Measure:

how many there arehow big each ishow important each is to the user

Test them in any combination such as to exhibit bugs Identify inconsistencies Know the limits of coverage


5

Problems (you knew we’d get here sometime)

The existence and extent of the feature depends on: Who is defining it (management, users, sales staff,

business analysts, tech. writers. Er, testers?) How they are defining it (text, UML, memoranda,

backs of envelopes, user stories) How it is expected to be used, and by whom

Lehman: the existence of the tool changes the nature of the job


7

Test coverage by structure

You need to be sure the developers have exercised some minimum part of the code using several approaches.

Code coverage has been the subject of much academic interest: there are lots of measures: Lines Decision points Definition-Use paths Linear Code Sequence and Jumps

Great hammers for great nails


8

Test coverage by feature

Do we have a test (set) for every feature plus installation, deinstallation, start-up and shut-down?

Can we decompose every feature into testable sub-bits? Has every one got a test? Does every test include at least one negative case?

Objection: No we don’t have any spec. worthy of the name nor

any time to write it.


9

Test coverage by GUI icon

The user interface has a number of screens, buttons, pull-downs, tabs, menus etc. Do we have them all listed, and with tests which exercise every one?

Objections: Q: d’you know how many such icons there are in the

application? A: if it’s that big, you need a list to be sure you’ve hit them all

Q: just pulling down icons, hitting buttons and entering text in fields doesn’t exercise the system. A: true but if any of those don’t work you need to know asap.


10

Test coverage by scenario

Users have goals they want to achieve. They achieve them using a number of (parts of) features. This

sets up subtle feature interactions which no other coverage approach will mimic.

Problems: P: The system is new and the users don’t know how they will use it.

S: use a model office or a simulator. P: The release adds new features - have we got to test all the old

ones as well? S: What is the risk if they interact in unexpected ways and you haven’t tested all these ways? (aka “yes”)

P: User management doesn’t want us to talk to users. S: Have management define a manager as a model user and put him or her in the model office.


11

Test coverage by transition

Web and conventional applications have “paths” a user may take through the system to achieve a goal.

Identify the paths in the form of a state transition diagram (typically from URL to URL in the case of a web test) such that a minimum number of paths can be identified and traversed.

Objections: O: Far too many paths to do this. R: What’s the risk of

something going wrong? Model your paths at whatever level you find useful to test against

O: The whole point of a web app. is to be agile. R: It doesn’t matter how agile your development is, if users click out because the app.’s unusable, because you haven’t tested all the paths. If the app is well-made there won’t be many paths.


13

Test coverage by web script, web page, application, and component

Web sites are built from bits.Testing each bit returns us to an older,

component-based test modelHaving identified the risk level of the web site,

decide the level of coverage of each component.It doesn’t matter how stable the web site is if the user experience is (to use a technical term) crap


15

More questions

Question: where does a feature start and end?Question: how d’you measure the size of a

feature? (See the IFPUG counting practices manual)Question: the number of states the system

can be in is astronomic - how do we decide what sets of variables to use?

Question: are there coverage questions in user interface testing?


16

Conclusions

Coverage must:answer a question conclusivelyhave a baseline,be user-, and business-relevant at some level Coverage is very definition-dependant: Invent a new coverage type and every manager

will want to know why you haven’t used it. If the features are ill-defined, the coverage will be.Coverage remains a major question

why?

Documents

jobtest management summit

bugtest management summit

test set

code coverage

feature coveragequestion

big question

user storieshow

screen icons