Writing a Test Strategy

Tor Stålhane

Strategy vs. Plan• Strategy

– tackles the question of why– looks at all influencing factor– comes to terms with the whole situation, not just one end result.

• Plan– is a list of steps taken to accomplish a goal.– tackles questions like how, when, where, who, and what

• Plan vs. StrategyA plan says, “Here are the steps,” while a strategy says, “Here are the best steps.” Strategy speaks to the reasons why, while the plan is focused on how.

Why a testing strategyWe need a testing strategy to help• System and software testers plus Test-and-

Evaluation staff to determine the overall test strategy when developing or modifying a software intensive system

• Project stakeholders – customers and senior management – to approve the test strategy

• Testers and system and software analysis to determine– Test objectives– Qualification requirements– Verification and validation criteria

Testing strategy conceptsWe will discuss the following concepts:• Purpose of a test strategy• Testing focus• Contents of a test strategy• Software integrity levels• Test objectives and priorities

Purpose of a test strategy – 1

The test strategy is important in order to• Obtain consensus on test goals and objectives

from stakeholders – e.g. management, developers, testers, customers and users

• Manage expectations right from the start• Be sure that we are heading in the right

direction• Identify the type of tests to be conducted at

all test levels

Purpose of a test strategy – 2

When we write a test strategy is important to remember that:

• Whatever we do, some kind of test strategy will emerge. Thus, we might as well specify the one we think is the best one

• A documented strategy is the most effective way to get an early agreement on goals and objectives

• We need to address:– Human factors – usability– Interoperability, except for stand-alone systems

Testing focus Our focus will depend on which stakeholder we

are considering at the moment:• Users – acceptance test and operational tests• Analysts – systems test, qualification tests• Designer – integration tests• Programmer – unit testsThe main point is that we need to define the

stakeholder first – then the tests to be run.

Contents of a test strategy – 1 The following is a list of what can be specified in

a test strategy. Not all of it is needed in all cases – only use what is necessary.

• Project plan, risks and activities• Relevant regulations – depending ofn

application area• Required processes and standards• Supporting guide lines

Contents of a test strategy – 2• Stakeholders – e.g. users, testers, maintainers

– and their objectives• Necessary resources – people, computers• Test levels and phases• Test environment – e.g. lab equipment• Completion criteria for each phase• Required documentation and review method

for each document

Software integrity levelThere are several ways to define software

integrity levels. When we choose an integrity level this will strongly influence the way we do testing.

We will look at three definitions of integrity levels:

• IEEE 1012 – general software• ISO 26262 – automotive software• IEC 61508 – general safety critical software

Warning The next slides are taken from the two standards

ISO 26262 (automotive) and IEC 61508 (industrial automation).

They show the accepted techniques for some testing activities – integration testing, unit testing and module testing.

Note that these lists• Show what is recommended depending on SIL or

ASIL level.• Do not include any techniques related to TDD or

agile techniques.

IEEE 1012 – general software

• 4, High – some functions affect critical system performance.

• 3, Major – some functions affects important system performance

• 2, Moderate – some functions effect system performance but workarounds can be implemented to compensate.

• 1, Low – some functions have noticeable effect on system performance but creates only inconveniences

V&V Activities

V&V activityDevelopment requirements

level

Design level

Implementation level

Test level

SW Integrity Level 4 3 2 1 4 3 2 1 4 3 2 1 4 3 2 1

Acceptance test execution X X X

Acceptance test plan X X X

Interface analysis X X X X X X X X X

Management and review support

X X X X X X X X

Management review of V&V

X X X X X X X X X X X

ISO 26262 – automotive software

The ASIL level – A, B, C or D – is the outcome of the combination of three factors:

• S – Severity. How dangerous is a event• E – Probability. How likely is the event• C – Controllability. How easy the event to

control if it occurs

Finding the ASIL levelSeverity Probability C1 C2 C3

S1

E1 QM QM QM

E2 QM QM QM

E3 QM QM A

E4 QM A B

S2

E1 QM QM QM

E2 QM QM A

E3 QM A B

E4 A B C

S3

E1 QM QM A

E2 QM A B

E3 A B C

E4 B C D

ACC system• High controllability – Class C1 system• High severity, e.g., collisions – Severity S3• Low probability errors – no testing needed, only

quality management during development• High probability errors – software class A or B

Methods for software integration testing

Methods and Measures Accordingto req.

ASIL

A B C D

1 Requirements based test 9.4.4 ++ ++ ++ ++

2 External interface test 9.4.4 + ++ ++ ++

3 Fault injection test 9.4.4 + + ++ ++

4 Error guessing test 9.4.4 + + ++ ++

Methods for software unit testing

Methods and Measures Accordingto req.

ASIL

A B C D

1 Functional tests 8.4.2 See table 8.2

2 Structural coverage 8.4.2 See table 8.3

3 Resource usage measurement 8.4.2 + + + ++

4 Back-to-back test between simulation model and code, if applicable

8.4.2 + + ++ ++

IEC 61508 – safety critical softwareThe level of criticality uses two parameters:• Ft = tolerable risk frequency• Fnp = protection system demand rate

PFDavg = Ft / Fnp .

E.g. we can accept one dangerous situation per year, thus Ft =10-4. Problems will cause the protection system to be invoked 10 times per hour.

Thus Fpn = 10 => PFDavg = 10-6 => SIL 1

This value decides the SIL level – see table on the next slide

Safety integrity

level

High demand or continuous mode of operation

(Probability of a dangerous failure per hour)

4 10 -9 to 10 -8

3 10 -8 to 10 -7

2 10 -7 to 10 -6

1 10 -6 to 10 -5

IEC 61508 – safety critical software

Detailed design Technique/Measure Ref SIL1 SIL2 SIL3 SIL4

1a Structured methods including for example, JSD, MASCOT, SADT and Yourdon

C.2.1HR HR HR HR

1b Semi-formal methods Table B.7 R HR HR HR

1c Formal methods including for example, CCS, CSP, HOL, LOTOS, OBJ, temporal logic, VDM and Z

C.2.4 --- R R HR

2 Computer-aided design tools B.3.5 R R HR HR

3 Defensive programming C.2.5 --- R HR HR

4 Modular approach Table B.9 HR HR HR HR

5 Design and coding standards Table B.1 R HR HR HR

6 Structured programming C.2.7 HR HR HR HR

7 Use of trusted/verified software modules and components (if available)

C.2.10C.4.5

R HR HR HR

Appropriate techniques/measures shall be selected according to the safety integrity level. Alternate or equivalent techniques/measures are indicated by a letter following the number. Only one of the alternate or equivalent techniques/measures has to be satisfied.

Module testing and integration Technique/Measure Ref SIL1 SIL2 SIL3 SIL4

1 Probabilistic testing C.5.1 --- R R HR

2 Dynamic analysis and testing B.6.5Table B.2

R HR HR HR

3 Data recording and analysis C.5.2 HR HR HR HR

4 Functional and black box testing B.5.1B.5.2

Table B.3

HR HR HR HR

5 Performance testing C.5.20Table B.6

R R HR HR

6 Interface testing C.5.3 R R HR HR

a) Software module and integration testing are verification activities (see table A.9).b) A numbered technique/measure shall be selected according to the safety integrity level.c) Appropriate techniques/measures shall be selected according to the safety integrity level.

Test objectives and priorities

Only in rather special cases can we test all input – binary input / output and few parameters. Thus, we need to know

• The overall objective of testing• The objective of every test case• The test case design techniques needed to

achieve our goals in a systematic way.The test objectives are our requirements

specification for testing.

Test data selection

One of the important decisions in selecting a test strategy is how to select test data. We will look at five popular methods

• Random testing• Domain partition testing• Risk based testing• User profile testing• Bach’s heuristic risk-based testing

Random testingThe idea of random testing is simple:1. Define all input parameters – e.g. integer, real, string2. Use a random test / number generator to produce

inputs to the SUTThe main problem with this method is the lack of an

oracle to check the results against. Thus, manual checking is necessary.

The method is mostly used for crash testing (robustness testing) – will the system survive this input?

Domain partition testing – 1

Definitions:• A domain is a set of input values for which the

program performs the same computation for every number of the set. We want to define the domains so that the program performs different computations on adjacent domains

• A program is said to have a domain error if the program incorrectly performs input classification – selects the wrong domain

Domain partition testing – 2The main challenge with domain partition is to find the

domains. For small components we can identify the domains from the code.

For large components or when we do not have access to the code, we may use the algorithm that is used for the implementation, e.g.X interval 1 => F1X interval 2 => F2X interval 3 => F3will give us three input domains defined by the three

intervals

Domain partition testing – 3Textual use case can also be used to identify

input domains. Important sources of information are

• Preconditions and triggers. Each set may identify a new input domain

• Alternate flows. Each alternate flow may identify a new input domain

It is important to remember that domain testing is not the same as function coverage.

Domain partition testing – 4

When we discuss testing, it is reasonable to also include erroneous inputs.

If we have several input parameters, incorrect values for each may identify a new domain since the error handling, triggered by a test or an exception will activate a new path throuh the code.

Domain testing – simple example

02 cbXaX

If a >< 0, this equation has the following solution

a

c

a

b

a

bX

2

2

42

Otherwise we have that

b

cX

Testing domains – 1

A = 0a = 0

A = 0a >< 0

A = 0

A = 0 04 2

2

a

c

a

b

04 2

2

a

c

a

b

D3

D2

D1

Testing domains – 2

We have four input domains:•D0: a = b = 0, which is a line in the three-dimensional space •D1: a = 0 and b >< 0, which a plane in the three-dimensional space •D3: b*b >= 4*ac and a >< 0. •D2: b*b < 4*ac and a >< 0.

Testing domains – 3

a

b

D0

D1

D1

Domains for a fixed c - value

D3

D2

D3

D2

2* sqrt(ac )

a

b

D

Domains for a fixed c - value

D3

D3

2* sqrt(ac )

Risk based testing

The idea of risk based testing is to 1.Identify the risk or cost of not delivering a

certain functionality.2.Use this info to prioritize tests. We will cover

this in more details later under “Test prioritation”

User profile testingThe main idea with this type of testing is to

generate tests that mirror the user’s way of using the system.

Consider a situation where we know that the users in 80% of all case

• Fetch a table from the database• Update one or more info items• Save the table back to the database

Then 80% of all tests should test these three actions.

Bach’s risk-based testingBach’s heuristics is based on his experience as a

tester. Based on this experience he has identified

• A generic risk list – things that are important to test

• A risk catalogue – things that often go wrong

We will give a short summary of the first of Bach’s lists.

Bach’s generic risk list – 1 Look out for anything that is:• Complex – large, intricate or convoluted• New – no past history in this product• Changed – anything that has been tampered with or

“improved”• Upstream dependency – a failure here will cascade

through the system• Downstream dependency – sensitive to failure in the

rest of the system• Critical – a failure here will cause serious damage

Bach’s generic risk list – 2• Precise – must meet the requirements exactly• Popular – will be used a lot• Strategic – of special importance to the users or

customers• Third-party – developed outside the project• Distributed – spread out over time or space but still

required to work together• Buggy – known to have a lot of problems• Recent failure – has a recent history of failures.

Test and system level – 1

Test and system level – 2From the diagram on the previous slide we see

that we can test on the• Electronics level – e.g. DoorActuator sends the

right signal• State / signal level – e.g. door is closed iff

DoorStateClosed• Logical level – e.g. the door remain closed as

long as the speed is non-zero• Safety level – e.g. the door remain closed as

long as the train is moving

Acknowledgement

The first part of this presentation is mainly taken from Gregory T. Daich’s presentation “Defining a Software Testing Strategy”, 30 April 2002.