comp8130 and 4130adrian marshall 8130 and 4130 test design & documentation adrian marshall

COMP8130 and 4130 Adrian Marshall

8130 and 4130

Test Design

&

Documentation

Adrian Marshall


Test Design

Logical grouping of tests

Test size

Modular test design

Test class checklists

Prioritisation of tests

Test coverage


Test Design

Group Tests by• Test classes

White, black, grey box

• Focus areasFunctionalityUsability Reliability Performance

• D, I, A, T (demo, inspection, analysis, test)


Test Design

Requirements testing• Group by:

• Similar functions

• Similar set-up conditions

• Minimise verification of a requirement across multiple tests / test cases

• Manage requirements coverage (and traceability) in the design


Test Design

Size of tests:• Aim to have self-contained tests that can be

executed within an hour• Where possible have pre-established test data

Ensure tests have set-up and re-set conditions at the end of the test

Use (develop) test class checklists for test design guidance


Test Design

Prioritisation of tests• Functional criticality• Build sequence

Test coverage• Criticality• Complexity• Product history


Test Design

Test class checklists

• Knowledge base of verification• Environment characteristics• Functional focus• Identifies testing at various levels


Test Design

In small groups develop some checklists for the following test classes:

• HMI (Human Machine Interface) functionality

• COTS (Commercial off the Shelf) software• Security (encryption, authentication,

control)


Test Documentation

Why do we document?

•Communication

•Repeatability

•Records of what needs to be done and what was done

•To manage test coverage


IEEE Test Documentation Set

IEEE 829/ AS 4006

Planning & preparation • Test Plans

• Test Design Specifications

• Test Case Specifications

• Test Procedure Specifications

Execution & reporting• Test Item Transmittal Report

• Test Log

• Test Incident Report

• Test Summary Report

Other useful documents• Test Verification Matrix

• Test Activity Measurement Report….


Test Documentation

Other terms for test documents:

Test Specifications

Test Descriptions


Test Documentation

Test Plans• Master Test Plan

• Define testing at all levels• Avoid needless overlaps and

dangerous gaps• Defines subordinate test plans• Consists of test concept and summary

test plan• Can be delivered in multiple phases


Test Documentation

Other Test PlansPlan tests for:• sub-systems, • major functional groups, and/or • test harnesses


Test Documentation

Test Design / Test Case specification

• Defines test design and major test cases• Describes how the tests are to be run• Details the test environment• Describes the test cases (not to a

procedural level), aims and objectives


Test Documentation

Test Design and Case specification

1 INTRODUCTION 32 REFERENCES 33 TEST APPROACH 33.1Test design33.2Test Conduct 33.2.1 Test Readiness 33.2.2 Test Execution Strategy 33.2.3 Pass / Fail determination 33.3Test Reporting 3


Test Documentation

Test Case spec (contd)4TEST ENVIRONMENT3

5REQUIREMENTS3

6 TEST DESCRIPTION 3

6.1Test Name 1

6.2Test Name 2

7DEFINITIONS, ABBREVIATIONS AND ACRONYMS3


Test Documentation

Test Case spec (contd)APPENDIX A: HARDWARE/SOFTWARE CONFIGURATIONS

3

APPENDIX B: SOFTWARE SETUP 3

APPENDIX C: TEST DATA 3


Test Documentation

Test Procedures• Include in Test Case (or have as a

separate document)• Can have delivery in multiple phases• Describes the test steps• Can be very prescriptive (beware of

maintenance of such procedures)• High level procedures may not be very

repeatable (good and bad with this)


Test Documentation

Test Logs / test record• Identify actual test schedule and test

duration• Describe software and hardware

versions• Identify any deviations from the test

procedures• List who conducted the tests and any

witnesses (eg customer, QA)• Can be included in the case/ procedure

document


Test Documentation

Incident/defect reports• Describe the issues raised• To include:

• Title• Adequate description of the issue• Test reference• Configuration info (software/

hardware versions, test data)


Test Documentation

Incident/defect reports• To include:

• Priority / severity information• Who assigned to• Status of the issue (open, under

investigation, being fixed, fixed, closed etc.

• Dates (open, closed)


Test Documentation

Incident/defect reports

• Should not have restrictions on creation

• Should only be closed the “raiser” or delegate


Test Documentation

Test Reports• Identify:

What was testedWhenBy whomOn whatOutcomes (pass / fail)Issues (reference defect number)


Test Documentation

Test Reports• Written on conclusion of test

• Aimed to report upwards

• Report recommendations should be documented separately


Test Verification Matrix

Maps the test

• Plans/designs

• cases

• procedures

to the project requirements to ensure that all testable requirements are verified and to manage requirement changes….

Test Documentation


Activity Measurement Report Quantifies the testing activities and their results :

• e.g. hours per activity• start and end dates• number of defects detected• defect trends• analysis of testing activities against plans….

Test Documentation


Summary

DesignDefine methods, priorities, logical groupings, size of tests, test

coverage, and test environment / tools

Use / develop checklists

DocumentationPre execution - Plans, cases, procedures

Execution – logs

Post execution – reports

Documents communicate, provide repeatability, aid in management of test coverage (needless overlaps and dangerous gaps)

Aim for the document is to support test execution not be a distraction to it.

comp8130 and 4130adrian marshall 8130 and 4130 test design & documentation adrian marshall

Documents

test slide

test documents

test environment

test concept

test data

test use

test design guidance

subordinate test plans