fujitsu computer systems testing considerations (past experience, methodology, options…) camp f2f...

Fujitsu Computer Systems

Testing Considerations(Past experience, Methodology, Options…)

CAMP F2F Redwood City,November 2012

Jacques DurandFujitsu

1


Testing serves different purposes

• Validating a specification

• Verifying an implementation

See: https://wiki.oasis-open.org/tab/TestingPolicy

2


Testing serves different purposes

• Validating a specification• ambiguities, fuzziness• gaps (unspecified features)• inconsistencies / conflicts

• Verifying an implementation• conformance• interoperability: two+ implementations interoperable (or conforming artifacts portable?)• maintenance: is the implementation still conforming/interoperable over time? (changes in config, etc.)

See: https://wiki.oasis-open.org/tab/TestingPolicy

3

Fujitsu Computer SystemsFujitsu America, Inc.

TestCase

TestCase

Testassertion

Testassertion

SpecificationTest

assertions TestCase

is

addressing

ActualTest suite

Normativestatement

Normativestatement

Normativestatements

derivedfrom

Test assertions: a common test foundation

Test Assertion:• A “measurable/testable statement of behavior for an implementation”• Blueprint for an executable test unit

• see: https://wiki.oasis-open.org/tab/TestabilityGuide• see: https://www.oasis-open.org/committees/download.php/44696/taguidelines-v1.0-wd02.pdf

4


Example: a specification on “widgets”

[requirement 1023] “A widget MUST be of cuboid shape”.

Here is a test assertion addressing this requirement:

TA id: widget-TA100-1 Target: widget Normative Source: “widget specification”, requirement 1023 Predicate: [the widget] has six facets, and each one of these is of rectangular shape. Prescription Level: mandatory

Worded:

• in knowledge of the testing ability

• understandable by a non-expert

5


An example TA (CIMI)- The normative source is an interpretation (summary) of several normative statements in the original specification- The “SUT” tag identifies the larger entity “responsible” for conformance, while the “target” is a sequence of messages

6


An example TA (SCA)- If the “prerequisite” is false, the “target” is not qualified for further evaluation of the “predicate”.- The tags here are just keywords used to classify this TA.

7


What is to be tested

PaaSProvider

Black-box testing: through the testing of concrete test “targets” being exchanges: artifacts, messages

Major CAMP entities or roles

AppUser#1

Messages

Artifacts

8


Conformance aspects to verify:

• (1) Message & Artifact well-formedness• Protocol aspects• schema validation, HTTP headers…• Request/Response basic consistency

• (2) Consistency between exchanges and what is currently deployed on the PaaS platform

• Right set of resources created, after operations?• message responses consistent with platform state?

9


Benefits of Test Assertions (1):

• Improve specifications• by “translating” normative requirements in precise implementation behaviors, discover :

• gaps / ambiguities / inconsistencies

• Align implementations (before testing)• lead to a common interpretation of the spec.

• Facilitate Testing• starting point for Test Suites

10


When to write Test Assertions ?

POC workTest SuitedeveloptProduct

dev

Implementationwork

Technical workOn Specification

Standardization(WDCS OS)

? ? ?

specificationwork

Best for maximum benefit:

while writing the spec

11


A look at 3 Testing Set-ups: ( all useful )

PaaSProvider(SystemUnder Test)

TestDriver

- a single implementation of System Under Test (SUT) (CAMP) is being tested.- Primary goal: conformance assessment

1. Isolation Testing

12



2. MatrixTesting

AppUser#1

- a test scenario is repeated for each pair of implementations(or pair PaaS Consumer-Provider)- main goal: interoperability assessment

AppUser#1

PaaSProvider A

AppUser#1AppUser#2

AppUser#1AppUser#3

PaaSProvider B

PaaSProvider C

13



PaaSProvider(SystemUnder Test)

TestDriver

- concerns in-production systems- main goal: check for regressions- 2 variants: (a) driven tests (b) real exchange monitoring

3. Maintenance Testing

(b)

(a)

AppUser#1 In-production

Provider

14

Fujitsu Computer Systems 15

Separating Test Operation from Test Analysis• allows for reusing analysis test tool in various set-ups

Phase 1: Test operations

TestAnalyzer

Testassertions

Phase 2: Test analysis

ExecutionTrace

TestReport

Fujitsu America, Inc.

SUTTestDriver

SUT#1

SUT#2

SUT#3

users SUT

In-productionProvider

Isolation testing

Matrix testing

Maintenance testing


Test Tools: overview

WebService

ANALYZER

MONITORInterceptor

Logger

Messageartifacts

Example of 2-phase testing: WS-Interoperability Testing

Execution report

Test report

Fujitsu America, Inc.

Phase 1

Phase 2

TestSCENARIOS

Client code

16


Benefits of Test Assertions (2): when implementing test suites

• Single Test Tool/Suite for conformance logic• usable in all test set-ups

• Visibility of conformance logic• in test suite (blueprint for test cases)• in test report (each report item refers to a T.A.)

17

fujitsu computer systems testing considerations (past experience, methodology, options…) camp f2f...

Documents