slide 1. slide 2 outline protocol verification / conformance testing ttcn automated validation...

Outline

• Protocol Verification / Conformance Testing

• TTCN

• Automated validation

• Formal design and specification

• Traffic Theory

• Application Debugging

Testing of the Telecommunications Systems

• Functional Tests

• Formal System Tests against system level requirements

• Protocol Verification

• Performance/Load Tests

• Interoperability Tests- IOT (Inter-working Tests)

• Automatic Test Suites

Protocol Testing

• Conformance Testing– Performed utilising test equipment

– Based on Conformance Test Suites/Cases specified by standard body

– Are pretty detailed in terms of protocol implementation verification

– Not good in testing performance/load issues

• Inter-working Trials – Testing of the real equipment inter-working

– Good for performance/load testing

– Hard to completely verify protocol

Conformance Testing

• Based on ISO/IEC 9646 (or X.290) – "Framework and Methodology of Conformance Testing of Implementations

of OSI and CCITT Protocols.”

• Abstract Test Suites (ATS) consisting of Abstract Test Cases

• Test Cases are defined using “Black Box”model, only by controlling and observing using the external interfaces.

• Provide basis for Generic Test Tools and Methods for verification of Telecommunication Standards and Protocols

ITU Conformance Testing Standards

• X.290 - OSI conformance testing methodology and framework for protocol recommendations

• X.291 Abstract test suite specification

• X.292 The Tree and Tabular Combined Notation (TTCN)

• X.293 Test realization

• X.294 Requirements on test laboratories and clients for the conformance assessment process

• X.295 Protocol profile test specification

• X.296 Implementation conformance statements

Conformance Testing

• Verification that protocols are conforming to Standard Requirements

• PICS - Protocol Implementation Conformance Statement– Information provided by protocol implementator on the extent of the

implementation: what optional items are implemented, is there any restrictions …

– based on the PICS questioner form provided by standard body

• PIXIT - Protocol implementation extra information– provides information regarding the physical configuration of unit under

test: eg. Telephone numbers, socket numbers ...

Tree and Tabular Combined Notation TTCN

• Defined as part of ISO/IEC 9646 (X.290)

• Used as a language for formal definition of test cases

• Each test case is an event tree in which external behavior such as: "If we send the message 'connect request,' either 'connect confirm' or 'disconnect indication' will be received" are described.

• Messages can be defined using either the TTCN-type notation or ASN.1.

TTCN Tests Reactive Systems

IUT

Stimulus Response

PCO

PCO = Point of Control and Observation

IUT = Implementation Under Test

A Typical Test Architecture

IUTL

U

Lower Tester Upper Tester

Underlying Service Provider

Parallel Test Architecture

IUT

Underlying Service Provider(s)

CPMTC

MTC = Main Test Component

PTC = Parallel Test Component

CP = Coordination Point

CP CPCP CP

PTC

PTC PTC

PTC PTC

TTCN Formats

• TTCN is provided in two forms:

• a graphical form (TTCN.GR) suitable for human readability

• a machine processable form (TTCN.MP) suitable for transmission of TTCN descriptions between machines and possibly suitable for other automated processing.

Main Components of TTCN

• Test Suite Structure

• Data declarations (mainly PDUs)– TTCN data types

– ASN.1 data types

• Constraints– i.e., actual instances of PDUs

• Dynamic behaviour trees (Test Cases)– Send, Receive, Timers, Expressions, Qualifiers

– Test Steps

Test Suite Structure

• TTCN Test suite consist of 4 major parts– suite overview part

– declarations part

– constraints part

– dynamic part

Suite Overview Part

• The suite overview part is a documentary feature comprised of indexes and page references.

• This is part of the heritage of TTCN as a paper-oriented language.

• It contains a table of contents and a description of the test suite, and its purpose is mainly to document the test suite to increase clarity and readability.

• In this part of the test suite, a quick overview of the entire test suite is possible.

Test Suite Overview Proforma

Test Suite Structure

Suite Name : SuiteIdentifier

Standards Ref : Free Text

PICS Ref : Free Text

PIXIT Ref : Free Text

Test Method(s) : FreeText

Comments : [FreeText]

Test Group Reference Selection Ref Test Group Objective Page No.

TestGroupReference [SelectExprIdentifier] FreeText Number

Detailed Comments: [FreeText]

Suite Declarations Part

• The declarations part is used for declaring– types,

– variables,

– timers,

– points of control and observation (PCOs),

– test components.

• The types can be declared using either TTCN or ASN.1 type notation.

• Graphical table is used for declarations

TTCN Data Types

• There is no equivalent to pointer types and, as a consequence of that, types cannot be directly or indirectly recursive.

• Two structured types that are specific – protocol data unit (PDU) – data packets sent between peer entities in

the protocol stack

– abstract service primitive (ASP) - - a data type in which to embed a PDU for sending and receiving

Example of ASN.1 Type Definition

ASN.1 Type Definition

Type Name : DATE_type

Comments : To illustrate the structure of ASN.1 type definitions

Type Definition

SEQUENCE {

day DAY_type,

month MONTH_type,

year YEAR_type

}

-- local DAY_type

DAY_type ::= INTEGER {first(), last(31)}

-- MONTH_type and YEAR_type are defined ASN.1 Type Definitions tables

Example of TTCN Type Definition

TTCN Type Definition

Type Name : DATE_type

Comments : To illustrate the structure of TTCN type definitions

Parameter Name Parameter Type Comments day month year

DAY_type MONTH_type YEAR_type

Detailed Comments:

Variable Declarations

Test Suite Variable Declarations

Variable Name Type Value Comments

state IA5String "idle" Used to indicate the final stable state of the previous Test Case, if any, in order to help determine which preamble to use.

Constraints Part

PDU Constraint Declaration

Constraint Name : C1

PDU Type : PDU_A

Derivation Path :

Comments :

Field Name Field Value Comments

FIELD1 (4 .. INFINITY)

FIELD2 TRUE

FIELD3 "A STRING"

In this part constraints are used to for describing the values sent or received.

The instances used for sending must be complete, but for receiving there is the possibility to define incomplete values using wild cards, ranges and lists.

Then a possible constraint is:

ASN.1 PDU Type Definition

PDU Name : XY_PDU

PCO Type :

Comments :

Type Definition

SET { field_1 INTEGER OPTIONAL, field_2 BOOLEAN, field_3 INTEGER OPTIONAL, field_4 INTEGER OPTIONAL }

ASN.1 PDU Constraint Declaration

Constraint Name : CONS1

PDU Type : XY_PDU

Derivation Path :

Comments :

Constraint Value

{ field_1 5, field_2 TRUE, field_3 3 }

-- field_4 is not specified=>omitted when sending -- if identifier field_3 was not used it would be ambiguous whether 3 was the value of field_3 of -- field_4, since both are OPTIONAL.

Dynamic Part

• In this part actual tests are defined

• Contains Test Suite with – test groups – a grouping of test cases. It might, for example, be

convenient to group all test cases concerning connection establishment and transport into a separate test group

– test steps – a grouping of test events, similar to a subroutine or procedure in other programming languages

– test events – the smallest, indivisible unit of a test suite. Typically, it corresponds to sending or receiving a message and the operations for manipulating timers

TTCN Behavior Tree

• Suppose that the following sequence of events can occur during a test whose purpose is to establish a connection, exchange some data, and close the connection.

– a) CONNECTrequest, CONNECTconfirm, DATArequest, DATAindication, DISCONNECTrequest.

• Possible alternatives to “valid behaviour” are– b) CONNECTrequest, CONNECTconfirm, DATArequest,

DISCONNECTindication.

– c) CONNECTrequest, DISCONNECTindication.

TTCN Behavior Tree

T0731090-98\d08

L!CONNECTrequest

L?CONNECTconfirm

L!DATArequest

L?DATAindication

L!DISCONNECTrequest a)

L?DISCONNECTindication b)

L?DISCONNECTindication c)

Progression of time

alternatives

EXAMPLE-TREE (L:NSAP)

TTCN Behavior TableThe behavior tables build up the tree by defining the events in lines on different indention levels. The rows on the same indention are events that describe alternatives, and row on the next indention are the continuation of the previous line.The line can consist of one or a number of the following:

– send statement (!) – states that a message is being sent– receive statement (?) – states that a message is being received– assignment – there is an assignment of values– timer operations – start, stop or cancel a timer– time-out statement– Boolean operations, qualifying the execution– attachment (+) – acts as a procedure call

TTCN Behavior TableAll the leaves in the event tree are assigned a verdict that can be:

– pass – the test case completed without the detection of any errors

– fail – an error was detected

– inconclusive – there was insufficient evidence for a conclusive verdict to be assigned, but that the behavior was valid

A verdict can be final or preliminary – it will not terminate the active test case execution.

To describe what is happening in the test case, the dynamic behavior can be explained in plain language.

Dynamic Behaviour Table

A TTCN behaviour tree:

Test Step Dynamic Behaviour

Test Step Name : TREE_EX_1(L:NSAP)

Group : TTCN_EXAMPLES/TREE_EXAMPLE_1/

Objective : To illustrate the use of trees.

Default : NOTE – This example can be simplified by using Defaults.

No. Label Behaviour Description Constraints Ref Verdict Comments

1 L!CONNECTrequest CR1 Request

2 L?CONNECTconfirm CC1 Confirm

3 L!DATArequest DTR1 Send Data

4 L?DATAindication DTI1 Receive Data

5 L!DISCONNECTrequest DSCR1 PASS Accept

6 L?DISCONNECTindication DSCI1 INCONC Premature

7 L?DISCONNECTindication DSCR1 INCONC Premature

TTCN MP Form$Begin_TestCase$TestCaseId TC43__S0_17$TestGroupRef V52NWKAN/BCC/BV/$TestPurpose /* On receipt of an AUDIT message containing an UP_ID IE,the IUT shall send an AUDIT COMPLETE message containing theconnection_incomplete IE "incomplete normal". */$DefaultsRef DEF_BCC_ANY_BODY$BehaviourDescription$BehaviourLine$LabelId$Line [0] +Preamble_bcc_AN20$Cref$VerdictId$Comment /* */$End_BehaviourLine$BehaviourLine$LabelId$Line [1] +STEP_bcc_init$Cref$VerdictId$Comment /* */$End_BehaviourLine$BehaviourLine$LabelId

Testing and validation.

To test an implementation of a protocol with an SDL specification the following approach can be taken.

• Get the interface into the appropriate state using a preamble.

• Send the appropriate message.

• Check all outputs

• Check the new state the protocol is in.

• Perform a postamble to return the protocol to the null state.

TTCN What do you need to know

• To understand Terminology

• To understand test principles and the test environment

• You don’t need to know how to write TTCN scripts, but you have to understand them so you can read test reports and analyse problems

Various Test Equipment

• Numerous Test Equipment supports the TTCN

• Protocol Analyzers

• Protocol Simulators

• Protocol Emulators

Protocol Analysers (Monitors)

– Monitors of the ‘live’ communication between equipment and interprets messages in human readable form.

– Same H/W may support monitoring of different protocol types

– It may provide some statistical analysis capability

– It may have advanced search/filtering capabilities.

– You may define event on which logging would start or stop.

Some Equipment

Some Equipment

ProtocolAnalyser

Protocol Simulators

– Allows Users to write a scripts (similar to TTCN) which will send a messages and react on received messages

– It can be used for testing various protocols

– It may be quite big effort to write required amount of scripts

– Suppliers often provide a set of test scripts targeting particular protocol

EquipmentUnder TestProtocol

Simulator

Protocol Emulators

– Emulates operation of equipment (not full implementation)

– No script writing required

– It provides some control of behavior and configuration

– Not flexible as simulator

– Usually one box can contain Protocol Analyser, Simulator and Emulator

EquipmentUnder TestProtocol

Emulator

Interoperability Testing (IOT)

• Testing what users may expect from product, which may include more then demanded by standards: performance, robustness and reliability

• Proper functioning in the system where product is finally installed

• Interoperability with applications which use the product

• IOT is usually performed in addition to Formal Protocol Conformance Testing

Pros & Cons of IOT

• Potential IOT Problems– Not covering all possible protocol scenarios

– Difficult to reproduce detected problems

– Often little possibility to actively control the test environment so investigation can be done

• The pros of IOT– Standard may be ambiguous so different manufactures implement

differently

– Performance/Load aspects easier to test

– IOT may be cheaper and faster

– It gives the confidence to user in the system operational capabilities

The Case for Automated Validation

• If a standard is ratified by a standards body that has not been validated and contains errors the following may occur:

– Some implementations of the protocol may exist with serious errors that will affect service operation and service assurance

– Different implementations of the protocol , that are bug free will have proprietary solutions to overcome protocol errors and thus may not inter-work

– Some vendors may choose to implement only a subset of standardised protocols, relying on proprietary protocols.

The Case for Automated Validation

• Even with simple protocols containing a small number of functional entities, messages and states the number of possibilities will be more than most people will have time to verify by hand.

• Communications protocol implementations are so complex, that it is likely to be impossible to test all protocol combinations on a system that has implemented a protocol.

• In reality standards committees are design by committee, and although some members perform automatic validation, most don’t, and protocols are modified to cope with bugs as they appear

State Explosion Problem

• Even with simple protocols containing a small number of functional entities, messages and states the number of possibilities will be more than most people will have time to verify by hand.

• Some problems have more permutations than a computer can go through, due to limitations on processing speed and memory.

Travelling salesman problem• A salesman spends his time visiting n cities (or nodes)

cyclically. In one tour he visits each city just once, and finishes up where he started. In what order should he visit them to minimise the distance travelled?

• If every city is connected to every other city, then the number of possible combinations is (n-1)!/ 2. For only 100 cities the number of possible combinations could be as high as 4.67 x 10155

• There is a mathematical proof to show that for the number of towns in the US that the time required to list all combinations would be greater than the estimated length of the universe, and would require more memory than there are atoms in the universe, according to current physics theory.

Automated analysis tools.

• Through the use of automatic tools, that traverse through all the possible sets of states number of errors can be easily detected:

– livelock and deadlock

– output with no receiver

– output with multiple receivers

– over the maximum number of instances for a process

– decision value not expected in the set of answers

– unreachable states

Automated analysis tools.

• Errors related to data access, for example: – variable usage not compliant with its type

– array overflow

• Automated tools will not be able to detect undesirable behaviour due to poor specification

Formal Design Vs Traditional Design

• Traditional design involves the following design cycle

• High Level Design– Requirements are written usually in English (or equivalent),

– Protocol is then defined using a formal techniques such as SDL, MSC and ASN.1

• Low Level Design

• Coding and testing– Most the effort occurs here. Coding and debugging thus becomes the focus

of the protocol design

Early attempts at Automated Validation

• First attempt at automated validation was of by IBM in Zurich in late ‘70s.

• Used a technique known as pertubation analysis, which involves starting at a state, determining all of the possible next states, and then using those next states in inputs for further pertubations.

Early attempts at Automated Validation

• Early attempts at automated validation had several drawbacks:

– There was no high level formal notation way to define protocols in a generic way, that could be used as input to validation software.

– Automated validation software required a large amount of human input.

– Software had to be written for each new protocol being validated

Formal Design

• The later an error is detected the more expensive it is to fix.

• Formal design techniques are used to verify the correctness of the protocol high level design phase.

• This involves testing the protocol description as defined in a specification languages such as SDL.

• This requires– An unambiguous notation

– Effective validation tools

Formal Protocol Specification Languages and Protocol Validation

• Formal protocol definition languages such as SDL can be used as input to protocol validation tools.

• Three common description languages used for this approach are SDL, Lotos and Estelle.

FDTs are also intended to satisfy objectives such as:

• a basis for analyzing specifications for correctness, efficiency, etc.;

• a basis for determining completeness of specifications;

• a basis for verification of specifications against the requirement of the Recommendation;

• a basis for determining conformance of implementations to Recommendations;

• a basis for determining consistency of specifications between Recommendations;

• a basis for implementation support.

Other Formal Specification Languages

• These other languages are discussed in the literature, but are beyond the scope of this subject and are not used by ITU-T:

• CCS (the Calculus of Communicating Systems)

• Lotos (Language for Temporal Ordering Systems): mathematically defined language

• Z Specification Language: a language based upon set theory and first-order predicate calculus.

• VDM (Vienna Development Method)

ITU Programming Languages

• ITU standardises the following formal description techniques

– Z.100 CCITT Specification and description language (SDL)

– Z.105 SDL combined with ASN.1 (SDL/ASN.1)

– Z.120 Message sequence chart (MSC)

• ITU also define the following telecommunications languages

– Z.200 CCITT high level language (CHILL) Common text with ISO/IEC

– Z.30x Introduction to the CCITT man-machine language

Performance analysis

• When a new protocol or service is introduced into a network, there needs to be an understanding what resources are required to implement the service.

• Performance analysis is the task of finding a services resource requirements are.

• This is used to cost the service at the cheapest price, with the highest probability that the targeted quality of service is delivered.

• There is a branch of statistics devoted to understanding the use of communications networks known as traffic theory.

Traffic Theory• Traffic theory is well understood for circuit switched,

unencrypted voice calls.

• Traffic theory uses a unit of measurement known as the Erlang.

• 1 Erlang is 1 Hour of calls

• Busy Hour Traffic rate is expressed in Erlangs

• Eg: If 350 calls are made on a trunk group, and the average call duration is 180 seconds,

– BHT = Average call duration (s) * Calls per hour / 3600

– BHT = 180 * 350 / 3600

– BHT = 17.5 Erlangs

Multiplexing

• If every person in the country tried to make a call at the same time, only a portion would get through.

• This is because lines are multiplexed.

• Traffic theory enables telecommunication planner to put in enough plant to ensure high probability that every call gets through while minimizing the cost in infrastructure

Multiplexing

Multiplexer

Each link has 1/n erlang of traffic

This link has 1 Erlang of traffic

n terminals or links

Erlang B

• The Erlang B distribution is used for dimensioning trunk routes. It is based on the following assumptions:

• There are an infinite number of sources;

• Calls arrive at random;

• Calls are served in order of arrival;

• Blocked calls are lost; and

• Holding times are exponentially distributed.

Erlang B

• The Erlang B formula is used to predict the probability that a call will be blocked. The Erlang B formula is:

where:B=Erlang B loss probabilityN=Number of trunks in full availability groupA=Traffic offered to group in Erlangs

Erlang B Example

• I am planning a remote PABX (private automatic branch exchange) connected by a tieline that will be used for all inbound calls to that PABX which will have 780 active ends.

• I estimate 30mE of inbound traffic per active end, and GOS should be better than 0.002.

• How many trunks do I need in the tie line route?

• Answer: to carry 780*0.03E=23.4E, I need 38 trunks, and the actual grade of service should be 0.0014

Erlang C

• The Erlang C distribution is used for dimensioning server pools where requests for service wait on a first in, first out (FIFO) queue until an idle server is available. It is based on the following assumptions:

– There are an infinite number of sources;

– Calls are served in order of arrival;

– Blocked calls are delayed; and

– Holding times are exponentially distributed.

Erlang C Formula• The Erlang C formula is used to predict the probability

that a call will be delayed, and can be used to predict the probability that a call will be delayed more than a certain time. From that other key queue performance metrics can be calculated. The Erlang C formula is:

where:P(>0)=Probability of delay greater than zeroN=Number of servers in full availability groupA=Traffic offered to group in Erlangs

Busy Hour Call Attempts(BHCA)

• Erlang is concerned with occupation of resources

• 1 call of 1 hour duration is same as 6 calls of 10 minutes duration

• For evaluation dynamic performance of modern telecommunications equipment the load on the system of messages flowing trough the system is important

• BHCA is more relevant for evaluating dynamic load

Debugging Embedded Real Time Applications

• Unix or PC Debuggers and Tools not applicable

• Hardware environment may not be stable

• Difficult to differentiate between Hardware of Software problems

• Problems may be difficult to reproduce and capture

• CPUs used are quite frequently new and not fully debugged

Firmware Debugging Tools

• Romulators

• Logic Analysers

• FW Simulation on Unix/DOS

• FW Application Monitor

• In-Circuit Emulator ICE

• CPU Performance Analysers

• On Chip Debugger (BDM)

ROMULATORS

• Replacement of the ROM with RAM which is controlled by the PC (for example)

• Instead of burning the ROM every time code has to be modified the new executable is downloaded to RAM

• Very cheap

• No support for debugging – no control or the CPU

– no visibility or CPU activity in addition to what is available trough the application interface

Logic Analysers

• Device capturing all signal activities on the Processor Pins

• With addition of specific SW signal trace may be represented in form of C/C++ Code

• Logic Analyser is very common HW Test Tool

• Doesn’t provide ability to control the target CPU

• No monitoring of internal CPU registers available

FW Simulators

• Customised (cut-down) version of the FW running on the UNIX or DOS

• This allows use of UNIX/DOS tools– Full blown Debuggers

– Run time error detection tools like PURIFY

» detection of the memory leakage

» detection of uninitialised variables

» stack overflows/underflows

» Reading or writing past the boundary of an array

» Using Freed memory

– Performance Analysers (Quantify, ...)

Built-In Debug Monitors

• Debug facility integrated with the application

• It may print-out relevant debugging info

• It may also be interactive where user can partially control the application and request some info from FW

• Assumes that HW is working well and that FW is stable enough.

• It is not good for debugging applications which are crashing - resetting

• Use of Monitor may affect the speed of application which is sometimes not acceptable

In-Circuit Emulators (ICE)

• Provides full control of FW (CPU)– Break Points on different conditions (HW or SW)

– Modification of Registers and Memory areas

– Single Step execution

– Source Level Tracing

• Allows running FW at full speed

• Application may run from “Emulation” RAM instead from ROM (this allows fast modification)

• RTOS Support is issue

• May be expensive (20-50 K)

CPU Performance Analysers

• Provides ability to determine how much time CPU is spending executing particular area of code.

• This allows to perform the performance analysis and tuning

• In case of Analysers which are RTOS aware CPU utilization per task can be displayed

Performance Analyser Example

CPU Background Debug Mode (BDM)

• Some microprocessors provide debugger implemented in CPU micro-code (BDM or JTAG)

• Full debug support available: registers can be viewed and modified, memory can be read or written ...

• On chip HW break-point support

• Control via dedicated serial interface on chip connected to PC running debugger application

• This is relatively cheep alternative to ICE (2-5K)

• No Tracing of application execution available

• For tracing combination of Logic Analyser and BDM is required

Debugging Tools Summary

• Cheep but very limited tools (no control of CPU)– Romulators

– Logic Analyser

• Built-in monitor allows some control of CPU but requires stable HW & FW platform

• FW simulation provides good flexible environment but it may not represent real-time aspect of real target

• Best Solutions for Low Level Debugging– In Circuit Emulator (ICE) - It can be expensive

– BDM with addition of Logic Analyser for tracing

slide 1. slide 2 outline protocol verification / conformance testing ttcn automated validation...

Documents

test slide

protocol slide

conformance test suitescases

test realization

test laboratories

abstract test cases

utilising test equipment

generic test tools