protocol testing

IHA præsentation 1

Protocol Testing

Lesson 5

IHA præsentation 2

Outline for today

• Guidelines for testing protocols

• TTCN – A Standard Language For Testing Protocols• Motivation

• The Language

• The usage

IHA præsentation 3

Guidelines for protocol testing

IHA præsentation 4

Guidelines

• Unit Testing• tests if individual units of source code are fit for use.• A unit is the smallest testable part of an application

• Integration Testing• individual software modules are combined and tested

as a group • Occurs after unit testing

• Coverage Testing• describes the degree to which the source code has

been tested

IHA præsentation 5

Guidelines

• Smoke Testing• Collection of written tests that are performed on a

system prior to being accepted for further testing

• Regression Testing• seeks to uncover software regressions• previously working software functionality stops working

as intended

• Conformance Testing• to determine whether a system meets some specified

standard

IHA præsentation 6

Conformance Testing Framework

• Conformance testing is the process of testing the extent to which implementations adhere to the requirements stated in relevant standard or specification

• Conformance Testing is functional black-box testing

• Functional refers to the correct functional behavior of an Implementation Under Test (IUT)

• Black-box means that the internal structure of the IUT remains hidden

IHA præsentation 7


IUT – Implementation Under Test

UT – Upper Tester function

LT – Lower Tester function

PCO – Point of Control & Observation

TCP – Test Coordination Procedures

ASP – Abstract Service Primitive

IHA præsentation 8


UT plays the role of a user that makes use of the service provided by the IUT

LT plays the role of the peer entity of the IUT, i.e. the LT and the IUT communicate in order to provide the service to the IUT

IHA præsentation 9


PCO is a standardized interface.

Typically the lower interface of an IUT is accessible only from the remote

Communication is always meant to be asynchronous => PCO is modeled as two FIFO queues

IHA præsentation 10


TCP can be used to coordinate the actions of LT and UT.

LT and UT can be on separate testers.

IUT and LT communicates by means of ASP.

PDUs are encoded in ASPs of the underlying service.



• Test case development starts with:

• Identification of a test purpose

• Description of a single requirement or set of related requirements

• A test case is an implementation of a test purpose



Test Case SchemeA test case starts and ends in stable testing states.

Preamble to take the IUT from stable testing state to test state.

Test body is performed in order to check the test purpose.

Verification to check test body

Postamble to drive the IUT into a stable testing state again


Protocol Testing

• Until now we have:• Developed our protocol layer including:

– Timers– State - event model– PDUs to be exchanged

• Interfaces to upper and lower layers

HOW DO WE TEST OUR PROTOCOL LAYER?


Protocol Testing

• Protocol Testing is mainly about:• Testing the state – event model.

PDU (message)Our protocol layerTest System

??

Observe response


Protocol Testing

• Testing the state – event model

• Difficult part is when we send an unexpected message into our layer

Unexpected PDU (message)

Our protocol layerTest System

??

•Ignore – take no action

•Reject – return an explicit denial to the sender

•Allow – take action to process the event

Standard testing languages and tools available


Test & Testing Control Notation Edition 3(TTCN-3)


• Introduction• What is TTCN? • Why use TTCN?

• The TTCN-3 Language• Background• Example

• TTCN-3 Usage• Where is it used?• Future Outlook


What is TTCN?

• Internationally standardized testing language for formally defining test scenarios and their implementation.

• Designed purely for testing

• Tool independent

• Flexible and powerful language• Combined use with other languages like ASN.1• Well-defined syntax, interchange format• Different presentation formats (e.g. tabular and graphical)• Support text-based protocols• Support for IP protocols


What is TTCN?

• In TTCN we can….

• Define Test Suites

• Define Test Cases

• Define Test steps

• Declare variables (test suite & test case variables)

• Declare timers

• Create PDUs

• Etc..


What is TTCN?

• TTCN Edition 3 is documented in a serie of ETSI Specification

• ES 201 873-1 (TTCN-3 Core Language)

• ES 201 873-2 (Tabular presentation format)

• ES 201 873-3 (Graphical presentation format)

• etc.

• etc.


Why Use TTCN?

• Quality• a well-tested product increases confidence in the product’s quality

• Time/Cost Improvement• TTCN is a language designed for protocol conformance testing• TTCN is standardized• TTCN is fairly simple

• Formal• strict rules, complete BNF and operational semantics

• Abstract• TTCN test suites are independent of an implementation


Application Areas

• New application areas• Software Testing

• Text-based protocols…

• Additional communication paradigm• Message-based communication

• Procedure-based communication

• Different kinds of testing

• Functional testing

• Conformance testing (behavioral testing)

• Scalability testing…


SUT

TTCN Basics

PASS, FAIL orINCONCLUSIVE

TTCNTest Suite

Result

IUTImplementation

Under TestObserve Responses

Send Stimuli

• Behavioral testing

• send stimulus, observe response

• was response received within specified time?

Test System


TTCN – Example

testcase MyTestCase() runs on DNSTester {

P.send(query);

P.receive(answer);

setverdict(pass);

stop;

}

Tester

MTC

Verdict = pass

P P SUT

query = (www.iha.dk, A)

answer = (www.iha.dk, 193.65.199.110,A)

Minimal Test Configuration


TTCN-3 test systems in a nutshell

• TTCN-3 specifies a test but a test system is needed for test execution

• TRI and TCI standards define test system architecture

• TTCN- 3 tools are required to support internal interfaces

• Allows reuse of test platforms with different tools but also for different SUTs

• A test system requires• A TTCN-3 tool = TTCN-3

compiler and execution environment

• A test platform for a specific device under test

Note: Tools come with default Test Control & Logging

Test System ExecutorTest System Executor

SUT Adapter Platform Adapter

Test Control

System Under Test (SUT)

[compiled]TTCN-3 Executable

CodecsLogging

TTCN-3 Test System

TCI

TRI

( s + s)

( )

TCI = TTCN-3 Control InterfaceTRI = TTCN-3 Runtime Interface


An example adaptation: A IPv6 test system

Test System ExecutorTest System Executor

Ethernet Adapter Real-time Adapter

ParameterFile

Open Source FreeBSD Router

[compiled]

IPv6 ATS TTCN-3 Executable

IPv6 Codecs

ToolLogging

TTCN-3 Test System

TCI

TRI


User’s view of TTCN-3

TTCN-3 CoreLanguage

ASN.1 Types & Values

IIDL Types

XML Types

C/C++ Types

Other Types & Valuesn

Tabularformat

Graphicalformat

Presentationformatn

TTCN-3 User

Text format


The Core Language and Other Presentation Formats

TTCN-3 Core Language

Text format

PresentationFormat3

PresentationFormatn

Graphical Format

Tabular Format

• Core format is text based (most popular)

• TTCN-3 can be edited or viewed in other formats

• Tabular format (for TTCN-2 people)

• Graphical format (good for visual overview)

• Other standardized formats in the future?

• Proprietary formats possible


Example Core (Text) Format

testcase TC_resolveEtsiWww() runs on DnsClient {

timer t_ack;serverPort.send(m_dnsQuestion("www.etsi.org"));t_ack.start(1.0);alt {

[] serverPort.receive(mw_dnsAnswer("172.26.1.17")) {setverdict (pass);

}

[] serverPort.receive { // any other message setverdict(fail); }[] t_ack.timeout { setverdict(inconc);

}

} t_ack.stop;}


testcase TC_resolveEtsiWww() runs on DnsClient

Example Graphical Format

DnsClientmtc

DnsPortserverPort

m_dnsQuestion("www.etsi.org")

timer t_ack

t_ack

altmw_dnsAnswer("172.26.1.17")

?

t_ack

inconc

fail

pass

t_ack


Example Tabular FormatTestcase

Name TC_resolveEtsiWww()

Group

Purpose

System Interface

MTC Type DnsClient

Comments

Local Def Name Type Initial value Comments

t_ack timer

Behavior

serverPort.send(m_dnsQuestion("www.etsi.org"));

t_ack.start(1.0);

alt {

[] serverPort.receive(mw_dnsAnswer("172.26.1.17")) { setverdict (pass); }

[] serverPort.receive // any other message { setverdict(fail); }

[] t_ack.timeout { setverdict(inconc); }

}

t_ack.stop;

Detailed Comments:


Use of TTCN-3 With Other Languages

• TTCN can be integrated with types systems of other languages

• Fully harmonized with ASN.1 (1997)

• Harmonized with other languages

• IDL, XML, C/C++

TTCN-3 Core Language

IDL Types & Values

Other types & Valuesn

XML Types & Values

ASN.1 Types & Values


ASN.1

• Abstract Syntax Notation One• language for describing structured information• standardized internationally (ISO/IEC 8824, ITU-T X.680)

• TTCN includes ASN.1• used for creating data descriptions and constraints• can be part of TTCN script or imported as a module

• SDL also includes ASN.1• ITU-T Z.105

• ASN.1 can be re-used across design and test!

ASN.1

SDLTTCN


TTCN – Example

testcase MyTestCase() runs on DNSTester {P.send(query);P.receive(answer); setverdict(pass);

stop;}

Tester

MTC

Verdict = ???

P P SUT

query = (www.iha.dk, A)

???

• P.receive(answer) blocks until it receives a message that matches answer

• Any other message does not unblock the tester, which then blocks forever

• If no message is received, the tester will also block forever


TTCN Test Case Example

testcase MyTestCase2() runs on DNSTester {

timer t := 5.0; P.send(query);

t.start;

alt { [] P.receive(answer); { setverdict(pass); } [] P.receive { // any message setverdict(fail); } [] t.timeout { setverdict(inconc); } } stop;}

testcase MyTestCase2() runs on DnsTester

Testermtc

SUTserverPort

query("www.iha.dk")

timer t_ack

t_ack

altanswer("172.26.1.17")

?

t_ack

inconc

fail

pass

t_ack


Test Suite

Building blocks of a TTCN-3 Test Suite

Test Data Types

Data types which specify• Structure of messages or calls and their information elements (fields, parameters)• Internal data structures (e.g., for computation)• Possibly encoding or display informationBuilt-in basic types integer, boolean, float, bitstring, hexstring, octetstring, charstring, universal charstring... and structured types record, record of, set, set of union, enumerated

... and special types such ascomponent, port, verdicttype, default, etc


Test Suite


Actual Test Data

Test Data Types

Actual test data (values) used during testing • Constants or Templates for specific message or call parameter values• Matching expressions for allowing multiple message or call parameter values

• value range, value list, wildcards, presence, length, size, permutation• regular expressions

• Using also template decomposition, parameterization and modification


Test Suite


Test Configuration

Actual Test Data

Test Data Types

Static aspects

• Test component and port types

Dynamic aspects

• Dynamic instantiation and management of test components

• Mappings of test components to abstract test system interfaces

• Connections between test component interfaces

• Management of test components


Test Suite


Test Behaviour

Test Configuration

Actual Test Data

Test Data Types

test cases

• specify sending/receiving messages, computation (e.g., checksums), and verdict assignment

• can be decomposed with functions and altsteps

• can (re)use default behaviour

• can use timers and timeouts

test execution control (optional)

• order, repetition, conditions, etc


TTCN-3 Module

module EtsiDnsTests{

// Test definition part

control{

// Test execution part// (optional)

}}

Module (…)

Module Control

Module Definitions


Definitions Part


// Message structure// Actual test data// Test configuration// Test Case definitions

}



group MessageStructure{

// Defintions of message types }group TestData{

// Templates for messages instances} group TestSystemConfiguration{

// Port and component types and mappings}group TestCases{

// Test case definitions}

}

Structured Definitions Part


Message Structure and Test Datatype record DnsMsg // simplified message structure!{ DnsMsgKind kind, charstring question, charstring answer optional}type enumerated DnsMsgKind {e_query, e_response}

template DnsMsg m_dnsQuestion( charstring p_question ){ kind := e_query, question := p_question, answer := omit // no answer}

template DnsMsg mw_dnsAnswer( charstring p_answer ){ kind := e_answer, question := ?, // any question ok answer := p_answer }


Test Configuration

type port DnsPort message { inout DnsMsg} // Note: port types may also allow multiple different // message types or restrict the direction

type component DnsClient { port DnsPort serverPort} // Note: component types can also define multiple port // instances of the same or different port type and // declare timers, constants or variables


Test Behaviourquery

responseserverPort


timer t_ack;serverPort.send(m_dnsQuestion("www.etsi.org"));t_ack.start(1.0);alt {

[] serverPort.receive(mw_dnsAnswer("172.26.1.17")) {setverdict(pass);

}[] serverPort.receive { // any other message setverdict(fail); }[] t_ack.timeout { setverdict(inconc); }

} t_ack.stop;}

mtc


The Control Part

Module (…)

Module Control

Module Definitions


// Test definition partmodulepar boolean mp_example;


// .. as in previous slide }

// Test execution partcontrol { if (mp_example) {

execute(TC_resolveEtsiWww()); }

}}


TTCN-3 Usage


TTCN in 3GPP

• Lessons from GSM• TTCN ATSs contribute to interoperability• powerful competitive tool for entry to the global GSM mobile market• TTCN ATSs widely used within members’ development processes

• 3GPP• TTCN in Protocol Conformance Specs

• 34.123-1 Prose description of tests• 34.123-2 Implementation Conformance Statement (ICS)• 34.123-3 TTCN Abstract Test Suite

• TTCN will be the normative specification


TTCN and Bluetooth

• Bluetooth SIG selected TTCN for...

• definition of Protocol Requirements tests

• interoperability

• definition of Profile Requirements tests

• application interoperability

• end-user expectations

RF

Baseband

AudioLink Manager

L2CAP

TCP/IP HID RFCOMM

Applications

Data

Con

trol

protocol testing

Documents

testing protocolsttcn

stable testing states

test purposedescription

test bodypostamble

test iutblackbox

protocol testinglesson

state event modeldifficult

lower interface