testing telecoms software with quickcheck

Testing Telecoms Software with QuickCheck

Thomas ArtsJohn HughesChalmers/ITU

Joakim JohanssonUlf Wiger

Ericsson

Session Border Gateway

Media Proxy

Opens media pin-holes for calls in

progress

2x10G Ethernet

Hardware for packet forwarding

Media Gateway Controller

Monitors signalling

traffic; opens and closes pinholes

2x1G Ethernet

Linux ”blade” processor

Session Border Gateway

Media Proxy

Media Gateway Controller

ITU standard protocol H.248

(Megaco) 150KLOC Erlang

1.5MLOC C

A QuickCheck Property

prop_reverse() -> ?FORALL(Xs,list(int()), lists:reverse(lists:reverse(Xs)) == Xs).

6> eqc:quickcheck(rev:prop_reverse()).....................................................................................................OK, passed 100 teststrue

A Wrong Property

prop_reverse_wrong() -> ?FORALL(Xs,list(int()), lists:reverse(Xs) == Xs).

10> eqc:quickcheck(rev:prop_reverse_wrong())......................Failed! After 22 tests.[-2,4,0]Shrinking......(6 times)[1,0]false

Testing the Media Proxy Control Software

Media Proxy

QuickCheckSend

random command sequences

Check the responses

Generate random messages in a random command sequence

But not completely random!

MediaDescriptor ::= SEQUENCE{

termStateDescr TerminationStateDescriptor OPTIONAL,streams CHOICE{

oneStream StreamParms,multiStream SEQUENCE OF StreamDescriptor

} OPTIONAL,…

}

Message Example: ASN.1 Description

• A media descriptor contains a list of streams

…with IWD Restrictions

• A media descriptor contains a list of streams

MediaDescriptor ::= SEQUENCE{

termStateDescr TerminationStateDescriptor OPTIONAL,streams CHOICE{

oneStream StreamParms,multiStream SEQUENCE OF StreamDescriptor

} OPTIONAL,…

}

QuickCheck Generator for Erlang Records

mediadescriptor(Streams) when Streams=/=[]-> {mediaDescriptor, #'MediaDescriptor'{ streams = case Streams of [{Id,Mode}] -> oneof([{oneStream,streamParms(Mode)}, {multiStream,[stream(Id,Mode)]}]); _ -> {multiStream, [stream(I,M) || {I,M}<-Streams]} end}}.

stream(I,Mode) -> #'StreamDescriptor'{ streamID = I, streamParms = streamParms(Mode)}.

Generators can be embedded in any data

structure

Message constructionLogic from the IWD

QuickCheck

Local Control DescriptorsAdd Request

Desc. Desc. Desc. Properties, Package M Comment

Media M

Stream M Multiple Stream descriptors can be included.

Localcontrol

O LocalControl will be included in all cases except when no media (m-line) is defined in the remote SDP.

mode O The default value of the mode property is “Inactive”, the property is not mandatory if the wanted value is “Inactive”.

StreamParms ::= SEQUENCE{

localControlDescriptor LocalControlDescriptor OPTIONAL,localDescriptor LocalRemoteDescriptor OPTIONAL,remoteDescriptor LocalRemoteDescriptor OPTIONAL,…,statisticsDescriptorStatisticsDescriptor OPTIONAL

}

streamParms(Mode) -> ?LET(RemoteMediaDefined, probably(), if RemoteMediaDefined ->

#'StreamParms'{ localControlDescriptor = localControl(Mode),

localDescriptor = localDescriptor(RemoteMediaDefined),

remoteDescriptor = remoteDescriptor(RemoteMediaDefined)}; true -> …… end).

Random boolean with

high probability

Two Cases: With and Without Remote Media

Megaco Commands

ContextContext

Megaco Commands

Context

ContextContext

Termination

Add

Returns Context ID &

Termination ID

Megaco Commands

Context

ContextContext

TerminationTermination

Add

Uses context ID

Megaco Commands

Context

ContextContext

TerminationTermination

StreamStream

Modify

Uses termination ID

Megaco Commands

Context

ContextContext

TerminationTermination

StreamStream

Subtract

Megaco Commands

Context

ContextContext

Termination

StreamStream

Subtract

Megaco Commands

ContextContext

Symbolic Test Cases

• Generated wrt an abstract state machine tracking contexts, terminations & streams

[{set,1,…,send_add, [?choose_cxt_id,Streams,Req]}, {assert,…,check_add,[{var,1}], {set,2,…,send_add, [{call,…,get_reply_cxt,[{var,1}]}, Streams2,Req2]}, {assert,…,check_add,[{var,2}]}]

Encode/Decode Error

• Recall the stream parameters…

StreamParms ::= SEQUENCE{

localControlDescriptor LocalControlDescriptor OPTIONAL,localDescriptor LocalRemoteDescriptor OPTIONAL,remoteDescriptor LocalRemoteDescriptor OPTIONAL,…,statisticsDescriptor StatisticsDescriptor OPTIONAL

}

Add/Modify Error

Add

Modify

Add/Subtract Error

• There one day, gone the next…

Add

Subtract

Add/Add/Modify Error

• If the terminations have different numbers of streams

Add

Modify

Add

Add/Add/Sub/Add… Error

• Extracted from a test case 160 commands long

Add

Subtract

Add

Add

Add

Subtract

Subtract