1

Master Test Specification

Simple Rail Road Command Protocol (SRCP)

Submitted by:

Rishu Seth

High Integrity Systems

Fachhochschule Frankfurt am Main

Matriculation number-936161

2

Table of Contents 1.Introduction ...................................................................................................... 4

1.1 Overview ............................................................................................... 4

1.2 Test Object ............................................................................................ 4

1.3 Test Environment .................................................................................. 5

1.4 Test Plan ................................................................................................ 6

1.5 Test Organisation .................................................................................. 7

1.6 Test Resources ...................................................................................... 7

2. Test Planning ................................................................................................... 8

2.1 Test Areas ............................................................................................. 8

2.2 Test Metrics .......................................................................................... 9

2.2.1 Defect Severity ...................................................................... 10

2.2.2 Defect Priority ....................................................................... 10

2.3 Test Requirement ................................................................................ 12

2.3.1 Network Connection ............................................................. 12

2.3.2 Lexical Units ......................................................................... 12

2.3.3 Commands ............................................................................. 13

2.3.4 Replies ................................................................................... 13

2.3.5 State of Server ....................................................................... 13

2.3.6 Hand Shake Mode ................................................................. 14

2.3.7 Command Mode .................................................................... 15

3. Test Specification .......................................................................................... 16

3.1 Equivalence Classes ........................................................................... 16

3.1.1 Feedback Sensors .................................................................. 16

3.1.2 Generic Accessories .............................................................. 17

3.1.3 Generic Loco ......................................................................... 17

3

3.2 Black Box Test Cases ......................................................................... 17

3.2.1 Hand Shake Area ................................................................... 18

3.2.2 Command Area ..................................................................... 19

3.2.3 Info Area ................................................................................ 20

3.3 White Box Testing .............................................................................. 20

3.3.1 SRCPD Source ...................................................................... 20

3.3.2 Masking of Condition ............................................................ 26

3.3.3 Cause Effect Graph ............................................................... 26

4. Findings .......................................................................................................... 30

5. Conclusion ...................................................................................................... 30

References .......................................................................................................... 30

4

1. Introduction

1.1 Overview

This document is Master Test Specification for Simple Railroad Command

Protocol (SRCP: Version 0.8.3). The initial focus of this document is to analyze

SRCP, it's specification and provide test cases & scenarios, that will

eventually lead to errors in the system and then after that also to look inside the

code and find anomalies and errors. Although to provide the full test coverage

is not exactly possible. The main objectives of this document are:

1. It provides a status report of the test object in comparison to the

requirements. Is the product fit for release? Adding value to the product by

finding defects.

2. To Test the most prioritized elements of the test-object. The realistic use of

the software should be considered.

3. Testing should be unbiased and from different point of views considering the

needs of different users.

4. Test process is well structured. Is it repeatable and predictable?

1.2 Test Object

SRCP is an internet protocol for controlling and programming (digital) model

railway systems. It produces client server communication. The server is the

component that is connected with the plant.

The client can be run on the same computer as the server. The server has no

clue about how the plant looks like. It is the server’s task to translate the

commands of SRCP into action.

A SRCP server is unaware of the exact concrete railway system. And the

topology and even it is not being informed about the installed equipment. Its

5

task is to collect all available information as precisely as possible handling

knowledge about the possibilities and limits of the controlled plant. Its data

have always to be understood as the best knowledge about the state of the

railway system.

Limitation of SRCP

SRCP does not fulfill any formal real time demands.

Merits of SRCP

1. A SRCP server abstracts the control of the construction.

2. Paying attention to different safety strategies.

3. Scaling Capability.

An SRCP server represents one central unit (CU). A CU provides at least one

bus over which information is read or commands are executed. A client contacts

the server in order to send commands to the plant and to receive information

from there. So there are SRCP channels between client and server, where the

client sends commands to the server and SRCP channel between server and the

plant, where the server has to execute the command on the plant. SRCP

connection is based on TCP (Transmission Control Protocol).

The Test object will be the protocol version 0.8.3 of SRCP and the SRCP

Daemon 2.0.12, which will emulate the SCRP environment.

1.3 Test Environment

The SRCP Test Environment consists of:

1. SRCPD - will emulate the SRCP server and a digital plant.

2. Server - A machine with OS Linux. SRCP Daemon will run on the server.

6

3. Client - The Client represents a working station with a python written

script, which will connect to the server. OS can be either Linux or

Windows.

4. Java - will provide a client program that sends commands to the SRCP

server.

5. Documentation – MS (Microsoft) Word, Excel, Latex.

1.4 Test Plan

The overall test strategy consists of 6 main phases or levels .The levels are

given below

Test-Planning

• To Analyze the SCRP specification.

• It provides test metrics to measure the quality of the test object.

• It formulates the requirements of the test object.

Test-Case-Generation

• Use the test metrics & requirements to generate test cases for black box

testing. Also divide the test cases in different test areas.

• Provide a python written client program which will help us to execute the

test cases on the test object.

• Inspect the source code of SRCP Daemon and specify white box test

cases.

Test-Case-Execution

• Execute test cases & log results.

Analysis Test Result

7

• To analyze test result and compare it with desired result.

Bugs Analysis

• To find the bugs that means deviation from our previous stag.

Test-Documentation

• Visualize test results.

• Defect Reporting.

Final-Test-Result

• Finalized the test result after Bugs Analysis.

1.5 Test Organization

• Developers: Dr. Schönfelder.

• Testers: Students.

• Execution of Tests: Workstations at the university.

1.6 Test Resources

• SCRPD 2.0.12 – source code.

• Specification of SRCP 0.8.3

• Configurations file "srcpd.conf".

• A client program written in Java will be used to connect to the server and

execute test cases.

8

2. Test Planning

2.1 Test Areas There are mainly three main test areas:

• Hand Shake Area: On initial connection hand Shake Mode is activated.

Then after this connection, its according to the demand of the client

whether to activate Command Mode or Info Mode.

• Command Area

• Info Area

There are some different Sub Test Areas also which are used in combination

with these three Test Areas:

1. Server: A SRCP server represents exactly one central unit (CU) in the

usual sense. Every CU disposes of at least one bus for addressing on

which information is read and/or commands are executed. If a SRCP

server can manage and provide several CU it is valid.

9

2. Feed Back (FB): Feedback devices are encoders that signal an occurrence

on construction

3. Generic Accessoire: A generic Accessoire generally indicates a decoder

that can serve one or more ports under one address. Often these are

switch decoders or signal decoders working as impulse decoders.

• Generic Loco (GL): It indicates engine decoders in the engine

address room of the hardware.

• Locks: Devices locking other devices. The LOCK devices are

devices providing a lock over another device. They are optional.

2.2 Test Metrices

It is the measure of software quality and confidence of our test object. Test

metrices have various uses:

1. It can reduce working time - once we have a matrix for a specific issue, it

will take our less time to test it or to think how to test it.

2. It is logical and testing challenging - It is a challenge to make our own

matrix and to find common issues that are repeatable from project to project.

For Testing of this project, used metrics will look at:

• Functional areas

• Requirements Check

For Test coverage and consistency of Test Effort

During Test Preparation this metric uses:

• Number of Test requirements compared to Functional Area/

Requirements

10

• Number of Test Cases planned compared to Test Cases ready for

execution

During Test Execution this metric uses :

• Number of Test Cases Passed, Failed or Blocked.

• Number of Test Cases Executed compared to Test Cases Planned

Severity: Grouping by potential danger caused.

2.2.1 Defect Severity

It is the measure of how lethal a defect can be and how strong is the effect of

failure on software user.

• Critical: It is the highest order of severity and results in the failure of

complete software system of a subsystem or of a software unit within the

system.

• Major: It is almost relevant to “Critical” but there are still some chances

of processing some acceptable alternatives to get desired result.

• Minor: This is referred to as some common normal defects which are not

as lethal as other two but somehow effect the normal working of a

software.

DEFECT SEVERITY:

TEST CLASS CRITERIA

DEFINITION STANDARD % PAS RATE

CRITICAL ESSENTIAL 100 %

MAJOR NECESSARY 90%

MINOR NOT IMMEDIATE 70%

2.2.2 Defect Priority

11

It is the measure of priority that is set in regard to addressing defects and

according to its severity, to set minimum time within which it should be fixed.

• High: This is the most prioritized area and defect falling in this category

severely affect the system’s working and should be fixed as soon as

possible.

• Medium: This is having a bit less priority and the defects can be fixed in

normal course of development activities.

• Low: This is having the lowest priority and the defects can be fixed after

the more severe n priortised defects are take care of.

For example when a client connects to the SRCP server, the server generates a

welcome message containing information about the server. The defect of not

generating the welcome would have minor severity, but the process is repeated

every time we connect to the server and that is the first thing we see. So the

priority for this defect would be high and this would be test cases we should

definitely consider.

Another example would be that all commands are case sensitive and have to be

written in uppercase letters. In case some commands would be accepted with

lower case letters that would result in a defect, which doesn't cause a failure and

doesn't really impair usability. So the defect would have a minor severity and

minor priority. So a test case for this defect would not be relevant, considering

our matrices.

DEFECT PRIORITY:

SEVERITY LEVEL DEFINITION

High High system crash data loss

Medium Wrong result operation error

Low Low minor problems

12

The table tells us that we will only consider relevant combinations for our test

cases. We cannot provide full test coverage of the Test-Object, because the

number of all possible test cases could be from high to astronomically high.

Implementing all possible test cases would consume too much time, so that by

the time we are finished will all possible test cases, new versions of our Test-

Object would be available, which also need to be tested. So for the test cases we

only focus only on the important ones with the help of our matrices

A test-area from the main test areas (Chapter 2.1) has passed the test process if

it does not contain any defects of the severity level "Critical“. It is essential to

the product. Here need 100% pass rate.

1. A test-area has passed the test process if it does not contain less than 90%

pass rate. It is called important.

2. A test-area has passed the test process if must not be less than 70% pass rate

and it is called desirable test class.

2.3 Test Requirements

Test-Requirements tell us what requirements the Test-Object has to full fill,

by analyzing the specification of the Test-Object. They will help us to derive

test cases.

2.3.1 Network Connection

The SRPC is based on the data transfer technologies of TCP (Transmission

control Protocol). One port is occupied. At present the port 12345 SHALL

be used. Registering with IANA can change this for the future and give the

port a name, too. The temporary name is srcp 12345/tcp

2.3.2 Lexical Units

1. In this case, all character codes are described in their decimal form as #

followed by the numeric value of the character.

13

• The entire communication consists of the characters of the 7bit ASCII

character set in the range from #32 to #127 (including the borders).

• Again the characters #9 (TAB) is used for space, #10 and #13 (CR, LF)

for end of line are valid.

• All further characters (esp. with a code value >127) are ignored in

incoming data and removed.

• In outgoing data they can be included, but is not used within SRCP.

2. The characters #9and #13 are seen as white space.

2.3.3 Commands

Commands are send in the “Command Mode” and in “Hand Shake” from client

to server. The SRCP server processes the command and generates a response

that is to be sent to the client. Command consists of:

1 Commands consist of a command word, followed by a command

parameter list separated by white space.

2 The end of a command is the end of line.

3 The end of line always consists of the character #10(\n, LF).

4 A prefixed #13(\r, CR) is valid. The character #13 is ignored.

2.3.4 Replies

A server HAS TO reply to every command of a client. That is to be sent to the

client on the same connection.

A reply SHALL be single-line. With the sign \(Backslash, #92) immediately

before the LF(resp CRLF) it is marked that the reply contains another line.

A single line including the end of the line MUST NOT exceed the length of

1000 characters.

If the result of a command can be ascertained due to communication with

connected model railway devices this opportunity SHALL be used. In case there

are insufficient results for a task the answer is “416 ERROR no data”.

14

2.3.5 State Of Server

SRCP server is always in one of three possible states towards the client: hand

shake, command mode or info mode.

Hand Shake Mode

In this Mode, after establishing a connection and sending the welcome message

the server waits for a command from the client. The server executes it and sends

a single line reply to the client. Then the server waits for the next command.

That way information and commands are being exchanged. Some example

commands used in this mode are:

1 SET CONNECTIONMODE SRCP <MODE> The type of connection is

agreed upon. Valid are INFO for the unidirectional information mode and

COMMAND for the bidirectional command mode. The server sends either 202

OK or the error message 401 ERROR unsupported connection mode to the

client.

2 SET PROTOCOL SRCP <VERSION> The desired protocol that is to be used

is arranged. If this command is missing the SRCP version given during

15

welcome is accepted. Every released version number from and including 0.8 is

valid as <VERSION>.

2.3.6 Command Mode

During command mode the server waits for commands from the client and

executes these. As the result of the execution the server always creates a reply

due to be sent to the client in the same TCP session. Subsequently the next

command is executed.

The server‘s replies to the client in the command mode always consist of a time

stamp at the beginning, a numerical answer code and further data. The

numerical answer code is structured in groups. 100-199 marks information and

results, 200-299 includes receipts confirming the processing according to the

rules, 400-499 marks error while executing commands, 500-599 errors by the server itself, 600-699 specific error codes in implementation.

1xx INFO: Informations, results

2xx OK: Command is accepted and brought to execution. Attention: This is no

confirmation for actually executing the command.

4xx/5xx ERROR: An error condition has occurred. The command is ignored

and not executed.

6xx ERROR: A server specific error has occurred. The command in question is

not executed.

The significance of the time stamp arises from the kind of reply: For quittances

(OK, ERROR) it specifies the point in time when the command is processed and

the receipt OK or ERROR is generated.

Some conditions in command mode are :

1. Replies of the commands by the server are to be sent to the client in the

same TCP session.

2. GET, SET, CHECK, WAIT, INIT, TERM, RESET, VERIFY are

commands that are defined in command mode.

16

3. Valid commands must always be executed.

4. It is not possible to go to info mode out of command mode.

2.4 Result of Planning Phase

PROJECT PLAN:

APRIL MAY JUNE Test planning

Test case generation

Test case execution

Analysis test result documentation

Bugs analysis documentation (Repeat until desire level)

Final result of test cases documentation

3. Test Specification

3.1 Equivalence Class

Equivalence class method will help us to develop test cases in a structured way

and Assists finding test data to provoke a special expected behaviour.

3.1.1 Feedback Sensors

1. Class of addresses allowed: 1 ≤ x ≤ 10.

Representative element = 1.

2. Class of addresses not allowed: x < 1 && x > 10.

Representative element = 11

17

3.1.2 Generic Accessoires

1. Class of all possible addresses allowed: 0 ≤ x ≤ 324.

Representative element = 1.

2. Class of addresses not allowed: x < 0 and x > 324.

Representative element = 325

3.1.3 Generic Loco

1. Class of all possible addresses allowed: 1 ≤ x ≤ 81.

Representative element = 1.

2. Class of addresses not allowed: x < 1 and x > 81.

Representative element = 82

3.2 Black Box Test Cases

A python-written client program will run input/output driven tests on the SRCP

server. The client program will communicate with the server through a TCP

based connection. Input data will be provided through the specification of

SRCP.

Because of the configuration of SRCPD we will be able to use bus 1 for most of

the test cases and bus 0 for "SERVER", "SESSION", "LOCK",

"DESCRIPTION" or "TIME".

18

FIGURE: Server Client Communication

19

ID Command Description Output Severity/Priority

1 Set Connection Mode SRCP Command

Enter command

mode in session

1

202 OK

Connection

Mode OK

Critical / High

2 SET

PROTOCOL

SRCP 0.8.3

Set protocol

version for

server

201 OK Protocol SRCP

Critical / High

3 Set Connection

Mode SRCP

Try enter

invalid mode

401 Error Unsupported

Connection

Mode

Critical / High

4 Set Connection Mode SRCP INFO

Enter info mode 202 OK

Connection

Mode OK

Critical / Major

5 GO Complete Hand

Shake

200 OK GO Critical / High

ID Command Description Output Severity/Priority 1 RESET 0 SERVER Reinitiate

server 200 OK Critical /High

2 INIT 1 POWER Initiate power supply on bus 1

200 OK Critical /High

3 SET 1 POWER ON

Set power supply on

200 OK Critical / High

4 GET 1 FB 1 Get info from updated FB on bus 1 with address 1

INFO 1 FB 2 123

Major /Medium

5 SET 0 POWER Get server out 415 ERROR Major/ High

20

OFF of power supply (not allowed)

forbidden

ID Command Description Output Severity/Priority 1 TERM 0

SERVER Close connection to server

100 INFO 0 SERVER

Critical / High

2 INIT 1 POWER

Initiate power supply on bus 1

101 INFO 1 POWER

Critical/ Medium

3 INIT 1 FB S88 888 1

Initialize a FB on bus 1 with address 1

101 INFO 1 FB

Major /Medium

4 INIT 1 GA 0 S 1 1

Initialize a GA on bus 1 with address 0 over protocol S with port & value 1

101 INFO 1 GA 0 1 1

Critical / Medium

5 TERM 1 GL 1 Set engine to default state & take it out of communication

102 INFO 1 GL 1

Major / Medium

Figure Test Case pass and fail ratio Total 45 Pass 18 Fail 27

0

5

10

15

20

25

30

35

40

45

50

Total Test Cases Pass Fail

Series1

21

3.3 White Box Testing

3.3.1 SRCPD Source

White Box Testing is a method of testing software that tests internal structures

or workings of an application, as opposed to its functionality. In white-box

testing an internal perspective of the system, as well as programming skills, are

required and used to design test cases. The tester chooses inputs to exercise

paths through the code and determine the appropriate outputs.

Main Criteria in White Box Testing is code coverage, which means all the

branches in the control flow graph are going to be covered. The White box

Testing is going to be performed on the Specific modules of the SRCP server to

ensure safety of the tested models.

First we need to look at the most important source files for out test process. For

each module of SRCPD there are 2 different files. The *.h files contain the

defined functions and constants. The *.c files contain the exact definition of the

functions which were defined in the corresponding *.h files.

I have performed white box testing on the following module of the SRCP:

Followings are the bugs findings performed by doing statement and logic

coverage white – box testing.

MODULE: srcp-fb.c 1. Function: initfb Bug: Here bus number is not checked and ‘fb’ is invalid under under bus number zero.

22

Suggestion: Bus number should be checked before initializing the ‘fb’. 2. Function: queue_reset_fb Bug: While on sleep it does not show any error message.

Suggestion: It should return an error message or result in case an error occurs.

3. Function: enqueueInfoFB

Bug: Here declared message length is constant i.e. 1000 and if another parameter is passed , it may crash.

Suggestion: It should be made Dynamic.

Problem FB: Take FB on bus 1 with address 1 out of communication.

It is not running on bus 1.

Suggestion: It should be running on bus 1.

Problem: SET 2 FB 1 SET FB with too short parameter list. Address is not mentioned.

Suggestion: Parameter list is checked and also address of bus should be checked

Flow graph:

It is a representation of all paths that have been traversed by a program during its execution, using graph notation. Using this flow chart, we can compute the number of independent paths through the code. In a control flow graph each node represents a basic block. Direct edges are used to represent jumps in control flow. There are two specially designated blocks:

Entry block: Through which control enters into flow graph.

Exit block: Through which all control flow leaves the graph.

1. Function: initfb Its code part and corresponding flow graph is given below:

CODE:

23

FLOW GRAPH:

2. Function: queue_reset_fb

Its code part and corresponding flow graph is given below:

24

CODE:

FLOW GRAPH:

25

3. Function: enqueueInfoFB

Its code part and corresponding flow graph is given below:

26

CODE:

FLOW GRAPH:

3.3.2 Masking of Conditions:

It is necessary for efficiently analyzing the test paths. In this case multi-condition decision is made separately to analyse the test paths efficiently.

In my white box testing I have made a masking of condition graph for ‘init_fb’ function in the module ‘srcp-fb.c’.

Masking of condition:

27

3.3.3 Cause Effect Graph

This is basically a hardware testing technique adapted to software testing. It

considers only the desired external behavior of a system. This is a testing

technique that aids in selecting test cases that logically relate Causes (inputs)

to Effects (outputs) to produce test cases.

A “Cause” represents a distinct input condition that brings about an

internal change in the system. An “Effect” represents an output

Steps to design the cause effect Graph

Step – 1: For a module, identify the input conditions (causes) and actions (effect).

28

Step – 2: Develop a cause-effect graph.

Step – 3: Transform cause-effect graph into a decision table.

Step – 4: Convert decision table rules to test cases. Each column of the decision table represents a test case.

ESTABLISHING A CONNECTION:

CAUSE EFFECT

(1) Client connects to the server by TCP/IP connection

(E1) Welcome string sent by the server

(2) Client desired operation mode (E2) Server reply according to operation mode demand by client

(3) Server – Client sent GO command (E3) Server confirmed GO command

CEG 1:

CEG 2:

29

CEG 3:

30

DECISION TABLE:

TEST CASE 1 2 3 4 (1) Client connects to the server by TCP/IP connection

1 0 1 0

(2) Client desired operation mode

1 1 1 0

(3) Client sends GO command

0 1 1 0

4. Findings

• Delayed Response from server

• Not proficient details in Specification

• Time Out – Server gets stuck after certain period of time

5. Conclusion

In our master test case specification, the Black box test cases are selected according to the SRCP protocol and our assumptions stated earlier. White Box test case findings were done while analyzing code and it is very important to do both Black Box and White Box testing together so as to find more bugs and be more sure about working of the software.

6. References

[1] Class notes and slides of Dr. Schoenfelder

[2] The Art of Software Testing , Second Edition, Glenford J. Myers