Experimental Dependability Assessment using a Faultload Specification Tool

Ruthiano S. Munaretti Bruno C. Fiss Taisy S. WeberSergio L. Cechin

Institute of Informatics - Federal University of Rio Grande do SulCaixa Postal 15064 - 90501-970 - Porto Alegre, RS, Brazil

{rsmunaretti, bcfiss, taisy, cechin}@inf.ufrgs.br

Abstract

Use of two or more fault injection tools in a test cam-paign enriches the scenario obtained from a test execution.Faultloads represent the main input for these tools but theirspecification mechanisms lack usability and expressiveness.This paper presents a full test scenario featuring the use ofjFaultload, which applies Java for the specification of fault-loads and translates them to specific formats that are ap-propriate to each available fault injector. FIRMAMENT, afault injector for communication systems, was integrated inthe environment and completes the test scenario. The ser-vice under test used to demonstrate the usability and expres-siveness of our solution is a video streaming session usingRTP Protocol.

1. Introduction

Testing is an essential task to evaluate systems. It allows

detection of implementation errors, as well as other depend-

ability impairments. Test engineers are responsible for all

issues related to test execution. It involves a lot of tasks,

such as the definition of a test process, the choice of test-

ing tools, the creation of test experiments, and so on. These

tasks must reach a reasonable coverage of errors, in order to

warrant a minimum level of reliability for the system under

test.

There are several testing techniques. We are particu-

larly interested in techniques that allow validating the sys-

tem behavior under faults. Among them, fault injection [5]

presents the advantage of high controllability. In that sense,

each fault that will be injected during a given test experi-

ment is predictable and easy to control, permitting light ob-

servation of system behavior under faults.

To effectively apply fault injection, a test engineer uses

tools, the fault injectors. The main goal of such tools is the

insertion of faults into the target system under test or into

the infrastructure supporting this target, as the communica-

tion network.

Without fault injectors, a test engineer may have to ex-

ecute hard tasks. For instance, he could develop his own

faulty procedures. Considering he could access the target

source code, he can manually include these procedures into

the code. But the inclusion of these faulty procedures is

highly intrusive and can deeply modify the features of the

target. This can potentially damage its consistence and in-

validate the test results. On the other hand, empirical tech-

niques (i.e., a disconnection of a network cable) can hide

a real fault condition and is also totally uncontrollable re-

specting the moment of activation and target state.

The existence of several fault injectors makes difficult

the choice of the most appropriate tool. At the same time,

the use of two or more fault injectors in the same test cam-

paign enriches the obtainable results. Moreover, a compar-

ison between these tools becomes possible, where the best

test approaches will be elected in a straightforward man-

ner. However, using fault injectors the test engineer must

also perform some arduous tasks. He must select the ap-

propriated injector, or more than one if the first injector he

chooses does not support all kinds of faults he needs. He

must also describe the faults to be injected using the format

each injector can handle.

A fault injector uses faultloads as the main input. Fault-

loads permit the specification of what faults will be injected,

when they will be injected and where they will be injected.

This specification must be provided by test engineers, be-

ing of high importance concerning the results that can be

obtained from fault injection tests.

Each fault injector has its own faultload specification

mechanism. These mechanisms may also differ highly. A

good level of usability (related to how easy a tool can be

applied) has a direct relation with this mechanism, affect-

ing how efficiently a test engineer uses it as well. Faultload

expressiveness is another key point. So, the specification of

rich test scenarios by a test engineer depends directly on the

availability of a high expressive specification mechanism.

978-1-4244-7785-2/10/$26.00 ©2010 IEEE

On the other hand, such mechanisms may also be complex,

depending on the functionalities involved.

If the mechanism to specify faultloads lacks usability and

expressiveness, the test engineer will probably avoid the use

of a fault injector, returning to manual and inefficient meth-

ods to inject faults. We propose a solution to this problem

developing jFaultload, a framework to specify faultloads.

This paper presents jFaultload and applies it to a com-

plete fault injection experiment to show its usability and ex-

pressiveness. The main feature of jFaultload is the use of

Java for faultload specification. We argue that using Java,

this specification can be easier and less error prone than

learning specific formats of each available injector. jFault-

load translates a given Java faultload to specific faultloads

that are then ready to be used by each injector the test en-

gineer wants for a given test campaign. The same faultload

can then attend different injectors, enhancing usability and

simplifying the use of these tools.

For the sample test experiment, we choose FIRMA-

MENT [10], a Linux kernel fault injector developed fol-

lowing ComFIRM concepts [2], that is already integrated

into our environment. FIRMAMENT has a high expressive

faultload mechanism, though it lacks usability, since it is

similar to assembly language. Using jFaultload, the fault-

load initially written in Java can be translated to the low

level fault specification format of FIRMAMENT.

The service under test, i.e. the target, is a video stream-

ing session using Real-time Transmission Protocol (RTP)

[9]. This protocol is very sensitive to timing faults, mainly

delay and jitter. Besides that, video communication in com-

munication systems is becoming very common and there-

fore it is a relevant test target.

The paper is divided into the following sections: section

2 explains some fault injector tools that made some effort

to ease the specification of faultloads. Section 3 shows the

jFaultload compiler structure. Section 4 executes an exper-

iment using jFaultload and FIRMAMENT. Finally, Section

5 concludes the paper.

2. Experiments Using Fault Injectors

Fault injectors offer all features needed to test a system

under faults. On the other hand, the known fault injectors

limit the test experiment scope, because they are also re-

sponsible to offer the mechanisms to define the faults that

can be injected.

We are focusing in fault injectors that can emulate com-

munication faults. There is a significant amount of commu-

nication fault injectors in the literature like DOCTOR [3],

NFTAPE [11] and ORCHESTRA [1].

The main drawback concerning the tools cited above is

related to their unavailability for practical use. Both source

code and binary files are not available. Also, these tools

are considered too old to be operating well nowadays (all of

them have been developed up to ten years ago).

For these reasons, new fault injector tools are continu-

ally been developed. Further, other requirements may be

attended, enriching test experiments. Next paragraphs de-

scribes three recent tools: MENDOSUS [6], FIRMAMENT

[10] and FAIL/FCI [4].

MENDOSUS implements fault injection into emulated

networks. The faults that can be injected are related to

crash and delay, applied to network components. To de-

scribe faultloads, this tool uses a script language.

FIRMAMENT is a fault injector that works at operating

system level. The faults that can be injected are drop and

delay of packets, besides arbitrary modification and dupli-

cation of packets. Considering faultload description, FIR-

MAMENT uses an assembler like language to create scripts

called faultlets.

FAIL/FCI injects faults into grid environments. Related

to the fault model implemented in this tool, only crash is

supported. Faultload description is implemented through a

high level language that uses a state machine approach.

Also considering recent tools, there is an interesting ap-

proach [8] grounded on a model based framework. Its main

goal is the use of software engineering modeling tools (such

as UML) in order to improve fault injection tests. This

model is implemented applying tools that are commonly

used, as well as graphical interfaces that promote its effec-

tive use.

This model-based approach and the concepts used by

FAIL/FCI are the closest related to the tool presented in this

paper. However, while the former does not consider a real

environment for an experiment (just a model), the latter has

a specific focus (grid environments). In that case, the pur-

pose of the current work is to offer a real experiment envi-

ronment and to preserve independence of the target system

focus.

Differently from classical ones, these recent tools are

widely available for use, in both source and binary forms.

Considering the first three injectors explained above, the

source codes of all of them are available. This feature turns

not only an advanced configuration of these tools easy, but

also any customization that may be needed, concerning test

experiments.

So, for an execution of a real test experiment, a test en-

gineer must make some efforts. First of all, a set of fault

injector tools may be chosen. As example, he may choose

the three recent fault injectors mentioned above (MENDO-

SUS, FIRMAMENT and FAIL/FCI). In that case, the test

engineer is responsible for all the tasks related to the cre-

ation of different faultloads for each fault injector, as well

as their settings.

However, using jFaultload, the test engineer tasks be-

come simpler. In that case (figure 1), jFaultload is respon-

sible for all the steps related to faultload generation, as well

as for the configuration of each tool that will be used. The

test engineer only needs to specify a Java faultload, as well

as a topology and application inputs that are independent of

any fault injection tool. This approach eases the tasks the

test engineer must perform, so that he can focus on the test

itself more than on less relevant test environment details.

ConventionalApproach

Figure 1. Experiment scenario

3. Compiler Environment

This section presents the main issues concerning jFault-

load.

3.1. Framework Layers

jFaultload is divided into layers. This permits a better

separation of concerns, turning easier to extend the frame-

work to cover new fault injectors. The framework layers

defined previously [7] have been improved in this paper.

Figure 2 illustrates these layers.

Core

Topology + Application

Intermediate Language

Faultload

Interface

Java

Mapping Script

Figure 2. Framework layers

Interface represents the extension point of the frame-

work. At this point, the environment administrator may de-

fine one or more mapping scripts, one for each new fault

injector that join the framework. The mapping scripts al-

low associating fault injector and framework functionali-

ties. jFaultload already provides mapping scripts for FIR-

MAMENT, MENDOSUS and FAIL/FCI.

Faultload is the main input to the framework. So, a Javafaultload is specified into this layer, together with the calls

needed for its execution.

Core includes the internal components. The core pro-

cesses an intermediate language and information related to

topology and application, both of them dependent on the

experiment.

3.2. Architecture

A preliminary model of this architecture [7] defined two

user profiles, illustrated in figure 3.

Experiment User is the main user of the framework,

commonly the test engineer, that will be responsible to plan

and execute test experiments. This user creates and main-

tains the faultloads.

Environment Administrator is responsible for the frame-

work instantiation. So, this user creates and tests the map-ping scripts related to each new fault injector that join the

environment. Unlike faultloads that must be create each

time an experiment is to be performed, mapping scripts

need to be developed just once. This task demand a deep

knowledge of the faultload description language the new in-

jector recognizes.

Step 1: Compiler – Faultload (Java)

Compilador de Injetores

Compilador de Injetores

Compilador de InjetoresFault Injectors

Inputs to Injectors

Faultload(Java)

Step 2: Compiler – Core

Step 3: Compiler – Interface

Intermediate Language

Intermediate Language (Topology + Application)

Topology

ApplicationMapping

Script

EnvironmentAdministrator

User Experiment

Figure 3. Environment architecture

The environment divides the compilation process into

three steps. Figure 3 also illustrates these steps.

Step 1 translates a Java faultload into an intermediatelanguage. It is close to assembly, in order to facilitate the

following compilation steps. In step 2, the intermediate lan-

guage receives the information related to topology and tar-

get application. Both are inputs to the framework and de-

pend on the test experiment itself. Finally, step 3 generates

faultloads considering each fault injector tool that will be

used later in the test experiment. To execute this last step,

the complete intermediate language from step 2 and map-

ping scripts are used.

4. Fault Injection Experiment

The current section presents an experiment using jFault-

load. The main goal is to show faultload specifications, as

well as the steps involved in the compilation process, in

order to demonstrate that the proposed framework meets

the requirements of high usability and expressiveness in de-

scribing faultloads.

4.1. jFaultload Settings

This section shows the settings of a complete test using

jFaultload. To perform the test experiment, the test engi-

neer must initially describe a faultload. The objective of the

test is to observe the behavior of a streaming video trans-

mission under delay faults. During the test experiment the

test engineer wants that some packets of RTP protocol will

be delayed. He needs to observe what happens when 3%

packets are delayed 0.1 seconds.

Figure 4 shows the related Java faultload. In this ex-

periment, all RTP packets sending from a given node (ob-

tained from the message queue calling “getHeadProtocol()”

method) will be delayed 0.1 seconds, following a uniform

probability distribution of 3%.

class Delay { void main() { Node n = new Node($Topology); Packet p;

Distrib d = new Distrib("3%");while(p=n.getHeadPacket()){ if(p.getRTPProtocol()){ if(d.sort()) {

p.delay(0.1); } } // if(p.getRTPProtocol()) } // while } // main()}

Figure 4. Delay fault description in Java

The methods getHeadProtocol() and getRTPProtocol()

are provided by jFaultload. The line ”Node n = new

Node($Topology);” models a node which identification is

later attached according the topology description. ”Packet

p;” creates a reference to a message packet. ”Distrib d =

new Distrib(”3%”);” defines a random distribution with 3%

probability. The line ”‘while(p=n.getHeadPacket())” indi-

cates that packets will be analysed while the sending queue

is not empty. ”if (p.getRTPProtocol())” verifies if the packet

is a RTP packet. ”if (d.sort())” chooses if this packet will be

delayed. Finally ”p.delay(0.1)” delays the packet if it have

been chosen. It is easy to note the high expressiveness of

this fault description; it is very similar to an algorithm de-

scription.

Considering the compilation process, steps 1 and 2 gen-

erate intermediate scripts without and with topology infor-

mation, respectively. Figure 5 illustrates the intermediate

script generated after step 1. In step 2, the label $Topologywill be changed by the appropriate label, that depends on

the test experiment.

1:Start1 INSERT n Node $Topology2 INSERT p Packet 3 INSERT d Distrib 3%4:Loop1 EVAL p=n.getHeadPacket() ? GOTO 5 ! GOTO LoopFalse15:If1 EVAL p.getRTPProtocol() ? GOTO 6 ! GOTO EndIf1

6:If2 EVAL d.sort() ? GOTO 7 ! GOTO EndIf27 DELAY 0.18:EndIf2 GOTO 99:EndIf1 GOTO 1010:EndLoop1 GOTO 411:LoopFalse1 GOTO 12

1:Start1 INSERT n Node $Topology2 INSERT p Packet 3 INSERT d Distrib 3%4:Loop1 EVAL p=n.getHeadPacket() ? GOTO 5 ! GOTO LoopFalse15:If1 EVAL p.getRTPProtocol() ? GOTO 6 ! GOTO EndIf1

Figure 5. Intermediate language

Before the execution of step 3, a framework administra-

tor must have been created mapping scripts to each fault in-

jector available in the environment. These mapping scripts

associate functionalities from both the framework and the

fault injector. jFaultload just contains all the scripts needed

for FIRMAMENT, MENDOSUS and FAIL/FCI, so they do

no need to be created again. Figure 6 exhibits a mapping

script for use with FIRMAMENT.

getHeadPacket() { PacketSelection(9); }

getRTPProtocol() {getUDPProtocol();PacketSelection(0);"SET 0x0f R0 AND R0 R1 SET 4 R0 MUL R1 R0 READS R0 R1";ProtocolSelection(6970);

}

getUDPProtocol() { ProtocolSelection(17); }

Figure 6. Mapping script

Finally, the faultload needed for a test experiment apply-

ing FIRMAMENT is produced. This faultload (figure 7) is

complete and reflects all the features the original Java fault-

load specifies. Also, this generated faultload is easy to read.

It can also be modified easily and reused in other experi-

ments improving the usability.

START:

LOOP_1: SET 9 R0 READB R0 R1 SET 17 R0 SUB R0 R1 JMPZ R1 PROTOCOL_17 JMP START

PROTOCOL_17: SET 0 R0 READB R0 R1 SET 0x0f R0 AND R0 R1 SET 4 R0 MUL R1 R0 READS R0 R1 SET 6970 R0 SUB R0 R1 JMPZ R1 PROTOCOL_6970 ACP

PROTOCOL_6970: SET 500 R0 RND R0 R1 SET 470 R0 ADD R0 R1 JMPN R1 DISTRIB_30_OK JMP START

DISTRIB_30_OK: SET 0.1 R0 DLY R0 JMP LOOP_1

Figure 7. Faultload for FIRMAMENT

4.2. Test Experiment

Now that the appropriated faultload was generated by

jFaultload, the test experiment is ready to proceed. The

faultlet shown in figure 7 that jFaultload created compiling

the Java faultload is then loaded into FIRMAMENT.

Before FIRMAMENT executes the faultlet on every

packet that leaves the protocol stack, the Darwin Stream-

ing Server application was set on the server machine, and a

looping trailer of the movie American Casino was readied

and began playing. The movie corresponds to the workload

of the test. The Darwin Streaming Server application is the

target that will be observed under faults. The state playingof a movie indicates that it can be requested by any client

and that this action will begin its streaming, as long as a

connection between the nodes is regularly established.

After a short period of time, FIRMAMENT receives

the command to begin executing the faultload defined by

the loaded faultlet, that was previously translated from a

generic Java faultload description.

The results of injecting delay faults into 3% of the outgo-

ing packets in the server machine is shown visually in figure

8. A capture of the QuickTime streaming monitor is repre-

sented in figure 9. The loss presented by the monitor indi-

cates that there was a portion of the analysed packets that

were effectivelly delayed by FIRMAMENT. The number of

analysed packets at each moment is not specified by the ap-

plication, but it can be assumed that it is a low number, con-

sidering that the percentage of packet losses changes radi-

cally from one sample to other. The data shown indicates

that there was at least one delayed packet inside the anal-

ysed packets.

The best indicator of video quality is the subjective crite-

ria that is the viewer’s opinion. To obtain that, the preferred

Figure 8. Capture of movie under delay faults

method is to show the final results of the streaming video to

the test engineer, just like it is presented in figure 8. He then

can judge if the behavior under faults is acceptable.

This test experiment shows that a generic faultload writ-

ten in Java without concerning about a specific fault injector

particularities could be automatically translated into a spe-

cific format, and that his generated faultload was perfectly

executed by FIRMAMENT.

Clearly, the steps to execute the whole experiment using

jFaultload and FIRMAMENT are mostly parameterizable

in terms of the target, its fault tolerance mechanisms and the

workload. To test a generic target application, it is sufficient

to build a Java script defining a relevant faultload according

to the features the test engineer wants to test, configure the

hardware and software so that they represent important situ-

ations to the target, execute the faultload translation tool and

fault injector properly, and monitor the experiments collect-

ing results.

The main effort to perform a full fault injection test is

elaborating a useful faultload. Using jFaultload no time

is spent configuring and translating the faultload descrip-

tion to the chosen fault injector. Besides saving time, this

can improve the ability of the test engineer in creating rel-

evant tests, since he now can worry only on how to build

high level faultload descriptions, not depending on how they

might be described in the actual fault injection tools. Not

only that, now the test engineer can use multiple fault in-

jectors on test targets, improving the experiment reliability

and helping developers to detect software errors and other

implementation inadequacies.

5. Final Remarks

This paper presented jFaultload, a framework for speci-

fication of communication faultloads using Java. Initially,

Figure 9. Quicktime’s monitor

several approaches that fault injectors use in test experi-

ments are explained. Next, we show the general infrastruc-

ture of the tool, with emphasis on the framework layers and

the architecture as a whole. Finally, a experiment was per-

formed with the proposed tool.

In terms of usability, one of the greatest advantages

brought by the framework is related to separation of con-cerns. In that sense, the adoption of two user profiles (i.e.,

administrator and test engineer) is of paramount importance

to reduce the complexity inherent to test experiments. Then,

while administrator realizes the configuration and instanti-

ation of the framework, the test engineer works with the

features that really matters - the test experiment itself. Even

so, in cases that two profiles is not possible, the user may

also act as administrator, because the administration tasks

are seldom performed, they are needed only when a new

injector join the environment.

Also, the approach adopted in the proposed tool is inde-

pendent of target system and with focus on real systems.

These two features are important, because all of the ex-

isted approaches have some degree of dependence on a

specific target application, such as FAIL/FCI (with focus

on grid systems) and MENDOSUS (focused on emulated

networks). Yet at this point, the use of Java for faultload

specification enhances usability, because Java is a language

widely known and next to the world of developers. Besides

this, Java is highly expressive and easy to learn, turning fea-

sible the specification of complex faultload in a short time,

compared with other approaches.

References

[1] S. Dawson, F. Jahanian, T. Mitton, and T.-L. Tung. Testing ofFault-Tolerant and Real-Time Distributed Systems via Pro-tocol Fault Injection. In Symposium on Fault-Tolerant Com-puting, pages 404–414, 1996.

[2] R. J. Drebes, F. O. Leite, G. Jacques-Silva, and T. S. Weber.

Comfirm: a communication fault injector for protocol test-

ing and validation. Latin American Test Workshop (LATW),2005.

[3] S. Han, K. Shin, and H. Rosenberg. DOCTOR: An Inte-grated Software Fault Injection Environment for DistributedReal-Time Systems. In Int. Computer Performance and De-pendability Symposium. (IPDS’95), pages 204–213, Erlan-

gen, Germany, 1995. IEEE Computer Society Press.[4] W. Hoarau and S. Tixeuil. A Language-Driven Tool for

Fault Injection in Distributed Systems. In Proc. of the 6thIEEE/ACM Intl. Workshop on Grid Computing, pages 194–

201, Grand Large, Franca, 2005.[5] M.-C. Hsueh, T. K. Tsai, and R. K. Iyer. Fault injection

techniques and tools. IEEE Computer, 30(4):75–82, 1997.[6] X. Li, R. Martin, K. Nagaraja, T. D. Nguyen, and B. Zhang.

Mendosus: A SAN-Based Fault-Injection Test-Bed for theConstruction of Highly Available Network Services. In Pro-ceedings of the 1st Workshop on Novel Uses of System AreaNetworks (SAN-1), 2002.

[7] R. Munaretti, T. Weber, and S. Cechin. Detailed Description

of Communication Faultloads to Improve the Usability of

Fault Injection Testing Tools. In XXXV Latin American In-formatics Conference, volume 1, pages 1–10, Pelotas, 2009.

UFPEL.[8] J. Olah and I. Majzik. A Model Based Framework for Spec-

ifying and Executing Fault Injection Experiments. IEEEFourth International Conference on Dependability of Com-puter Systems, pages 107–114, 2009.

[9] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacob-

son. RTP: A Transport Protocol for Real-Time Applications.

RFC 3550 (Standard), July 2003. Updated by RFC 5506.[10] T. Siqueira, B. Fiss, R. Weber, S. Cechin, and T. Weber. Ap-

plying firmament to test the sctp communication protocol

under network faults. Test Workshop, 2009. LATW ’09. 10thLatin American, pages 1–6, March 2009.

[11] D. Stott, B. Floering, D. Burke, Z. Kalbarczyk, and R. K.

Iyer. NFTAPE: a Framework for Assessing Dependabil-ity in Distributed Systems with Lightweight Fault Injectors.

In Proceedings of the IEEE International Computer Perfor-mance and Dependability Symposium, pages 91–100, Mar.

2000.