Event-Based Data Dissemination onInter-Administrative Domains: Is It Viable?∗

Roberto BALDONI, Leonardo QUERZONI, Sirio SCIPIONI

Dipartimento di Informatica e Sistemistica “A. Ruberti”Sapienza Universita di Roma

{baldoni,querzoni,scipioni}@dis.uniroma1.it

Abstract

Middleware for timely and reliable data disseminationis a fundamental building block of the Event Driven Archi-tecture (EDA), an ideal platform for developing air trafficcontrol, defense systems, etc. Many of these middlewaresare compliant to the Data Distribution Service (DDS) spec-ification and they have been traditionally designed to bedeployed on managed environments where they show pre-dictable behaviors. However, the enterprise setting can beunmanaged and characterized by geographic inter-domainscale and heterogeneous resources.

In this paper we present a study aimed at assessing thestrengths and weaknesses of a commercial DDS implemen-tation deployed on an unmanaged setting. Our experimentscampaign outlines that, if the application manages a smallnumber of homogeneous resources, this middleware per-form timely and reliably. In a more general setting withfragmentation and heterogeneous resources, reliability andtimeliness rapidly degenerate pointing out a need of re-search in self-configuring scalable event dissemination withQoS guarantee on unmanaged settings.

1. Introduction

It is a common belief that sense-and-react applications(including air traffic control, defense systems, fraud detec-tion etc) cannot be built without using Event-Driven Archi-tecture (EDA) technologies [3]. This is due to the neces-sity of sub-second responses to changing conditions in theenvironment and this can only be achieved if the sensingportion of the application polls the environment rather thanhaving data pushed to it [4]. Timeliness and reliability indata distribution are essential to guarantee the correctness

∗This work was partially supported by the European Network of Excel-lence ReSIST and by a grant from an agreement CINI-Finmeccanica.

and safety of an application in such domain. If the systemfails in delivering data on time, instability may arise, pos-sibly resulting in threats to either infrastructures or humanlives.

Historically, most of the standards for data distributionmiddleware, e.g. CORBA Event [9] and Notification [10]services, Java Message Service (JMS) [7] etc., as well asmost proprietary solutions, lacked the support necessary forreal-time, mission and safety critical systems. The maindrawbacks of these solutions are related to a limited (or notexisting) support for Quality of Service (QoS) and to thelack of architectural properties needed to promote depend-ability and survivability. Recently, in order to fill this gap,the Object Management Group (OMG) proposed the DataDistribution Service (DDS) [6] specification. This standardgathers the experience of proprietary real-time pub/sub mid-dleware solutions independently engineered and evolvedwithin the industrial process control and in the defense sys-tems application domains. The resulting standard is basedon a completely decentralized architecture and provides anextremely rich set of configurable properties for QoS. Cur-rent DDS implementations are designed for managed andcontrolled environments where they deliver predictable per-formance characterized by high throughput, low latency andloss rates close to zero even in stress conditions as remarkedby several studies [8, 12]. These studies compare, in LANscenarios, various DDS implementations and other pub-lish/subscribe softwares such as OpenSplice, OpenDDS,TAO DDS, and RTI DDS.

However, the enterprise setting is radically differentfrom a managed one: it is often characterized by an inter-administrative (i.e. unmanaged) geographic scale, sharednetwork channels, heterogeneous resources with the conse-quent unpredictability of, for example, end-to-end latencyand load.

In this paper we present a summary of results of anextensive campaign of experiments whose aim was to as-

12th IEEE International Workshop on Future Trends of Distributed Computing Systems

1071-0485/08 $25.00 © 2008 IEEE

DOI 10.1109/FTDCS.2008.14

44

sess the strengths and weaknesses of current DDS imple-mentations on such unmanaged environment. During thecampaign we tested three DDS implementations deployedon PlanetLab on national and continental scenarios. Allof them shown similar behaviors and shortcomings, this iswhy, in the next sections, we report for the sake of brevityonly the results obtained from the DDS middleware byReal-Time Innovations Inc. [1] as it shown the better perfor-mance coupled with the more stable behavior. These mid-dlewares can perform timely and reliably on an unmanagedsetting only if there is no event fragmentation at the DDSlevel and if resources are in a small number and homoge-neous. In a more general unmanaged setting, reliabilityand timeliness of event dissemination rapidly degenerate.Lessons learned from the campaign point out shortcomingsin these middleware systems that turn out in research chal-lenges. These challenges include: the need of designingself-tuning mechanisms within the middleware to cope witha constantly changing setting (e.g., end-to-end latency, loadon a single node), the design of scalable algorithms for effi-cient event routing also in the presence of event fragmenta-tion, adaptive mechanisms to compensate possible diversityof resources in terms of computational power. Answering tothese questions is mandatory for enlarging as more as pos-sible the zone in which the performance of the event dis-semination perform timely and reliably on an unmanagedsetting.

The paper is organized as follows: Section 2 introducesthe testbed we realized, and Section 3 shows and discussesthe results; finally, Section 4 concludes the paper resumingthe lessons we learned from this practical experience.

2 Testbed

The testbed we developed1 for this study is based onPlanetLab [11], an open overlay network composed bymore than 800 nodes, located in 400 sites. PlanetLab al-lowed us to deploy a testbed where congestion, latency andother network parameters are similar to a real environment.

We defined two different scenarios scenarios for ourtests: national and european. They are composed (respec-tively) by 1 publisher and 2 subscribers, 1 publisher and 20subscribers. Another important difference between thesescenarios is the geographic location of nodes: in the na-tional one all nodes are located in Italy (in particular weused nodes in Rome, Milan and Naples), while in the euro-pean scenario nodes have been selected from several euro-pean countries (e.g. four nodes in France, two in Germany,etc.). In both scenarios we tested a simple application where

1The code needed to run the tests, together with a document includingall configuration details, is available at the following url: http://www.dis.uniroma1.it/˜midlab/dds_testbed.zip

a single publisher distributes data to various subscribers em-bodied by the remaining nodes.

All the results reported in the next sections were obtainedusing an application written in C++ that exploits RTI DDSlibraries. These libraries enable publish-subscribe commu-nications and several services, as flow control or reliablecommunication. In our test, we configured several QoSpolicies and transport protocol parameters in order to op-timize performance of RTI DDS in the considered scenar-ios. Here we provide only a basic list of the most importantQoS policies and parameters settings used for our tests. Thefirst and we think the most important QoS policy is Relia-bility, that specifies what kind of communication paradigmwe want to use in our test: reliable or best-effort. Other rele-vant QoS policies are related to reliable communication; forexample History.kind controls the behavior of the Servicewhen the value of an instance changes before it is finallycommunicated to some of its existing DataReader (The re-ceiving component on a subscriber node) entities, we setthis policy to KEEP ALL to maintain all values. A lastfundamental parameter is the heartbeat period that controlsthe time interval between two subsequent send of heartbeatmessages. These messages announce to the DataReader thatit should have received all events up to the current one andcan also be used by the DataWriter (the sender componenton a publisher node) to request the DataReader to send backan acknowledgement. In our tests this parameters is set to1 second. Other configuration parameters are related tothe transport layer. RTI DDS provides a plugin mechanismthat can be used to select different kind of transport prim-itives: shared memory, unicast, multicast, etc. Due to theimpossibility to change the network infrastructure (Internet)of our testbed we were forced to use UNICAST IPv4 basedon UDP/IP. A fundamental transport layer parameter is thefragment size of UDP packets. The maximum allowed frag-ment size defines the number of UDP messages in which anevent is fragmented. We will show in the next section thatthis parameters have a critical impact on loss rate and la-tency in reliable communications.

During our tests we measured both the data distributionlatency (time needed for an event to travel from the pub-lisher to the subscriber, i.e. half Round Trip Time) andmaximum throughput (i.e. number of events received persecond) allowed by the tested. During tests conducted withthe best-effort policy we also kept track of loss rate (i.e.number of events lost). Every point plotted in the graphsreports the average value measured in 20 independent runsof our test application executed with a time interval of 12hours from the preceding one. For latency tests we col-lected 1000 samples, produced at a rate of 1 per second, foreach run. For throughput tests we executed, in each run, 10independent event production bursts, each with a durationof 30 seconds, separated by a time interval of 30 seconds.

45

During each burst the application try to send as much eventsevents as possible and measures the number of events sentby the publisher and received by the subscribers.

3 Experiments

3.1 Fragment Size vs. Loss Rate

First, we want to focus on loss rate and show the impact ithas on the latency of reliable communications. We analyzedthe behaviour of RTI DDS using different fragment sizes.This parameter defines the maximum size of UDP packetgenerated and sent by middleware, therefore, it also definesthe number of packets each event will be fragmented in.

0

10

20

30

40

50

60

70

80

90

100

16

32

64

128

256

512

1024

2048

4096

8192

1638

4

3276

8

6553

6

1310

72

Lo

ss R

ate (

%)

Event Size (B)

Frag Size = 1KB

Frag Size = 4KB

Frag Size = 8KB

Frag Size = 16KB

Frag Size = 64KB

Figure 1. National scenario: impact of frag-ment size on event loss rate

Figure 1 reports RTI DDS behaviour with best-effortcommunication in the national scenario varying both frag-ment size and event size. The first interesting outcome ofthis test is that the number of events lost during each run isquite small (under 4%) as long as we consider small eventsizes. When events become bigger (right half of the graph),reducing fragment size have a drastic effect on loss rate. Inthis case, the middleware must retransmit each event var-ious time in order to guarantee a reliable communication,therefore introducing huge latency.

Figure 2 reports the latency for the same experiments us-ing reliable communication. This figure shows that loss ratehave an impact on latency in reliable communications: la-tency grows rapidly up to an unacceptable level when lossrate is high. Fragmentation is a vulnerable point this kindof middleware systems because fragmenting an event in-creases the probability of a loss.

0

20

40

60

80

100

120

140

16

32

64

128

256

512

1024

2048

4096

8192

1638

4

3276

8

Laten

cy (

ms)

Event Size (B)

Frag Size = 8KB

Frag Size = 64KB

Figure 2. National Scenario: impact of frag-ment size on latency

Latency RTT µsec StdDev µsecBest-Effort 128B 148 12Best-Effort 16KB 610 13

Reliable 128B 152 15Reliable 16KB 640 18

Throughput msg/sec MBytes/secBest-Effort 128B 67904.51 8.289Best-Effort 16KB 7415.12 115.861

Reliable 128B 64918.30 7.192Reliable 16KB 7315.12 114.29875

Table 1. Summary of LAN results

3.2 Best-Effort/Reliable communicationvs. Latency

Table 1 reports the results obtained in a LAN-based set-ting. The numbers clearly show how both best-effort andreliable communications experience similar latency withsmall (one order of magnitude) standard deviation when themiddleware is deployed on a controlled, high-performancesetting. The conditions at the basis of these results arehardly met in an unmanaged WAN setting where latencyand standard deviation have often the same order of magni-tude.

Figure 3 reports both latency and its standard deviationwhen the middleware is deployed in the national scenario.Interestingly, latencies experienced for both best-effort andreliable communication remains low and stable for a widerange of event sizes (up to 16KBytes) with a standard de-viation of the same order of magnitude. This result is quiteimportant as it shows how RTI DDS can, in principle, befruitfully employed in unmanaged, shared scenarios as itdelivers consistent performance (see the detail in Figure 3).

46

Figure 3. National scenario: latency and itsstandard deviation vs. event size

This encouraging result is however limited to smallevents. Starting from event of size 32KBytes or larger, ex-perienced latency and loss rate start to grow rapidly. Thisphenomenon is evident in the right part of Figure 3 where,for events of 64KBytes and 128 KBytes, latency for reli-able communication becomes rapidly unacceptable. In fact,we experienced that, in our scenarios, the probability of apacket loss is larger for packets of 64KBytes than for packetof 8KBytes. Consequently sending events larger than 64KBcan be problematic and should be avoided when possible.

0

10

20

30

40

50

60

70

80

90

100

16

32

64

128

256

512

1024

2048

4196

8192

16384

32768

65536

131072

Lo

ss R

ate (

%)

Event Size (B)

Sub 1

Sub 2

Figure 4. National scenario: loss rate of twodifferent subscribers

3.3 Heterogenous nodes

The results showed in previous section are even more ev-ident if we consider the different performance experiencedby the two subscriber nodes in the national scenario. These

two nodes are characterized by widely different capabilitiesin terms of bandwidth and link reliability. Figure 4 reportsthe loss rate experienced at each node. Subscriber 1 showslow loss rates, even with large messages, while subscriber2 has consistently large loss rates that get worse when wereach large event sizes. This subscriber is the responsiblefor introducing the large latency reported by the graphs inthe previous section.

0

50

100

150

200

250

300

350

400

16

32

64

128

256

512

1024

2048

4196

8192

16384

32768

65536

131072

Th

ro

ug

hp

ut (

Mb

it/

s)

Event Size (B)

Sub 1

Sub 2

Pub BE

Pub RE

Figure 5. National scenario: throughput forreliable and best-effort communications

Figure 5 shows more clearly how a slow node can slowdown throughput of the publisher for the topic it is sub-scribed to. The curves showing the behaviour at subscriberside remarks how the two subscribers have clearly differentcapabilities: subscriber 2 is, in fact, not able to sustain largethroughput due to its limited resources. With best-effortcommunication this is not a real problem, as the publishergenerates events at its maximum rate, and only subscriber 2will be affected by its poor performance. With reliable com-munication, instead, the graph clearly shows how through-put at publisher site (and, therefore, the whole throughputfor the topic) is limited by the performance of subscriber2. This problem is caused by the fact that the DDS Spec-ification defines a transport layer based on UDP protocol.Consequently reliability must be implemented at an upperlayer in the middleware. In this case implementing reliablecommunication over UDP channels requires the definitionof reliability in terms of events: events have to be storedin a queue at publisher side until they are acknowledged byevery subscriber. A queue stores produced events up to amaximum: when the queue is full the publisher stops pro-ducing further events. Figure 5 shows a slow subscriber isable to limits throughput of the publisher belonging to sametopic. A parameter max blocking time can be configured inRTI DDS to discard events in send queues after a predefinedtime interval. This parameter have to be manually config-ured and if it is too small it can drastically reduce the level

47

of reliability. Moreover frequent retransmissions decreasedrastically throughput, based on the number of event trans-mitted. This aspect can be a crucial point if you want touse a DDS product in an unmanaged setting, similar to theone we are considering here, where nodes can have widelydifferent capabilities.

3.4 Scalability in terms of numbers ofusers

0

50

100

150

200

250

300

350

400

16

32

64

128

256

512

1024

2048

4196

8192

16384

32768

65536

131072

Th

ro

ug

hp

ut (

Mb

it/

s)

Event Size (B)

2 Sub

20 Sub

Figure 6. European Scenario: Impact of Num-ber of Subscriber on Throughput

In figure 6 we analyze the european scenario where wehave 20 subscribers deployed on several sites in the eu-ropean area. In this scenario we want to show the be-haviour of RTI DDS in terms of throughput when a pub-lisher have to distribute the same event to a large set, 20 inour tests, of subscribers. We have to remember that in ourscenario we can not use IP Multicast because we can’t con-figure routers belonging to different administrative domains(a common condition in an unmanaged environment). Con-sequently, we have to use Unicast IPv4 as a transport layerfor RTI DDS. Using UDP-based unicast to transfer events,20 subscribers require 20 “data send” therefore the over-all throughput suffers a strong degradation of performance.This is evident from Figure 6 where the global throughputfor best-effort communication is clearly negatively affectedby a larger number of recipients.

3.5 Impact of Static Configuration

During our tests the lack of self-organization and on-the-fly adaptiveness characteristics clearly emerged. We hadto manually configure and test several parameters that canstrongly impact performance. Moreover, due to the lackof control over the environment and its inherent dynam-ics, these parameters must also be adapted and changed

from test to test. For example the performances of reliablecommunication in RTI DDS are strictly related to heart-beat frequency. Slow heartbeat frequency did not allow ourtest application to maintain a strict reliable communicationparadigm or introduced large latencies. Moreover, in RTIDDS, there is an automatica discovery mechanism that isunable to work in our inter-administrative scenario; there-fore a publisher does not have to know directly each sub-scriber but it can automatically add a subscriber to its list ifit is contacted by the subscriber itself.

4 Conclusions and Lessons Learned

Future distributed enterprise computing systems will mixwith very high probability SOA an EDA. There are indeedtwo main reasons to foresee this merge. Firstly, SOA can beeffectively used for normal operations within an enterprisecomputing system. But what about behaviors that deviatefrom expectations? This is for example a typical event inmonitoring, auditing or stock management. For these op-erations EDA offer much better capabilities than SOA toquickly react to these asynchronous events [4]. Secondly,we are close to the advent of “mashups” enterprises [2] thatfuse stream of data from available sources on the Internetfor creating added value streams [5] 2. Reliable and timelyevent dissemination on unmanaged system is an essentialelement for enabling this picture.

The DDS is the only standard at the moment that speci-fies QoS policies such as reliability and timeliness for eventdissemination. Even though implementations have been de-signed for a managed environment we positively remarkedthat in some limited settings, they can be used as-is. Inparticular during our experiments campaign RTI DDS hasshown the best attitude in this porting from managed tounmanaged settings. If there is no event fragmentation atthe DDS level ( i.e., event size is smaller than the maxi-mum payload of a UDP packet) and if resources are in asmall number and homogeneous (i.e., every computer hasthe same computational power), RTI DDS can be effectivelyused on the top of an unmanaged environment. Unfortu-nately in a more general settings performance rapidly de-generate and the data dissemination becomes unpredictable.

To enlarge the setting in which data dissemination canbe done in timely and reliable ways on the top of an unman-aged environment, our study shows that there is a need ofintroducing innovative solutions on at least three importantarchitectural aspects each of them representing a challeng-ing research area:Adaptivity and Self-configuration capability: during thesetup phase of out tests we were forced to configure and to

2Amazon and eBay already offer Web Services that are used by thirdparties to create value-added services.

48

readapt the software differently on each node, adapting thesoftware configuration to a continuously changing setting.

The same problem was also evidenced by the lack of anautomatic mechanism for node discovery. These issues area consequence of the complete lack of self-configuration ca-pabilities in the software and of its inability to automaticallyadapt to changed environmental conditions.Efficient event routing primitives: the main event rout-ing and transport mechanism employed by RTI DDS (andmany of its siblings) is IP multicast. In our unmanagedinter-administrative scenario, the middleware can not use IPmulticast so it resorts to a point-to-point UDP-based com-munication primitive. This limitations have a strong impacton scalability: it introduces high latency in best-effort com-munications and strongly reduces the overall throughput inreliable communications. From this point of view we advo-cate the exploitation of application based multicast services,able to avoid the pitfalls imposed by the unreliable sharedchannels.Management of heterogeneous resources: the unman-aged and heterogeneous environment that characterizes oursetting clearly created some difficulties to the tested soft-ware. Some of these problems are related to the strong di-versity among the capabilities of various nodes. For ex-ample, the software was not able to efficiently support datadistribution among a very slow and a very fast node, justadapting the whole process to the slowest participant. Thisbehavior is certainly not acceptable in our setting of inter-est. Therefore, we think that future middleware products inthis field should implement mechanism able to exploit nodediversity, without simply adapting to the worst case.

5 Acknowledgements

Authors would to thank Stefania della Bina e Fabio Fal-cinelli for their help in implementing test applications, us-ing several publish/subscribe softwares such as OpenSpliceand RTI DDS, for LAN and PlanetLab environment.

References

[1] Real-time innovations inc., http://www.rti.com/products/data_distribution/index.html.

[2] D. Butler. Mashups mix data into global service. Nature,439(7072):6–7, 2006.

[3] K. M. Chandy. Sense and respond systems. In InternationalComputer Measurement Group Conference, pages 59–66,2005.

[4] K. M. Chandy. Event-driven applications: Costs, benefitsand design approaches. In Gartner Application Integrationand Web Services Summit, 2006.

[5] K. M. Chandy, L. Tian, and D. M. Zimmerman. Enterprisecomputing systems as information factories. In Proceed-

ings of 10th IEEE International Enterprise Distributed Ob-ject Computing Conference (EDOC), 2006.

[6] O. M. Group. Data distribution service for real-time systemsspecification, 2002.

[7] S. M. Inc. Java message service api rev 1.1, 2002.[8] B. McCormick and L. Madden. Open architecture publish-

subscribe benchmarking. In OMG Real-Time EmbeddedSystem WorkShop 2005, 2005.

[9] Object Management Group. CORBA event service speci-fication, version 1.1. OMG Document formal/2000-03-01,2001.

[10] Object Management Group. CORBA notification servicespecification, version 1.0.1. OMG Document formal/2002-08-04, 2002.

[11] PlanetLab Consortium. http://www.planet-lab.org/.

[12] Vanderbilt University. RT-DEEP project, http://www.dre.vanderbilt.edu/DDS/.

49

Appendix A RTI DDS Configuration Param-eters and QoS policies

The following Table reports the complete list of configura-tion parameters and QoS policies used for our tests with RTIDDS.

RTI DDS Participant QoS Valuetransport builtin mask UDPv4

receiver pool buffer size 64KB

RTI DDS Transport Properties QoS Valuemessage size max 64KB

send socket buffer size 64KBreceive socket buffer size 64KB

RTI DDS Flow controller Properties Valuemax tokens Unlimited

tokens leaked per period Ulimitedperiod 10ms

bytes per token 64KB

RTI DDS Topic QoS Valuedurability TRANSIENT LOCALreliability BEST EFFORT

or RELIABLEhistory KEEP ALL

RTI DDS ValueData Reader / Data Writer QoS

publish mode.kind DDS ASYNCHRONOUSPUBLISH MODE QOS

Max blocking time 10 ssize send queue 400 packets

size output queue 400 packetssize receive queue 400 packets

initial instance 1heartbeat period 30 ms

heartbeat per max samples 400max byte per nack response 131072 Byte

max heartbeat retries 25 msmax nack response delay 25 msmin nack response delay 0 ms

min/max heartbeat response delay 0 msfast heartbeat period (throughput) 30 ms

fast heartbeat period (latency) 100 ms

Appendix B PlanetLab nodes employed in thetests

PlanetLab nodes in the national scenario:

Publisher planetlab1.elet.polimi.it;

Subscriber 1 planetlab-1.dis.uniroma1.it;

Subscriber 2 planetlab02.dis.unina.it;

PlanetLab nodes in the european scenario:

Publisher planet1.zib.de;

Subscriber 1 merkur.planetlab.haw-hamburg.de;

Subscriber 2 planetlab2.diku.dk;

Subscriber 3 planetlab1.cesnet.cz;

Subscriber 4 planetlab2.info.ucl.ac.be;

Subscriber 5 planck227.test.ibbt.be;

Subscriber 6 planetlab02.mpi-sws.mpg.de;

Subscriber 7 planetlab02.ethz.ch;

Subscriber 8 planetlab02.dis.unina.it;

Subscriber 9 planetlab1.csg.uzh.ch;

Subscriber 10 planetlab0.cs.stir.ac.uk;

Subscriber 11 planetlab02.cnds.unibe.ch;

Subscriber 12 planetlab2.informatik.uni-kl.de;

Subscriber 13 planetlab2.itwm.fhg.de;

Subscriber 14 plab2-c703.uibk.ac.at;

Subscriber 15 planetlab1.ifi.uio.no;

Subscriber 16 planet2.manchester.ac.uk;

Subscriber 17 planetlab1.exp-math.uni-essen.de;

Subscriber 18 planetlab1.hiit.fi;

Subscriber 19 planet2.colbud.hu;

Subscriber 20 planetlab1.eecs.iu-bremen.de;

50