IMPROVING PERFORMANCE OF PARALLEL SIMULATION KERNEL FOR WIRELESS NETWORKSIMULATIONS

M. Thoppian, S. Venkatesan, H. Vu, R. Prakash, N. MittalDepartment of Computer Science, The University of Texas at Dallas,

Richardson, TXand

J. AndersonPACE Lab, Rockwell Collins,

Richardson, TX

ABSTRACT

For decades, simulations have been the means to study thebehavior and analyze the performance of engineering, computerscience, and military applications. In the sequential discrete-eventdriven simulations, the events are executed sequentially in anincreasing order of their timestamp. Simulation of large complexapplications using sequential simulators often turns out to beinfeasible due to computer resource limitations. Parallelizingdiscrete-event simulation is acknowledged as the most promisingapproach for simulating large complex applications. Paralleldiscrete-event simulation refers to the execution of events from asingle discrete-event simulationprogram in parallel. In this paperwefocus on extracting parallel events for wireless ad hoc networksimulations based on the distance and terrain information.

In sequential simulations, the events areone. Consider a sample event list shown

El

E2

E3

E4

E5

E6

executed one byin Figure 1. The

Time 0.1

Time 0.1

Tirne 0.3

Time 0.4

Time 0.4

Time 3.0

Fig. 1. Sample Event List

I. INTRODUCTIONWith recent growth in the area of wireless communi-

cations, there is an increasing need to develop techniquesto improve the simulation tools used to model large-scalewireless ad hoc networks. Sequential discrete-event drivensimulation traditionally refers to the use of a global eventlist, a global clock, and a single processor to simulatethe entire application. In sequential discrete-event drivensimulation, the events are executed in an increasing orderof their timestamp, one after another. Simulation of largecomplex applications often turns out to be beyond the ca-pability of sequential simulators. Parallel simulations is oneway of overcoming the resource limitations and achievingsimulation speed-up.

In this paper we propose techniques to extract events thatare non-conflicting, i.e. events that are causally independent.These non-conflicting events can be executed in parallelwithout violating the correctness of the simulation.

In a discrete event-driven simulation, the nodes scheduleevents and insert them in a global event list. The schedulerexecutes events in the order of their timestamp. Upon exe-cution, the event is removed from the list and the simulationclock advances to the time of next event in the event list.

events El, E2 .... E6 are scheduled by different nodes inthe network. These events are sorted in the increasing orderof their timestamp. At the begining of the simulation, eventswith timestamp 0 are executed. If there are no such events inthe event list, the simulator advances the simulation clock tothe timestamp of the next event in the event list. If there aremultiple events scheduled for the same time, the schedulerexecutes these events one-by-one in the case of sequentialsimulations. In our example, after executing events El andE2, the simulator advances the clock to 0.3 seconds. It thenexecutes event E3 and so on. However in the case of parallelsimulations, events that are independent of each other canbe scheduled in parallel. In our example, using parallelkernel the scheduler can execute both the events El andE2 simultaneously without affecting the correctness of thesimulation. In addition to the events occuring at the sameinstance of time, there could be many other independentevents in the simulation that can be processed in parallel.

Suppose events Ei and Ej had timestamps Ti and Tj,respectively. Let Ti < Tj. If event Ei changes the state ofa variable referenced by Ej, then these two events are saidto be causally related and executing Ej before Ei wouldresult in a causal error. For example consider the Figure 2.Node N1 sends messages Ml and M2 to node N2, such

1 of 6

that Ml is sent before M2. The receipt of message Mlhas to precede the receipt of message M2 at node N2 toavoid causal order violation. In the past, several algorithmsfor parallel simulation have been proposed. The parallelsimulation techniques proposed in the literature [1] havebeen classified into the following two categories:

1) Conservative approach: In conservative approach, allpossible causality errors are prevented by strictlyadhering to the local causality constraint.

2) Optimistic approach: In optimistic approach, the localcausality constraint is not strictly adhered to. Anoptimistic approach instead guarantees to detect andrecover from causality errors.

Node 1 Node 2

Ml

Node 1

11

Ei

T2

Ej

Node 2

Ml

M2Ej

Ei

execution window. If two events occur within the sameparallel execution time window and if they belong todifferent process models or different nodes, then the eventsare candidates for parallel execution. The current release ofOPNET Modeler 11.5 has only the MAC and physical layermade parallel safe. As all the events within the window arecandidates for parallel execution, the choice of this windowvalue is cruicial. Ideally, the size of the window should bedynamically set in the simulations based on the amountof available parallelism. However, the value of parallelexecution time window in OPNET Modeler has to be setbefore running the simulation experiment and is a staticvalue.

In Qualnet [5], the entire network is divided into par-titions with each partition acting as a logical process.Zhengrong et al. [6] use MAC layer, PHY layer and cross-layer interaction between the MAC sub-layer and the PHYlayer to extract lookahead values. At the MAC sub-layer theback off timers, inter frame spaces and duration of currentframe being received are used to determine lookahead. Atthe PHY layer lookahead is extracted by considering thetransition time between receiving and transmitting and bylooking at interference signals. By exploiting the cross-layerinteraction, a larger lookahead is extracted.

(b) Incorrect Order of Execution

Fig. 2. Causal Relationship between EB and E.: (a) Correct orderexecution, (b) Incorrect order of execution resulting in causality error

In this paper we focus on the conservative approach forparallel simulations. To execute events in parallel it isnecessary to know if the events are causally independent.The biggest challenge faced is: how does one identify all theevents in the event list that are independent of each other? Inthis paper we enumerate ways of extracting events that are

safe for parallel execution in static wireless ad hoc networksimulations.

II. RELATED WORKSeveral parallel simulation techniques have been pro-

posed in the literature [1]. The simulation tools such as

OPNET, GloMoSim, and Qualnet employ conservative par-

allel simulation techniques.In GloMoSim [2], each protocol layer in a node is

represented as an entity. Each of this entity could be a

logical process. Meyer et al. [3] present ideas to improvelookahead based on the message paths through a protocolstack. This was termed as "path lookahead". Each layermaintains a lookahead value with its neighbors in the stackand uses null messages to advance the lookahead.

The OPNET parallel simulation kernel [4] executesevents in parallel if they occur within the same parallel

III. EXTRACTING PARALLELISM IN WIRELESSNETWORKS

In this section we present techniques to extract eventsthat are non-conflicting. Two events are said to be non-

conflicting if they are independent: the order in whichthey are executed does not affect the correctness of thesimulation. Given a list of events, our objective is to find a

set of events (say 7P) that can be executed in parallel withoutviolating the correctness of the simulations. The set P can

be extracted at the following three levels (i) Event level, (ii)Node level, and (iii) Group level.

A. Extracting Parallelism at Event Level

At the event level, events in the event list that are

independent of each other can form the set P. A simplealgorithm to do this would be to pick the event with thesmallest timestamp in the event list. All the events thatare causally independent of this smallest event form theset P. Once these events are executed, the next smallestevent from the list is chosen and all the events that are

causally independent of this event forms the next set P.This is repeated until there are no more events in the eventlist. To determine if two events are causally independent,the application, routing, MAC, PHY layer information can

be used. For example if there are two application layer

2 of 6

(a) Correct Order of Execution

sessions running simultaneously at a node, then events fromthese two sessions can be executed in parallel such thatthe causal relationship is not violated. One way to ensurethat causal relationship between events are not violated isby using lookahead. Lookahead represents a time intervalwithin which the events are considered independent of eachother. If the lookahead value is say 6, then the eventswithin the time interval (t, t + S) are considered to beindependent of each other and can be executed in any orderwithout violating the correctness of the simulation. Thus,lookahead gives a lower bound on when an action at anode would affect the state of another node. At the eventlevel the lookahead value is computed by using cross-layerinteraction information and node level interaction. Thus,events that belong to different layers within the same nodeor events that belong to different nodes can be executedin parallel as long as they are within this pre-determinedlookahead value.

B. Extracting Parallelism at Node LevelAt the node level, each event has a node ID attached to it.

Lookahead between each pair of nodes is computed using(i) distance information between nodes, (ii) message inter-arrival time between nodes, (iii) terrain information, and(iv) channel characteristics. Based on the lookahead valuecomputed at the node level, set P can extracted from theevent list.

1) Lookahead based on Distance Information. Usingthe routing information at nodes, the hop distancebetween nodes can be computed. If the number ofhops between two nodes A and B is say x and ift is the lower bound on the propagation delay for amessage to travel one-hop, then xt is the value oflookahead between nodes A and B. Events at nodeA and node B can be executed in parallel as long asthey are within xt time units of each other.

2) Lookahead based on Inter-arrival times. The trafficgenerators at the nodes can be used to computethe time interval between successive communicationbetween a pair of nodes. For example suppose node Asends an application packet every T seconds to nodeB. Using the traffic generator at the node A, the valueof T can be computed and this value can be used asa the lookahead between nodes A and B.

3) Lookahead based on Terrain Information. Based onthe terrain information, nodes can determine if theycan communicate to each other directly. If nodes Aand B are separated by an intervening terrain, thenevents from these nodes can be executed in parallelwithout violating the correctness. If a multi-hop routeexists between node A and node B, then the hop

distance and/or interval of communication betweennodes A and B can be used as a lookahead value.

4) Lookahead based on Channel Characteristics. In net-work scenarios where nodes have access to multiplechannels, nodes using mutually orthogonal channelsdo not interfere with each other. Events at nodes usingdifferent channel for communication can be executedin parallel without violating the correctness of simu-lation. Based on the hop distance and/or interval ofcommunication between nodes, a lookahead value canbe derived upon.

C. Extracting Parallelism at Group Level

At the group level, each event has a logical process IDassociated with it. The entire network is divided into groups,with each group acting as a logical process. Each logicalprocess can be simulated on an separate physical processor.A network can be partitioned based on the following (i) Dis-tance information, (ii) Terrain information, and (iii) Channelcharacteristics. Between each pair of logical processes, aseparate value of lookahead is computed. The lookaheadvalue can be determined by computing the hop distancesbetween the logical processes or by using the interval ofinter-group communication. Two events from two differentlogical processes can be executed in parallel if they arewithin the lookahead value of each other.

1) Distance Information. We present a simple algorithmfor partitioning a wireless network into logical processesbased on the distance between nodes in the network.We assume that the nodes are assigned unique IDs fromNI ... Nk, with k being the number of nodes simulatedin the network.

Partitioning AlgorithmA node is said to be covered if it has been assigned a logicalprocess ID. The logical process ID indicates the logicalprocess to which the node belongs.

1) Node with node id NI and all its neighbors form alogical process (say LPI). Thus all of these nodesare assigned logical process ID LPI.

2) The lowest id uncovered node (say Ni) in the two-hop neighborhood of the node N1 forms the nextlogical process LP2. All the nodes that have not beenassigned logical process ID and are neighbors of Niwould be assigned the logical process ID LP2.

3) This procedure is repeated until all the nodes in thenetwork are covered.

Figure 3 shows the result of partitioning in a samplenetwork.

3 of 6

nodes N4, N5, and N6 use channel 2 for communication.Thus, nodes NI, N2 and N3 can form a logical process sayLPI and nodes N4, N5, and N6 can form another logicalprocess LP2. After partitioning the network into logical

Fig. 3. Partitioning due to distance

2) Terrain Information. Based on the terrain informa-tion, nodes in a network can be partitioned into logicalprocesses. Consider the example given in Figure 4. In thisnetwork scenario, two groups of nodes are seperated bya hill (obstruction). Direct communication between thesegroups of nodes is not possible because of the obstruc-tion. A router placed on the hill is used for inter-groupcommunication. Most of the communication is taking placewithin the group and very infrequently the router at the topof the hill is used for the inter-group communication. Inmilitary applications, such scenarios occur often where twobattalions are seperated by an obstruction. Most of the timethe soldiers in a battalion communicate among themselvesand occasionally a message is sent to the soldiers in theother battalion. In parallel simulations, each of these groupscan be simulated as a separate logical process.

Group A Moiuntain(obshtru ction) Group B

Fig. 4. Partitioning due to Terrain

3) Channel Characteristics. In network scenarios wherenodes have access to multiple orthogonal channels, thechannel characteristics information can be used to partitionthe network into groups. The network can be partitionedinto groups, such that each group uses different channel forcommunication. Each of these groups can be considered asa logical process. Figure 5 gives an example of networkusing multiple channels. In this example nodes NI, N2 andN3 communicate with each other using channel 1, while

N3

Channel 1 Channel 2 Channel 3

Fig. 5. Partitioning due to channel characteristics

processes, the lookahead value is determined as follows:* Based on hop distance: The hop distance between

logical processes is used to determine the lookahead.Suppose logical processes LPI and LP2 are x hopsapart. The events from LPI and LP2 can be executedin parallel if they are with xt time units, where t isthe lower bound on the one-hop propagation delay.

* Based on inter-group interaction: The lower bound onthe time interval between inter-group communicationcan be used to determine the lookahead. Suppose anode within a logical process LPI sends a messageto a node within a logical process LP2 every T timeunits, then the value of T is an upper bound on thelookahead between LPI and LP2.

IV. EXAMPLE

To illustrate how parallel events can be extracted atdifferent levels, consider the network scenario in Figure 3.A sample event list is given in Figure 6.

,El, LP1, Nl, 0.2

E2, LP1, Nl, 0.24

E3, LP5, N12, 0.3

E4, LP1, N3, 0.35

E5, LP5, N12, 0.4

E6, LP5, N12, 0.44

E7, LP5, N10, 0.5

E8, LP5, N10, 0.54

E9, LP2, N5, 0.63

El10, LP2, N4, 0.71

Ell, LP1, Nl, 0.8

Fig. 6. Sample event list for the example network

Each event entry in the list is a four tuple <EID, LID, NID, t >, where EID represents the event ID,

4 of 6

LID represents the logical process ID, NID represents thenode for which the event is scheduled, and t represents thetime at which the event is to be executed. Let the lowerbound on the one-hop propagation delay be p = 0.1 seconds.Let us a choose a value of 0.05 seconds as a lookaheadwithin a node.

The Figure 7 shows the order in which events areexecuted in parallel simulation at the event level. At eventlevel, if an event e occurs at node n at time t and anotherevent c occurs at the same node within the interval (t, t+6),where 6 is the pre-determined lookahead value, then boththe events can be executed in parallel. An event e occuringat node N, at time ti can depend on an event c occuringat time t¼ at node Nb, if t + (Hops,b x p) < ti, whereHops,b is the number of hops between nodes Na and Nb.Hence these two events should be executed in parallel. Inour example, events El and E2 occur at nodes NI and N12,respectively. An event at time t at node N12 can changethe state of node NI not before t + (6 x p), where thenumber of hops between N12 and Nt is 6. Thus using thepropagation delay value and the hop distance value, it isclear that events El and E2 are independent of each otherand hence can be executed in parallel. Events El and E3occuring at node NI can be executed in parallel, as theyfall within the lookahead value of 0.05 seconds. Event E4cannot be executed with events El, E2, and E3 because(0.2 +p) < 0.35) and (0.24 +p) < 0.35. Similarly, the restof the events can be processed as shown in the Figure 7.

Events execution order

E1,E2,E3

E4,E5,E6

E7,E8,E9,EIO

Ell

Events E3 and E4 occur 0.1 1 seconds apart and at nodes thatare one hop away of each other. Hence there is a possibilitythat they are dependent events, so we cannot execute theseevents in parallel. The figure shows how remaining eventscan be processed at the node level.

Events execution order

El,E3

E2

E4,E5

E6,E7,E9,EIO

E8,Eli

Fig. 8. Event execution order (node level)

Figure 9 shows the order in which events are executed inparallel at the group level. At the group level, the lookaheadis computed based on the hop distance between logicalgroups. At group level, the lookahead between nodes NIand N12 is 2p whereas at the node level it is 6p.

Events execution order

El,E3

E2,E5

E4, E6

E7

E8, E9

EI, ElI

Fig. 9. Event execution order (group level)

Fig. 7. Event execution order (event level)

Figure 8 shows the order in which events are executedin parallel at the node level. At the node level, we onlyconsider the events occuring at different nodes. The eventswithin the same node would be executed sequentially.Events El and E2 occur 0.1 seconds apart and at nodes thatare six hops away of each other, so they are independentof each other. Hence this two events can be executed inparallel. Whereas events El and E3 occur at the same nodeand at the node level, we cannot execute them in parallel.

V. DISCUSSION

At the event level in addition to the events belongingto different nodes, the events belonging to different layerswithin the nodes can be executed in parallel. Thus, thenumber of events executed in parallel could be more atthe event level when compared to the number of eventsexecuted at the node or group level. However, the compu-tational overhead to compute the set P at the event levelcould be high.At the node level, a higher value of lookahead can be

extracted between nodes when compared to the lookahead

5 of 6

Number of Nodes Total # Total # of Events % of Reduction % of Eventsof Events in Parallel in Execution Time in Parallel

4 Node Network 2199 1420 32.3 64

10 Node Network 4855 1554 16 32

TABLE IEXPERIMENT RESULTS.

value extracted at the group level. However the computa-tional overhead of modeling each node as a logical processis higher when compared to modeling each group as alogical process.

VI. SIMULATION EXPERIMENTSWe conducted simulation experiments to evaluate the

performance in terms of the amount of parallelism thatcan be extracted. We ran the experiments on a sequentialsimulator and logged all the events that occured during thesequential run. In this section we show what percentage oftotal events can be executed in parallel at the group level.The network is divided into two groups. A node in onegroup sends a stream of packets to a node in another group.This is a constant bit-rate traffic with packets sent every0.5 seconds. The packet size is 512 bytes. We conductedexperiments for a 4 node and a 10 node network. TheTable I shows the percentage of events that can be executedin parallel over a dual processor system. The table alsoshows the total number of events that were simulated. Thepercentage of events that are executed in a 4 node network isseen to be higher than a 10 node network. This is because, atthe group level, the number of events extracted for parallelexecution mainly depends on the number of partitions. Inboth the experiments, we had partitioned the network intwo partitions, hence for a four node network we get largerpercentage for parallel execution.

violating causality constraints. In the future we intend toevaluate each of the techniques presented in this paper byconducting simulation experiments.

REFERENCES[1] Fujimoto, R.M. Parallel discrete-event simulation. In Communica-

tions of the ACM, October 1990.[2] L. Bajaj, M. Takai, R. Ahuja, K. Tang, R. Bagrodia, and M. Gerla.

GloMoSim: A scalable network simulation environment. TechnicalReport 990027, UCLA, Computer Science Department, 1999.

[3] Meyer, R.A and Bagrodia, R.L. Path Lookahead: A Data Flow Viewof PDES Models. In Proceedings of the 13th Workshop on Paralleland Distributed Simulation (PADS'99), 1999.

[4] OPNET. OPNET AModeler Documentationhttp://www. opnet. comlsupport.

[5] Qualnet. Qualnet Parallel Developer Documentationhttp://www.scalable-networks.coml.

[6] Zhengrong Ji, Junlan Zhou, Mineo Takai, Jay Martin, and RajivBagrodia. Optimizing Parallel Execution of Detailed WirelessNetwork Simulation. In 18th Workshop on Parallel and DistributedSimulation (PADS'2004), 2004.

VII. CONCLUSIONWith unprecendented growth in the usage of military

wireless communication devices, there is an increasing needfor simulating high fidelity, large-scale wireless networks inshorter time. In this paper, we presented ways of extractingparallelism available in wireless network simulations at (i)event level, (ii) node level, and (iii) group level. Using theterrain information, distance information, medium charac-teristics, and cross-layer interaction, a set of non-conflictingevents can be extraced and executed in parallel without

6 of 6