On the Accuracy of MANET Simulators

David Cavin

Yoav Sasson

& André Schiper

Presented by

Michael W. Totaro

Mobile Computing and Wireless Systems (MoCWiS) group

UL Lafayette - CACS

Topics

Overview Introduction Related Work Flooding Algorithm The Simulators Simulations Conclusions Q & A

Overview

The simulation phase of MANET applications or protocol deployment requires meaningful simulation results

The model on which the simulator is based should match as closely as possible to reality

Simulation results of a straightforward algorithm using several popular simulators are presented, whereby significant divergences exist between the simulators

Introduction

Context– Interest in MANETs (Mobile Ad-hoc Networks)

requires adaptation of solutions from the traditional wired networks to the wireless environment

– Simulation is a tool that can often help to improve or validate protocols

– Generally speaking, all simulators provide a complete toolkit to developers that facilitates metrics collection and various wireless network diagnostics

Introduction (2)

Accuracy of simulation results– Popular simulators such as NS-2, OPNET

Modeler, and GloMoSim provide advanced simulation environments to test and debug networking protocols, including wireless applications

– It is essential that the simulated behaviors match as closely as possible the reality

– This latter requirement assumes that several issues are sufficiently addressed

Introduction (3)

Accuracy of simulation results (cont’d)– First Issue

Application is likely to rely on components such as a collision detection module, as well as radio propagation or MAC protocols

Correct definitions of these components in the simulator is critical

Typically, the algorithm being evaluated is modeled in detail; however, cross-layer interactions are very rarely taken into account

Introduction (4)

Accuracy of simulation results (cont’d)– Second Issue

Simulation parameters and the environment (e.g., mobility schemes, power ranges, connectivity) must be realistic

Incorrect initial conditions may lead to unexpected results that are not realizable in a real network

Introduction (5)

Accuracy of simulation results (cont’d)– Focus of research

The research presented in this paper shows the results of a set of measures collected during the simulation of a flooding algorithm on three different simulators: OPNET, NS-2, and GloMoSim

Special attention was given to setting the same parameters and considering the same scenarios in each simulator; nevertheless, very different results—barely compatible—were collected

Related Work

The research literature offers an abundance of papers on the efficiency of wireless algorithms comparing relative performances of each by means of simulation

Few of these papers, however, focus on possible divergences that may occur between simulators, probably because the researchers work with only a single simulator—one with which they are most familiar—and thus do not expect nor anticipate significant differences among simulators

Related Work (2)

The physical layer and the important parameters that influence its behavior have been modeled in NS-2 and OPNET– Results suggest that the configuration affects

seriously the absolute performance of a protocol, and can even change the relative ranking among protocols for the same scenario

Related Work (3)

The effect of detail in MANET simulations has been studied– Appropriate levels of detail in simulation models

for radio propagation and energy consumption remain questionable

Simulations that are too detailed may not be easily adapted to expeditiously explore alternatives

Conversely, simulations that lack detail can lead to misleading or incorrect results

Flooding Algorithm

Introduction– A frequently used operation to spread information

to the whole network is the broadcast of messages

– The performance of the broadcast is likely to affect the global efficiency of any protocol using it; hence, the broadcast should be implemented in the most efficient way

Flooding Algorithm (2)

Introduction– Simulations

Peer-to-peer wireless network, roughly 50 nodes randomly placed on a 1km x 1km area

Ad-hoc mode, without any central access point (infrastructureless)

Every node (peer) has the same possibilities and functionalities

Flooding Algorithm (3)

Flooding– Flooding a message over the network is a simple

way to implement broadcast Node initiates a broadcast Message is transmitted to its neighborhood (i.e., all

nodes within the sender’s transmission range) When the message is received by a recipient for the first

time, the recipient re-broadcasts it

Flooding Algorithm (4)

Flooding example

Flooding Algorithm (5)

Drawback of Flooding → overhead of flooded messages in the network

Under ideal conditions (i.e., all nodes received the broadcast) in a network of N nodes, a single broadcast will generate exactly N copies of itself

– Likely to increase probability of collisions– Most nodes will receive the same message several times,

thus keeping the shared medium unnecessarily busy

Flooding Algorithm (6)

Architecture

Algorithm protocol

stack

Assume thatevery

messagehas unique

ID

The Simulators

Introduction– The way a new algorithm is integrated can be

considerably different from one simulator to another

– A summary of the different implementation approaches for each simulator is presented, along with particular requirements and challenges

The Simulators (2)

OPNET Modeler– Can simulate many kinds of wired networks, and a 802.11

compliant MAC layer implementation is also provided– Phases of OPNET deployment process

1. Choose and configure node models to use in simulations—for example, a wireless node, a workstation, a router, and so on

2. Build and organize network by connecting the different entities

3. Select the statistics you wish to collect during simulations

The Simulators (3)

OPNET Modeler (cont’d)– In this experiment, the authors created a new node model

which encapsulates 802.11 MAC layer of OPNET, as well as an application process that implements the flooding algorithm Flooding algorithm process model is described as a state

machine, whereby each state has code that is executed upon state activation

A transition that links two states is followed whenever a certain condition carried by the transition is true

Difficulty with OPNET is actually building the state machine for each level of the protocol stack

The Simulators (4)

NS-2– Discrete event network simulator that supports both wired

and wireless networks, including most MANET routing protocols as well as an 802.11 MAC layer implementation

– Source code is split between C++ for its core engine, and OTcl, an object-oriented version of PCL for configuration and simulation scripts

– Implementation and simulation steps1. Implement the protocol by adding a combination of C++ and

OTcl code to NS-2’s source base2. Describe the simulation in an OTcl script3. Run the simulation4. Analyze the generated trace file

The Simulators (5)

NS-2 (cont’d)– In this experiment, the authors adapted the

implementation of flooding provided in NS-2 An Agent (which, in NS-2, represents an endpoint

where packets are constructed, processed, or consumed) was implemented at the Application layer for the broadcast source, and the simulation trace was collected at the MAC layer

Major challenges with NS-2 include: a substantial learning curve; difficult debugging; a large memory footprint; and, a lack of scalability

The Simulators (6)

GloMoSim– Scalable simulation environment for wireless and wired

networks, developed initially at UCLA Computing Laboratory

– Provides various applications (CBR, ftp, telnet), transport protocols (tcp, udp), routing protocols (AODV, flooding), and mobility schemes (random waypoint, random drunken)

– User must define specific scenarios in text configuration files app file—contains description of traffic to generate (e.g., app

type, bit rate, and so on) Config file—contains description of other (remaining)

parameters

The Simulators (3)

GloMoSim (cont’d)– Statistics collected can be either textual or

graphical– According to the authors, compared to OPNET,

GloMoSim’s architecture is much less flexible

Simulations

Static parameters Varying parameters

Common constant parametersof the simulations

Varying parameters thatdescribe the behavior ofan ad-hoc network and

that can be set in acontrolled way

Simulations (2)

Metrics

– First metric gives information about the time needed to flood a message

Time delay: For a node n, this is the average time needed for one packet to reach n

– Second metric measures the general efficiency of the algorithm Success rate: For a node n, this is the difference between the

expected and the actual number of messages received at n

– Third metric stores the overhead of messages that are unnecessarily flooded in the network

Overhead: For a node n, this is the sum of duplicated packets received by n

Simulations (3)

Results– Only the most striking graphs are provided in the

paper– Several scenarios are defined by varying one or

more parameters from the previous table labeled “Varying parameters”

– For each scenario, the set of varied parameters is given in the table just above the graph

Simulations (4)

This scenario depicts a critical factor that influences the success rate in MANETs: the effective transmission range

Notice the apparent differences in trend between the simulators

Success rate vs. Power range

GloMoSim

OPNETNS-2

Success rate vs. Power range

Simulations (5)

This scenario evaluates the effects of node mobility on the flooding’s ability to deliver packets reliably

Again, we see a significant difference in success rate

Success rate vs. Mobility

GloMoSim

OPNET

NS-2

Success rate vs. Mobility

Simulations (6)

This scenario presents the average overhead of messages flooded in the network for a single simulation run

This metric is related to the mean number of reachable neighbors (that is, within transmission range

OPNETNS-2

GloMoSim

Overhead vs. Mobility

Overhead vs. Mobility

Simulations (7)

The final scenario compares average time delay needed to flood a message throughout the whole network

This metric increases with the number of hops from source to destination and also whenever collisions occur

Time delay vs. Mobility

OPNET

NS-2

GloMoSim

Time delay vs. Mobility

Simulations (8)

Analysis and interpretation– Simulation results of the flooding algorithm demonstrate

that modeling of the MAC protocol and of the physical layer can lead to different results, depending upon the simulator

– Possible reasons Differing physical layer implementations Implementation of a new protocol is itself difficult to transpose

from one simulator to another Given that successive releases provide bug fixes, it is

reasonable to assume that MANET simulators still contain errors or incompatibilities to IEEE 802.11 standard

Conclusions (Authors)

Instead of simulations, a more realistic scheme might entail a hybrid approach in which only the lowest layers—MAC and physical—and the mobility model are simulated and all the upper layers (from transport to application) are executed on a cluster of machines

There is an important lack of real experiments the prove the feasibility of wireless protocols

Questions?

?